draft-ietf-ippm-reporting-metrics-01.txt   draft-ietf-ippm-reporting-metrics-02.txt 
Network Working Group A. Morton Network Working Group A. Morton
Internet-Draft G. Ramachandran Internet-Draft G. Ramachandran
Intended status: Informational G. Maguluri Intended status: Informational G. Maguluri
Expires: September 6, 2010 AT&T Labs Expires: December 1, 2010 AT&T Labs
March 5, 2010 May 30, 2010
Reporting Metrics: Different Points of View Reporting Metrics: Different Points of View
draft-ietf-ippm-reporting-metrics-01 draft-ietf-ippm-reporting-metrics-02
Abstract Abstract
Consumers of IP network performance metrics have many different uses Consumers of IP network performance metrics have many different uses
in mind. This memo categorizes the different audience points of in mind. This memo categorizes the different audience points of
view. It describes how the categories affect the selection of metric view. It describes how the categories affect the selection of metric
parameters and options when seeking info that serves their needs. parameters and options when seeking info that serves their needs.
The memo then proceeds to discuss "long-term" reporting The memo then proceeds to discuss "long-term" reporting
considerations (e.g, days, weeks or months, as opposed to 10 considerations (e.g, days, weeks or months, as opposed to 10
seconds). seconds).
Requirements Language Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
Status of this Memo Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF). Note that other groups may also distribute
other groups may also distribute working documents as Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at This Internet-Draft will expire on December 1, 2010.
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on September 6, 2010.
Copyright Notice Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the BSD License. described in the Simplified BSD License.
This document may contain material from IETF Documents or IETF This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this 10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process. modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other it for publication as an RFC or to translate it into languages other
than English. than English.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 4 2. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 4
3. Effect of POV on the Loss Metric . . . . . . . . . . . . . . . 5 3. Reporting Results . . . . . . . . . . . . . . . . . . . . . . 5
3.1. Loss Threshold . . . . . . . . . . . . . . . . . . . . . . 5 3.1. Overview of Metric Statistics . . . . . . . . . . . . . . 5
3.1.1. Network Characterization . . . . . . . . . . . . . . . 5 3.2. Long-Term Reporting Considerations . . . . . . . . . . . . 6
3.1.2. Application Performance . . . . . . . . . . . . . . . 7 4. Effect of POV on the Loss Metric . . . . . . . . . . . . . . . 8
3.2. Errored Packet Designation . . . . . . . . . . . . . . . . 7 4.1. Loss Threshold . . . . . . . . . . . . . . . . . . . . . . 8
3.3. Causes of Lost Packets . . . . . . . . . . . . . . . . . . 7 4.1.1. Network Characterization . . . . . . . . . . . . . . . 8
3.4. Summary for Loss . . . . . . . . . . . . . . . . . . . . . 8 4.1.2. Application Performance . . . . . . . . . . . . . . . 10
4. Effect of POV on the Delay Metric . . . . . . . . . . . . . . 8 4.2. Errored Packet Designation . . . . . . . . . . . . . . . . 10
4.1. Treatment of Lost Packets . . . . . . . . . . . . . . . . 8 4.3. Causes of Lost Packets . . . . . . . . . . . . . . . . . . 10
4.1.1. Application Performance . . . . . . . . . . . . . . . 9 4.4. Summary for Loss . . . . . . . . . . . . . . . . . . . . . 11
4.1.2. Network Characterization . . . . . . . . . . . . . . . 9 5. Effect of POV on the Delay Metric . . . . . . . . . . . . . . 11
4.1.3. Delay Variation . . . . . . . . . . . . . . . . . . . 10 5.1. Treatment of Lost Packets . . . . . . . . . . . . . . . . 11
4.1.4. Reordering . . . . . . . . . . . . . . . . . . . . . . 11 5.1.1. Application Performance . . . . . . . . . . . . . . . 11
4.2. Preferred Statistics . . . . . . . . . . . . . . . . . . . 11 5.1.2. Network Characterization . . . . . . . . . . . . . . . 12
4.3. Summary for Delay . . . . . . . . . . . . . . . . . . . . 12 5.1.3. Delay Variation . . . . . . . . . . . . . . . . . . . 13
5. Effect of POV on Raw Capacity Metrics . . . . . . . . . . . . 12 5.1.4. Reordering . . . . . . . . . . . . . . . . . . . . . . 14
5.1. Type-P Parameter . . . . . . . . . . . . . . . . . . . . . 12 5.2. Preferred Statistics . . . . . . . . . . . . . . . . . . . 14
5.2. a priori Factors . . . . . . . . . . . . . . . . . . . . . 13 5.3. Summary for Delay . . . . . . . . . . . . . . . . . . . . 15
5.3. IP-layer Capacity . . . . . . . . . . . . . . . . . . . . 13 6. Effect of POV on Raw Capacity Metrics . . . . . . . . . . . . 15
5.4. IP-layer Utilization . . . . . . . . . . . . . . . . . . . 13 6.1. Type-P Parameter . . . . . . . . . . . . . . . . . . . . . 15
5.5. IP-layer Available Capacity . . . . . . . . . . . . . . . 14 6.2. a priori Factors . . . . . . . . . . . . . . . . . . . . . 16
5.6. Variability in Utilization and Avail. Capacity . . . . . . 15 6.3. IP-layer Capacity . . . . . . . . . . . . . . . . . . . . 16
6. Test Streams and Sample Size . . . . . . . . . . . . . . . . . 15 6.4. IP-layer Utilization . . . . . . . . . . . . . . . . . . . 17
6.1. Test Stream Characteristics . . . . . . . . . . . . . . . 15 6.5. IP-layer Available Capacity . . . . . . . . . . . . . . . 17
6.2. Sample Size . . . . . . . . . . . . . . . . . . . . . . . 15 6.6. Variability in Utilization and Avail. Capacity . . . . . . 18
7. Reporting Results . . . . . . . . . . . . . . . . . . . . . . 16 7. Test Streams and Sample Size . . . . . . . . . . . . . . . . . 18
7.1. Overview of Metric Statistics . . . . . . . . . . . . . . 16 7.1. Test Stream Characteristics . . . . . . . . . . . . . . . 18
7.2. Long-Term Reporting Considerations . . . . . . . . . . . . 17 7.2. Sample Size . . . . . . . . . . . . . . . . . . . . . . . 19
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19
9. Security Considerations . . . . . . . . . . . . . . . . . . . 18 9. Security Considerations . . . . . . . . . . . . . . . . . . . 19
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 18 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 19
11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.1. Normative References . . . . . . . . . . . . . . . . . . . 19 11.1. Normative References . . . . . . . . . . . . . . . . . . . 20
11.2. Informative References . . . . . . . . . . . . . . . . . . 19 11.2. Informative References . . . . . . . . . . . . . . . . . . 21
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 21
1. Introduction 1. Introduction
When designing measurements of IP networks and presenting the When designing measurements of IP networks and presenting the
results, knowledge of the audience is a key consideration. To results, knowledge of the audience is a key consideration. To
present a useful and relevant portrait of network conditions, one present a useful and relevant portrait of network conditions, one
must answer the following question: must answer the following question:
"How will the results be used?" "How will the results be used?"
There are two main audience categories: There are two main audience categories:
1. Network Characterization - describes conditions in an IP network 1. Network Characterization - describes conditions in an IP network
for quality assurance, troubleshooting, modeling, etc. The for quality assurance, troubleshooting, modeling, Service Level
point-of-view looks inward, toward the network, and the consumer Agreements (SLA), etc. The point-of-view looks inward, toward
intends their actions there. the network, and the consumer intends their actions there.
2. Application Performance Estimation - describes the network 2. Application Performance Estimation - describes the network
conditions in a way that facilitates determining affects on user conditions in a way that facilitates determining affects on user
applications, and ultimately the users themselves. This point- applications, and ultimately the users themselves. This point-
of-view looks outward, toward the user(s), accepting the network of-view looks outward, toward the user(s), accepting the network
as-is. This consumer intends to estimate a network-dependent as-is. This consumer intends to estimate a network-dependent
aspect of performance, or design some aspect of an application's aspect of performance, or design some aspect of an application's
accommodation of the network. (These are *not* application accommodation of the network. (These are *not* application
metrics, they are defined at the IP layer.) metrics, they are defined at the IP layer.)
skipping to change at page 4, line 46 skipping to change at page 4, line 46
2. Purpose and Scope 2. Purpose and Scope
The purpose of this memo is to clearly delineate two points-of-view The purpose of this memo is to clearly delineate two points-of-view
(POV) for using measurements, and describe their effects on the test (POV) for using measurements, and describe their effects on the test
design, including the selection of metric parameters and reporting design, including the selection of metric parameters and reporting
the results. the results.
The current scope of this memo primarily covers the design and The current scope of this memo primarily covers the design and
reporting of the loss and delay metrics [RFC2680] [RFC2679]. It will reporting of the loss and delay metrics [RFC2680] [RFC2679]. It will
also discuss the delay variation and reordering metrics where also discuss the delay variation [RFC3393] and reordering metrics
applicable. [RFC4737] where applicable.
With capacity metrics growing in relevance to the industry, the memo With capacity metrics growing in relevance to the industry, the memo
also covers POV and reporting considerations for metrics resulting also covers POV and reporting considerations for metrics resulting
from the Bulk Transfer Capacity Framework [RFC3148] and Network from the Bulk Transfer Capacity Framework [RFC3148] and Network
Capacity Definitions [RFC5136]. These memos effectively describe two Capacity Definitions [RFC5136]. These memos effectively describe two
different categories of metrics, [RFC3148] with congestion flow- different categories of metrics, [RFC3148] with congestion flow-
control and the notion of unique data bits delivered, and [RFC5136] control and the notion of unique data bits delivered, and [RFC5136]
using a definition of raw capacity without the restrictions of data using a definition of raw capacity without the restrictions of data
uniqueness or congestion-awareness. It might seem at first glance uniqueness or congestion-awareness. It might seem at first glance
that each of these metrics has an obvious audience (Raw = Network that each of these metrics has an obvious audience (Raw = Network
Characterization, Restricted = Application Performance), but reality Characterization, Restricted = Application Performance), but reality
is more complex and consistent with the overall topic of capacity is more complex and consistent with the overall topic of capacity
measurement and reporting. The Raw and Restricted capacity metrics measurement and reporting. For example, TCP is usually used in
will be treated in separate sections, although they share one common Restricted capacity measurement methods, while UDP appears in Raw
capacity measurement. The Raw and Restricted capacity metrics will
be treated in separate sections, although they share one common
reporting issue: representing variability in capacity metric results. reporting issue: representing variability in capacity metric results.
Sampling, or the design of the active packet stream that is the basis Sampling, or the design of the active packet stream that is the basis
for the measurements, is also discussed. for the measurements, is also discussed.
3. Effect of POV on the Loss Metric 3. Reporting Results
This section gives an overview of recommendations, followed by
additional considerations for reporting results in the "long-term".
3.1. Overview of Metric Statistics
This section gives an overview of reporting recommendations for the
loss, delay, and delay variation metrics based on the discussion and
conclusions of the preceding sections.
The minimal report on measurements MUST include both Loss and Delay
Metrics.
For Packet Loss, the loss ratio defined in [RFC2680] is a sufficient
starting point, especially the guidance for setting the loss
threshold waiting time. We have calculated a waiting time above that
should be sufficient to differentiate between packets that are truly
lost or have long finite delays under general measurement
circumstances, 51 seconds. Knowledge of specific conditions can help
to reduce this threshold, but 51 seconds is considered to be
manageable in practice.
We note that a loss ratio calculated according to [Y.1540] would
exclude errored packets form the numerator. In practice, the
difference between these two loss metrics is small if any, depending
on whether the last link prior to the destination contributes errored
packets.
For Packet Delay, we recommend providing both the mean delay and the
median delay with lost packets designated undefined (as permitted by
[RFC2679]). Both statistics are based on a conditional distribution,
and the condition is packet arrival prior to a waiting time dT, where
dT has been set to take maximum packet lifetimes into account, as
discussed above. Using a long dT helps to ensure that delay
distributions are not truncated.
For Packet Delay Variation (PDV), the minimum delay of the
conditional distribution should be used as the reference delay for
computing PDV according to [Y.1540] or [RFC3393]. A useful value to
report is a pseudo range of delay variation based on calculating the
difference between a high percentile of delay and the minimum delay.
For example, the 99.9%-ile minus the minimum will give a value that
can be compared with objectives in [Y.1541].
3.2. Long-Term Reporting Considerations
[I-D.ietf-ippm-reporting] describes methods to conduct measurements
and report the results on a near-immediate time scale (10 seconds,
which we consider to be "short-term").
Measurement intervals and reporting intervals need not be the same
length. Sometimes, the user is only concerned with the performance
levels achieved over a relatively long interval of time (e.g, days,
weeks, or months, as opposed to 10 seconds). However, there can be
risks involved with running a measurement continuously over a long
period without recording intermediate results:
o Temporary power failure may cause loss of all the results to date.
o Measurement system timing synchronization signals may experience a
temporary outage, causing sub-sets of measurements to be in error
or invalid.
o Maintenance may be necessary on the measurement system, or its
connectivity to the network under test.
For these and other reasons, such as
o the constraint to collect measurements on intervals similar to
user session length, or
o the dual-use of measurements in monitoring activities where
results are needed on a period of a few minutes,
there is value in conducting measurements on intervals that are much
shorter than the reporting interval.
There are several approaches for aggregating a series of measurement
results over time in order to make a statement about the longer
reporting interval. One approach requires the storage of all metric
singletons collected throughout the reporting interval, even though
the measurement interval stops and starts many times.
Another approach is described in [RFC5835] as "temporal aggregation".
This approach would estimate the results for the reporting interval
based on many individual measurement interval statistics (results)
alone. The result would ideally appear in the same form as though a
continuous measurement was conducted. A memo to address the details
of temporal aggregation is yet to be prepared.
Yet another approach requires a numerical objective for the metric,
and the results of each measurement interval are compared with the
objective. Every measurement interval where the results meet the
objective contribute to the fraction of time with performance as
specified. When the reporting interval contains many measurement
intervals it is possible to present the results as "metric A was less
than or equal to objective X during Y% of time.
NOTE that numerical thresholds are not set in IETF performance work
and are explicitly excluded from the IPPM charter.
In all measurement, it is important to avoid unintended
synchronization with network events. This topic is treated in
[RFC2330] for Poisson-distributed inter-packet time streams, and
[RFC3432] for Periodic streams. Both avoid synchronization through
use of random start times.
There are network conditions where it is simply more useful to report
the connectivity status of the Source-Destination path, and to
distinguish time intervals where connectivity can be demonstrated
from other time intervals (where connectivity does not appear to
exist). [RFC2678] specifies a number of one-way and two connectivity
metrics of increasing complexity. In this memo, we RECOMMEND that
long term reporting of loss, delay, and other metrics be limited to
time intervals where connectivity can be demonstrated, and other
intervals be summarized as percent of time where connectivity does
not appear to exist. We note that this same approach has been
adopted in ITU-T Recommendation [Y.1540] where performance parameters
are only valid during periods of service "availability" (evaluated
according to a function based on packet loss, and sustained periods
of loss ratio greater than a threshold are declared "unavailable").
4. Effect of POV on the Loss Metric
This section describes the ways in which the Loss metric can be tuned This section describes the ways in which the Loss metric can be tuned
to reflect the preferences of the two audience categories, or to reflect the preferences of the two audience categories, or
different POV. The waiting time to declare a packet lost, or loss different POV. The waiting time to declare a packet lost, or loss
threshold is one area where there would appear to be a difference, threshold is one area where there would appear to be a difference,
but the ability to post-process the results may resolve it. but the ability to post-process the results may resolve it.
3.1. Loss Threshold 4.1. Loss Threshold
RFC 2680 [RFC2680] defines the concept of a waiting time for packets RFC 2680 [RFC2680] defines the concept of a waiting time for packets
to arrive, beyond which they are declared lost. The text of the RFC to arrive, beyond which they are declared lost. The text of the RFC
declines to recommend a value, instead saying that "good engineering, declines to recommend a value, instead saying that "good engineering,
including an understanding of packet lifetimes, will be needed in including an understanding of packet lifetimes, will be needed in
practice." Later, in the methodology, they give reasons for waiting practice." Later, in the methodology, they give reasons for waiting
"a reasonable period of time", and leaving the definition of "a reasonable period of time", and leaving the definition of
"reasonable" intentionally vague. "reasonable" intentionally vague.
3.1.1. Network Characterization 4.1.1. Network Characterization
Practical measurement experience has shown that unusual network Practical measurement experience has shown that unusual network
circumstances can cause long delays. One such circumstance is when circumstances can cause long delays. One such circumstance is when
routing loops form during IGP re-convergence following a failure or routing loops form during IGP re-convergence following a failure or
drastic link cost change. Packets will loop between two routers drastic link cost change. Packets will loop between two routers
until new routes are installed, or until the IPv4 Time-to-Live (TTL) until new routes are installed, or until the IPv4 Time-to-Live (TTL)
field (or the IPv6 Hop Limit) decrements to zero. Very long delays field (or the IPv6 Hop Limit) decrements to zero. Very long delays
on the order of several seconds have been measured [Casner] [Cia03]. on the order of several seconds have been measured [Casner] [Cia03].
Therefore, network characterization activities prefer a long waiting Therefore, network characterization activities prefer a long waiting
skipping to change at page 7, line 11 skipping to change at page 10, line 5
calculation would not truncate the delay distribution (possibly calculation would not truncate the delay distribution (possibly
causing a change in its mathematical properties), because the packets causing a change in its mathematical properties), because the packets
that might arrive have been given sufficient time to traverse the that might arrive have been given sufficient time to traverse the
network. network.
It is worth noting that packets that are stored and deliberately It is worth noting that packets that are stored and deliberately
forwarded at a much later time constitute a replay attack on the forwarded at a much later time constitute a replay attack on the
measurement system, and are beyond the scope of normal performance measurement system, and are beyond the scope of normal performance
reporting. reporting.
3.1.2. Application Performance 4.1.2. Application Performance
Fortunately, application performance estimation activities are not Fortunately, application performance estimation activities are not
adversely affected by the estimated worst-case transfer time. adversely affected by the estimated worst-case transfer time.
Although the designer's tendency might be to set the Loss Threshold Although the designer's tendency might be to set the Loss Threshold
at a value equivalent to a particular application's threshold, this at a value equivalent to a particular application's threshold, this
specific threshold can be applied when post-processing the specific threshold can be applied when post-processing the
measurements. A shorter waiting time can be enforced by locating measurements. A shorter waiting time can be enforced by locating
packets with delays longer than the application's threshold, and re- packets with delays longer than the application's threshold, and re-
designating such packets as lost. Thus, the measurement system can designating such packets as lost. Thus, the measurement system can
use a single loss threshold and support both application and network use a single loss threshold and support both application and network
performance POVs simultaneously. performance POVs simultaneously.
3.2. Errored Packet Designation 4.2. Errored Packet Designation
RFC 2680 designates packets that arrive containing errors as lost RFC 2680 designates packets that arrive containing errors as lost
packets. Many packets that are corrupted by bit errors are discarded packets. Many packets that are corrupted by bit errors are discarded
within the network and do not reach their intended destination. within the network and do not reach their intended destination.
This is consistent with applications that would check the payload This is consistent with applications that would check the payload
integrity at higher layers, and discard the packet. However, some integrity at higher layers, and discard the packet. However, some
applications prefer to deal with errored payloads on their own, and applications prefer to deal with errored payloads on their own, and
even a corrupted payload is better than no packet at all. even a corrupted payload is better than no packet at all.
To address this possibility, and to make network characterization To address this possibility, and to make network characterization
more complete, it is recommended to distinguish between packets that more complete, it is recommended to distinguish between packets that
do not arrive (lost) and errored packets that arrive (conditionally do not arrive (lost) and errored packets that arrive (conditionally
lost). lost).
3.3. Causes of Lost Packets 4.3. Causes of Lost Packets
Although many measurement systems use a waiting time to determine if Although many measurement systems use a waiting time to determine if
a packet is lost or not, most of the waiting is in vain. The packets a packet is lost or not, most of the waiting is in vain. The packets
are no-longer traversing the network, and have not reached their are no-longer traversing the network, and have not reached their
destination. destination.
There are many causes of packet loss, including: There are many causes of packet loss, including:
1. Queue drop, or discard 1. Queue drop, or discard
2. Corruption of the IP header, or other essential header info 2. Corruption of the IP header, or other essential header info
3. TTL expiration (or use of a TTL value that is too small) 3. TTL expiration (or use of a TTL value that is too small)
4. Link or router failure 4. Link or router failure
After waiting sufficient time, packet loss can probably be attributed After waiting sufficient time, packet loss can probably be attributed
to one of these causes. to one of these causes.
3.4. Summary for Loss 4.4. Summary for Loss
Given that measurement post-processing is possible (even encouraged Given that measurement post-processing is possible (even encouraged
in the definitions of IPPM metrics), measurements of loss can easily in the definitions of IPPM metrics), measurements of loss can easily
serve both points of view: serve both points of view:
o Use a long waiting time to serve network characterization and o Use a long waiting time to serve network characterization and
revise results for specific application delay thresholds as revise results for specific application delay thresholds as
needed. needed.
o Distinguish between errored packets and lost packets when possible o Distinguish between errored packets and lost packets when possible
to aid network characterization, and combine the results for to aid network characterization, and combine the results for
application performance if appropriate. application performance if appropriate.
4. Effect of POV on the Delay Metric 5. Effect of POV on the Delay Metric
This section describes the ways in which the Delay metric can be This section describes the ways in which the Delay metric can be
tuned to reflect the preferences of the two consumer categories, or tuned to reflect the preferences of the two consumer categories, or
different POV. different POV.
4.1. Treatment of Lost Packets 5.1. Treatment of Lost Packets
The Delay Metric [RFC2679] specifies the treatment of packets that do The Delay Metric [RFC2679] specifies the treatment of packets that do
not successfully traverse the network: their delay is undefined. not successfully traverse the network: their delay is undefined.
" >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined " >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined
(informally, infinite)<< means that Src sent the first bit of a (informally, infinite)<< means that Src sent the first bit of a
Type-P packet to Dst at wire-time T and that Dst did not receive that Type-P packet to Dst at wire-time T and that Dst did not receive that
packet." packet."
It is an accepted, but informal practice to assign infinite delay to It is an accepted, but informal practice to assign infinite delay to
lost packets. We next look at how these two different treatments lost packets. We next look at how these two different treatments
align with the needs of measurement consumers who wish to align with the needs of measurement consumers who wish to
characterize networks or estimate application performance. Also, we characterize networks or estimate application performance. Also, we
look at the way that lost packets have been treated in other metrics: look at the way that lost packets have been treated in other metrics:
delay variation and reordering. delay variation and reordering.
4.1.1. Application Performance 5.1.1. Application Performance
Applications need to perform different functions, dependent on Applications need to perform different functions, dependent on
whether or not each packet arrives within some finite tolerance. In whether or not each packet arrives within some finite tolerance. In
other words, a receivers' packet processing takes one of two other words, a receivers' packet processing takes one of two
directions (or "forks" in the road): directions (or "forks" in the road):
o Packets that arrive within expected tolerance are handled by o Packets that arrive within expected tolerance are handled by
processes that remove headers, restore smooth delivery timing (as processes that remove headers, restore smooth delivery timing (as
in a de-jitter buffer), restore sending order, check for errors in in a de-jitter buffer), restore sending order, check for errors in
payloads, and many other operations. payloads, and many other operations.
skipping to change at page 9, line 28 skipping to change at page 12, line 17
that attempt recovery from the apparent loss, such as that attempt recovery from the apparent loss, such as
retransmission requests, loss concealment, or forward error retransmission requests, loss concealment, or forward error
correction to replace the missing packet. correction to replace the missing packet.
So, it is important to maintain a distinction between packets that So, it is important to maintain a distinction between packets that
actually arrive, and those that do not. Therefore, it is preferable actually arrive, and those that do not. Therefore, it is preferable
to leave the delay of lost packets undefined, and to characterize the to leave the delay of lost packets undefined, and to characterize the
delay distribution as a conditional distribution (conditioned on delay distribution as a conditional distribution (conditioned on
arrival). arrival).
4.1.2. Network Characterization 5.1.2. Network Characterization
In this discussion, we assume that both loss and delay metrics will In this discussion, we assume that both loss and delay metrics will
be reported for network characterization (at least). be reported for network characterization (at least).
Assume packets that do not arrive are reported as Lost, usually as a Assume packets that do not arrive are reported as Lost, usually as a
fraction of all sent packets. If these lost packets are assigned fraction of all sent packets. If these lost packets are assigned
undefined delay, then network's inability to deliver them (in a undefined delay, then network's inability to deliver them (in a
timely way) is captured only in the loss metric when we report timely way) is captured only in the loss metric when we report
statistics on the Delay distribution conditioned on the event of statistics on the Delay distribution conditioned on the event of
packet arrival (within the Loss waiting time threshold). We can say packet arrival (within the Loss waiting time threshold). We can say
skipping to change at page 10, line 36 skipping to change at page 13, line 31
We note that a Delay CDF that is conditioned on packet arrival would We note that a Delay CDF that is conditioned on packet arrival would
not exhibit this apparent overlap with loss. not exhibit this apparent overlap with loss.
Although infinity is a familiar mathematical concept, it is somewhat Although infinity is a familiar mathematical concept, it is somewhat
disconcerting to see any time-related metric reported as infinity, in disconcerting to see any time-related metric reported as infinity, in
the opinion of the authors. Questions are bound to arise, and tend the opinion of the authors. Questions are bound to arise, and tend
to detract from the goal of informing the consumer with a performance to detract from the goal of informing the consumer with a performance
report. report.
4.1.3. Delay Variation 5.1.3. Delay Variation
[RFC3393] excludes lost packets from samples, effectively assigning [RFC3393] excludes lost packets from samples, effectively assigning
an undefined delay to packets that do not arrive in a reasonable an undefined delay to packets that do not arrive in a reasonable
time. Section 4.1 describes this specification and its rationale time. Section 4.1 describes this specification and its rationale
(ipdv = inter-packet delay variation in the quote below). (ipdv = inter-packet delay variation in the quote below).
"The treatment of lost packets as having "infinite" or "undefined" "The treatment of lost packets as having "infinite" or "undefined"
delay complicates the derivation of statistics for ipdv. delay complicates the derivation of statistics for ipdv.
Specifically, when packets in the measurement sequence are lost, Specifically, when packets in the measurement sequence are lost,
simple statistics such as sample mean cannot be computed. One simple statistics such as sample mean cannot be computed. One
possible approach to handling this problem is to reduce the event possible approach to handling this problem is to reduce the event
space by conditioning. That is, we consider conditional statistics; space by conditioning. That is, we consider conditional statistics;
namely we estimate the mean ipdv (or other derivative statistic) namely we estimate the mean ipdv (or other derivative statistic)
conditioned on the event that selected packet pairs arrive at the conditioned on the event that selected packet pairs arrive at the
destination (within the given timeout). While this itself is not destination (within the given timeout). While this itself is not
without problems (what happens, for example, when every other packet without problems (what happens, for example, when every other packet
is lost), it offers a way to make some (valid) statements about ipdv, is lost), it offers a way to make some (valid) statements about ipdv,
at the same time avoiding events with undefined outcomes." at the same time avoiding events with undefined outcomes."
4.1.4. Reordering We note that the argument above applies to all forms of packet delay
variation that can be constructed using the "selection function"
concept of [RFC3393]. In recent work the two main forms of delay
variation metrics have been compared and the results are summarized
in [RFC5481].
5.1.4. Reordering
[RFC4737]defines metrics that are based on evaluation of packet [RFC4737]defines metrics that are based on evaluation of packet
arrival order, and include a waiting time to declare a packet lost arrival order, and include a waiting time to declare a packet lost
(to exclude them from further processing). (to exclude them from further processing).
If packets are assigned a delay value, then the reordering metric If packets are assigned a delay value, then the reordering metric
would declare any packets with infinite delay to be reordered, would declare any packets with infinite delay to be reordered,
because their sequence numbers will surely be less than the "Next because their sequence numbers will surely be less than the "Next
Expected" threshold when (or if) they arrive. But this practice Expected" threshold when (or if) they arrive. But this practice
would fail to maintain orthogonality between the reordering metric would fail to maintain orthogonality between the reordering metric
and the loss metric. Confusion can be avoided by designating the and the loss metric. Confusion can be avoided by designating the
delay of non-arriving packets as undefined, and reserving delay delay of non-arriving packets as undefined, and reserving delay
values only for packets that arrive within a sufficiently long values only for packets that arrive within a sufficiently long
waiting time. waiting time.
4.2. Preferred Statistics 5.2. Preferred Statistics
Today in network characterization, the sample mean is one statistic Today in network characterization, the sample mean is one statistic
that is almost ubiquitously reported. It is easily computed and that is almost ubiquitously reported. It is easily computed and
understood by virtually everyone in this audience category. Also, understood by virtually everyone in this audience category. Also,
the sample is usually filtered on packet arrival, so that the mean is the sample is usually filtered on packet arrival, so that the mean is
based a conditional distribution. based a conditional distribution.
The median is another statistic that summarizes a distribution, The median is another statistic that summarizes a distribution,
having somewhat different properties from the sample mean. The having somewhat different properties from the sample mean. The
median is stable in distributions with a few outliers or without median is stable in distributions with a few outliers or without
skipping to change at page 12, line 5 skipping to change at page 15, line 5
ile have value when summarizing a distribution. ile have value when summarizing a distribution.
When both the sample mean and median are available, a comparison will When both the sample mean and median are available, a comparison will
sometimes be informative, because these two statistics are equal only sometimes be informative, because these two statistics are equal only
when the delay distribution is perfectly symmetrical. when the delay distribution is perfectly symmetrical.
Also, these statistics are generally useful from the Application Also, these statistics are generally useful from the Application
Performance POV, so there is a common set that should satisfy Performance POV, so there is a common set that should satisfy
audiences. audiences.
4.3. Summary for Delay Plots of the delay distribution may also be useful when single-value
statistics indicate that new conditions are present. An empirically-
derived probability distribution function will usually describe
multiple modes more efficiently than any other form of result.
5.3. Summary for Delay
From the perspectives of: From the perspectives of:
1. application/receiver analysis, where subsequent processing 1. application/receiver analysis, where subsequent processing
depends on whether the packet arrives or times-out, depends on whether the packet arrives or times-out,
2. straightforward network characterization without double-counting 2. straightforward network characterization without double-counting
defects, and defects, and
3. consistency with Delay variation and Reordering metric 3. consistency with Delay variation and Reordering metric
definitions, definitions,
the most efficient practice is to distinguish between truly lost and the most efficient practice is to distinguish between truly lost and
delayed packets with a sufficiently long waiting time, and to delayed packets with a sufficiently long waiting time, and to
designate the delay of non-arriving packets as undefined. designate the delay of non-arriving packets as undefined.
5. Effect of POV on Raw Capacity Metrics 6. Effect of POV on Raw Capacity Metrics
This section describes the ways that raw capacity metrics can be This section describes the ways that raw capacity metrics can be
tuned to reflect the preferences of the two audiences, or different tuned to reflect the preferences of the two audiences, or different
Points-of-View (POV). Raw capacity refers to the metrics defined in Points-of-View (POV). Raw capacity refers to the metrics defined in
[RFC5136] which do not include restrictions such as data uniqueness [RFC5136] which do not include restrictions such as data uniqueness
or flow-control response to congestion. or flow-control response to congestion.
In summary, the metrics considered are IP-layer Capacity, Utilization In summary, the metrics considered are IP-layer Capacity, Utilization
(or used capacity), and Available Capacity, for individual links and (or used capacity), and Available Capacity, for individual links and
complete paths. These three metrics form a triad: knowing one metric complete paths. These three metrics form a triad: knowing one metric
constrains the other two (within their allowed range), and knowing constrains the other two (within their allowed range), and knowing
two determines the third. The link metrics have another key aspect two determines the third. The link metrics have another key aspect
in common: they are single-measurement-point metrics at the egress of in common: they are single-measurement-point metrics at the egress of
a link. The path Capacity and Available Capacity are derived by a link. The path Capacity and Available Capacity are derived by
examining the set of single-point link measurements and taking the examining the set of single-point link measurements and taking the
minimum value. minimum value.
5.1. Type-P Parameter 6.1. Type-P Parameter
The concept of "packets of type-P" is defined in [RFC2330]. The The concept of "packets of type-P" is defined in [RFC2330]. The
type-P categorization has critical relevance in all forms of capacity type-P categorization has critical relevance in all forms of capacity
measurement and reporting. The ability to categorize packets based measurement and reporting. The ability to categorize packets based
on header fields for assignment to different queues and scheduling on header fields for assignment to different queues and scheduling
mechanisms is now common place. When un-used resources are shared mechanisms is now common place. When un-used resources are shared
across queues, the conditions in all packet categories will affect across queues, the conditions in all packet categories will affect
capacity and related measurements. This is one source of variability capacity and related measurements. This is one source of variability
in the results that all audiences would prefer to see reported in a in the results that all audiences would prefer to see reported in a
useful and easily understood way. useful and easily understood way.
Type-P in OWAMP and TWAMP is essentially confined to the Diffserv Type-P in OWAMP and TWAMP is essentially confined to the Diffserv
Codepoint [ref]. DSCP is the most common qualifier for type-P. Codepoint [ref]. DSCP is the most common qualifier for type-P.
Each audience will have a set of type-P qualifications and value Each audience will have a set of type-P qualifications and value
combinations that are of interest. Measurements and reports SHOULD combinations that are of interest. Measurements and reports SHOULD
have the flexibility to per-type and aggregate performance. have the flexibility to per-type and aggregate performance.
5.2. a priori Factors 6.2. a priori Factors
The audience for Network Characterization may have detailed The audience for Network Characterization may have detailed
information about each link that comprises a complete path (due to information about each link that comprises a complete path (due to
ownership, for example), or some of the links in the path but not ownership, for example), or some of the links in the path but not
others, or none of the links. others, or none of the links.
There are cases where the measurement audience only has information There are cases where the measurement audience only has information
on one of the links (the local access link), and wishes to measure on one of the links (the local access link), and wishes to measure
one or more of the raw capacity metrics. This scenario is quite one or more of the raw capacity metrics. This scenario is quite
common, and has spawned a substantial number of experimental common, and has spawned a substantial number of experimental
measurement methods [ref to CAIDA survey page, etc.]. Many of these measurement methods [ref to CAIDA survey page, etc.]. Many of these
methods respect that their users want a result fairly quickly and in methods respect that their users want a result fairly quickly and in
a one-trial. Thus, the measurement interval is kept short (a few a one-trial. Thus, the measurement interval is kept short (a few
seconds to a minute). seconds to a minute).
5.3. IP-layer Capacity 6.3. IP-layer Capacity
For links, this metric's theoretical maximum value can be determined For links, this metric's theoretical maximum value can be determined
from the physical layer bit rate and the bit rate reduction due to from the physical layer bit rate and the bit rate reduction due to
the layers between the physical layer and IP. When measured, this the layers between the physical layer and IP. When measured, this
metric takes additional factors into account, such as the ability of metric takes additional factors into account, such as the ability of
the sending device to process and forward traffic under various the sending device to process and forward traffic under various
conditions. For example, the arrival of routing updates may spawn conditions. For example, the arrival of routing updates may spawn
high priority processes that reduce the sending rate temporarily. high priority processes that reduce the sending rate temporarily.
Thus, the measured capacity of a link will be variable, and the Thus, the measured capacity of a link will be variable, and the
maximum capacity observed applies to a specific time, time interval, maximum capacity observed applies to a specific time, time interval,
and other relevant circumstances. and other relevant circumstances.
For paths composed of a series of links, it is easy to see how the For paths composed of a series of links, it is easy to see how the
sources of variability for the results grow with each link in the sources of variability for the results grow with each link in the
path. Results variability will be discussed in more detail below. path. Results variability will be discussed in more detail below.
5.4. IP-layer Utilization 6.4. IP-layer Utilization
The ideal metric definition of Link Utilization [RFC5136] is based on The ideal metric definition of Link Utilization [RFC5136] is based on
the actual usage (bits successfully received during a time interval) the actual usage (bits successfully received during a time interval)
and the Maximum Capacity for the same interval. and the Maximum Capacity for the same interval.
In practice, Link Utilization can be calculated by counting the IP- In practice, Link Utilization can be calculated by counting the IP-
layer (or other layer) octets received over a time interval and layer (or other layer) octets received over a time interval and
dividing by the theoretical maximum of octets that could have been dividing by the theoretical maximum of octets that could have been
delivered in the same interval. A commonly used time interval is 5 delivered in the same interval. A commonly used time interval is 5
minutes, and this interval has been sufficient to support network minutes, and this interval has been sufficient to support network
skipping to change at page 14, line 21 skipping to change at page 17, line 30
single (average) Utilization value for each 5 minute interval. Some single (average) Utilization value for each 5 minute interval. Some
performance management systems have begun to make 1 minute averages performance management systems have begun to make 1 minute averages
available. available.
There is also a limit on the smallest useful measurement interval. There is also a limit on the smallest useful measurement interval.
Intervals on the order of the serialization time for a single Maximum Intervals on the order of the serialization time for a single Maximum
Transmission Unit (MTU) packet will observe on/off behavior and Transmission Unit (MTU) packet will observe on/off behavior and
report 100% or 0%. The smallest interval needs to be some multiple report 100% or 0%. The smallest interval needs to be some multiple
of MTU serialization time for averaging to be effective. of MTU serialization time for averaging to be effective.
5.5. IP-layer Available Capacity 6.5. IP-layer Available Capacity
The Available Capacity of a link can be calculated using the Capacity The Available Capacity of a link can be calculated using the Capacity
and Utilization metrics. and Utilization metrics.
When Available capacity of a link or path is estimated through some When Available capacity of a link or path is estimated through some
measurement technique, the following parameters SHOULD be reported: measurement technique, the following parameters SHOULD be reported:
o Name and reference to the exact method of measurement o Name and reference to the exact method of measurement
o IP packet length, octets (including IP header) o IP packet length, octets (including IP header)
skipping to change at page 15, line 5 skipping to change at page 18, line 12
actual Available capacity of the link or path. Therefore, it is actual Available capacity of the link or path. Therefore, it is
important to know the capacity value beyond which there will be no important to know the capacity value beyond which there will be no
measured improvement. measured improvement.
The Application Design audience may have a target capacity value and The Application Design audience may have a target capacity value and
simply wish to assess whether there is sufficient Available Capacity. simply wish to assess whether there is sufficient Available Capacity.
This case simplifies measurement of link and path capacity to some This case simplifies measurement of link and path capacity to some
degree, as long as the measurable maximum exceeds the target degree, as long as the measurable maximum exceeds the target
capacity. capacity.
5.6. Variability in Utilization and Avail. Capacity 6.6. Variability in Utilization and Avail. Capacity
As with most metrics and measurements, assessing the consistency or As with most metrics and measurements, assessing the consistency or
variability in the results gives a the user an intuitive feel for the variability in the results gives a the user an intuitive feel for the
degree (or confidence) that any one value is representative of other degree (or confidence) that any one value is representative of other
results, or the underlying distribution from which these singleton results, or the underlying distribution from which these singleton
measurements have come. measurements have come.
Two questions are raised here for further discussion: Two questions are raised here for further discussion:
What ways can Utilization be measured and summarized to describe the What ways can Utilization be measured and summarized to describe the
potential variability in a useful way? potential variability in a useful way?
How can the variability in Available Capacity estimates be reported, How can the variability in Available Capacity estimates be reported,
so that the confidence in the results is also conveyed? so that the confidence in the results is also conveyed?
6. Test Streams and Sample Size 7. Test Streams and Sample Size
This section discusses two key aspects of measurement that are This section discusses two key aspects of measurement that are
sometimes omitted from the report: the description of the test stream sometimes omitted from the report: the description of the test stream
on which the measurements are based, and the sample size. on which the measurements are based, and the sample size.
6.1. Test Stream Characteristics 7.1. Test Stream Characteristics
Network Characterization has traditionally used Poisson-distributed Network Characterization has traditionally used Poisson-distributed
inter-packet spacing, as this provides an unbiased sample. The inter-packet spacing, as this provides an unbiased sample. The
average inter-packet spacing may be selected to allow observation of average inter-packet spacing may be selected to allow observation of
specific network phenomena. Other test streams are designed to specific network phenomena. Other test streams are designed to
sample some property of the network, such as the presence of sample some property of the network, such as the presence of
congestion, link bandwidth, or packet reordering. congestion, link bandwidth, or packet reordering.
If measuring a network in order to make inferences about applications If measuring a network in order to make inferences about applications
or receiver performance, then there are usually efficiencies derived or receiver performance, then there are usually efficiencies derived
from a test stream that has similar characteristics to the sender. from a test stream that has similar characteristics to the sender.
In some cases, it is essential to synthesize the sender stream, as In some cases, it is essential to synthesize the sender stream, as
with Bulk Transfer Capacity estimates. In other cases, it may be with Bulk Transfer Capacity estimates. In other cases, it may be
sufficient to sample with a "known bias", e.g., a Periodic stream to sufficient to sample with a "known bias", e.g., a Periodic stream to
estimate real-time application performance. estimate real-time application performance.
6.2. Sample Size 7.2. Sample Size
Sample size is directly related to the accuracy of the results, and Sample size is directly related to the accuracy of the results, and
plays a critical role in the report. Even if only the sample size plays a critical role in the report. Even if only the sample size
(in terms of number of packets) is given for each value or summary (in terms of number of packets) is given for each value or summary
statistic, it imparts a notion of the confidence in the result. statistic, it imparts a notion of the confidence in the result.
In practice, the sample size will be selected taking both statistical In practice, the sample size will be selected taking both statistical
and practical factors into account. Among these factors are: and practical factors into account. Among these factors are:
1. The estimated variability of the quantity being measured 1. The estimated variability of the quantity being measured
skipping to change at page 16, line 24 skipping to change at page 19, line 33
4. etc. 4. etc.
A sample size may sometimes be referred to as "large". This is a A sample size may sometimes be referred to as "large". This is a
relative, and qualitative term. It is preferable to describe what relative, and qualitative term. It is preferable to describe what
one is attempting to achieve with their sample. For example, stating one is attempting to achieve with their sample. For example, stating
an implication may be helpful: this sample is large enough such that an implication may be helpful: this sample is large enough such that
a single outlying value at ten times the "typical" sample mean (the a single outlying value at ten times the "typical" sample mean (the
mean without the outlying value) would influence the mean by no more mean without the outlying value) would influence the mean by no more
than X. than X.
7. Reporting Results
This section gives an overview of recommendations, followed by
additional considerations for reporting results in the "long-term".
7.1. Overview of Metric Statistics
This section gives an overview of reporting recommendations for the
loss, delay, and delay variation metrics based on the discussion and
conclusions of the preceding sections.
The minimal report on measurements MUST include both Loss and Delay
Metrics.
For Packet Loss, the loss ratio defined in [RFC2680] is a sufficient
starting point, especially the guidance for setting the loss
threshold waiting time. We have calculated a waiting time above that
should be sufficient to differentiate between packets that are truly
lost or have long finite delays under general measurement
circumstances, 51 seconds. Knowledge of specific conditions can help
to reduce this threshold, but 51 seconds is considered to be
manageable in practice.
We note that a loss ratio calculated according to [Y.1540] would
exclude errored packets form the numerator. In practice, the
difference between these two loss metrics is small if any, depending
on whether the last link prior to the destination contributes errored
packets.
For Packet Delay, we recommend providing both the mean delay and the
median delay with lost packets designated undefined (as permitted by
[RFC2679]). Both statistics are based on a conditional distribution,
and the condition is packet arrival prior to a waiting time dT, where
dT has been set to take maximum packet lifetimes into account, as
discussed above. Using a long dT helps to ensure that delay
distributions are not truncated.
For Packet Delay Variation (PDV), the minimum delay of the
conditional distribution should be used as the reference delay for
computing PDV according to [Y.1540] or [RFC3393]. A useful value to
report is a pseudo range of delay variation based on calculating the
difference between a high percentile of delay and the minimum delay.
For example, the 99.9%-ile minus the minimum will give a value that
can be compared with objectives in [Y.1541].
7.2. Long-Term Reporting Considerations
[I-D.ietf-ippm-reporting] describes methods to conduct measurements
and report the results on a near-immediate time scale (10 seconds,
which we consider to be "short-term").
Measurement intervals and reporting intervals need not be the same
length. Sometimes, the user is only concerned with the performance
levels achieved over a relatively long interval of time (e.g, days,
weeks, or months, as opposed to 10 seconds). However, there can be
risks involved with running a measurement continuously over a long
period without recording intermediate results:
o Temporary power failure may cause loss of all the results to date.
o Measurement system timing synchronization signals may experience a
temporary outage, causing sub-sets of measurements to be in error
or invalid.
o Maintenance may be necessary on the measurement system, or its
connectivity to the network under test.
For these and other reasons, such as
o the constraint to collect measurements on intervals similar to
user session length, or
o the dual-use of measurements in monitoring activities where
results are needed on a period of a few minutes,
there is value in conducting measurements on intervals that are much
shorter than the reporting interval.
There are several approaches for aggregating a series of measurement
results over time in order to make a statement about the longer
reporting interval. One approach requires the storage of all metric
singletons collected throughout the reporting interval, even though
the measurement interval stops and starts many times.
Another approach is described in [I-D.ietf-ippm-framework-compagg] as
"temporal aggregation". This approach would estimate the results for
the reporting interval based on many individual measurement interval
statistics (results) alone. The result would ideally appear in the
same form as though a continuous measurement was conducted. A memo
to address the details of temporal aggregation is yet to be prepared.
Yet another approach requires a numerical objective for the metric,
and the results of each measurement interval are compared with the
objective. Every measurement interval where the results meet the
objective contribute to the fraction of time with performance as
specified. When the reporting interval contains many measurement
intervals it is possible to present the results as "metric A was less
than or equal to objective X during Y% of time.
NOTE that numerical thresholds are not set in IETF performance work
and are explicitly excluded from the IPPM charter.
8. IANA Considerations 8. IANA Considerations
This document makes no request of IANA. This document makes no request of IANA.
Note to RFC Editor: this section may be removed on publication as an Note to RFC Editor: this section may be removed on publication as an
RFC. RFC.
9. Security Considerations 9. Security Considerations
The security considerations that apply to any active measurement of The security considerations that apply to any active measurement of
live networks are relevant here as well. See [RFC4656]. live networks are relevant here as well. See [RFC4656].
10. Acknowledgements 10. Acknowledgements
The authors would like to thank Phil Chimento for his suggestion to The authors thank: Phil Chimento for his suggestion to employ
employ conditional distributions for Delay, and Steve Konish Jr. for conditional distributions for Delay, Steve Konish Jr. for his careful
his careful review and suggestions. review and suggestions, Dave Mcdysan and Don McLachlan for useful
comments based on their long experience with measurement and
reporting.
11. References 11. References
11.1. Normative References 11.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
May 1998. May 1998.
[RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring
Connectivity", RFC 2678, September 1999.
[RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Delay Metric for IPPM", RFC 2679, September 1999. Delay Metric for IPPM", RFC 2679, September 1999.
[RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Packet Loss Metric for IPPM", RFC 2680, September 1999. Packet Loss Metric for IPPM", RFC 2680, September 1999.
[RFC3148] Mathis, M. and M. Allman, "A Framework for Defining [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining
Empirical Bulk Transfer Capacity Metrics", RFC 3148, Empirical Bulk Transfer Capacity Metrics", RFC 3148,
July 2001. July 2001.
[RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation
Metric for IP Performance Metrics (IPPM)", RFC 3393, Metric for IP Performance Metrics (IPPM)", RFC 3393,
November 2002. November 2002.
[RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network
performance measurement with periodic streams", RFC 3432,
November 2002.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
Zekauskas, "A One-way Active Measurement Protocol Zekauskas, "A One-way Active Measurement Protocol
(OWAMP)", RFC 4656, September 2006. (OWAMP)", RFC 4656, September 2006.
[RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
S., and J. Perser, "Packet Reordering Metrics", RFC 4737, S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
November 2006. November 2006.
[RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", [RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity",
RFC 5136, February 2008. RFC 5136, February 2008.
11.2. Informative References 11.2. Informative References
[Casner] "A Fine-Grained View of High Performance Networking, NANOG [Casner] "A Fine-Grained View of High Performance Networking, NANOG
22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May 22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May
20-22 2001. 20-22 2001.
[Cia03] "Standardized Active Measurements on a Tier 1 IP Backbone, [Cia03] "Standardized Active Measurements on a Tier 1 IP Backbone,
IEEE Communications Mag., pp 90-97.", June 2003. IEEE Communications Mag., pp 90-97.", June 2003.
[I-D.ietf-ippm-framework-compagg]
Morton, A., "Framework for Metric Composition",
draft-ietf-ippm-framework-compagg-09 (work in progress),
December 2009.
[I-D.ietf-ippm-reporting] [I-D.ietf-ippm-reporting]
Shalunov, S. and M. Swany, "Reporting IP Performance Shalunov, S. and M. Swany, "Reporting IP Performance
Metrics to Users", draft-ietf-ippm-reporting-04 (work in Metrics to Users", draft-ietf-ippm-reporting-04 (work in
progress), July 2009. progress), July 2009.
[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
Applicability Statement", RFC 5481, March 2009.
[RFC5835] Morton, A. and S. Van den Berghe, "Framework for Metric
Composition", RFC 5835, April 2010.
[Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data [Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data
communication service - IP packet transfer and communication service - IP packet transfer and
availability performance parameters", December 2002. availability performance parameters", December 2002.
[Y.1541] ITU-T Recommendation Y.1540, "Network Performance [Y.1541] ITU-T Recommendation Y.1540, "Network Performance
Objectives for IP-Based Services", February 2006. Objectives for IP-Based Services", February 2006.
Authors' Addresses Authors' Addresses
Al Morton Al Morton
 End of changes. 42 change blocks. 
192 lines changed or deleted 231 lines changed or added

This html diff was produced by rfcdiff 1.38. The latest version is available from http://tools.ietf.org/tools/rfcdiff/