draft-ietf-ippm-reporting-metrics-09.txt   rfc6703.txt 
Network Working Group A. Morton Internet Engineering Task Force (IETF) A. Morton
Internet-Draft G. Ramachandran Request for Comments: 6703 G. Ramachandran
Intended status: Informational G. Maguluri Category: Informational G. Maguluri
Expires: November 11, 2012 AT&T Labs ISSN: 2070-1721 AT&T Labs
May 10, 2012 August 2012
Reporting IP Network Performance Metrics: Different Points of View Reporting IP Network Performance Metrics: Different Points of View
draft-ietf-ippm-reporting-metrics-09
Abstract Abstract
Consumers of IP network performance metrics have many different uses Consumers of IP network performance metrics have many different uses
in mind. The memo provides "long-term" reporting considerations in mind. This memo provides "long-term" reporting considerations
(e.g., hours, days, weeks or months, as opposed to 10 seconds), based (e.g., hours, days, weeks, or months, as opposed to 10 seconds),
on analysis of the two key audience points-of-view. It describes how based on analysis of the points of view of two key audiences. It
the audience categories affect the selection of metric parameters and describes how these audience categories affect the selection of
options when seeking info that serves their needs. metric parameters and options when seeking information that serves
their needs.
Status of this Memo
This Internet-Draft is submitted in full conformance with the Status of This Memo
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering This document is not an Internet Standards Track specification; it is
Task Force (IETF). Note that other groups may also distribute published for informational purposes.
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 5741.
This Internet-Draft will expire on November 11, 2012. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc6703.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 7 skipping to change at page 3, line 7
modifications of such material outside the IETF Standards Process. modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other it for publication as an RFC or to translate it into languages other
than English. than English.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Introduction ....................................................4
2. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 4 2. Purpose and Scope ...............................................4
3. Reporting Results . . . . . . . . . . . . . . . . . . . . . . 5 3. Reporting Results ...............................................5
3.1. Overview of Metric Statistics . . . . . . . . . . . . . . 5 3.1. Overview of Metric Statistics ..............................5
3.2. Long-Term Reporting Considerations . . . . . . . . . . . . 6 3.2. Long-Term Reporting Considerations .........................6
4. Effect of POV on the Loss Metric . . . . . . . . . . . . . . . 8 4. Effect of POV on the Loss Metric ................................8
4.1. Loss Threshold . . . . . . . . . . . . . . . . . . . . . . 8 4.1. Loss Threshold .............................................8
4.1.1. Network Characterization . . . . . . . . . . . . . . . 8 4.1.1. Network Characterization ............................8
4.1.2. Application Performance . . . . . . . . . . . . . . . 11 4.1.2. Application Performance ............................11
4.2. Errored Packet Designation . . . . . . . . . . . . . . . . 11 4.2. Errored Packet Designation ................................11
4.3. Causes of Lost Packets . . . . . . . . . . . . . . . . . . 11 4.3. Causes of Lost Packets ....................................11
4.4. Summary for Loss . . . . . . . . . . . . . . . . . . . . . 12 4.4. Summary for Loss ..........................................12
5. Effect of POV on the Delay Metric . . . . . . . . . . . . . . 12 5. Effect of POV on the Delay Metric ..............................12
5.1. Treatment of Lost Packets . . . . . . . . . . . . . . . . 12 5.1. Treatment of Lost Packets .................................12
5.1.1. Application Performance . . . . . . . . . . . . . . . 13 5.1.1. Application Performance ............................13
5.1.2. Network Characterization . . . . . . . . . . . . . . . 13 5.1.2. Network Characterization ...........................13
5.1.3. Delay Variation . . . . . . . . . . . . . . . . . . . 14 5.1.3. Delay Variation ....................................14
5.1.4. Reordering . . . . . . . . . . . . . . . . . . . . . . 15 5.1.4. Reordering .........................................15
5.2. Preferred Statistics . . . . . . . . . . . . . . . . . . . 15 5.2. Preferred Statistics ......................................15
5.3. Summary for Delay . . . . . . . . . . . . . . . . . . . . 16 5.3. Summary for Delay .........................................16
6. Reporting Raw Capacity Metrics . . . . . . . . . . . . . . . . 16 6. Reporting Raw Capacity Metrics .................................16
6.1. Type-P Parameter . . . . . . . . . . . . . . . . . . . . . 17 6.1. Type-P Parameter ..........................................17
6.2. A priori Factors . . . . . . . . . . . . . . . . . . . . . 17 6.2. A priori Factors ..........................................17
6.3. IP-layer Capacity . . . . . . . . . . . . . . . . . . . . 17 6.3. IP-Layer Capacity .........................................17
6.4. IP-layer Utilization . . . . . . . . . . . . . . . . . . . 18 6.4. IP-Layer Utilization ......................................18
6.5. IP-layer Available Capacity . . . . . . . . . . . . . . . 18 6.5. IP-Layer Available Capacity ...............................18
6.6. Variability in Utilization and Available Capacity . . . . 19 6.6. Variability in Utilization and Available Capacity .........19
6.6.1. General Summary of Variability . . . . . . . . . . . . 19 6.6.1. General Summary of Variability .....................19
7. Reporting Restricted Capacity Metrics . . . . . . . . . . . . 20 7. Reporting Restricted Capacity Metrics ..........................20
7.1. Type-P Parameter and Type-C Parameter . . . . . . . . . . 21 7.1. Type-P Parameter and Type-C Parameter .....................21
7.2. A priori Factors . . . . . . . . . . . . . . . . . . . . . 21 7.2. A Priori Factors ..........................................21
7.3. Measurement Interval . . . . . . . . . . . . . . . . . . . 21 7.3. Measurement Interval ......................................22
7.4. Bulk Transfer Capacity Reporting . . . . . . . . . . . . . 22 7.4. Bulk Transfer Capacity Reporting ..........................22
7.5. Variability in Bulk Transfer Capacity . . . . . . . . . . 23 7.5. Variability in Bulk Transfer Capacity .....................23
8. Reporting on Test Streams and Sample Size . . . . . . . . . . 23 8. Reporting on Test Streams and Sample Size ......................23
8.1. Test Stream Characteristics . . . . . . . . . . . . . . . 23 8.1. Test Stream Characteristics ...............................23
8.2. Sample Size . . . . . . . . . . . . . . . . . . . . . . . 24 8.2. Sample Size ...............................................24
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 9. Security Considerations ........................................25
10. Security Considerations . . . . . . . . . . . . . . . . . . . 24 10. Acknowledgements ..............................................25
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 25 11. References ....................................................25
12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25 11.1. Normative References .....................................25
12.1. Normative References . . . . . . . . . . . . . . . . . . . 25 11.2. Informative References ...................................26
12.2. Informative References . . . . . . . . . . . . . . . . . . 26
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26
1. Introduction 1. Introduction
When designing measurements of IP networks and presenting a result, When designing measurements of IP networks and presenting a result,
knowledge of the audience is a key consideration. To present a knowledge of the audience is a key consideration. To present a
useful and relevant portrait of network conditions, one must answer useful and relevant portrait of network conditions, one must answer
the following question: the following question:
"How will the results be used?" "How will the results be used?"
There are two main audience categories for the report of results: There are two main audience categories for the report of results:
1. Network Characterization - describes conditions in an IP network 1. Network Characterization - describes conditions in an IP network
for quality assurance, troubleshooting, modeling, Service Level for quality assurance, troubleshooting, modeling, Service Level
Agreements (SLA), etc. The point-of-view looks inward toward the Agreements (SLAs), etc. This point of view (POV) looks inward
network where the report consumer intends their actions. toward the network where the report consumer intends their
actions.
2. Application Performance Estimation - describes the network 2. Application Performance Estimation - describes the network
conditions in a way that facilitates determining effects on user conditions in a way that facilitates determining effects on user
applications, and ultimately the users themselves. This point- applications, and ultimately the users themselves. This POV
of-view looks outward, toward the user(s), accepting the network looks outward, toward the user(s), accepting the network as is.
as-is. This report consumer intends to estimate a network- This report consumer intends to estimate a network-dependent
dependent aspect of performance, or design some aspect of an aspect of performance or design some aspect of an application's
application's accommodation of the network. (These are *not* accommodation of the network. (These are *not* application
application metrics, they are defined at the IP layer.) metrics; they are defined at the IP layer.)
This memo considers how these different points-of-view affect both This memo considers how these different POVs affect both the
the measurement design (parameters and options of the metrics) and measurement design (parameters and options of the metrics) and
statistics reported when serving their needs. statistics reported when serving the report consumer's needs.
The IPPM framework [RFC2330] and other RFCs describing IPPM metrics The IP Performance Metrics (IPPM) Framework [RFC2330] and other RFCs
provide a background for this memo. describing IPPM provide a background for this memo.
2. Purpose and Scope 2. Purpose and Scope
The purpose of this memo is to clearly delineate two points-of-view The purpose of this memo is to clearly delineate two POVs for using
(POV) for using measurements, and describe their effects on the test measurements and describe their effects on the test design, including
design, including the selection of metric parameters and reporting the selection of metric parameters and reporting the results.
the results.
The scope of this memo primarily covers the design and reporting of The scope of this memo primarily covers the test design and reporting
the loss and delay metrics [RFC2680] [RFC2679]. It will also discuss of the loss and delay metrics [RFC2680] [RFC2679]. It will also
the delay variation [RFC3393] and reordering metrics [RFC4737] where discuss the delay variation [RFC3393] and reordering metrics
applicable. [RFC4737] where applicable.
With capacity metrics growing in relevance to the industry, the memo With capacity metrics growing in relevance to the industry, the memo
also covers POV and reporting considerations for metrics resulting also covers POV and reporting considerations for metrics resulting
from the Bulk Transfer Capacity Framework [RFC3148] and Network from the Bulk Transfer Capacity Framework [RFC3148] and Network
Capacity Definitions [RFC5136]. These memos effectively describe two Capacity Definitions [RFC5136]. These memos effectively describe two
different categories of metrics, different categories of metrics:
o Restricted: [RFC3148] includes restrictions of congestion control o Restricted [RFC3148]: includes restrictions of congestion control
and the notion of unique data bits delivered, and and the notion of unique data bits delivered, and
o Raw: [RFC5136] using a definition of raw capacity without the o Raw [RFC5136]: uses a definition of raw capacity without the
restrictions of data uniqueness or congestion awareness. restrictions of data uniqueness or congestion awareness.
It might seem at first glance that each of these metrics has an It might seem, at first glance, that each of these metrics has an
obvious audience (Raw = Network Characterization, Restricted = obvious audience (raw = network characterization, restricted =
Application Performance), but reality is more complex and consistent application performance), but reality is more complex and consistent
with the overall topic of capacity measurement and reporting. For with the overall topic of capacity measurement and reporting. For
example, TCP is usually used in Restricted capacity measurement example, TCP is usually used in restricted capacity measurement
methods, while UDP appears in Raw capacity measurement. The Raw and methods, while UDP appears in raw capacity measurement. The raw and
Restricted capacity metrics will be treated in separate sections, restricted capacity metrics will be treated in separate sections,
although they share one common reporting issue: representing although they share one common reporting issue: representing
variability in capacity metric results as part of a long-term report. variability in capacity metric results as part of a long-term report.
Sampling, or the design of the active packet stream that is the basis Sampling, or the design of the active packet stream that is the basis
for the measurements, is also discussed. for the measurements, is also discussed.
3. Reporting Results 3. Reporting Results
This section gives an overview of recommendations, followed by This section gives an overview of recommendations, followed by
additional considerations for reporting results in the "long term", additional considerations for reporting results in the "long term",
based on the discussion and conclusions of the major sections that based on the discussion and conclusions of the major sections that
follow. follow.
3.1. Overview of Metric Statistics 3.1. Overview of Metric Statistics
This section gives an overview of reporting recommendations for the This section gives an overview of reporting recommendations for all
loss, delay, and delay variation metrics. the metrics considered in this memo.
The minimal report on measurements must include both Loss and Delay The minimal report on measurements must include both loss and delay
Metrics. metrics.
For Packet Loss, the loss ratio defined in [RFC2680] is a sufficient For packet loss, the loss ratio defined in [RFC2680] is a sufficient
starting point, especially the existing guidance for setting the loss starting point -- especially the existing guidance for setting the
threshold waiting time. We have calculated a waiting time in section loss threshold waiting time. In Section 4.1.1, we have calculated a
4.1.1 that should be sufficient to differentiate between packets that waiting time -- 51 seconds -- that should be sufficient to
are truly lost or have long finite delays under general measurement differentiate between packets that are truly lost or have long finite
circumstances, 51 seconds. Knowledge of specific conditions can help delays under general measurement circumstances. Knowledge of
to reduce this threshold, and a waiting time of approximately 50 specific conditions can help to reduce this threshold, and a waiting
seconds is considered to be manageable in practice. time of approximately 50 seconds is considered to be manageable in
practice.
We note that a loss ratio calculated according to [Y.1540] would We note that a loss ratio calculated according to [Y.1540] would
exclude errored packets from the numerator. In practice, the exclude errored packets from the numerator. In practice, the
difference between these two loss metrics is small if any, depending difference between these two loss metrics is small, if any, depending
on whether the last link prior to the destination contributes errored on whether the last link prior to the Destination contributes errored
packets. packets.
For Packet Delay, we recommend providing both the mean delay and the For packet delay, we recommend providing both the mean delay and the
median delay with lost packets designated undefined (as permitted by median delay with lost packets designated as undefined (as permitted
[RFC2679]). Both statistics are based on a conditional distribution, by [RFC2679]). Both statistics are based on a conditional
and the condition is packet arrival prior to a waiting time dT, where distribution, and the condition is packet arrival prior to a waiting
dT has been set to take maximum packet lifetimes into account, as time dT, where dT has been set to take maximum packet lifetimes into
discussed above for loss. Using a long dT helps to ensure that delay account, as discussed above for loss. Using a long dT helps to
distributions are not truncated. ensure that delay distributions are not truncated.
For Packet Delay Variation (PDV), the minimum delay of the For Packet Delay Variation (PDV), the minimum delay of the
conditional distribution should be used as the reference delay for conditional distribution should be used as the reference delay for
computing PDV according to [Y.1540] or [RFC5481] and [RFC3393]. A computing PDV according to [Y.1540] or [RFC5481] and [RFC3393]. A
useful value to report is a pseudo range of delay variation based on useful value to report is a "pseudo" range of delay variation based
calculating the difference between a high percentile of delay and the on calculating the difference between a high percentile of delay and
minimum delay. For example, the 99.9 percentile minus the minimum the minimum delay. For example, the 99.9th percentile minus the
will give a value that can be compared with objectives in [Y.1541]. minimum will give a value that can be compared with objectives in
[Y.1541].
For Capacity, both Raw and Restricted, reporting the variability in a For both raw capacity and restricted capacity, reporting the
useful way is identified as the main challenge. The Min, Max, and variability in a useful way is identified as the main challenge. The
Range statistics are suggested along with a ratio of Max to Min and min, max, and range statistics are suggested along with a ratio of
moving averages. In the end, a simple plot of the singleton results max to min and moving averages. In the end, a simple plot of the
over time may succeed where summary metrics fail, or serve to confirm singleton results over time may succeed where summary metrics fail or
that the summaries are valid. may serve to confirm that the summaries are valid.
3.2. Long-Term Reporting Considerations 3.2. Long-Term Reporting Considerations
[I-D.ietf-ippm-reporting] describes methods to conduct measurements [IPPM-RPT] describes methods to conduct measurements and report the
and report the results on a near-immediate time scale (10 seconds, results on a near-immediate time scale (10 seconds, which we consider
which we consider to be "short-term"). to be "short-term").
Measurement intervals and reporting intervals need not be the same Measurement intervals and reporting intervals need not be the same
length. Sometimes, the user is only concerned with the performance length. Sometimes, the user is only concerned with the performance
levels achieved over a relatively long interval of time (e.g., days, levels achieved over a relatively long interval of time (e.g., days,
weeks, or months, as opposed to 10 seconds). However, there can be weeks, or months, as opposed to 10 seconds). However, there can be
risks involved with running a measurement continuously over a long risks involved with running a measurement continuously over a long
period without recording intermediate results: period without recording intermediate results:
o Temporary power failure may cause loss of all the results to date. o Temporary power failure may cause loss of all results to date.
o Measurement system timing synchronization signals may experience a o Measurement system timing synchronization signals may experience a
temporary outage, causing sub-sets of measurements to be in error temporary outage, causing subsets of measurements to be in error
or invalid. or invalid.
o Maintenance may be necessary on the measurement system, or its o Maintenance on the measurement system or on its connectivity to
connectivity to the network under test. the network under test may be necessary.
For these and other reasons, such as For these and other reasons, such as
o the constraint to collect measurements on intervals similar to o the constraint to collect measurements on intervals similar to
user session length, user session length,
o the dual-use of measurements in monitoring activities where o the dual use of measurements in monitoring activities where
results are needed on a period of a few minutes, or results are needed on a period of a few minutes, or
o the ability to inspect results of a single measurement interval o the ability to inspect results of a single measurement interval
for deeper analysis, for deeper analysis,
there is value in conducting measurements on intervals that are much there is value in conducting measurements on intervals that are much
shorter than the reporting interval. shorter than the reporting interval.
There are several approaches for aggregating a series of measurement There are several approaches for aggregating a series of measurement
results over time in order to make a statement about the longer results over time in order to make a statement about the longer
reporting interval. One approach requires the storage of all metric reporting interval. One approach requires the storage of all metric
singletons collected throughout the reporting interval, even though singletons collected throughout the reporting interval, even though
the measurement interval stops and starts many times. the measurement interval stops and starts many times.
Another approach is described in [RFC5835] as "temporal aggregation". Another approach is described in [RFC5835] as "temporal aggregation".
This approach would estimate the results for the reporting interval This approach would estimate the results for the reporting interval
based on many individual measurement interval statistics (results) based on combining many individual short-term measurement interval
alone. The result would ideally appear in the same form as though a statistics to yield a long-term result. The result would ideally
continuous measurement had been conducted. A memo addressing the appear in the same form as though a continuous measurement had been
details of temporal aggregation is yet to be prepared. conducted. A memo addressing the details of temporal aggregation is
yet to be prepared.
Yet another approach requires a numerical objective for the metric, Yet another approach requires a numerical objective for the metric,
and the results of each measurement interval are compared with the and the results of each measurement interval are compared with the
objective. Every measurement interval where the results meet the objective. Every measurement interval where the results meet the
objective contribute to the fraction of time with performance as objective contribute to the fraction of time with performance as
specified. When the reporting interval contains many measurement specified. When the reporting interval contains many measurement
intervals it is possible to present the results as "metric A was less intervals, it is possible to present the results as "metric A was
than or equal to objective X during Y% of time". less than or equal to objective X during Y% of time".
NOTE that numerical thresholds of acceptability are not set in IETF NOTE that numerical thresholds of acceptability are not set in
performance work and are therefore excluded from the scope of this IETF performance work and are therefore excluded from the scope of
memo. this memo.
In all measurement, it is important to avoid unintended In all measurements, it is important to avoid unintended
synchronization with network events. This topic is treated in synchronization with network events. This topic is treated in
[RFC2330] for Poisson-distributed inter-packet time streams, and [RFC2330] for Poisson-distributed inter-packet time streams and in
[RFC3432] for Periodic streams. Both avoid synchronization through [RFC3432] for Periodic streams. Both avoid synchronization by using
use of random start times. random start times.
There are network conditions where it is simply more useful to report There are network conditions where it is simply more useful to report
the connectivity status of the Source-Destination path, and to the connectivity status of the Source-Destination path, and to
distinguish time intervals where connectivity can be demonstrated distinguish time intervals where connectivity can be demonstrated
from other time intervals (where connectivity does not appear to from other time intervals (where connectivity does not appear to
exist). [RFC2678] specifies a number of one-way and two-way exist). [RFC2678] specifies a number of one-way and two-way
connectivity metrics of increasing complexity. In this memo, we connectivity metrics of increasing complexity. In this memo, we
RECOMMEND that long term reporting of loss, delay, and other metrics recommend that long-term reporting of loss, delay, and other metrics
be limited to time intervals where connectivity can be demonstrated, be limited to time intervals where connectivity can be demonstrated,
and other intervals be summarized as percent of time where and that other intervals be summarized as the percent of time where
connectivity does not appear to exist. We note that this same connectivity does not appear to exist. We note that this same
approach has been adopted in ITU-T Recommendation [Y.1540] where approach has been adopted in ITU-T Recommendation [Y.1540] where
performance parameters are only valid during periods of service performance parameters are only valid during periods of service
"availability" (evaluated according to a function based on packet "availability" (evaluated according to a function based on packet
loss, and sustained periods of loss ratio greater than a threshold loss, and sustained periods of loss ratio greater than a threshold
are declared "unavailable"). are declared "unavailable").
4. Effect of POV on the Loss Metric 4. Effect of POV on the Loss Metric
This section describes the ways in which the Loss metric can be tuned This section describes the ways in which the loss metric can be tuned
to reflect the preferences of the two audience categories, or to reflect the preferences of the two audience categories, or
different POV. The waiting time to declare a packet lost, or loss different POVs. The waiting time before declaring that a packet is
threshold is one area where there would appear to be a difference, lost -- the loss threshold -- is one area where there would appear to
but the ability to post-process the results may resolve it. be a difference, but the ability to post-process the results may
resolve it.
4.1. Loss Threshold 4.1. Loss Threshold
RFC 2680 [RFC2680] defines the concept of a waiting time for packets RFC 2680 [RFC2680] defines the concept of a waiting time for packets
to arrive, beyond which they are declared lost. The text of the RFC to arrive, beyond which they are declared lost. The text of the RFC
declines to recommend a value, instead saying that "good engineering, declines to recommend a value, instead saying that "good engineering,
including an understanding of packet lifetimes, will be needed in including an understanding of packet lifetimes, will be needed in
practice." Later, in the methodology, they give reasons for waiting practice". Later, in the methodology, they give reasons for waiting
"a reasonable period of time", and leaving the definition of "a reasonable period of time" and leave the definition of
"reasonable" intentionally vague. We estimate a practical bound on "reasonable" intentionally vague. Below, we estimate a practical
waiting time below. bound on waiting time.
4.1.1. Network Characterization 4.1.1. Network Characterization
Practical measurement experience has shown that unusual network Practical measurement experience has shown that unusual network
circumstances can cause long delays. One such circumstance is when circumstances can cause long delays. One such circumstance is when
routing loops form during IGP re-convergence following a failure or routing loops form during IGP re-convergence following a failure or
drastic link cost change. Packets will loop between two routers drastic link cost change. Packets will loop between two routers
until new routes are installed, or until the IPv4 Time-to-Live (TTL) until new routes are installed or until the IPv4 Time-to-Live (TTL)
field (or the IPv6 Hop Limit) decrements to zero. Very long delays field (or the IPv6 Hop Limit) decrements to zero. Very long delays
on the order of several seconds have been measured [Casner] [Cia03]. on the order of several seconds have been measured [Casner] [Cia03].
Therefore, network characterization activities prefer a long waiting Therefore, network characterization activities prefer a long waiting
time in order to distinguish these events from other causes of loss time in order to distinguish these events from other causes of loss
(such as packet discard at a full queue, or tail drop). This way, (such as packet discard at a full queue, or tail drop). This way,
the metric design helps to distinguish more reliably between packets the metric design helps to distinguish more reliably between packets
that might yet arrive, and those that are no longer traversing the that might yet arrive and those that are no longer traversing the
network. network.
It is possible to calculate a worst-case waiting time, assuming that It is possible to calculate a worst-case waiting time, assuming that
a routing loop is the cause. We model the path between Source and a routing loop is the cause. We model the path between Source and
Destination as a series of delays in links (t) and queues (q), as Destination as a series of delays in links (t) and queues (q), as
these are the dominant contributors to delay (in active measurement, these are the dominant contributors to delay (in active measurement,
the Source and Destination hosts contribute minimal delay). The the Source and Destination hosts contribute minimal delay). The
normal path delay, D, across n queues (where TTL is decremented at a normal path delay, D, across n queues (where TTL is decremented at a
node with a queue) and n+1 links without encountering a loop, is node with a queue) and n+1 links without encountering a loop, is
Path model with n=5 Path model with n=5
Source --- q1 --- q2 --- q3 --- q4 --- q5 --- Destination Source --- q1 --- q2 --- q3 --- q4 --- q5 --- Destination
t0 t1 t2 t3 t4 t5 t0 t1 t2 t3 t4 t5
n n
--- ---
\ \
D = t + > (t + q ) D = t + > (t + q)
0 / i i 0 / i i
--- ---
i = 1 i = 1
Figure 1: Normal Path Delay Figure 1: Normal Path Delay
and the time spent in the loop with L queues, is and the time spent in the loop with L queues is
Path model with n=5 and L=3 Path model with n=5 and L=3
Time in one Loop = (qx+tx + qy+ty + qz+tz) Time in one loop = (qx+tx + qy+ty + qz+tz)
qy -- qz qy -- qz
| ?/exit? | ?/exit?
qx--/\ qx--/\
Src --- q1 --- q2 ---/ q3 --- q4 --- q5 --- Dst Src --- q1 --- q2 ---/ q3 --- q4 --- q5 --- Dst
t0 t1 t2 t3 t4 t5 t0 t1 t2 t3 t4 t5
j + L-1 j + L-1
--- ---
\ (TTL - n) \ (TTL - n)
R = C > (t + q ) where C = --------- R = C > (t + q) where C = ---------
/ i i max L / i i max L
--- ---
i=j i=j
Figure 2: Delay due to Rotations in a Loop Figure 2: Delay Due to Rotations in a Loop
where n is the total number of queues in the non-loop path (with n+1 where n is the total number of queues in the non-loop path (with n+1
links), j is the queue number where the loop begins, C is the number links), j is the queue number where the loop begins, C is the number
of times a packet circles the loop, and TTL is the packet's initial of times a packet circles the loop, and TTL is the packet's initial
Time-to-Live value at the source (or Hop Count in IPv6). Time-to-Live value at the Source (or Hop Count in IPv6).
If we take the delays of all links and queues as 100ms each, the If we take the delays of all links and queues as 100 ms each, the
TTL=255, the number of queues n=5 and the queues in the loop L=4, TTL=255, the number of queues n=5, and the queues in the loop L=4,
then using C_max: then using C_max:
D = 1.1 sec and R ~= 50 sec, and D + R ~= 51.1 seconds D = 1.1 seconds and R ~= 50 seconds, and D + R ~= 51.1 seconds
We note that the link delays of 100ms would span most continents, and We note that the link delays of 100 ms would span most continents,
a constant queue length of 100ms is also very generous. When a loop and a constant queue length of 100 ms is also very generous. When a
occurs, it is almost certain to be resolved in 10 seconds or less. loop occurs, it is almost certain to be resolved in 10 seconds or
The value calculated above is an upper limit for almost any real- less. The value calculated above is an upper limit for almost any
world circumstance. real-world circumstance.
A waiting time threshold parameter, dT, set consistent with this A waiting time threshold parameter, dT, set consistent with this
calculation would not truncate the delay distribution (possibly calculation, would not truncate the delay distribution (possibly
causing a change in its mathematical properties), because the packets causing a change in its mathematical properties), because the packets
that might arrive have been given sufficient time to traverse the that might arrive have been given sufficient time to traverse the
network. network.
It is worth noting that packets that are stored and deliberately It is worth noting that packets that are stored and deliberately
forwarded at a much later time constitute a replay attack on the forwarded at a much later time constitute a replay attack on the
measurement system, and are beyond the scope of normal performance measurement system and are beyond the scope of normal performance
reporting. reporting.
4.1.2. Application Performance 4.1.2. Application Performance
Fortunately, application performance estimation activities are not Fortunately, application performance estimation activities are not
adversely affected by the long estimated limit on waiting time, adversely affected by the long estimated limit on waiting time,
because most applications will use shorter time thresholds. Although because most applications will use shorter time thresholds. Although
the designer's tendency might be to set the Loss Threshold at a value the designer's tendency might be to set the loss threshold at a value
equivalent to a particular application's threshold, this specific equivalent to a particular application's threshold, this specific
threshold can be applied when post-processing the measurements. A threshold can be applied when post-processing the measurements. A
shorter waiting time can be enforced by locating packets with delays shorter waiting time can be enforced by locating packets with delays
longer than the application's threshold, and re-designating such longer than the application's threshold and re-designating such
packets as lost. Thus, the measurement system can use a single loss packets as lost. Thus, the measurement system can use a single loss
waiting time and support both application and network performance waiting time and support both application and network performance
POVs simultaneously. POVs simultaneously.
4.2. Errored Packet Designation 4.2. Errored Packet Designation
RFC 2680 designates packets that arrive containing errors as lost RFC 2680 designates packets that arrive containing errors as lost
packets. Many packets that are corrupted by bit errors are discarded packets. Many packets that are corrupted by bit errors are discarded
within the network and do not reach their intended destination. within the network and do not reach their intended destination.
This is consistent with applications that would check the payload This is consistent with applications that would check the payload
integrity at higher layers, and discard the packet. However, some integrity at higher layers and discard the packet. However, some
applications prefer to deal with errored payloads on their own, and applications prefer to deal with errored payloads on their own, and
even a corrupted payload is better than no packet at all. even a corrupted payload is better than no packet at all.
To address this possibility, and to make network characterization To address this possibility, and to make network characterization
more complete, it is recommended to distinguish between packets that more complete, distinguishing between packets that do not arrive
do not arrive (lost) and errored packets that arrive (conditionally (lost) and errored packets that arrive (conditionally lost) is
lost). recommended.
4.3. Causes of Lost Packets 4.3. Causes of Lost Packets
Although many measurement systems use a waiting time to determine if Although many measurement systems use a waiting time to determine
a packet is lost or not, most of the waiting is in vain. The packets whether or not a packet is lost, most of the waiting is in vain. The
are no-longer traversing the network, and have not reached their packets are no longer traversing the network and have not reached
destination. their destination.
There are many causes of packet loss, including: There are many causes of packet loss, including the following:
1. Queue drop, or discard 1. Queue drop, or discard
2. Corruption of the IP header, or other essential header info 2. Corruption of the IP header, or other essential header
information
3. TTL expiration (or use of a TTL value that is too small) 3. TTL expiration (or use of a TTL value that is too small)
4. Link or router failure 4. Link or router failure
5. Layers below the source-to-destination IP layer can discard
5. Layers below the Source-to-Destination IP layer can discard
packets that fail error checking, and link-layer checksums often packets that fail error checking, and link-layer checksums often
cover the entire packet cover the entire packet
It is reasonable to consider a packet that has not arrived after a It is reasonable to consider a packet that has not arrived after a
large amount of time to be lost (due to one of the causes above) large amount of time to be lost (due to one of the causes above)
because packets do not "live forever" in the network, or have because packets do not "live forever" in the network or have infinite
infinite delay. delay.
4.4. Summary for Loss 4.4. Summary for Loss
Given that measurement post-processing is possible (even encouraged Given that measurement post-processing is possible (even encouraged
in the definitions of IPPM metrics), measurements of loss can easily in the definitions of IPPM), measurements of loss can easily serve
serve both points of view: both POVs:
o Use a long waiting time to serve network characterization and o Use a long waiting time to serve network characterization and
revise results for specific application delay thresholds as revise results for specific application delay thresholds as
needed. needed.
o Distinguish between errored packets and lost packets when possible o Distinguish between errored packets and lost packets when possible
to aid network characterization, and combine the results for to aid network characterization, and combine the results for
application performance if appropriate. application performance if appropriate.
5. Effect of POV on the Delay Metric 5. Effect of POV on the Delay Metric
This section describes the ways in which the Delay metric can be This section describes the ways in which the delay metric can be
tuned to reflect the preferences of the two consumer categories, or tuned to reflect the preferences of the two consumer categories, or
different POV. different POVs.
5.1. Treatment of Lost Packets 5.1. Treatment of Lost Packets
The Delay Metric [RFC2679] specifies the treatment of packets that do The delay metric [RFC2679] specifies the treatment of packets that do
not successfully traverse the network: their delay is undefined. not successfully traverse the network: their delay is undefined.
" >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined
(informally, infinite)<< means that Src sent the first bit of a (informally, infinite)<< means that Src sent the first bit of a
Type-P packet to Dst at wire-time T and that Dst did not receive that Type-P packet to Dst at wire-time T and that Dst did not receive
packet." that packet.
It is an accepted, but informal practice to assign infinite delay to It is an accepted but informal practice to assign infinite delay to
lost packets. We next look at how these two different treatments lost packets. We next look at how these two different treatments
align with the needs of measurement consumers who wish to align with the needs of measurement consumers who wish to
characterize networks or estimate application performance. Also, we characterize networks or estimate application performance. Also, we
look at the way that lost packets have been treated in other metrics: look at the way that lost packets have been treated in other metrics:
delay variation and reordering. delay variation and reordering.
5.1.1. Application Performance 5.1.1. Application Performance
Applications need to perform different functions, dependent on Applications need to perform different functions, dependent on
whether or not each packet arrives within some finite tolerance. In whether or not each packet arrives within some finite tolerance. In
other words, a receiver's packet processing takes only one of two other words, a receiver's packet processing takes only one of two
alternative directions (a "fork" in the road): alternative directions (a "fork" in the road):
o Packets that arrive within expected tolerance are handled by o Packets that arrive within expected tolerance are handled by
removing headers, restoring smooth delivery timing (as in a de- removing headers, restoring smooth delivery timing (as in a
jitter buffer), restoring sending order, checking for errors in de-jitter buffer), restoring sending order, checking for errors in
payloads, and many other operations. payloads, and many other operations.
o Packets that do not arrive when expected lead to attempted o Packets that do not arrive when expected lead to attempted
recovery from the apparent loss, such as retransmission requests, recovery from the apparent loss, such as retransmission requests,
loss concealment, or forward error correction to replace the loss concealment, or forward error correction to replace the
missing packet. missing packet.
So, it is important to maintain a distinction between packets that So, it is important to maintain a distinction between packets that
actually arrive, and those that do not. Therefore, it is preferable actually arrive and those that do not. Therefore, it is preferable
to leave the delay of lost packets undefined, and to characterize the to leave the delay of lost packets undefined and to characterize the
delay distribution as a conditional distribution (conditioned on delay distribution as a conditional distribution (conditioned on
arrival). arrival).
5.1.2. Network Characterization 5.1.2. Network Characterization
In this discussion, we assume that both loss and delay metrics will In this discussion, we assume that both loss and delay metrics will
be reported for network characterization (at least). be reported for network characterization (at least).
Assume packets that do not arrive are reported as Lost, usually as a Assume that packets that do not arrive are reported as lost, usually
fraction of all sent packets. If these lost packets are assigned as a fraction of all sent packets. If these lost packets are
undefined delay, then the network's inability to deliver them (in a assigned an undefined delay, then the network's inability to deliver
timely way) is relegated only in the Loss metric when we report them (in a timely way) is relegated only in the loss metric when we
statistics on the Delay distribution conditioned on the event of report statistics on the delay distribution conditioned on the event
packet arrival (within the Loss waiting time threshold). We can say of packet arrival (within the loss waiting time threshold). We can
that the Delay and Loss metrics are orthogonal, in that they convey say that the delay and loss metrics are orthogonal in that they
non-overlapping information about the network under test. This is a convey non-overlapping information about the network under test.
valuable property, whose absence is discussed below. This is a valuable property whose absence is discussed below.
However, if we assign infinite delay to all lost packets, then: However, if we assign infinite delay to all lost packets, then
o The delay metric results are influenced both by packets that o The delay metric results are influenced both by packets that
arrive and those that do not. arrive and those that do not.
o The delay singleton and the loss singleton do not appear to be o The delay singleton and the loss singleton do not appear to be
orthogonal (Delay is finite when Loss=0, Delay is infinite when orthogonal (delay is finite when loss=0; delay is infinite when
Loss=1). loss=1).
o The network is penalized in both the loss and delay metrics, o The network is penalized in both the loss and delay metrics,
effectively double-counting the lost packets. effectively double-counting the lost packets.
As further evidence of overlap, consider the Cumulative Distribution As further evidence of overlap, consider the Cumulative Distribution
Function (CDF) of Delay when the value "positive infinity" is Function (CDF) of delay when the value "positive infinity" is
assigned to all lost packets. Figure 3 shows a CDF where a small assigned to all lost packets. Figure 3 shows a CDF where a small
fraction of packets are lost. fraction of packets are lost.
1 | - - - - - - - - - - - - - - - - - -+ 1 | - - - - - - - - - - - - - - - - - -+
| | | |
| _..----'''''''''''''''''''' | _..----''''''''''''''''''''
| ,-'' | ,-''
| ,' | ,'
| / Mass at | / Mass at
| / +infinity | / +infinity
| / = fraction | / = fraction
|| lost || lost
|/ |/
0 |_____________________________________ 0 |_____________________________________
0 Delay +o0 0 Delay +o0
Figure 3: Cumulative Distribution Function for Delay when Loss = Figure 3: Cumulative Distribution Function for Delay
+Infinity When Loss = +Infinity
We note that a Delay CDF that is conditioned on packet arrival would We note that a delay CDF that is conditioned on packet arrival would
not exhibit this apparent overlap with loss. not exhibit this apparent overlap with loss.
Although infinity is a familiar mathematical concept, it is somewhat Although infinity is a familiar mathematical concept, it is somewhat
disconcerting to see any time-related metric reported as infinity. disconcerting to see any time-related metric reported as infinity.
Questions are bound to arise, and tend to detract from the goal of Questions are bound to arise and tend to detract from the goal of
informing the consumer with a performance report. informing the consumer with a performance report.
5.1.3. Delay Variation 5.1.3. Delay Variation
[RFC3393] excludes lost packets from samples, effectively assigning [RFC3393] excludes lost packets from samples, effectively assigning
an undefined delay to packets that do not arrive in a reasonable an undefined delay to packets that do not arrive in a reasonable
time. Section 4.1 of [RFC3393] describes this specification and its time. Section 4.1 of [RFC3393] describes this specification and its
rationale (ipdv = inter-packet delay variation in the quote below). rationale (ipdv = inter-packet delay variation in the quote below).
"The treatment of lost packets as having "infinite" or "undefined" The treatment of lost packets as having "infinite" or "undefined"
delay complicates the derivation of statistics for ipdv. delay complicates the derivation of statistics for ipdv.
Specifically, when packets in the measurement sequence are lost, Specifically, when packets in the measurement sequence are lost,
simple statistics such as sample mean cannot be computed. One simple statistics such as sample mean cannot be computed. One
possible approach to handling this problem is to reduce the event possible approach to handling this problem is to reduce the event
space by conditioning. That is, we consider conditional statistics; space by conditioning. That is, we consider conditional
namely we estimate the mean ipdv (or other derivative statistic) statistics; namely we estimate the mean ipdv (or other derivative
conditioned on the event that selected packet pairs arrive at the statistic) conditioned on the event that selected packet pairs
destination (within the given timeout). While this itself is not arrive at the Destination (within the given timeout). While this
without problems (what happens, for example, when every other packet itself is not without problems (what happens, for example, when
is lost), it offers a way to make some (valid) statements about ipdv, every other packet is lost), it offers a way to make some (valid)
at the same time avoiding events with undefined outcomes." statements about ipdv, at the same time avoiding events with
undefined outcomes.
We note that the argument above applies to all forms of packet delay We note that the argument above applies to all forms of packet delay
variation that can be constructed using the "selection function" variation that can be constructed using the "selection function"
concept of [RFC3393]. In recent work the two main forms of delay concept of [RFC3393]. In recent work, the two main forms of delay
variation metrics have been compared and the results are summarized variation metrics have been compared, and the results are summarized
in [RFC5481]. in [RFC5481].
5.1.4. Reordering 5.1.4. Reordering
[RFC4737] defines metrics that are based on evaluation of packet [RFC4737] defines metrics that are based on evaluation of packet
arrival order, and include a waiting time to declare a packet lost arrival order and that include a waiting time before declaring that a
(to exclude them from further processing). packet is lost (to exclude the packet from further processing).
If packets are assigned a delay value, then the reordering metric If packets are assigned a delay value, then the reordering metric
would declare any packets with infinite delay to be reordered, would declare any packets with infinite delay to be reordered,
because their sequence numbers will surely be less than the "Next because their sequence numbers will surely be less than the "Next
Expected" threshold when (or if) they arrive. But this practice Expected" threshold when (or if) they arrive. But this practice
would fail to maintain orthogonality between the reordering metric would fail to maintain orthogonality between the reordering metric
and the loss metric. Confusion can be avoided by designating the and the loss metric. Confusion can be avoided by designating the
delay of non-arriving packets as undefined, and reserving delay delay of non-arriving packets as undefined and reserving delay values
values only for packets that arrive within a sufficiently long only for packets that arrive within a sufficiently long waiting time.
waiting time.
5.2. Preferred Statistics 5.2. Preferred Statistics
Today in network characterization, the sample mean is one statistic Today in network characterization, the sample mean is one statistic
that is almost ubiquitously reported. It is easily computed and that is almost ubiquitously reported. It is easily computed and
understood by virtually everyone in this audience category. Also, understood by virtually everyone in this audience category. Also,
the sample is usually filtered on packet arrival, so that the mean is the sample is usually filtered on packet arrival, so that the mean is
based on a conditional distribution. based on a conditional distribution.
The median is another statistic that summarizes a distribution, The median is another statistic that summarizes a distribution,
having somewhat different properties from the sample mean. The having somewhat different properties from the sample mean. The
median is stable in distributions with a few outliers or without median is stable in distributions with a few outliers or without
them. However, the median's stability prevents it from indicating them. However, the median's stability prevents it from indicating
when a large fraction of the distribution changes value. 50% or more when a large fraction of the distribution changes value. 50% or more
values would need to change for the median to capture the change. values would need to change for the median to capture the change.
Both the median and sample mean have difficulty with bimodal Both the median and sample mean have difficulty with bimodal
distributions. The median will reside in only one of the modes, and distributions. The median will reside in only one of the modes, and
the mean may not lie in either mode range. For this and other the mean may not lie in either mode range. For this and other
reasons, additional statistics such as the minimum, maximum, and 95 reasons, additional statistics such as the minimum, maximum, and 95th
percentile have value when summarizing a distribution. percentile have value when summarizing a distribution.
When both the sample mean and median are available, a comparison will When both the sample mean and median are available, a comparison will
sometimes be informative, because these two statistics are equal only sometimes be informative, because these two statistics are equal only
under unusual circumstances, such as when the delay distribution is under unusual circumstances, such as when the delay distribution is
perfectly symmetrical. perfectly symmetrical.
Also, these statistics are generally useful from the Application Also, these statistics are generally useful from the application
Performance POV, so there is a common set that should satisfy performance POV, so there is a common set that should satisfy
audiences. audiences.
Plots of the delay distribution may also be useful when single-value Plots of the delay distribution may also be useful when single-value
statistics indicate that new conditions are present. An empirically- statistics indicate that new conditions are present. An empirically
derived probability distribution function will usually describe derived probability distribution function will usually describe
multiple modes more efficiently than any other form of result. multiple modes more efficiently than any other form of result.
5.3. Summary for Delay 5.3. Summary for Delay
From the perspectives of: From the perspectives of
1. application/receiver analysis, where subsequent processing 1. application/receiver analysis, where subsequent processing
depends on whether the packet arrives or times-out, depends on whether the packet arrives or times out,
2. straightforward network characterization without double-counting 2. straightforward network characterization without double-counting
defects, and defects, and
3. consistency with Delay variation and Reordering metric 3. consistency with delay variation and reordering metric
definitions, definitions,
the most efficient practice is to distinguish between packets that the most efficient practice is to distinguish between packets that
are truly lost and those that are delayed packets with a sufficiently are truly lost and those that are delayed packets with a sufficiently
long waiting time, and to designate the delay of non-arriving packets long waiting time, and to designate the delay of non-arriving packets
as undefined. as undefined.
6. Reporting Raw Capacity Metrics 6. Reporting Raw Capacity Metrics
Raw capacity refers to the metrics defined in [RFC5136] which do not Raw capacity refers to the metrics defined in [RFC5136], which do not
include restrictions such as data uniqueness or flow-control response include restrictions such as data uniqueness or flow-control response
to congestion. to congestion.
The metrics considered are IP-layer Capacity, Utilization (or used The metrics considered are IP-layer capacity, utilization (or used
capacity), and Available Capacity, for individual links and complete capacity), and available capacity, for individual links and complete
paths. These three metrics form a triad: knowing one metric paths. These three metrics form a triad: knowing one metric
constrains the other two (within their allowed range), and knowing constrains the other two (within their allowed range), and knowing
two determines the third. The link metrics have another key aspect two determines the third. The link metrics have another key aspect
in common: they are single-measurement-point metrics at the egress of in common: they are single-measurement-point metrics at the egress of
a link. The path Capacity and Available Capacity are derived by a link. The path capacity and available capacity are derived by
examining the set of single-point link measurements and taking the examining the set of single-point link measurements and taking the
minimum value. minimum value.
6.1. Type-P Parameter 6.1. Type-P Parameter
The concept of "packets of Type-P" is defined in [RFC2330]. The The concept of "packets of Type-P" is defined in [RFC2330]. The
Type-P categorization has critical relevance in all forms of capacity Type-P categorization has critical relevance in all forms of capacity
measurement and reporting. The ability to categorize packets based measurement and reporting. The ability to categorize packets based
on header fields for assignment to different queues and scheduling on header fields for assignment to different queues and scheduling
mechanisms is now common place. When un-used resources are shared mechanisms is now commonplace. When unused resources are shared
across queues, the conditions in all packet categories will affect across queues, the conditions in all packet categories will affect
capacity and related measurements. This is one source of variability capacity and related measurements. This is one source of variability
in the results that all audiences would prefer to see reported in a in the results that all audiences would prefer to see reported in a
useful and easily understood way. useful and easily understood way.
Communication of Type-P within the One-way Active Measurement Communication of Type-P within the One-Way Active Measurement
Protocol (OWAMP) and the Two-way Active Measurement Protocol (TWAMP) Protocol (OWAMP) and the Two-Way Active Measurement Protocol (TWAMP)
is essentially confined to the Diffserv Codepoint [RFC4656]. DSCP is is essentially confined to the Diffserv Code Point (DSCP) [RFC4656].
the most common qualifier for Type-P. DSCP is the most common qualifier for Type-P.
Each audience will have a set of Type-P qualifications and value Each audience will have a set of Type-P qualifications and value
combinations that are of interest. Measurements and reports should combinations that are of interest. Measurements and reports should
have the flexibility to report per-type and aggregate performance. have the flexibility to report per-type and aggregate performance.
6.2. A priori Factors 6.2. A priori Factors
The audience for Network Characterization may have detailed The audience for network characterization may have detailed
information about each link that comprises a complete path (due to information about each link that comprises a complete path (due to
ownership, for example), or some of the links in the path but not ownership, for example), or some of the links in the path but not
others, or none of the links. others, or none of the links.
There are cases where the measurement audience only has information There are cases where the measurement audience only has information
on one of the links (the local access link), and wishes to measure on one of the links (the local access link) and wishes to measure one
one or more of the raw capacity metrics. This scenario is quite or more of the raw capacity metrics. This scenario is quite common
common, and has spawned a substantial number of experimental and has spawned a substantial number of experimental measurement
measurement methods (e.g., http://www.caida.org/tools/taxonomy/ ). methods (e.g., http://www.caida.org/tools/taxonomy/). Many of these
Many of these methods respect that their users want a result fairly methods respect that their users want a result fairly quickly and in
quickly and in a one-trial. Thus, the measurement interval is kept one trial. Thus, the measurement interval is kept short (a few
short (a few seconds to a minute). For long-term reporting, a sample seconds to a minute). For long-term reporting, a sample of
of short term results need to be summarized. short-term results needs to be summarized.
6.3. IP-layer Capacity 6.3. IP-Layer Capacity
For links, this metric's theoretical maximum value can be determined For links, this metric's theoretical maximum value can be determined
from the physical layer bit rate and the bit rate reduction due to from the physical-layer bit rate and the bit rate reduction due to
the layers between the physical layer and IP. When measured, this the layers between the physical layer and IP. When measured, this
metric takes additional factors into account, such as the ability of metric takes additional factors into account, such as the ability of
the sending device to process and forward traffic under various the sending device to process and forward traffic under various
conditions. For example, the arrival of routing updates may spawn conditions. For example, the arrival of routing updates may spawn
high priority processes that reduce the sending rate temporarily. high-priority processes that reduce the sending rate temporarily.
Thus, the measured capacity of a link will be variable, and the Thus, the measured capacity of a link will be variable, and the
maximum capacity observed applies to a specific time, time interval, maximum capacity observed applies to a specific time, time interval,
and other relevant circumstances. and other relevant circumstances.
For paths composed of a series of links, it is easy to see how the For paths composed of a series of links, it is easy to see how the
sources of variability for the results grow with each link in the sources of variability for the results grow with each link in the
path. Results variability will be discussed in more detail below. path. Variability of results will be discussed in more detail below.
6.4. IP-layer Utilization 6.4. IP-Layer Utilization
The ideal metric definition of Link Utilization [RFC5136] is based on The ideal metric definition of link utilization [RFC5136] is based on
the actual usage (bits successfully received during a time interval) the actual usage (bits successfully received during a time interval)
and the Maximum Capacity for the same interval. and the maximum capacity for the same interval.
In practice, Link Utilization can be calculated by counting the IP- In practice, link utilization can be calculated by counting the
layer (or other layer) octets received over a time interval and IP-layer (or other layer) octets received over a time interval and
dividing by the theoretical maximum of octets that could have been dividing by the theoretical maximum number of octets that could have
delivered in the same interval. A commonly used time interval is 5 been delivered in the same interval. A commonly used time interval
minutes, and this interval has been sufficient to support network is 5 minutes, and this interval has been sufficient to support
operations and design for some time. 5 minutes is somewhat long network operations and design for some time. 5 minutes is somewhat
compared with the expected download time for web pages, but short long compared with the expected download time for web pages but short
with respect to large file transfers and TV program viewing. It is with respect to large file transfers and TV program viewing. It is
fair to say that considerable variability is concealed by reporting a fair to say that considerable variability is concealed by reporting a
single (average) Utilization value for each 5 minute interval. Some single (average) utilization value for each 5-minute interval. Some
performance management systems have begun to make 1 minute averages performance management systems have begun to make 1-minute averages
available. available.
There is also a limit on the smallest useful measurement interval. There is also a limit on the smallest useful measurement interval.
Intervals on the order of the serialization time for a single Maximum Intervals on the order of the serialization time for a single Maximum
Transmission Unit (MTU) packet will observe on/off behavior and Transmission Unit (MTU) packet will observe on/off behavior and
report 100% or 0%. The smallest interval needs to be some multiple report 100% or 0%. The smallest interval needs to be some multiple
of MTU serialization time for averaging to be effective. of MTU serialization time for averaging to be effective.
6.5. IP-layer Available Capacity 6.5. IP-Layer Available Capacity
The Available Capacity of a link can be calculated using the Capacity The available capacity of a link can be calculated using the capacity
and Utilization metrics. and utilization metrics.
When Available capacity of a link or path is estimated through some When available capacity of a link or path is estimated through some
measurement technique, the following parameters should be reported: measurement technique, the following parameters should be reported:
o Name and reference to the exact method of measurement o Name and reference to the exact method of measurement
o IP packet length, octets (including IP header) o IP packet length, octets (including IP header)
o Maximum Capacity that can be assessed in the measurement o Maximum capacity that can be assessed in the measurement
configuration configuration
o The time duration of the measurement o Time duration of the measurement
o All other parameters specific to the measurement method o All other parameters specific to the measurement method
Many methods of Available capacity measurement have a maximum Many methods of available capacity measurement have a maximum
capacity that they can measure, and this maximum may be less than the capacity that they can measure, and this maximum may be less than the
actual Available capacity of the link or path. Therefore, it is actual available capacity of the link or path. Therefore, it is
important to know the capacity value beyond which there will be no important to know the capacity value beyond which there will be no
measured improvement. measured improvement.
The Application Design audience may have a desired target capacity The application performance estimation audience may have a desired
value and simply wish to assess whether there is sufficient Available target capacity value and simply wish to assess whether there is
Capacity. This case simplifies measurement of link and path capacity sufficient available capacity. This case simplifies the measurement
to some degree, as long as the measurable maximum exceeds the target of link and path capacity to some degree, as long as the measurable
capacity. maximum exceeds the target capacity.
6.6. Variability in Utilization and Available Capacity 6.6. Variability in Utilization and Available Capacity
As with most metrics and measurements, assessing the consistency or As with most metrics and measurements, assessing the consistency or
variability in the results gives the user an intuitive feel for the variability in the results gives the user an intuitive feel for the
degree (or confidence) that any one value is representative of other degree (or confidence) that any one value is representative of other
results, or the spread of the underlying distribution of the results, or the spread of the underlying distribution of the
singleton measurements. singleton measurements.
How can Utilization be measured and summarized to describe the How can utilization be measured and summarized to describe the
potential variability in a useful way? potential variability in a useful way?
How can the variability in Available Capacity estimates be reported, How can the variability in available capacity estimates be reported,
so that the confidence in the results is also conveyed? so that the confidence in the results is also conveyed?
We suggest some methods below: We suggest some methods below.
6.6.1. General Summary of Variability 6.6.1. General Summary of Variability
With a set of singleton Utilization or Available Capacity estimates, With a set of singleton utilization or available capacity estimates,
each representing a time interval needed to ascertain the estimate, each representing a time interval needed to ascertain the estimate,
we seek to describe the variation over the set of singletons as we seek to describe the variation over the set of singletons as
though reporting summary statistics of a distribution. Three useful though reporting summary statistics of a distribution. Three useful
summary statistics are: summary statistics are
o Minimum, o Minimum,
o Maximum, o Maximum, and
o Range o Range
An alternate way to represent the range is as a ratio of maximum to
An alternate way to represent the Range is as ratio of Maximum to minimum value. This enables an easily understandable statistic to
Minimum value. This enables an easily understandable statistic to describe the range observed. For example, when maximum = 3*minimum,
describe the range observed. For example, when Maximum = 3*Minimum, then the max/min ratio is 3, and users may see variability of this
then the Max/Min Ratio is 3 and users may see variability of this order. On the other hand, capacity estimates with a max/min ratio
order. On the other hand, Capacity estimates with a Max/Min Ratio
near 1 are quite consistent and near the central measure or statistic near 1 are quite consistent and near the central measure or statistic
reported. reported.
For an on-going series of singleton estimates, a moving average of n For an ongoing series of singleton estimates, a moving average of n
estimates may provide a single value estimate to more easily estimates may provide a single value estimate to more easily
distinguish substantial changes in performance over time. For distinguish substantial changes in performance over time. For
example, in a window of n singletons observed in time interval, t, a example, in a window of n singletons observed in time interval t, a
percentage change of x% is declared to be a substantial change and percentage change of x% is declared to be a substantial change and
reported as an exception. reported as an exception.
Often, the most informative summary of the results is a two-axis plot Often, the most informative summary of the results is a two-axis plot
rather than a table of statistics, where time is plotted on the rather than a table of statistics, where time is plotted on the
x-axis and the singleton value on the y-axis. The time-series plot x-axis and the singleton value on the y-axis. The time-series plot
can illustrate sudden changes in an otherwise stable range, identify can illustrate sudden changes in an otherwise stable range, identify
bi-modality easily, and help quickly assess correlation with other bi-modality easily, and help quickly assess correlation with other
time-series. Plots of frequency of the singleton values are likewise time-series. Plots of frequency of the singleton values are likewise
useful tools to visualize the variation. useful tools to visualize the variation.
7. Reporting Restricted Capacity Metrics 7. Reporting Restricted Capacity Metrics
Restricted capacity refers to the metrics defined in [RFC3148] which Restricted capacity refers to the metrics defined in [RFC3148], which
include criteria of data uniqueness or flow-control response to include criteria of data uniqueness or flow-control response to
congestion. congestion.
In primary metric considered is Bulk Transfer Capacity (BTC) for One primary metric considered is Bulk Transfer Capacity (BTC) for
complete paths. [RFC3148] defines complete paths. [RFC3148] defines BTC as
BTC = data_sent / elapsed_time BTC = data_sent / elapsed_time
for a connection with congestion aware flow control, where data_sent for a connection with congestion-aware flow control, where data_sent
is the total of unique payload bits (no headers). is the total number of unique payload bits (no headers).
We note that this definition *differs* from the raw capacity We note that this definition *differs* from the raw capacity
definition in Section 2.3.1 of [RFC5136], where IP-layer Capacity definition in Section 2.3.1 of [RFC5136], where IP-layer capacity
*includes* all bits in the IP header and payload. This means that *includes* all bits in the IP header and payload. This means that
Restricted Capacity BTC is already operating at a disadvantage when restricted capacity BTC is already operating at a disadvantage when
compared to the raw capacity at layers below TCP. Further, there are compared to the raw capacity at layers below TCP. Further, there are
cases where one IP-layer is encapsulated in another IP-layer or other cases where one IP layer is encapsulated in another IP layer or other
form of tunneling protocol, designating more and more of the form of tunneling protocol, designating more and more of the
fundamental transport capacity as header bits that are pure overhead fundamental transport capacity as header bits that are pure overhead
to the BTC measurement. to the BTC measurement.
We also note that Raw and Restricted Capacity metrics are not We also note that raw and restricted capacity metrics are not
orthogonal in the sense defined in Section 5.1.2 above. The orthogonal in the sense defined in Section 5.1.2 above. The
information they covey about the network under test is certainly information they convey about the network under test is certainly
overlapping, but they reveal two different and important aspects of overlapping, but they reveal two different and important aspects of
performance. performance.
When thinking about the triad of raw capacity metrics, BTC is most When thinking about the triad of raw capacity metrics, BTC is most
akin to the "IP-Type-P Available Path Capacity", at least in the eyes akin to the "IP-Type-P Available Path Capacity", at least in the eyes
of a network user who seeks to know what transmission performance a of a network user who seeks to know what transmission performance a
path might support. path might support.
7.1. Type-P Parameter and Type-C Parameter 7.1. Type-P Parameter and Type-C Parameter
The concept of "packets of Type-P" is defined in [RFC2330]. The The concept of "packets of Type-P" is defined in [RFC2330]. The
considerations for Restricted Capacity are identical to the raw considerations for restricted capacity are identical to the raw
capacity section on this topic, with the addition that the various capacity section on this topic, with the addition that the various
fields and options in the TCP header must be included in the fields and options in the TCP header must be included in the
description. description.
The vast array of TCP flow control options are not well-captured by The vast array of TCP flow-control options are not well captured by
Type-P, because they do not exist in the TCP header bits. Therefore, Type-P, because they do not exist in the TCP header bits. Therefore,
we introduce a new notion here: TCP Configuration of "Type-C". The we introduce a new notion here: TCP Configuration of "Type-C". The
elements of Type-C describe all of the settings for TCP options and elements of Type-C describe all of the settings for TCP options and
congestion control algorithm variables, including the main form of congestion control algorithm variables, including the main form of
congestion control in use. Readers should consider the parameters congestion control in use. Readers should consider the parameters
and variables of [RFC3148] and [RFC6349] when constructing Type-C. and variables of [RFC3148] and [RFC6349] when constructing Type-C.
7.2. A priori Factors 7.2. A Priori Factors
The audience for Network Characterization may have detailed The audience for network characterization may have detailed
information about each link that comprises a complete path (due to information about each link that comprises a complete path (due to
ownership, for example), or some of the links in the path but not ownership, for example), or some of the links in the path but not
others, or none of the links. others, or none of the links.
There are cases where the measurement audience only has information There are cases where the measurement audience only has information
on one of the links (the local access link), and wishes to measure on one of the links (the local access link) and wishes to measure one
one or more BTC metrics. The discussion of Section 6.2 applies here or more BTC metrics. The discussion in Section 6.2 applies here
as well. as well.
7.3. Measurement Interval 7.3. Measurement Interval
There are limits on a useful measurement interval for BTC. Three There are limits on a useful measurement interval for BTC. Three
factors that influence the interval duration are listed below: factors that influence the interval duration are listed below:
1. Measurements may choose to include or exclude the 3-way handshake 1. Measurements may choose to include or exclude the 3-way handshake
of TCP connection establishment, which requires at least 1.5 * of TCP connection establishment, which requires at least 1.5 *
RTT and contains both the delay of the path and the host RTT (round-trip time) and contains both the delay of the path and
processing time for responses. However, user experience includes the host processing time for responses. However, user experience
the 3-way handshake for all new TCP connections. includes the 3-way handshake for all new TCP connections.
2. Measurements may choose to include or exclude Slow-Start, 2. Measurements may choose to include or exclude Slow-Start,
preferring instead to focus on a portion of the transfer that preferring instead to focus on a portion of the transfer that
represents "equilibrium" (which needs to be defined for represents "equilibrium" (which needs to be defined for
particular circumstances if used). However, user experience particular circumstances if used). However, user experience
includes the Slow-Start for all new TCP connections. includes the Slow-Start for all new TCP connections.
3. Measurements may choose to use a fixed block of data to transfer, 3. Measurements may choose to use a fixed block of data to transfer,
where the size of the block has a relationship to the file size where the size of the block has a relationship to the file size
of the application of interest. This approach yields variable of the application of interest. This approach yields variable
size measurement intervals, where a path with faster BTC is size measurement intervals, where a path with faster BTC is
measured for less time than a path with slower BTC, and this has measured for less time than a path with slower BTC, and this has
implications when path impairments are time-varying, or implications when path impairments are time-varying, or
transient. Users are likely to turn their immediate attention transient. Users are likely to turn their immediate attention
elsewhere when a very large file must be transferred, thus they elsewhere when a very large file must be transferred; thus, they
do not directly experience such a long transfer -- they see the do not directly experience such a long transfer -- they see the
result (success or fail) and possibly an objective measurement of result (success or failure) and possibly an objective measurement
the transfer time (which will likely include the 3-way handshake, of the transfer time (which will likely include the 3-way
Slow-start, and application file management processing time as handshake, Slow-Start, and application file management processing
well as the BTC). time as well as the BTC).
Individual measurement intervals may be short or long, but there is a Individual measurement intervals may be short or long, but there is a
need to report the results on a long-term basis that captures the BTC need to report the results on a long-term basis that captures the BTC
variability experienced between each interval. Consistent BTC is a variability experienced between each interval. Consistent BTC is a
valuable commodity along with the value attained. valuable commodity along with the value attained.
7.4. Bulk Transfer Capacity Reporting 7.4. Bulk Transfer Capacity Reporting
When BTC of a link or path is estimated through some measurement When BTC of a link or path is estimated through some measurement
technique, the following parameters should be reported: technique, the following parameters should be reported:
o Name and reference to the exact method of measurement o Name and reference to the exact method of measurement
o Maximum Transmission Unit (MTU) o Maximum Transmission Unit (MTU)
o Maximum BTC that can be assessed in the measurement configuration o Maximum BTC that can be assessed in the measurement configuration
o The time and duration of the measurement o Time and duration of the measurement
o Number of BTC connections used simultaneously
o The number of BTC connections used simultaneously
o *All* other parameters specific to the measurement method, o *All* other parameters specific to the measurement method,
especially the Congestion Control algorithm in use especially the congestion control algorithm in use
See also [RFC6349]. See also [RFC6349].
Many methods of Bulk Transfer Capacity measurement have a maximum Many methods of BTC measurement have a maximum capacity that they can
capacity that they can measure, and this maximum may be less than the measure, and this maximum may be less than the available capacity of
available capacity of the link or path. Therefore, it is important the link or path. Therefore, it is important to specify the measured
to specify the measured BTC value beyond which there will be no BTC value beyond which there will be no measured improvement.
measured improvement.
The Application Design audience may have a desired target capacity The application performance estimation audience may have a desired
value and simply wish to assess whether there is sufficient BTC. target capacity value and simply wish to assess whether there is
This case simplifies measurement of link and path capacity to some sufficient BTC. This case simplifies the measurement of link and
degree, as long as the measurable maximum exceeds the target path capacity to some degree, as long as the measurable maximum
capacity. exceeds the target capacity.
7.5. Variability in Bulk Transfer Capacity 7.5. Variability in Bulk Transfer Capacity
As with most metrics and measurements, assessing the consistency or As with most metrics and measurements, assessing the consistency or
variability in the results gives the user an intuitive feel for the variability in the results gives the user an intuitive feel for the
degree (or confidence) that any one value is representative of other degree (or confidence) that any one value is representative of other
results, or the underlying distribution from which these singleton results, or the underlying distribution from which these singleton
measurements have come. measurements have come.
With two questions looming: With two questions looming --
1. What ways can BTC be measured and summarized to describe the 1. What ways can BTC be measured and summarized to describe the
potential variability in a useful way? potential variability in a useful way?
2. How can the variability in BTC estimates be reported, so that the 2. How can the variability in BTC estimates be reported, so that the
confidence in the results is also conveyed? confidence in the results is also conveyed?
We suggest the methods of Section 6.6.1 above, and the additional -- we suggest the methods listed in Section 6.6.1 above, and the
results presentations given in [RFC6349]. additional results presentations given in [RFC6349].
8. Reporting on Test Streams and Sample Size 8. Reporting on Test Streams and Sample Size
This section discusses two key aspects of measurement that are This section discusses two key aspects of measurement that are
sometimes omitted from the report: the description of the test stream sometimes omitted from the report: the description of the test stream
on which the measurements are based, and the sample size. on which the measurements are based, and the sample size.
8.1. Test Stream Characteristics 8.1. Test Stream Characteristics
Network Characterization has traditionally used Poisson-distributed Network characterization has traditionally used Poisson-distributed
inter-packet spacing, as this provides an unbiased sample. The inter-packet spacing, as this provides an unbiased sample. The
average inter-packet spacing may be selected to allow observation of average inter-packet spacing may be selected to allow observation of
specific network phenomena. Other test streams are designed to specific network phenomena. Other test streams are designed to
sample some property of the network, such as the presence of sample some property of the network, such as the presence of
congestion, link bandwidth, or packet reordering. congestion, link bandwidth, or packet reordering.
If measuring a network in order to make inferences about applications If measuring a network in order to make inferences about applications
or receiver performance, then there are usually efficiencies derived or receiver performance, then there are usually efficiencies derived
from a test stream that has similar characteristics to the sender. from a test stream that has similar characteristics to the sender.
In some cases, it is essential to synthesize the sender stream, as In some cases, it is essential to synthesize the sender stream, as
with Bulk Transfer Capacity estimates. In other cases, it may be with BTC estimates. In other cases, it may be sufficient to sample
sufficient to sample with a "known bias", e.g., a Periodic stream to with a "known bias", e.g., a Periodic stream to estimate real-time
estimate real-time application performance. application performance.
8.2. Sample Size 8.2. Sample Size
Sample size is directly related to the accuracy of the results, and Sample size is directly related to the accuracy of the results and
plays a critical role in the report. Even if only the sample size plays a critical role in the report. Even if only the sample size
(in terms of number of packets) is given for each value or summary (in terms of number of packets) is given for each value or summary
statistic, it imparts a notion of the confidence in the result. statistic, it imparts a notion of the confidence in the result.
In practice, the sample size will be selected taking both statistical In practice, the sample size will be selected taking both statistical
and practical factors into account. Among these factors are: and practical factors into account. Among these factors are the
following:
1. The estimated variability of the quantity being measured 1. The estimated variability of the quantity being measured.
2. The desired confidence in the result (although this may be 2. The desired confidence in the result (although this may be
dependent on assumption of the underlying distribution of the dependent on assumption of the underlying distribution of the
measured quantity). measured quantity).
3. The effects of active measurement traffic on user traffic. 3. The effects of active measurement traffic on user traffic.
A sample size may sometimes be referred to as "large". This is a A sample size may sometimes be referred to as "large". This is a
relative, and qualitative term. It is preferable to describe what relative and qualitative term. It is preferable to describe what one
one is attempting to achieve with their sample. For example, stating is attempting to achieve with his sample. For example, stating an
an implication may be helpful: this sample is large enough such that implication may be helpful: this sample is large enough that a single
a single outlying value at ten times the "typical" sample mean (the outlying value at ten times the "typical" sample mean (the mean
mean without the outlying value) would influence the mean by no more without the outlying value) would influence the mean by no more
than X. than X.
The Appendix of [RFC2330] indicates that sample size of 128 The Appendix of [RFC2330] indicates that a sample size of 128
singletons worked well for goodness-of-fit testing, while a much singletons worked well for goodness-of-fit testing, while a much
larger size almost always failed (8192 singletons). larger size (8192 singletons) almost always failed.
9. IANA Considerations
This document makes no request of IANA.
Note to RFC Editor: this section may be removed on publication as an
RFC.
10. Security Considerations 9. Security Considerations
The security considerations that apply to any active measurement of The security considerations that apply to any active measurement of
live networks are relevant here as well. See [RFC4656] for mandatory live networks are relevant here as well. See the Security
to implement security features that intend to mitigate attacks Considerations section of [RFC4656] for mandatory-to-implement
described in the corresponding security considerations section. security features that intend to mitigate attacks.
Measurement systems conducting long-term measurements are more Measurement systems conducting long-term measurements are more
exposed to threats as a by-product of ports open longer to perform exposed to threats as a by-product of ports open longer to perform
their task, and more easily detected measurement activity on those their task, and more easily detected measurement activity on those
ports. Further, use of long packet waiting times affords an attacker ports. Further, use of long packet waiting times affords an attacker
a better opportunity to prepare and launch a replay attack. a better opportunity to prepare and launch a replay attack.
11. Acknowledgements 10. Acknowledgements
The authors thank: Phil Chimento for his suggestion to employ The authors thank Phil Chimento for his suggestion to employ
conditional distributions for Delay, Steve Konish Jr. for his careful conditional distributions for delay, Steve Konish Jr. for his careful
review and suggestions, Dave McDysan and Don McLachlan for useful review and suggestions, Dave McDysan and Don McLachlan for useful
comments based on their long experience with measurement and comments based on their long experience with measurement and
reporting, Daniel Genin for his observation of non-orthogonality reporting, Daniel Genin for his observation of non-orthogonality
between Raw and Restricted Capacity metrics (and our omission of this between raw and restricted capacity metrics (and for noticing our
fact), and Matt Zekauskas for suggestions on organizing the memo for previous omission of this fact), and Matt Zekauskas for suggestions
easier consumption. on organizing the memo for easier consumption.
12. References 11. References
12.1. Normative References 11.1. Normative References
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
May 1998. May 1998.
[RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring [RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring
Connectivity", RFC 2678, September 1999. Connectivity", RFC 2678, September 1999.
[RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Delay Metric for IPPM", RFC 2679, September 1999. Delay Metric for IPPM", RFC 2679, September 1999.
[RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Packet Loss Metric for IPPM", RFC 2680, September 1999. Packet Loss Metric for IPPM", RFC 2680, September 1999.
[RFC3148] Mathis, M. and M. Allman, "A Framework for Defining [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining
Empirical Bulk Transfer Capacity Metrics", RFC 3148, Empirical Bulk Transfer Capacity Metrics", RFC 3148,
July 2001. July 2001.
[RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay
Metric for IP Performance Metrics (IPPM)", RFC 3393, Variation Metric for IP Performance Metrics (IPPM)",
November 2002. RFC 3393, November 2002.
[RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network
performance measurement with periodic streams", RFC 3432, performance measurement with periodic streams", RFC 3432,
November 2002. November 2002.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
Zekauskas, "A One-way Active Measurement Protocol Zekauskas, "A One-way Active Measurement Protocol
(OWAMP)", RFC 4656, September 2006. (OWAMP)", RFC 4656, September 2006.
[RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
S., and J. Perser, "Packet Reordering Metrics", RFC 4737, S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
November 2006. November 2006.
[RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", [RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity",
RFC 5136, February 2008. RFC 5136, February 2008.
12.2. Informative References 11.2. Informative References
[Casner] "A Fine-Grained View of High Performance Networking, NANOG [Casner] Casner, S., Alaettinoglu, C., and C. Kuan, "A Fine-
22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May Grained View of High-Performance Networking",
20-22 2001. NANOG 22 Conf., May 20-22 2001,
<http://www.nanog.org/presentations/archive/index.php>.
[Cia03] "Standardized Active Measurements on a Tier 1 IP Backbone, [Cia03] Ciavattone, L., Morton, A., and G. Ramachandran,
IEEE Communications Mag., pp 90-97.", June 2003. "Standardized Active Measurements on a Tier 1 IP
Backbone", IEEE Communications Magazine, Vol. 41
No. 6, pp. 90-97, June 2003.
[I-D.ietf-ippm-reporting] [IPPM-RPT] Shalunov, S. and M. Swany, "Reporting IP Performance
Shalunov, S. and M. Swany, "Reporting IP Performance Metrics to Users", Work in Progress, March 2011.
Metrics to Users", draft-ietf-ippm-reporting-06 (work in
progress), March 2011.
[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
Applicability Statement", RFC 5481, March 2009. Applicability Statement", RFC 5481, March 2009.
[RFC5835] Morton, A. and S. Van den Berghe, "Framework for Metric [RFC5835] Morton, A., Ed., and S. Van den Berghe, Ed., "Framework
Composition", RFC 5835, April 2010. for Metric Composition", RFC 5835, April 2010.
[RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage, [RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage,
"Framework for TCP Throughput Testing", RFC 6349, "Framework for TCP Throughput Testing", RFC 6349,
August 2011. August 2011.
[Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data [Y.1540] International Telecommunication Union, "Internet protocol
communication service - IP packet transfer and data communication service - IP packet transfer and
availability performance parameters", December 2011. availability performance parameters", ITU-T
Recommendation Y.1540, March 2011.
[Y.1541] ITU-T Recommendation Y.1540, "Network Performance [Y.1541] International Telecommunication Union, "Network
Objectives for IP-Based Services", February 2011. performance objectives for IP-based services", ITU-T
Recommendation Y.1541, December 2011.
Authors' Addresses Authors' Addresses
Al Morton Al Morton
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown, NJ 07748 Middletown, NJ 07748
USA USA
Phone: +1 732 420 1571 Phone: +1 732 420 1571
Fax: +1 732 368 1192 Fax: +1 732 368 1192
Email: acmorton@att.com EMail: acmorton@att.com
URI: http://home.comcast.net/~acmacm/ URI: http://home.comcast.net/~acmacm/
Gomathi Ramachandran Gomathi Ramachandran
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown, New Jersey 07748 Middletown, New Jersey 07748
USA USA
Phone: +1 732 420 2353 Phone: +1 732 420 2353
Fax: EMail: gomathi@att.com
Email: gomathi@att.com
URI:
Ganga Maguluri Ganga Maguluri
AT&T Labs AT&T Labs
200 Laurel Avenue 200 Laurel Avenue South
Middletown, New Jersey 07748 Middletown, New Jersey 07748
USA USA
Phone: 732-420-2486 Phone: +1 732 420 2486
Fax: EMail: gmaguluri@att.com
Email: gmaguluri@att.com
URI:
 End of changes. 184 change blocks. 
442 lines changed or deleted 438 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/