draft-ietf-ippm-reporting-metrics-06.txt   draft-ietf-ippm-reporting-metrics-07.txt 
Network Working Group A. Morton Network Working Group A. Morton
Internet-Draft G. Ramachandran Internet-Draft G. Ramachandran
Intended status: Informational G. Maguluri Intended status: Informational G. Maguluri
Expires: July 10, 2012 AT&T Labs Expires: August 16, 2012 AT&T Labs
January 7, 2012 February 13, 2012
Reporting Metrics: Different Points of View Reporting Metrics: Different Points of View
draft-ietf-ippm-reporting-metrics-06 draft-ietf-ippm-reporting-metrics-07
Abstract Abstract
Consumers of IP network performance metrics have many different uses Consumers of IP network performance metrics have many different uses
in mind. The memo provides "long-term" reporting considerations in mind. The memo provides "long-term" reporting considerations
(e.g, days, weeks or months, as opposed to 10 seconds), based on (e.g, days, weeks or months, as opposed to 10 seconds), based on
analysis of the two key audience points-of-view. It describes how analysis of the two key audience points-of-view. It describes how
the audience categories affect the selection of metric parameters and the audience categories affect the selection of metric parameters and
options when seeking info that serves their needs. options when seeking info that serves their needs.
skipping to change at page 1, line 42 skipping to change at page 1, line 42
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 10, 2012. This Internet-Draft will expire on August 16, 2012.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 27 skipping to change at page 3, line 27
4.3. Causes of Lost Packets . . . . . . . . . . . . . . . . . . 10 4.3. Causes of Lost Packets . . . . . . . . . . . . . . . . . . 10
4.4. Summary for Loss . . . . . . . . . . . . . . . . . . . . . 11 4.4. Summary for Loss . . . . . . . . . . . . . . . . . . . . . 11
5. Effect of POV on the Delay Metric . . . . . . . . . . . . . . 11 5. Effect of POV on the Delay Metric . . . . . . . . . . . . . . 11
5.1. Treatment of Lost Packets . . . . . . . . . . . . . . . . 11 5.1. Treatment of Lost Packets . . . . . . . . . . . . . . . . 11
5.1.1. Application Performance . . . . . . . . . . . . . . . 11 5.1.1. Application Performance . . . . . . . . . . . . . . . 11
5.1.2. Network Characterization . . . . . . . . . . . . . . . 12 5.1.2. Network Characterization . . . . . . . . . . . . . . . 12
5.1.3. Delay Variation . . . . . . . . . . . . . . . . . . . 13 5.1.3. Delay Variation . . . . . . . . . . . . . . . . . . . 13
5.1.4. Reordering . . . . . . . . . . . . . . . . . . . . . . 14 5.1.4. Reordering . . . . . . . . . . . . . . . . . . . . . . 14
5.2. Preferred Statistics . . . . . . . . . . . . . . . . . . . 14 5.2. Preferred Statistics . . . . . . . . . . . . . . . . . . . 14
5.3. Summary for Delay . . . . . . . . . . . . . . . . . . . . 15 5.3. Summary for Delay . . . . . . . . . . . . . . . . . . . . 15
6. Effect of POV on Raw Capacity Metrics . . . . . . . . . . . . 15 6. Reporting Raw Capacity Metrics . . . . . . . . . . . . . . . . 15
6.1. Type-P Parameter . . . . . . . . . . . . . . . . . . . . . 15 6.1. Type-P Parameter . . . . . . . . . . . . . . . . . . . . . 15
6.2. a priori Factors . . . . . . . . . . . . . . . . . . . . . 16 6.2. A priori Factors . . . . . . . . . . . . . . . . . . . . . 16
6.3. IP-layer Capacity . . . . . . . . . . . . . . . . . . . . 16 6.3. IP-layer Capacity . . . . . . . . . . . . . . . . . . . . 16
6.4. IP-layer Utilization . . . . . . . . . . . . . . . . . . . 17 6.4. IP-layer Utilization . . . . . . . . . . . . . . . . . . . 17
6.5. IP-layer Available Capacity . . . . . . . . . . . . . . . 17 6.5. IP-layer Available Capacity . . . . . . . . . . . . . . . 17
6.6. Variability in Utilization and Avail. Capacity . . . . . . 18 6.6. Variability in Utilization and Avail. Capacity . . . . . . 18
6.6.1. General Summary of Variability . . . . . . . . . . . . 18 6.6.1. General Summary of Variability . . . . . . . . . . . . 18
7. Effect of POV on Restricted Capacity Metrics . . . . . . . . . 19 7. Reporting Restricted Capacity Metrics . . . . . . . . . . . . 19
7.1. Type-P Parameter and Type-C Parameter . . . . . . . . . . 20 7.1. Type-P Parameter and Type-C Parameter . . . . . . . . . . 20
7.2. a priori Factors . . . . . . . . . . . . . . . . . . . . . 20 7.2. A priori Factors . . . . . . . . . . . . . . . . . . . . . 20
7.3. Measurement Interval . . . . . . . . . . . . . . . . . . . 20 7.3. Measurement Interval . . . . . . . . . . . . . . . . . . . 20
7.4. Bulk Transfer Capacity Reporting . . . . . . . . . . . . . 21 7.4. Bulk Transfer Capacity Reporting . . . . . . . . . . . . . 21
7.5. Variability in Bulk Transfer Capacity . . . . . . . . . . 22 7.5. Variability in Bulk Transfer Capacity . . . . . . . . . . 22
8. Test Streams and Sample Size . . . . . . . . . . . . . . . . . 22 8. Reporting on Test Streams and Sample Size . . . . . . . . . . 22
8.1. Test Stream Characteristics . . . . . . . . . . . . . . . 22 8.1. Test Stream Characteristics . . . . . . . . . . . . . . . 22
8.2. Sample Size . . . . . . . . . . . . . . . . . . . . . . . 23 8.2. Sample Size . . . . . . . . . . . . . . . . . . . . . . . 22
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23
10. Security Considerations . . . . . . . . . . . . . . . . . . . 23 10. Security Considerations . . . . . . . . . . . . . . . . . . . 23
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 23 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 23
12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 24 12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 24
12.1. Normative References . . . . . . . . . . . . . . . . . . . 24 12.1. Normative References . . . . . . . . . . . . . . . . . . . 24
12.2. Informative References . . . . . . . . . . . . . . . . . . 25 12.2. Informative References . . . . . . . . . . . . . . . . . . 24
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25
1. Introduction 1. Introduction
When designing measurements of IP networks and presenting the When designing measurements of IP networks and presenting the
results, knowledge of the audience is a key consideration. To results, knowledge of the audience is a key consideration. To
present a useful and relevant portrait of network conditions, one present a useful and relevant portrait of network conditions, one
must answer the following question: must answer the following question:
"How will the results be used?" "How will the results be used?"
skipping to change at page 5, line 42 skipping to change at page 5, line 42
3.1. Overview of Metric Statistics 3.1. Overview of Metric Statistics
This section gives an overview of reporting recommendations for the This section gives an overview of reporting recommendations for the
loss, delay, and delay variation metrics. loss, delay, and delay variation metrics.
The minimal report on measurements MUST include both Loss and Delay The minimal report on measurements MUST include both Loss and Delay
Metrics. Metrics.
For Packet Loss, the loss ratio defined in [RFC2680] is a sufficient For Packet Loss, the loss ratio defined in [RFC2680] is a sufficient
starting point, especially the guidance for setting the loss starting point, especially the existing guidance for setting the loss
threshold waiting time. We have calculated a waiting time above that threshold waiting time. We have calculated a waiting time above that
should be sufficient to differentiate between packets that are truly should be sufficient to differentiate between packets that are truly
lost or have long finite delays under general measurement lost or have long finite delays under general measurement
circumstances, 51 seconds. Knowledge of specific conditions can help circumstances, 51 seconds. Knowledge of specific conditions can help
to reduce this threshold, but 51 seconds is considered to be to reduce this threshold, but 51 seconds is considered to be
manageable in practice. manageable in practice.
We note that a loss ratio calculated according to [Y.1540] would We note that a loss ratio calculated according to [Y.1540] would
exclude errored packets from the numerator. In practice, the exclude errored packets from the numerator. In practice, the
difference between these two loss metrics is small if any, depending difference between these two loss metrics is small if any, depending
on whether the last link prior to the destination contributes errored on whether the last link prior to the destination contributes errored
packets. packets.
For Packet Delay, we recommend providing both the mean delay and the For Packet Delay, we recommend providing both the mean delay and the
median delay with lost packets designated undefined (as permitted by median delay with lost packets designated undefined (as permitted by
[RFC2679]). Both statistics are based on a conditional distribution, [RFC2679]). Both statistics are based on a conditional distribution,
and the condition is packet arrival prior to a waiting time dT, where and the condition is packet arrival prior to a waiting time dT, where
dT has been set to take maximum packet lifetimes into account, as dT has been set to take maximum packet lifetimes into account, as
discussed below. Using a long dT helps to ensure that delay discussed above for loss. Using a long dT helps to ensure that delay
distributions are not truncated. distributions are not truncated.
For Packet Delay Variation (PDV), the minimum delay of the For Packet Delay Variation (PDV), the minimum delay of the
conditional distribution should be used as the reference delay for conditional distribution should be used as the reference delay for
computing PDV according to [Y.1540] or [RFC5481] and [RFC3393]. A computing PDV according to [Y.1540] or [RFC5481] and [RFC3393]. A
useful value to report is a pseudo range of delay variation based on useful value to report is a pseudo range of delay variation based on
calculating the difference between a high percentile of delay and the calculating the difference between a high percentile of delay and the
minimum delay. For example, the 99.9%-ile minus the minimum will minimum delay. For example, the 99.9%-ile minus the minimum will
give a value that can be compared with objectives in [Y.1541]. give a value that can be compared with objectives in [Y.1541].
For Capacity, both Raw and Restricted, reporting the variability in a
useful way is identified as the main challenge. The Min, Max, and
Range statistics are suggested along with a ratio of Max to Min and
moving averages. In the end, a simple plot of the singleton results
over time may succeed where summary metrics fail, or serve to confirm
that the summaries are valid.
3.2. Long-Term Reporting Considerations 3.2. Long-Term Reporting Considerations
[I-D.ietf-ippm-reporting] describes methods to conduct measurements [I-D.ietf-ippm-reporting] describes methods to conduct measurements
and report the results on a near-immediate time scale (10 seconds, and report the results on a near-immediate time scale (10 seconds,
which we consider to be "short-term"). which we consider to be "short-term").
Measurement intervals and reporting intervals need not be the same Measurement intervals and reporting intervals need not be the same
length. Sometimes, the user is only concerned with the performance length. Sometimes, the user is only concerned with the performance
levels achieved over a relatively long interval of time (e.g, days, levels achieved over a relatively long interval of time (e.g, days,
weeks, or months, as opposed to 10 seconds). However, there can be weeks, or months, as opposed to 10 seconds). However, there can be
skipping to change at page 9, line 36 skipping to change at page 9, line 41
and where C is the number of times a packet circles the loop. and where C is the number of times a packet circles the loop.
If we take the delays of all links and queues as 100ms each, the If we take the delays of all links and queues as 100ms each, the
TTL=255, the number of hops n=5 and the hops in the loop L=4, then TTL=255, the number of hops n=5 and the hops in the loop L=4, then
D = 1.1 sec and R ~= 50 sec, and D + R ~= 51.1 seconds D = 1.1 sec and R ~= 50 sec, and D + R ~= 51.1 seconds
We note that the link delays of 100ms would span most continents, and We note that the link delays of 100ms would span most continents, and
a constant queue length of 100ms is also very generous. When a loop a constant queue length of 100ms is also very generous. When a loop
occurs, it is almost certain to be resolved in 10 seconds or less. occurs, it is almost certain to be resolved in 10 seconds or less.
The value calculated above is an upper limit for almost any realistic The value calculated above is an upper limit for almost any real-
circumstance. world circumstance.
A waiting time threshold parameter, dT, set consistent with this A waiting time threshold parameter, dT, set consistent with this
calculation would not truncate the delay distribution (possibly calculation would not truncate the delay distribution (possibly
causing a change in its mathematical properties), because the packets causing a change in its mathematical properties), because the packets
that might arrive have been given sufficient time to traverse the that might arrive have been given sufficient time to traverse the
network. network.
It is worth noting that packets that are stored and deliberately It is worth noting that packets that are stored and deliberately
forwarded at a much later time constitute a replay attack on the forwarded at a much later time constitute a replay attack on the
measurement system, and are beyond the scope of normal performance measurement system, and are beyond the scope of normal performance
skipping to change at page 10, line 15 skipping to change at page 10, line 17
4.1.2. Application Performance 4.1.2. Application Performance
Fortunately, application performance estimation activities are not Fortunately, application performance estimation activities are not
adversely affected by the estimated worst-case transfer time. adversely affected by the estimated worst-case transfer time.
Although the designer's tendency might be to set the Loss Threshold Although the designer's tendency might be to set the Loss Threshold
at a value equivalent to a particular application's threshold, this at a value equivalent to a particular application's threshold, this
specific threshold can be applied when post-processing the specific threshold can be applied when post-processing the
measurements. A shorter waiting time can be enforced by locating measurements. A shorter waiting time can be enforced by locating
packets with delays longer than the application's threshold, and re- packets with delays longer than the application's threshold, and re-
designating such packets as lost. Thus, the measurement system can designating such packets as lost. Thus, the measurement system can
use a single loss threshold and support both application and network use a single loss waiting time and support both application and
performance POVs simultaneously. network performance POVs simultaneously.
4.2. Errored Packet Designation 4.2. Errored Packet Designation
RFC 2680 designates packets that arrive containing errors as lost RFC 2680 designates packets that arrive containing errors as lost
packets. Many packets that are corrupted by bit errors are discarded packets. Many packets that are corrupted by bit errors are discarded
within the network and do not reach their intended destination. within the network and do not reach their intended destination.
This is consistent with applications that would check the payload This is consistent with applications that would check the payload
integrity at higher layers, and discard the packet. However, some integrity at higher layers, and discard the packet. However, some
applications prefer to deal with errored payloads on their own, and applications prefer to deal with errored payloads on their own, and
skipping to change at page 12, line 24 skipping to change at page 12, line 29
delay distribution as a conditional distribution (conditioned on delay distribution as a conditional distribution (conditioned on
arrival). arrival).
5.1.2. Network Characterization 5.1.2. Network Characterization
In this discussion, we assume that both loss and delay metrics will In this discussion, we assume that both loss and delay metrics will
be reported for network characterization (at least). be reported for network characterization (at least).
Assume packets that do not arrive are reported as Lost, usually as a Assume packets that do not arrive are reported as Lost, usually as a
fraction of all sent packets. If these lost packets are assigned fraction of all sent packets. If these lost packets are assigned
undefined delay, then network's inability to deliver them (in a undefined delay, then the network's inability to deliver them (in a
timely way) is captured only in the loss metric when we report timely way) is relegated only in the Loss metric when we report
statistics on the Delay distribution conditioned on the event of statistics on the Delay distribution conditioned on the event of
packet arrival (within the Loss waiting time threshold). We can say packet arrival (within the Loss waiting time threshold). We can say
that the Delay and Loss metrics are Orthogonal, in that they convey that the Delay and Loss metrics are Orthogonal, in that they convey
non-overlapping information about the network under test. non-overlapping information about the network under test. This is a
valuable property, whose absence is discussed below.
However, if we assign infinite delay to all lost packets, then: However, if we assign infinite delay to all lost packets, then:
o The delay metric results are influenced both by packets that o The delay metric results are influenced both by packets that
arrive and those that do not. arrive and those that do not.
o The delay singleton and the loss singleton do not appear to be o The delay singleton and the loss singleton do not appear to be
orthogonal (Delay is finite when Loss=0, Delay is infinite when orthogonal (Delay is finite when Loss=0, Delay is infinite when
Loss=1). Loss=1).
skipping to change at page 13, line 35 skipping to change at page 13, line 36
Although infinity is a familiar mathematical concept, it is somewhat Although infinity is a familiar mathematical concept, it is somewhat
disconcerting to see any time-related metric reported as infinity, in disconcerting to see any time-related metric reported as infinity, in
the opinion of the authors. Questions are bound to arise, and tend the opinion of the authors. Questions are bound to arise, and tend
to detract from the goal of informing the consumer with a performance to detract from the goal of informing the consumer with a performance
report. report.
5.1.3. Delay Variation 5.1.3. Delay Variation
[RFC3393] excludes lost packets from samples, effectively assigning [RFC3393] excludes lost packets from samples, effectively assigning
an undefined delay to packets that do not arrive in a reasonable an undefined delay to packets that do not arrive in a reasonable
time. Section 4.1 describes this specification and its rationale time. Section 4.1 of [RFC3393] describes this specification and its
(ipdv = inter-packet delay variation in the quote below). rationale (ipdv = inter-packet delay variation in the quote below).
"The treatment of lost packets as having "infinite" or "undefined" "The treatment of lost packets as having "infinite" or "undefined"
delay complicates the derivation of statistics for ipdv. delay complicates the derivation of statistics for ipdv.
Specifically, when packets in the measurement sequence are lost, Specifically, when packets in the measurement sequence are lost,
simple statistics such as sample mean cannot be computed. One simple statistics such as sample mean cannot be computed. One
possible approach to handling this problem is to reduce the event possible approach to handling this problem is to reduce the event
space by conditioning. That is, we consider conditional statistics; space by conditioning. That is, we consider conditional statistics;
namely we estimate the mean ipdv (or other derivative statistic) namely we estimate the mean ipdv (or other derivative statistic)
conditioned on the event that selected packet pairs arrive at the conditioned on the event that selected packet pairs arrive at the
destination (within the given timeout). While this itself is not destination (within the given timeout). While this itself is not
skipping to change at page 14, line 30 skipping to change at page 14, line 32
delay of non-arriving packets as undefined, and reserving delay delay of non-arriving packets as undefined, and reserving delay
values only for packets that arrive within a sufficiently long values only for packets that arrive within a sufficiently long
waiting time. waiting time.
5.2. Preferred Statistics 5.2. Preferred Statistics
Today in network characterization, the sample mean is one statistic Today in network characterization, the sample mean is one statistic
that is almost ubiquitously reported. It is easily computed and that is almost ubiquitously reported. It is easily computed and
understood by virtually everyone in this audience category. Also, understood by virtually everyone in this audience category. Also,
the sample is usually filtered on packet arrival, so that the mean is the sample is usually filtered on packet arrival, so that the mean is
based a conditional distribution. based on a conditional distribution.
The median is another statistic that summarizes a distribution, The median is another statistic that summarizes a distribution,
having somewhat different properties from the sample mean. The having somewhat different properties from the sample mean. The
median is stable in distributions with a few outliers or without median is stable in distributions with a few outliers or without
them. However, the median's stability prevents it from indicating them. However, the median's stability prevents it from indicating
when a large fraction of the distribution changes value. 50% or more when a large fraction of the distribution changes value. 50% or more
values would need to change for the median to capture the change. values would need to change for the median to capture the change.
Both the median and sample mean have difficulty with bimodal Both the median and sample mean have difficulty with bimodal
distributions. The median will reside in only one of the modes, and distributions. The median will reside in only one of the modes, and
skipping to change at page 15, line 27 skipping to change at page 15, line 29
2. straightforward network characterization without double-counting 2. straightforward network characterization without double-counting
defects, and defects, and
3. consistency with Delay variation and Reordering metric 3. consistency with Delay variation and Reordering metric
definitions, definitions,
the most efficient practice is to distinguish between truly lost and the most efficient practice is to distinguish between truly lost and
delayed packets with a sufficiently long waiting time, and to delayed packets with a sufficiently long waiting time, and to
designate the delay of non-arriving packets as undefined. designate the delay of non-arriving packets as undefined.
6. Effect of POV on Raw Capacity Metrics 6. Reporting Raw Capacity Metrics
This section describes the ways that raw capacity metrics can be Raw capacity refers to the metrics defined in [RFC5136] which do not
tuned to reflect the preferences of the two audiences, or different include restrictions such as data uniqueness or flow-control response
Points-of-View (POV). Raw capacity refers to the metrics defined in to congestion.
[RFC5136] which do not include restrictions such as data uniqueness
or flow-control response to congestion.
In summary, the metrics considered are IP-layer Capacity, Utilization The metrics considered are IP-layer Capacity, Utilization (or used
(or used capacity), and Available Capacity, for individual links and capacity), and Available Capacity, for individual links and complete
complete paths. These three metrics form a triad: knowing one metric paths. These three metrics form a triad: knowing one metric
constrains the other two (within their allowed range), and knowing constrains the other two (within their allowed range), and knowing
two determines the third. The link metrics have another key aspect two determines the third. The link metrics have another key aspect
in common: they are single-measurement-point metrics at the egress of in common: they are single-measurement-point metrics at the egress of
a link. The path Capacity and Available Capacity are derived by a link. The path Capacity and Available Capacity are derived by
examining the set of single-point link measurements and taking the examining the set of single-point link measurements and taking the
minimum value. minimum value.
6.1. Type-P Parameter 6.1. Type-P Parameter
The concept of "packets of type-P" is defined in [RFC2330]. The The concept of "packets of type-P" is defined in [RFC2330]. The
skipping to change at page 16, line 14 skipping to change at page 16, line 15
across queues, the conditions in all packet categories will affect across queues, the conditions in all packet categories will affect
capacity and related measurements. This is one source of variability capacity and related measurements. This is one source of variability
in the results that all audiences would prefer to see reported in a in the results that all audiences would prefer to see reported in a
useful and easily understood way. useful and easily understood way.
Type-P in OWAMP and TWAMP is essentially confined to the Diffserv Type-P in OWAMP and TWAMP is essentially confined to the Diffserv
Codepoint [RFC4656]. DSCP is the most common qualifier for type-P. Codepoint [RFC4656]. DSCP is the most common qualifier for type-P.
Each audience will have a set of type-P qualifications and value Each audience will have a set of type-P qualifications and value
combinations that are of interest. Measurements and reports SHOULD combinations that are of interest. Measurements and reports SHOULD
have the flexibility to per-type and aggregate performance. have the flexibility to report per-type and aggregate performance.
6.2. a priori Factors 6.2. A priori Factors
The audience for Network Characterization may have detailed The audience for Network Characterization may have detailed
information about each link that comprises a complete path (due to information about each link that comprises a complete path (due to
ownership, for example), or some of the links in the path but not ownership, for example), or some of the links in the path but not
others, or none of the links. others, or none of the links.
There are cases where the measurement audience only has information There are cases where the measurement audience only has information
on one of the links (the local access link), and wishes to measure on one of the links (the local access link), and wishes to measure
one or more of the raw capacity metrics. This scenario is quite one or more of the raw capacity metrics. This scenario is quite
common, and has spawned a substantial number of experimental common, and has spawned a substantial number of experimental
measurement methods [ref to CAIDA survey page, etc.]. Many of these measurement methods (e.g., http://www.caida.org/tools/taxonomy/ ).
methods respect that their users want a result fairly quickly and in Many of these methods respect that their users want a result fairly
a one-trial. Thus, the measurement interval is kept short (a few quickly and in a one-trial. Thus, the measurement interval is kept
seconds to a minute). For long-term reporting, a sample of short short (a few seconds to a minute). For long-term reporting, a sample
term results need to be summarized. of short term results need to be summarized.
6.3. IP-layer Capacity 6.3. IP-layer Capacity
For links, this metric's theoretical maximum value can be determined For links, this metric's theoretical maximum value can be determined
from the physical layer bit rate and the bit rate reduction due to from the physical layer bit rate and the bit rate reduction due to
the layers between the physical layer and IP. When measured, this the layers between the physical layer and IP. When measured, this
metric takes additional factors into account, such as the ability of metric takes additional factors into account, such as the ability of
the sending device to process and forward traffic under various the sending device to process and forward traffic under various
conditions. For example, the arrival of routing updates may spawn conditions. For example, the arrival of routing updates may spawn
high priority processes that reduce the sending rate temporarily. high priority processes that reduce the sending rate temporarily.
skipping to change at page 17, line 45 skipping to change at page 17, line 45
When Available capacity of a link or path is estimated through some When Available capacity of a link or path is estimated through some
measurement technique, the following parameters SHOULD be reported: measurement technique, the following parameters SHOULD be reported:
o Name and reference to the exact method of measurement o Name and reference to the exact method of measurement
o IP packet length, octets (including IP header) o IP packet length, octets (including IP header)
o Maximum Capacity that can be assessed in the measurement o Maximum Capacity that can be assessed in the measurement
configuration configuration
o The time a duration of the measurement o The time duration of the measurement
o All other parameters specific to the measurement method o All other parameters specific to the measurement method
Many methods of Available capacity measurement have a maximum Many methods of Available capacity measurement have a maximum
capacity that they can measure, and this maximum may be less than the capacity that they can measure, and this maximum may be less than the
actual Available capacity of the link or path. Therefore, it is actual Available capacity of the link or path. Therefore, it is
important to know the capacity value beyond which there will be no important to know the capacity value beyond which there will be no
measured improvement. measured improvement.
The Application Design audience may have a target capacity value and The Application Design audience may have a desired target capacity
simply wish to assess whether there is sufficient Available Capacity. value and simply wish to assess whether there is sufficient Available
This case simplifies measurement of link and path capacity to some Capacity. This case simplifies measurement of link and path capacity
degree, as long as the measurable maximum exceeds the target to some degree, as long as the measurable maximum exceeds the target
capacity. capacity.
6.6. Variability in Utilization and Avail. Capacity 6.6. Variability in Utilization and Avail. Capacity
As with most metrics and measurements, assessing the consistency or As with most metrics and measurements, assessing the consistency or
variability in the results gives a the user an intuitive feel for the variability in the results gives the user an intuitive feel for the
degree (or confidence) that any one value is representative of other degree (or confidence) that any one value is representative of other
results, or the underlying distribution from which these singleton results, or the spread of the underlying distribution of the
measurements have come. singleton measurements.
What ways can Utilization be measured and summarized to describe the How can Utilization be measured and summarized to describe the
potential variability in a useful way? potential variability in a useful way?
How can the variability in Available Capacity estimates be reported, How can the variability in Available Capacity estimates be reported,
so that the confidence in the results is also conveyed? so that the confidence in the results is also conveyed?
We suggest some methods below: We suggest some methods below:
6.6.1. General Summary of Variability 6.6.1. General Summary of Variability
With a set of singleton Utilization or Available Capacity estimates, With a set of singleton Utilization or Available Capacity estimates,
skipping to change at page 19, line 17 skipping to change at page 19, line 17
reported as an exception. reported as an exception.
Often, the most informative summary of the results is a two-axis plot Often, the most informative summary of the results is a two-axis plot
rather than a table of statistics, where time is plotted on the rather than a table of statistics, where time is plotted on the
x-axis and the singleton value on the y-axis. The time-series plot x-axis and the singleton value on the y-axis. The time-series plot
can illustrate sudden changes in an otherwise stable range, identify can illustrate sudden changes in an otherwise stable range, identify
bi-modality easily, and help quickly assess correlation with other bi-modality easily, and help quickly assess correlation with other
time-series. Plots of frequency of the singleton values are likewise time-series. Plots of frequency of the singleton values are likewise
useful tools to visualize the variation. useful tools to visualize the variation.
7. Effect of POV on Restricted Capacity Metrics 7. Reporting Restricted Capacity Metrics
This section describes the ways that restricted capacity metrics can Restricted capacity refers to the metrics defined in [RFC3148] which
be tuned to reflect the preferences of the two audiences, or include criteria of data uniqueness or flow-control response to
different Points-of-View (POV). Raw capacity refers to the metrics congestion.
defined in [RFC3148] which include restrictions such as data
uniqueness or flow-control response to congestion.
In primary metric considered is Bulk Transfer Capacity (BTC) for In primary metric considered is Bulk Transfer Capacity (BTC) for
complete paths. [RFC3148] defines complete paths. [RFC3148] defines
BTC = data_sent / elapsed_time BTC = data_sent / elapsed_time
for a connection with congestion-aware flow control, where data_sent for a connection with congestion-aware flow control, where data_sent
is the total of unique payload bits (no headers). is the total of unique payload bits (no headers).
We note that this definition *differs* from the raw capacity We note that this definition *differs* from the raw capacity
definition in Section 2.3.1 of [RFC5136], where IP-layer Capacity definition in Section 2.3.1 of [RFC5136], where IP-layer Capacity
*includes* all bits in the IP header and payload. This means that *includes* all bits in the IP header and payload. This means that
Restricted Capacity BTC is already operating at a disadvantage when Restricted Capacity BTC is already operating at a disadvantage when
compared to the raw capacity at layers below TCP. Further, there are compared to the raw capacity at layers below TCP. Further, there are
cases where "THE IP-layer" is encapsulated in another IP-layer or cases where one IP-layer is encapsulated in another IP-layer or other
other form of tunneling protocol, designating more and more of the form of tunneling protocol, designating more and more of the
fundamental transport capacity as header bits that are pure overhead fundamental transport capacity as header bits that are pure overhead
to the BTC measurement. to the BTC measurement.
We also note that Raw and Restricted Capacity metrics are not
orthogonal in the sense defined in Section 5.1.2 above. The
information they covey about the network under test is certainly
overlapping, but they reveal two different and important aspects of
performance.
When thinking about the triad of raw capacity metrics, BTC is most When thinking about the triad of raw capacity metrics, BTC is most
akin to the "IP-Type-P Available Path Capacity", at least in the eyes akin to the "IP-Type-P Available Path Capacity", at least in the eyes
of a network user who seeks to know what transmission performance a of a network user who seeks to know what transmission performance a
path might support. path might support.
7.1. Type-P Parameter and Type-C Parameter 7.1. Type-P Parameter and Type-C Parameter
The concept of "packets of type-P" is defined in [RFC2330]. The The concept of "packets of type-P" is defined in [RFC2330]. The
considerations for Restricted Capacity are identical to the raw considerations for Restricted Capacity are identical to the raw
capacity section on this topic, with the addition that the various capacity section on this topic, with the addition that the various
fields and options in the TCP header MUST be included in the fields and options in the TCP header MUST be included in the
description. description.
The vast array of TCP flow control options are not well-captured by The vast array of TCP flow control options are not well-captured by
Type-P, because they do not exist in the TCP header bits. Therefore, Type-P, because they do not exist in the TCP header bits. Therefore,
we introduce a new notion here: TCP Configuration of "Type-C". The we introduce a new notion here: TCP Configuration of "Type-C". The
elements of Type-C describe all of the settings for TCP options and elements of Type-C describe all of the settings for TCP options and
congestion control algorithm variables, including the main form of congestion control algorithm variables, including the main form of
congestion control in use. congestion control in use.
7.2. a priori Factors 7.2. A priori Factors
The audience for Network Characterization may have detailed The audience for Network Characterization may have detailed
information about each link that comprises a complete path (due to information about each link that comprises a complete path (due to
ownership, for example), or some of the links in the path but not ownership, for example), or some of the links in the path but not
others, or none of the links. others, or none of the links.
There are cases where the measurement audience only has information There are cases where the measurement audience only has information
on one of the links (the local access link), and wishes to measure on one of the links (the local access link), and wishes to measure
one or more BTC metrics. This scenario is quite common, and has one or more BTC metrics. The discussion of Section 6.2 applies here
spawned a substantial number of experimental measurement methods [ref as well.
to CAIDA survey page, etc.]. Many of these methods respect that
their users want a result fairly quickly and in a one-trial. Thus,
the measurement interval is kept short (a few seconds to a minute).
For long-term reporting, a sample of short term results need to be
summarized.
7.3. Measurement Interval 7.3. Measurement Interval
There are limits on a useful measurement interval for BTC. Three There are limits on a useful measurement interval for BTC. Three
factors that influence the interval duration are listed below: factors that influence the interval duration are listed below:
1. Measurements may choose to include or exclude the 3-way handshake 1. Measurements may choose to include or exclude the 3-way handshake
of TCP connection establishment, which requires at least 1.5 * of TCP connection establishment, which requires at least 1.5 *
RTT and contains both the delay of the path and the host RTT and contains both the delay of the path and the host
processing time for responses. However, user experience includes processing time for responses. However, user experience includes
the 3-way handshake for all new TCP connections. the 3-way handshake for all new TCP connections.
2. Measurements may choose to include or exclude Slow-Start, 2. Measurements may choose to include or exclude Slow-Start,
preferring instead to focus on a portion of the transfer that preferring instead to focus on a portion of the transfer that
represents "equilibrium" <<<< which needs a definition for this represents "equilibrium" (which needs to be defined for
purpose >>>>. However, user experience includes the Slow-Start particular circumstances if used). However, user experience
for all new TCP connections. includes the Slow-Start for all new TCP connections.
3. Measurements may choose to use a fixed block of data to transfer, 3. Measurements may choose to use a fixed block of data to transfer,
where the size of the block has a relationship to the file size where the size of the block has a relationship to the file size
of the application of interest. This approach yields variable of the application of interest. This approach yields variable
size measurement intervals, where a path faster BTC is measured size measurement intervals, where a path with faster BTC is
for less time than a slower path, an this has implications when measured for less time than a path with slower BTC, and this has
path impairments are time-varying, or transient. Users are implications when path impairments are time-varying, or
likely to turn their immediate attention elsewhere when a very transient. Users are likely to turn their immediate attention
large file must be transferred, thus they do not directly elsewhere when a very large file must be transferred, thus they
experience such a long transfer -- they see the result (success do not directly experience such a long transfer -- they see the
or fail) and possibly an objective measurement of the transfer result (success or fail) and possibly an objective measurement of
time (which will likely include the 3-way handshake, Slow-start, the transfer time (which will likely include the 3-way handshake,
and application file management processing time as well as the Slow-start, and application file management processing time as
BTC). well as the BTC).
Individual measurement intervals may be short or long, but there is a Individual measurement intervals may be short or long, but there is a
need to report the results on a long-term basis that captures the BTC need to report the results on a long-term basis that captures the BTC
variability experienced between each interval. Consistent BTC is a variability experienced between each interval. Consistent BTC is a
valuable commodity along with the value attained. valuable commodity along with the value attained.
7.4. Bulk Transfer Capacity Reporting 7.4. Bulk Transfer Capacity Reporting
When BTC of a link or path is estimated through some measurement When BTC of a link or path is estimated through some measurement
technique, the following parameters SHOULD be reported: technique, the following parameters SHOULD be reported:
skipping to change at page 21, line 42 skipping to change at page 21, line 37
o Maximum BTC that can be assessed in the measurement configuration o Maximum BTC that can be assessed in the measurement configuration
o The time and duration of the measurement o The time and duration of the measurement
o The number of BTC connections used simultaneously o The number of BTC connections used simultaneously
o *All* other parameters specific to the measurement method, o *All* other parameters specific to the measurement method,
especially the Congestion Control algorithm in use especially the Congestion Control algorithm in use
See also See also [RFC6349].
[http://tools.ietf.org/wg/ippm/draft-ietf-ippm-tcp-throughput-tm/]
Many methods of Bulk Transfer Capacity measurement have a maximum Many methods of Bulk Transfer Capacity measurement have a maximum
capacity that they can measure, and this maximum may be less than the capacity that they can measure, and this maximum may be less than the
available capacity of the link or path. Therefore, it is important available capacity of the link or path. Therefore, it is important
to specify the measured BTC value beyond which there will be no to specify the measured BTC value beyond which there will be no
measured improvement. measured improvement.
The Application Design audience may have a target capacity value and The Application Design audience may have a desired target capacity
simply wish to assess whether there is sufficient BTC. This case value and simply wish to assess whether there is sufficient BTC.
simplifies measurement of link and path capacity to some degree, as This case simplifies measurement of link and path capacity to some
long as the measurable maximum exceeds the target capacity. degree, as long as the measurable maximum exceeds the target
capacity.
7.5. Variability in Bulk Transfer Capacity 7.5. Variability in Bulk Transfer Capacity
As with most metrics and measurements, assessing the consistency or As with most metrics and measurements, assessing the consistency or
variability in the results gives a the user an intuitive feel for the variability in the results gives the user an intuitive feel for the
degree (or confidence) that any one value is representative of other degree (or confidence) that any one value is representative of other
results, or the underlying distribution from which these singleton results, or the underlying distribution from which these singleton
measurements have come. measurements have come.
With two questions looming: With two questions looming:
1. What ways can BTC be measured and summarized to describe the 1. What ways can BTC be measured and summarized to describe the
potential variability in a useful way? potential variability in a useful way?
2. How can the variability in BTC estimates be reported, so that the 2. How can the variability in BTC estimates be reported, so that the
confidence in the results is also conveyed? confidence in the results is also conveyed?
we suggest the methods of Section 6.6.1 above, and the additional we suggest the methods of Section 6.6.1 above, and the additional
results presentations given in [RFC6349]. results presentations given in [RFC6349].
8. Test Streams and Sample Size 8. Reporting on Test Streams and Sample Size
This section discusses two key aspects of measurement that are This section discusses two key aspects of measurement that are
sometimes omitted from the report: the description of the test stream sometimes omitted from the report: the description of the test stream
on which the measurements are based, and the sample size. on which the measurements are based, and the sample size.
8.1. Test Stream Characteristics 8.1. Test Stream Characteristics
Network Characterization has traditionally used Poisson-distributed Network Characterization has traditionally used Poisson-distributed
inter-packet spacing, as this provides an unbiased sample. The inter-packet spacing, as this provides an unbiased sample. The
average inter-packet spacing may be selected to allow observation of average inter-packet spacing may be selected to allow observation of
skipping to change at page 23, line 21 skipping to change at page 23, line 15
In practice, the sample size will be selected taking both statistical In practice, the sample size will be selected taking both statistical
and practical factors into account. Among these factors are: and practical factors into account. Among these factors are:
1. The estimated variability of the quantity being measured 1. The estimated variability of the quantity being measured
2. The desired confidence in the result (although this may be 2. The desired confidence in the result (although this may be
dependent on assumption of the underlying distribution of the dependent on assumption of the underlying distribution of the
measured quantity). measured quantity).
3. The effects of active measurement traffic on user traffic 3. The effects of active measurement traffic on user traffic.
4. etc.
A sample size may sometimes be referred to as "large". This is a A sample size may sometimes be referred to as "large". This is a
relative, and qualitative term. It is preferable to describe what relative, and qualitative term. It is preferable to describe what
one is attempting to achieve with their sample. For example, stating one is attempting to achieve with their sample. For example, stating
an implication may be helpful: this sample is large enough such that an implication may be helpful: this sample is large enough such that
a single outlying value at ten times the "typical" sample mean (the a single outlying value at ten times the "typical" sample mean (the
mean without the outlying value) would influence the mean by no more mean without the outlying value) would influence the mean by no more
than X. than X.
9. IANA Considerations 9. IANA Considerations
skipping to change at page 23, line 49 skipping to change at page 23, line 41
10. Security Considerations 10. Security Considerations
The security considerations that apply to any active measurement of The security considerations that apply to any active measurement of
live networks are relevant here as well. See [RFC4656]. live networks are relevant here as well. See [RFC4656].
11. Acknowledgements 11. Acknowledgements
The authors thank: Phil Chimento for his suggestion to employ The authors thank: Phil Chimento for his suggestion to employ
conditional distributions for Delay, Steve Konish Jr. for his careful conditional distributions for Delay, Steve Konish Jr. for his careful
review and suggestions, Dave Mcdysan and Don McLachlan for useful review and suggestions, Dave McDysan and Don McLachlan for useful
comments based on their long experience with measurement and comments based on their long experience with measurement and
reporting, and Matt Zekauskas for suggestions on organizing the memo reporting, Daniel Genin for his observation of non-orthogonality
for easier consumption. between Raw and Restricted Capacity metrics (and our omission of this
fact), and Matt Zekauskas for suggestions on organizing the memo for
easier consumption.
12. References 12. References
12.1. Normative References 12.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
skipping to change at page 25, line 31 skipping to change at page 25, line 25
[RFC5835] Morton, A. and S. Van den Berghe, "Framework for Metric [RFC5835] Morton, A. and S. Van den Berghe, "Framework for Metric
Composition", RFC 5835, April 2010. Composition", RFC 5835, April 2010.
[RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage, [RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage,
"Framework for TCP Throughput Testing", RFC 6349, "Framework for TCP Throughput Testing", RFC 6349,
August 2011. August 2011.
[Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data [Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data
communication service - IP packet transfer and communication service - IP packet transfer and
availability performance parameters", December 2002. availability performance parameters", December 2011.
[Y.1541] ITU-T Recommendation Y.1540, "Network Performance [Y.1541] ITU-T Recommendation Y.1540, "Network Performance
Objectives for IP-Based Services", February 2006. Objectives for IP-Based Services", February 2011.
Authors' Addresses Authors' Addresses
Al Morton Al Morton
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown, NJ 07748 Middletown, NJ 07748
USA USA
Phone: +1 732 420 1571 Phone: +1 732 420 1571
 End of changes. 47 change blocks. 
93 lines changed or deleted 98 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/