draft-ietf-ippm-testplan-rfc2680-00.txt   draft-ietf-ippm-testplan-rfc2680-01.txt 
Network Working Group L. Ciavattone Network Working Group L. Ciavattone
Internet-Draft AT&T Labs Internet-Draft AT&T Labs
Intended status: Informational R. Geib Intended status: Informational R. Geib
Expires: January 9, 2013 Deutsche Telekom Expires: July 5, 2013 Deutsche Telekom
A. Morton A. Morton
AT&T Labs AT&T Labs
M. Wieser M. Wieser
Technical University Darmstadt Technical University Darmstadt
July 8, 2012 January 1, 2013
Test Plan and Results for Advancing RFC 2680 on the Standards Track Test Plan and Results for Advancing RFC 2680 on the Standards Track
draft-ietf-ippm-testplan-rfc2680-00 draft-ietf-ippm-testplan-rfc2680-01
Abstract Abstract
This memo proposes to advance a performance metric RFC along the This memo proposes to advance a performance metric RFC along the
standards track, specifically RFC 2680 on One-way Loss Metrics. standards track, specifically RFC 2680 on One-way Loss Metrics.
Observing that the metric definitions themselves should be the Observing that the metric definitions themselves should be the
primary focus rather than the implementations of metrics, this memo primary focus rather than the implementations of metrics, this memo
describes the test procedures to evaluate specific metric requirement describes the test procedures to evaluate specific metric requirement
clauses to determine if the requirement has been interpreted and clauses to determine if the requirement has been interpreted and
implemented as intended. Two completely independent implementations implemented as intended. Two completely independent implementations
skipping to change at page 2, line 4 skipping to change at page 2, line 4
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 9, 2013. This Internet-Draft will expire on July 5, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
skipping to change at page 3, line 8 skipping to change at page 3, line 8
Without obtaining an adequate license from the person(s) controlling Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other it for publication as an RFC or to translate it into languages other
than English. than English.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1. RFC 2680 Coverage . . . . . . . . . . . . . . . . . . . . 4 1.1. RFC 2680 Coverage . . . . . . . . . . . . . . . . . . . . 5
2. A Definition-centric metric advancement process . . . . . . . 5 2. A Definition-centric metric advancement process . . . . . . . 5
3. Test configuration . . . . . . . . . . . . . . . . . . . . . . 5 3. Test configuration . . . . . . . . . . . . . . . . . . . . . . 6
4. Error Calibration, RFC 2680 . . . . . . . . . . . . . . . . . 9 4. Error Calibration, RFC 2680 . . . . . . . . . . . . . . . . . 10
4.1. Clock Synchronization Calibration . . . . . . . . . . . . 9 4.1. Clock Synchronization Calibration . . . . . . . . . . . . 10
4.2. Packet Loss Determination Error . . . . . . . . . . . . . 9 4.2. Packet Loss Determination Error . . . . . . . . . . . . . 10
5. Pre-determined Limits on Equivalence . . . . . . . . . . . . . 10 5. Pre-determined Limits on Equivalence . . . . . . . . . . . . . 11
6. Tests to evaluate RFC 2680 Specifications . . . . . . . . . . 11 6. Tests to evaluate RFC 2680 Specifications . . . . . . . . . . 12
6.1. One-way Loss, ADK Sample Comparison . . . . . . . . . . . 11 6.1. One-way Loss, ADK Sample Comparison . . . . . . . . . . . 12
6.1.1. 340B/Periodic Cross-imp. results . . . . . . . . . . . 12 6.1.1. 340B/Periodic Cross-imp. results . . . . . . . . . . . 13
6.1.2. 64B/Periodic Cross-imp. results . . . . . . . . . . . 13 6.1.2. 64B/Periodic Cross-imp. results . . . . . . . . . . . 14
6.1.3. 64B/Poisson Cross-imp. results . . . . . . . . . . . . 14 6.1.3. 64B/Poisson Cross-imp. results . . . . . . . . . . . . 15
6.1.4. Conclusions on the ADK Results for One-way Packet 6.1.4. Conclusions on the ADK Results for One-way Packet
Loss . . . . . . . . . . . . . . . . . . . . . . . . . 15 Loss . . . . . . . . . . . . . . . . . . . . . . . . . 16
6.2. One-way Loss, Delay threshold . . . . . . . . . . . . . . 15 6.2. One-way Loss, Delay threshold . . . . . . . . . . . . . . 16
6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 16 6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 17
6.2.2. Perfas Results for Loss Threshold . . . . . . . . . . 17 6.2.2. Perfas Results for Loss Threshold . . . . . . . . . . 18
6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 17 6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 18
6.3. One-way Loss with Out-of-Order Arrival . . . . . . . . . . 17 6.3. One-way Loss with Out-of-Order Arrival . . . . . . . . . . 18
6.4. Poisson Sending Process Evaluation . . . . . . . . . . . . 18 6.4. Poisson Sending Process Evaluation . . . . . . . . . . . . 19
6.4.1. NetProbe Results . . . . . . . . . . . . . . . . . . . 19 6.4.1. NetProbe Results . . . . . . . . . . . . . . . . . . . 20
6.4.2. Perfas Results . . . . . . . . . . . . . . . . . . . . 20 6.4.2. Perfas Results . . . . . . . . . . . . . . . . . . . . 21
6.4.3. Conclusions for Goodness-of-Fit . . . . . . . . . . . 22 6.4.3. Conclusions for Goodness-of-Fit . . . . . . . . . . . 23
6.5. Implementation of Statistics for One-way Delay . . . . . . 22 6.5. Implementation of Statistics for One-way Delay . . . . . . 23
7. Security Considerations . . . . . . . . . . . . . . . . . . . 22 7. Conclusions for RFC 2680bis . . . . . . . . . . . . . . . . . 23
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 8. Security Considerations . . . . . . . . . . . . . . . . . . . 24
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 23 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 24
10.1. Normative References . . . . . . . . . . . . . . . . . . . 23 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 24
10.2. Informative References . . . . . . . . . . . . . . . . . . 24 11.1. Normative References . . . . . . . . . . . . . . . . . . . 24
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25 11.2. Informative References . . . . . . . . . . . . . . . . . . 26
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26
1. Introduction 1. Introduction
The IETF (IP Performance Metrics working group, IPPM) has considered The IETF (IP Performance Metrics working group, IPPM) has considered
how to advance their metrics along the standards track since 2001. how to advance their metrics along the standards track since 2001.
A renewed work effort sought to investigate ways in which the A renewed work effort sought to investigate ways in which the
measurement variability could be reduced and thereby simplify the measurement variability could be reduced and thereby simplify the
problem of comparison for equivalence. problem of comparison for equivalence.
There is consensus [I-D.ietf-ippm-metrictest] that the metric There is consensus [RFC6576] that the metric definitions should be
definitions should be the primary focus of evaluation rather than the the primary focus of evaluation rather than the implementations of
implementations of metrics, and equivalent results are deemed to be metrics, and equivalent results are deemed to be evidence that the
evidence that the metric specifications are clear and unambiguous. metric specifications are clear and unambiguous. This is the metric
This is the metric specification equivalent of protocol specification equivalent of protocol interoperability. The
interoperability. The advancement process either produces confidence advancement process either produces confidence that the metric
that the metric definitions and supporting material are clearly definitions and supporting material are clearly worded and
worded and unambiguous, OR, identifies ways in which the metric unambiguous, OR, identifies ways in which the metric definitions
definitions should be revised to achieve clarity. should be revised to achieve clarity.
The process should also permit identification of options that were The process should also permit identification of options that were
not implemented, so that they can be removed from the advancing not implemented, so that they can be removed from the advancing
specification (this is an aspect more typical of protocol advancement specification (this is an aspect more typical of protocol advancement
along the standards track). along the standards track).
This memo's purpose is to implement the current approach for This memo's purpose is to implement the current approach for
[RFC2680]. [RFC2680].
In particular, this memo documents consensus on the extent of In particular, this memo documents consensus on the extent of
skipping to change at page 4, line 49 skipping to change at page 4, line 49
Another aspect of the metric RFC advancement process is the Another aspect of the metric RFC advancement process is the
requirement to document the work and results. The procedures of requirement to document the work and results. The procedures of
[RFC2026] are expanded in[RFC5657], including sample implementation [RFC2026] are expanded in[RFC5657], including sample implementation
and interoperability reports. This memo follows the template in and interoperability reports. This memo follows the template in
[I-D.morton-ippm-advance-metrics] for the report that accompanies the [I-D.morton-ippm-advance-metrics] for the report that accompanies the
protocol action request submitted to the Area Director, including protocol action request submitted to the Area Director, including
description of the test set-up, procedures, results for each description of the test set-up, procedures, results for each
implementation and conclusions. implementation and conclusions.
Although the conclusion reached through testing is that [RFC2680]
should be advanced on the Standards Track with modifications, the
revised text of RFC 2680bis is not yet ready for review. Therefore,
this memo documents the information to support [RFC2680] advancement,
and the approval of RFC2680bis is left for future action.
1.1. RFC 2680 Coverage 1.1. RFC 2680 Coverage
This plan is intended to cover all critical requirements and sections This plan is intended to cover all critical requirements and sections
of [RFC2680]. of [RFC2680].
Note that there are only five instances of the requirement term Note that there are only five instances of the requirement term
"MUST" in [RFC2680] outside of the boilerplate and [RFC2119] "MUST" in [RFC2680] outside of the boilerplate and [RFC2119]
reference. reference.
Material may be added as it is "discovered" (apparently, not all Material may be added as it is "discovered" (apparently, not all
requirements use requirements language). requirements use requirements language).
2. A Definition-centric metric advancement process 2. A Definition-centric metric advancement process
The process described in Section 3.5 of [I-D.ietf-ippm-metrictest] The process described in Section 3.5 of [RFC6576] takes as a first
takes as a first principle that the metric definitions, embodied in principle that the metric definitions, embodied in the text of the
the text of the RFCs, are the objects that require evaluation and RFCs, are the objects that require evaluation and possible revision
possible revision in order to advance to the next step on the in order to advance to the next step on the standards track.
standards track.
IF two implementations do not measure an equivalent singleton or IF two implementations do not measure an equivalent singleton or
sample, or produce the an equivalent statistic, sample, or produce the an equivalent statistic,
AND sources of measurement error do not adequately explain the lack AND sources of measurement error do not adequately explain the lack
of agreement, of agreement,
THEN the details of each implementation should be audited along with THEN the details of each implementation should be audited along with
the exact definition text, to determine if there is a lack of clarity the exact definition text, to determine if there is a lack of clarity
that has caused the implementations to vary in a way that affects the that has caused the implementations to vary in a way that affects the
skipping to change at page 5, line 47 skipping to change at page 6, line 8
Finally, all the findings MUST be documented in a report that can Finally, all the findings MUST be documented in a report that can
support advancement on the standards track, similar to those support advancement on the standards track, similar to those
described in [RFC5657]. The list of measurement devices used in described in [RFC5657]. The list of measurement devices used in
testing satisfies the implementation requirement, while the test testing satisfies the implementation requirement, while the test
results provide information on the quality of each specification in results provide information on the quality of each specification in
the metric RFC (the surrogate for feature interoperability). the metric RFC (the surrogate for feature interoperability).
3. Test configuration 3. Test configuration
One metric implementation used was NetProbe version 5.8.5, (an One metric implementation used was NetProbe version 5.8.5, (an
earlier version is used in the WIPM system and deployed world-wide). earlier version is used in the WIPM system and deployed world-wide
NetProbe uses UDP packets of variable size, and can produce test [WIPM]). NetProbe uses UDP packets of variable size, and can produce
streams with Periodic [RFC3432] or Poisson [RFC2330] sample test streams with Periodic [RFC3432] or Poisson [RFC2330] sample
distributions. distributions.
The other metric implementation used was Perfas+ version 3.1, The other metric implementation used was Perfas+ version 3.1,
developed by Deutsche Telekom. Perfas+ uses UDP unicast packets of developed by Deutsche Telekom [Perfas]. Perfas+ uses UDP unicast
variable size (but supports also TCP and multicast). Test streams packets of variable size (but supports also TCP and multicast). Test
with periodic, Poisson or uniform sample distributions may be used. streams with periodic, Poisson or uniform sample distributions may be
used.
Figure 1 shows a view of the test path as each Implementation's test Figure 1 shows a view of the test path as each Implementation's test
flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2), flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2),
based on Figure 1 of [I-D.ietf-ippm-metrictest]. based on Figure 1 of [RFC6576].
+----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+
|Imp1| |Imp1| ,---. |Imp2| |Imp2| |Imp1| |Imp1| ,---. |Imp2| |Imp2|
+----+ +----+ / \ +-------+ +----+ +----+ +----+ +----+ / \ +-------+ +----+ +----+
| V100 | V200 / \ | Tunnel| | V300 | V400 | V100 | V200 / \ | Tunnel| | V300 | V400
| | ( ) | Head | | | | | ( ) | Head | | |
+--------+ +------+ | |__| Router| +----------+ +--------+ +------+ | |__| Router| +----------+
|Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet | |Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet |
|Switch |--|Head |-| | | |Switch | |Switch |--|Head |-| | | |Switch |
+-+--+---+ |Router| | | +---+---+--+--+--+----+ +-+--+---+ |Router| | | +---+---+--+--+--+----+
skipping to change at page 9, line 45 skipping to change at page 10, line 45
synchronization in Section 2.8.3 of [RFC2680] (also required in synchronization in Section 2.8.3 of [RFC2680] (also required in
Section 3.7 of [RFC2680] for sample metrics). Section 3.7 of [RFC2680] for sample metrics).
Also, it is recommended to report the probability that a packet Also, it is recommended to report the probability that a packet
successfully arriving at the destination network interface is successfully arriving at the destination network interface is
incorrectly designated as lost due to resource exhaustion in Section incorrectly designated as lost due to resource exhaustion in Section
2.8.3 of [RFC2680]. 2.8.3 of [RFC2680].
4.1. Clock Synchronization Calibration 4.1. Clock Synchronization Calibration
For NetProbe and Perfas clock synchronization test results, refer to For NetProbe and Perfas+ clock synchronization test results, refer to
Section 4 of [I-D.ietf-ippm-testplan-rfc2679]. Section 4 of [RFC6808].
4.2. Packet Loss Determination Error 4.2. Packet Loss Determination Error
Since both measurement implementations have resource limitations, it Since both measurement implementations have resource limitations, it
is theoretically possible that these limits could be exceeded and a is theoretically possible that these limits could be exceeded and a
packet that arrived at the destination successfully might be packet that arrived at the destination successfully might be
discarded in error. discarded in error.
In previous test efforts [I-D.morton-ippm-advance-metrics], NetProbe In previous test efforts [I-D.morton-ippm-advance-metrics], NetProbe
produced 6 multicast streams with an aggregate bit rate over 53 produced 6 multicast streams with an aggregate bit rate over 53
skipping to change at page 11, line 13 skipping to change at page 12, line 13
11.4 of [RFC2330]. This is equivalent to a 95% confidence factor. 11.4 of [RFC2330]. This is equivalent to a 95% confidence factor.
6. Tests to evaluate RFC 2680 Specifications 6. Tests to evaluate RFC 2680 Specifications
This section describes some results from production network (cross- This section describes some results from production network (cross-
Internet) tests with measurement devices implementing IPPM metrics Internet) tests with measurement devices implementing IPPM metrics
and a network emulator to create relevant conditions, to determine and a network emulator to create relevant conditions, to determine
whether the metric definitions were interpreted consistently by whether the metric definitions were interpreted consistently by
implementors. implementors.
The procedures are similar contained in Appendix A.1 of The procedures are similar contained in Appendix A.1 of [RFC6576] for
[I-D.ietf-ippm-metrictest] for One-way Delay. One-way Delay.
6.1. One-way Loss, ADK Sample Comparison 6.1. One-way Loss, ADK Sample Comparison
This test determines if implementations produce results that appear This test determines if implementations produce results that appear
to come from a common packet loss distribution, as an overall to come from a common packet loss distribution, as an overall
evaluation of Section 3 of [RFC2680], "A Definition for Samples of evaluation of Section 3 of [RFC2680], "A Definition for Samples of
One-way Packet Loss". Same-implementation comparison results help to One-way Packet Loss". Same-implementation comparison results help to
set the threshold of equivalence that will be applied to cross- set the threshold of equivalence that will be applied to cross-
implementation comparisons. implementation comparisons.
skipping to change at page 11, line 51 skipping to change at page 12, line 51
implementations, using identical options and network emulator implementations, using identical options and network emulator
settings (if used). settings (if used).
3. Measure a sample of one-way packet loss singletons with *four or 3. Measure a sample of one-way packet loss singletons with *four or
more* instances of the *same* implementations, using identical more* instances of the *same* implementations, using identical
options, noting that connectivity differences SHOULD be the same options, noting that connectivity differences SHOULD be the same
as for the cross implementation testing. as for the cross implementation testing.
4. If less than ten test streams are available, skip to step 7. 4. If less than ten test streams are available, skip to step 7.
5. Apply the ADK comparison procedures (see Appendix C of 5. Apply the ADK comparison procedures (see Appendix C of [RFC6576])
[I-D.ietf-ippm-metrictest]) and determine the resolution and and determine the resolution and confidence factor for
confidence factor for distribution equivalence of each same- distribution equivalence of each same-implementation comparison
implementation comparison and each cross-implementation and each cross-implementation comparison.
comparison.
6. Take the coarsest resolution and confidence factor for 6. Take the coarsest resolution and confidence factor for
distribution equivalence from the same-implementation pairs, or distribution equivalence from the same-implementation pairs, or
the limit defined in Section 5 above, as a limit on the the limit defined in Section 5 above, as a limit on the
equivalence threshold for these experimental conditions. equivalence threshold for these experimental conditions.
7. Compare the cross-implementation ADK performance with the 7. Compare the cross-implementation ADK performance with the
equivalence threshold determined in step 5 to determine if equivalence threshold determined in step 5 to determine if
equivalence can be declared. equivalence can be declared.
The common parameters used for tests in this section are: The common parameters used for tests in this section are:
The cross-implementation comparison uses a simple ADK analysis The cross-implementation comparison uses a simple ADK analysis
[Rtool] [Radk], where all NetProbe loss counts are compared with all [Rtool] [Radk], where all NetProbe loss counts are compared with all
Perfas loss results. Perfas+ loss results.
In the result analysis of this section: In the result analysis of this section:
o All comparisons used 1 packet resolution. o All comparisons used 1 packet resolution.
o No Correction Factors were applied. o No Correction Factors were applied.
o The 0.95 confidence factor (1.960 for cross-implementation o The 0.95 confidence factor (1.960 for cross-implementation
comparison) was used. comparison) was used.
skipping to change at page 17, line 7 skipping to change at page 18, line 7
In NetProbe, the Loss Threshold is implemented uniformly over all In NetProbe, the Loss Threshold is implemented uniformly over all
packets as a post-processing routine. With the Loss Threshold set at packets as a post-processing routine. With the Loss Threshold set at
3 seconds, all packets with one-way delay >3 seconds are marked 3 seconds, all packets with one-way delay >3 seconds are marked
"Lost" and included in the Lost Packet list with their transmission "Lost" and included in the Lost Packet list with their transmission
time (as required in Section 3.3 of [RFC2680]). This resulted in 342 time (as required in Section 3.3 of [RFC2680]). This resulted in 342
packets designated as lost in one of the test streams (with average packets designated as lost in one of the test streams (with average
delay = 3.091 sec). delay = 3.091 sec).
6.2.2. Perfas Results for Loss Threshold 6.2.2. Perfas Results for Loss Threshold
Perfas uses a fixed Loss Threshold which was not adjustable during Perfas+ uses a fixed Loss Threshold which was not adjustable during
this study. The Loss Threshold is approximately one minute, and this study. The Loss Threshold is approximately one minute, and
emulation of a delay of this size was not attempted. However, it is emulation of a delay of this size was not attempted. However, it is
possible to implement any delay threshold desired with a post- possible to implement any delay threshold desired with a post-
processing routine and subsequent analysis. Using this method, 195 processing routine and subsequent analysis. Using this method, 195
packets would be declared lost (with average delay = 3.091 sec). packets would be declared lost (with average delay = 3.091 sec).
6.2.3. Conclusions for Loss Threshold 6.2.3. Conclusions for Loss Threshold
Both implementations assume that any constant delay value desired can Both implementations assume that any constant delay value desired can
be used as the Loss Threshold, since all delays are stored as a pair be used as the Loss Threshold, since all delays are stored as a pair
skipping to change at page 20, line 43 skipping to change at page 21, line 43
data: a27ms$s2[2:1001] and pexp data: a27ms$s2[2:1001] and pexp
AD = 0.6913, p-value = 0.5661 AD = 0.6913, p-value = 0.5661
alternative hypothesis: NA alternative hypothesis: NA
We see that both 100 and 1000 packet sets from two different streams We see that both 100 and 1000 packet sets from two different streams
(s1 and s2) all passed the AD <= 2.492 criterion. (s1 and s2) all passed the AD <= 2.492 criterion.
6.4.2. Perfas Results 6.4.2. Perfas Results
Section 11.4 of [RFC2330] suggests three possible measurement points Section 11.4 of [RFC2330] suggests three possible measurement points
to evaluate the Poisson distribution. The Perfas analysis uses "wire to evaluate the Poisson distribution. The Perfas+ analysis uses
times for the packets as recorded using a packet filter". However, "wire times for the packets as recorded using a packet filter".
due to limited access at the Perfas side of the test setup, the However, due to limited access at the Perfas+ side of the test setup,
captures were made after the Perfas streams traversed the production the captures were made after the Perfas+ streams traversed the
network, adding a small amount of unwanted delay variation to the production network, adding a small amount of unwanted delay variation
wire times (and possibly error due to packet loss). to the wire times (and possibly error due to packet loss).
The statistical summary for two Perfas streams is below: The statistical summary for two Perfas+ streams is below:
> summary(a27pe$p1) > summary(a27pe$p1)
Min. 1st Qu. Median Mean 3rd Qu. Max. Min. 1st Qu. Median Mean 3rd Qu. Max.
0.004 0.347 0.788 1.054 1.548 4.231 0.004 0.347 0.788 1.054 1.548 4.231
> summary(a27pe$p2) > summary(a27pe$p2)
Min. 1st Qu. Median Mean 3rd Qu. Max. Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0010 0.2710 0.7080 0.9696 1.3740 7.1160 0.0010 0.2710 0.7080 0.9696 1.3740 7.1160
We see that both the Means are near the specified lambda = 1. We see that both the Means are near the specified lambda = 1.
skipping to change at page 22, line 22 skipping to change at page 23, line 22
AD = 0.3381, p-value = 0.9068 AD = 0.3381, p-value = 0.9068
alternative hypothesis: NA alternative hypothesis: NA
> >
We see that both 193, 100, and 93 packet sets from two different We see that both 193, 100, and 93 packet sets from two different
streams (p1 and p2) all passed the AD <= 2.492 criterion. streams (p1 and p2) all passed the AD <= 2.492 criterion.
6.4.3. Conclusions for Goodness-of-Fit 6.4.3. Conclusions for Goodness-of-Fit
Both NetProbe and Perfas implementations produce adequate Poisson Both NetProbe and Perfas+ implementations produce adequate Poisson
distributions when according to the Anderson-Darling Goodness-of-Fit distributions when according to the Anderson-Darling Goodness-of-Fit
at the 5% significance (1-alpha = 0.05, or 95% confidence level). at the 5% significance (1-alpha = 0.05, or 95% confidence level).
6.5. Implementation of Statistics for One-way Delay 6.5. Implementation of Statistics for One-way Delay
We check which statistics were implemented, and report on those We check which statistics were implemented, and report on those
facts, noting that Section 4 of [RFC2680] does not specify the facts, noting that Section 4 of [RFC2680] does not specify the
calculations exactly, and gives only some illustrative examples. calculations exactly, and gives only some illustrative examples.
NetProbe Perfas NetProbe Perfas
4.1. Type-P-One-way-Delay-Packet-Loss-Ave yes yes 4.1. Type-P-One-way-Delay-Packet-Loss-Ave yes yes
(this is more commonly referred to as loss ratio) (this is more commonly referred to as loss ratio)
Implementation of Section 4 Statistics Implementation of Section 4 Statistics
We note that implementations refer to this metric as a loss ratio, We note that implementations refer to this metric as a loss ratio,
and this is an area for likely revision of the text to make it more and this is an area for likely revision of the text to make it more
consistent with wide-spread usage. consistent with wide-spread usage.
7. Security Considerations 7. Conclusions for RFC 2680bis
This memo concludes that [RFC2680] should be advanced on the
standards track, and recommends the following edits to improve the
text (which are not deemed significant enough to affect maturity).
o Revise Type-P-One-way-Delay-Packet-Loss-Ave to Type-P-One-way-
Delay-Packet-Loss-Ratio
o Regarding implementation of the loss delay threshold (section
6.2), the assumption of post-processing is compliant, and the text
of RFC 2680bis should be revised slightly to include this point.
We note that there are at least two Eratta on [RFC2680] and these
should be processed as part of the editing process.
8. Security Considerations
The security considerations that apply to any active measurement of The security considerations that apply to any active measurement of
live networks are relevant here as well. See [RFC4656] and live networks are relevant here as well. See [RFC4656] and
[RFC5357]. [RFC5357].
8. IANA Considerations 9. IANA Considerations
This memo makes no requests of IANA, and the authors hope that IANA This memo makes no requests of IANA, and the authors hope that IANA
personnel will be able to use their valuable time in other worthwhile personnel will be able to use their valuable time in other worthwhile
pursuits. pursuits.
9. Acknowledgements 10. Acknowledgements
The authors thank Lars Eggert for his continued encouragement to The authors thank Lars Eggert for his continued encouragement to
advance the IPPM metrics during his tenure as AD Advisor. advance the IPPM metrics during his tenure as AD Advisor.
Nicole Kowalski supplied the needed CPE router for the NetProbe side Nicole Kowalski supplied the needed CPE router for the NetProbe side
of the test set-up, and graciously managed her testing in spite of of the test set-up, and graciously managed her testing in spite of
issues caused by dual-use of the router. Thanks Nicole! issues caused by dual-use of the router. Thanks Nicole!
The "NetProbe Team" also acknowledges many useful discussions on The "NetProbe Team" also acknowledges many useful discussions on
statistical interpretation with Ganga Maguluri. statistical interpretation with Ganga Maguluri.
10. References 11. References
10.1. Normative References
[I-D.ietf-ippm-metrictest]
Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IPPM
standard advancement testing",
draft-ietf-ippm-metrictest-05 (work in progress),
November 2011.
[I-D.ietf-ippm-testplan-rfc2679] 11.1. Normative References
Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test
Plan and Results for Advancing RFC 2679 on the Standards
Track", draft-ietf-ippm-testplan-rfc2679-02 (work in
progress), June 2012.
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision [RFC2026] Bradner, S., "The Internet Standards Process -- Revision
3", BCP 9, RFC 2026, October 1996. 3", BCP 9, RFC 2026, October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
May 1998. May 1998.
skipping to change at page 24, line 36 skipping to change at page 25, line 43
May 2008. May 2008.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
RFC 5357, October 2008. RFC 5357, October 2008.
[RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation
and Implementation Reports for Advancement to Draft and Implementation Reports for Advancement to Draft
Standard", BCP 9, RFC 5657, September 2009. Standard", BCP 9, RFC 5657, September 2009.
10.2. Informative References [RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP
Performance Metrics (IPPM) Standard Advancement Testing",
BCP 176, RFC 6576, March 2012.
[RFC6808] Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test
Plan and Results Supporting Advancement of RFC 2679 on the
Standards Track", RFC 6808, December 2012.
11.2. Informative References
[ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling
Tests of fit, for continuous and discrete cases", Tests of fit, for continuous and discrete cases",
University of Washington, Technical Report No. 81, University of Washington, Technical Report No. 81,
May 1986. May 1986.
[I-D.morton-ippm-advance-metrics] [I-D.morton-ippm-advance-metrics]
Morton, A., "Lab Test Results for Advancing Metrics on the Morton, A., "Lab Test Results for Advancing Metrics on the
Standards Track", draft-morton-ippm-advance-metrics-02 Standards Track", draft-morton-ippm-advance-metrics-02
(work in progress), October 2010. (work in progress), October 2010.
[Perfas] Heidemann, C., "Qualitaet in IP-Netzen Messverfahren",
published by ITG Fachgruppe, 2nd meeting 5.2.3 (NGN) http:
//www.itg523.de/oeffentlich/01nov/
Heidemann_QOS_Messverfahren.pdf , November 2001.
[RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling
Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005.
[Radgof] Bellosta, C., "ADGofTest: Anderson-Darling Goodness-of-Fit [Radgof] Bellosta, C., "ADGofTest: Anderson-Darling Goodness-of-Fit
Test. R package version 0.3.", http://cran.r-project.org/ Test. R package version 0.3.", http://cran.r-project.org/
web/packages/ADGofTest/index.html, December 2011. web/packages/ADGofTest/index.html, December 2011.
[Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and
Combinations of Such Tests. R package version 1.0.", , Combinations of Such Tests. R package version 1.0.", ,
2008. 2008.
[Rtool] R Development Core Team, "R: A language and environment [Rtool] R Development Core Team, "R: A language and environment
for statistical computing. R Foundation for Statistical for statistical computing. R Foundation for Statistical
Computing, Vienna, Austria. ISBN 3-900051-07-0, URL Computing, Vienna, Austria. ISBN 3-900051-07-0, URL
http://www.R-project.org/", , 2011. http://www.R-project.org/", , 2011.
[WIPM] "AT&T Global IP Network",
http://ipnetwork.bgtmo.ip.att.net/pws/index.html, 2012.
Authors' Addresses Authors' Addresses
Len Ciavattone Len Ciavattone
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown, NJ 07748 Middletown, NJ 07748
USA USA
Phone: +1 732 420 1239 Phone: +1 732 420 1239
Fax: Fax:
 End of changes. 30 change blocks. 
92 lines changed or deleted 118 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/