draft-ietf-ippm-testplan-rfc2680-02.txt   draft-ietf-ippm-testplan-rfc2680-03.txt 
Network Working Group L. Ciavattone Network Working Group L. Ciavattone
Internet-Draft AT&T Labs Internet-Draft AT&T Labs
Intended status: Informational R. Geib Intended status: Informational R. Geib
Expires: August 21, 2013 Deutsche Telekom Expires: January 9, 2014 Deutsche Telekom
A. Morton A. Morton
AT&T Labs AT&T Labs
M. Wieser M. Wieser
Technical University Darmstadt Technical University Darmstadt
February 17, 2013 July 8, 2013
Test Plan and Results for Advancing RFC 2680 on the Standards Track Test Plan and Results for Advancing RFC 2680 on the Standards Track
draft-ietf-ippm-testplan-rfc2680-02 draft-ietf-ippm-testplan-rfc2680-03
Abstract Abstract
This memo proposes to advance a performance metric RFC along the This memo proposes to advance a performance metric RFC along the
standards track, specifically RFC 2680 on One-way Loss Metrics. standards track, specifically RFC 2680 on One-way Loss Metrics.
Observing that the metric definitions themselves should be the Observing that the metric definitions themselves should be the
primary focus rather than the implementations of metrics, this memo primary focus rather than the implementations of metrics, this memo
describes the test procedures to evaluate specific metric requirement describes the test procedures to evaluate specific metric requirement
clauses to determine if the requirement has been interpreted and clauses to determine if the requirement has been interpreted and
implemented as intended. Two completely independent implementations implemented as intended. Two completely independent implementations
have been tested against the key specifications of RFC 2680. have been tested against the key specifications of RFC 2680.
In this version, the results are presented in the R-tool output form.
Beautification is future work.
Requirements Language Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 21, 2013.
This Internet-Draft will expire on January 9, 2014.
Copyright Notice Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 29 skipping to change at page 3, line 29
6.1.3. 64B/Poisson Cross-imp. results . . . . . . . . . . . . 15 6.1.3. 64B/Poisson Cross-imp. results . . . . . . . . . . . . 15
6.1.4. Conclusions on the ADK Results for One-way Packet 6.1.4. Conclusions on the ADK Results for One-way Packet
Loss . . . . . . . . . . . . . . . . . . . . . . . . . 16 Loss . . . . . . . . . . . . . . . . . . . . . . . . . 16
6.2. One-way Loss, Delay threshold . . . . . . . . . . . . . . 16 6.2. One-way Loss, Delay threshold . . . . . . . . . . . . . . 16
6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 17 6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 17
6.2.2. Perfas Results for Loss Threshold . . . . . . . . . . 18 6.2.2. Perfas Results for Loss Threshold . . . . . . . . . . 18
6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 18 6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 18
6.3. One-way Loss with Out-of-Order Arrival . . . . . . . . . . 18 6.3. One-way Loss with Out-of-Order Arrival . . . . . . . . . . 18
6.4. Poisson Sending Process Evaluation . . . . . . . . . . . . 19 6.4. Poisson Sending Process Evaluation . . . . . . . . . . . . 19
6.4.1. NetProbe Results . . . . . . . . . . . . . . . . . . . 20 6.4.1. NetProbe Results . . . . . . . . . . . . . . . . . . . 20
6.4.2. Perfas Results . . . . . . . . . . . . . . . . . . . . 21 6.4.2. Perfas+ Results . . . . . . . . . . . . . . . . . . . 21
6.4.3. Conclusions for Goodness-of-Fit . . . . . . . . . . . 23 6.4.3. Conclusions for Goodness-of-Fit . . . . . . . . . . . 23
6.5. Implementation of Statistics for One-way Delay . . . . . . 23 6.5. Implementation of Statistics for One-way Loss . . . . . . 23
7. Conclusions for RFC 2680bis . . . . . . . . . . . . . . . . . 23 7. Conclusions for RFC 2680bis . . . . . . . . . . . . . . . . . 23
8. Security Considerations . . . . . . . . . . . . . . . . . . . 24 8. Security Considerations . . . . . . . . . . . . . . . . . . . 24
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 24 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 24
11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 24 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25
11.1. Normative References . . . . . . . . . . . . . . . . . . . 24 11.1. Normative References . . . . . . . . . . . . . . . . . . . 25
11.2. Informative References . . . . . . . . . . . . . . . . . . 26 11.2. Informative References . . . . . . . . . . . . . . . . . . 26
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 27
1. Introduction 1. Introduction
The IETF (IP Performance Metrics working group, IPPM) has considered The IETF (IP Performance Metrics working group, IPPM) has considered
how to advance their metrics along the standards track since 2001. how to advance their metrics along the standards track since 2001.
A renewed work effort sought to investigate ways in which the A renewed work effort sought to investigate ways in which the
measurement variability could be reduced and thereby simplify the measurement variability could be reduced and thereby simplify the
problem of comparison for equivalence. problem of comparison for equivalence.
skipping to change at page 14, line 6 skipping to change at page 14, line 6
o Test duration = 1200 seconds (during April 7, 2011, EDT) o Test duration = 1200 seconds (during April 7, 2011, EDT)
The netem emulator was set for 100ms constant delay, with 10% loss The netem emulator was set for 100ms constant delay, with 10% loss
ratio. In this experiment, the netem emulator was configured to ratio. In this experiment, the netem emulator was configured to
operate independently on each VLAN and thus the emulator itself is a operate independently on each VLAN and thus the emulator itself is a
potential source of error when comparing streams that traverse the potential source of error when comparing streams that traverse the
test path in different directions. test path in different directions.
A07bps_loss <- c(114, 175, 138, 142, 181, 105) (NetProbe) A07bps_loss <- c(114, 175, 138, 142, 181, 105) (NetProbe)
A07per_loss <- c(115, 128, 136, 127, 139, 138) (Perfas) A07per_loss <- c(115, 128, 136, 127, 139, 138) (Perfas+)
> A07bps_loss <- c(114, 175, 138, 142, 181, 105) > A07bps_loss <- c(114, 175, 138, 142, 181, 105)
> A07per_loss <- c(115, 128, 136, 127, 139, 138) > A07per_loss <- c(115, 128, 136, 127, 139, 138)
> >
> A07cross_loss_ADK <- adk.test(A07bps_loss, A07per_loss) > A07cross_loss_ADK <- adk.test(A07bps_loss, A07per_loss)
> A07cross_loss_ADK > A07cross_loss_ADK
Anderson-Darling k-sample test. Anderson-Darling k-sample test.
Number of samples: 2 Number of samples: 2
Sample sizes: 6 6 Sample sizes: 6 6
skipping to change at page 15, line 5 skipping to change at page 15, line 5
o IP header + payload = 64 octets o IP header + payload = 64 octets
o Periodic sampling at 1 packet per second o Periodic sampling at 1 packet per second
o Test duration = 300 seconds (during March 24, 2011, EDT) o Test duration = 300 seconds (during March 24, 2011, EDT)
The netem emulator was set for 0ms constant delay, with 10% loss The netem emulator was set for 0ms constant delay, with 10% loss
ratio. ratio.
> M24per_loss <- c(42,34,35,35) (Perfas) > M24per_loss <- c(42,34,35,35) (Perfas+)
> M24apd_23BC_loss <- c(27,39,29,24) (NetProbe) > M24apd_23BC_loss <- c(27,39,29,24) (NetProbe)
> M24apd_loss23BC_ADK <- adk.test(M24apd_23BC_loss,M24per_loss) > M24apd_loss23BC_ADK <- adk.test(M24apd_23BC_loss,M24per_loss)
> M24apd_loss23BC_ADK > M24apd_loss23BC_ADK
Anderson-Darling k-sample test. Anderson-Darling k-sample test.
Number of samples: 2 Number of samples: 2
Sample sizes: 4 4 Sample sizes: 4 4
Total number of values: 8 Total number of values: 8
Number of unique values: 7 Number of unique values: 7
skipping to change at page 16, line 6 skipping to change at page 16, line 6
o Poisson sampling at lambda = 1 packet per second o Poisson sampling at lambda = 1 packet per second
o Test duration = 20 minutes (during April 27, 2011, EDT) o Test duration = 20 minutes (during April 27, 2011, EDT)
The netem configuration was 0ms delay and 10% loss, but there were The netem configuration was 0ms delay and 10% loss, but there were
two passes through an emulator for each stream, and loss emulation two passes through an emulator for each stream, and loss emulation
was present for 18 minutes of the 20 minute test . was present for 18 minutes of the 20 minute test .
A27aps_loss <- c(91,110,113,102,111,109,112,113) (NetProbe) A27aps_loss <- c(91,110,113,102,111,109,112,113) (NetProbe)
A27per_loss <- c(95,123,126,114) (Perfas) A27per_loss <- c(95,123,126,114) (Perfas+)
A27cross_loss_ADK <- adk.test(A27aps_loss, A27per_loss) A27cross_loss_ADK <- adk.test(A27aps_loss, A27per_loss)
> A27cross_loss_ADK > A27cross_loss_ADK
Anderson-Darling k-sample test. Anderson-Darling k-sample test.
Number of samples: 2 Number of samples: 2
Sample sizes: 8 4 Sample sizes: 8 4
Total number of values: 12 Total number of values: 12
Number of unique values: 11 Number of unique values: 11
skipping to change at page 19, line 5 skipping to change at page 19, line 5
packet's emulated delay is independent from others, and 10% loss. packet's emulated delay is independent from others, and 10% loss.
The tests described in this section used: The tests described in this section used:
o IP header + payload = 64 octets o IP header + payload = 64 octets
o Periodic sampling = 1 packet per second o Periodic sampling = 1 packet per second
o Test duration = 600 seconds (during May 2, 2011, EDT) o Test duration = 600 seconds (during May 2, 2011, EDT)
> Y02aps_loss <- c(53,45,67,55) (NetProbe) > Y02aps_loss <- c(53,45,67,55) (NetProbe)
> Y02per_loss <- c(59,62,67,69) (Perfas) > Y02per_loss <- c(59,62,67,69) (Perfas+)
> Y02cross_loss_ADK <- adk.test(Y02aps_loss, Y02per_loss) > Y02cross_loss_ADK <- adk.test(Y02aps_loss, Y02per_loss)
> Y02cross_loss_ADK > Y02cross_loss_ADK
Anderson-Darling k-sample test. Anderson-Darling k-sample test.
Number of samples: 2 Number of samples: 2
Sample sizes: 4 4 Sample sizes: 4 4
Total number of values: 8 Total number of values: 8
Number of unique values: 7 Number of unique values: 7
Mean of Anderson Darling Criterion: 1 Mean of Anderson Darling Criterion: 1
skipping to change at page 21, line 40 skipping to change at page 21, line 40
Anderson-Darling GoF Test Anderson-Darling GoF Test
data: a27ms$s2[2:1001] and pexp data: a27ms$s2[2:1001] and pexp
AD = 0.6913, p-value = 0.5661 AD = 0.6913, p-value = 0.5661
alternative hypothesis: NA alternative hypothesis: NA
We see that both 100 and 1000 packet sets from two different streams We see that both 100 and 1000 packet sets from two different streams
(s1 and s2) all passed the AD <= 2.492 criterion. (s1 and s2) all passed the AD <= 2.492 criterion.
6.4.2. Perfas Results 6.4.2. Perfas+ Results
Section 11.4 of [RFC2330] suggests three possible measurement points Section 11.4 of [RFC2330] suggests three possible measurement points
to evaluate the Poisson distribution. The Perfas+ analysis uses to evaluate the Poisson distribution. The Perfas+ analysis uses
"wire times for the packets as recorded using a packet filter". "wire times for the packets as recorded using a packet filter".
However, due to limited access at the Perfas+ side of the test setup, However, due to limited access at the Perfas+ side of the test setup,
the captures were made after the Perfas+ streams traversed the the captures were made after the Perfas+ streams traversed the
production network, adding a small amount of unwanted delay variation production network, adding a small amount of unwanted delay variation
to the wire times (and possibly error due to packet loss). to the wire times (and possibly error due to packet loss).
The statistical summary for two Perfas+ streams is below: The statistical summary for two Perfas+ streams is below:
skipping to change at page 23, line 26 skipping to change at page 23, line 26
We see that both 193, 100, and 93 packet sets from two different We see that both 193, 100, and 93 packet sets from two different
streams (p1 and p2) all passed the AD <= 2.492 criterion. streams (p1 and p2) all passed the AD <= 2.492 criterion.
6.4.3. Conclusions for Goodness-of-Fit 6.4.3. Conclusions for Goodness-of-Fit
Both NetProbe and Perfas+ implementations produce adequate Poisson Both NetProbe and Perfas+ implementations produce adequate Poisson
distributions when according to the Anderson-Darling Goodness-of-Fit distributions when according to the Anderson-Darling Goodness-of-Fit
at the 5% significance (1-alpha = 0.05, or 95% confidence level). at the 5% significance (1-alpha = 0.05, or 95% confidence level).
6.5. Implementation of Statistics for One-way Delay 6.5. Implementation of Statistics for One-way Loss
We check which statistics were implemented, and report on those We check which statistics were implemented, and report on those
facts, noting that Section 4 of [RFC2680] does not specify the facts, noting that Section 4 of [RFC2680] does not specify the
calculations exactly, and gives only some illustrative examples. calculations exactly, and gives only some illustrative examples.
NetProbe Perfas NetProbe Perfas
4.1. Type-P-One-way-Packet-Loss-Average yes yes 4.1. Type-P-One-way-Packet-Loss-Average yes yes
(this is more commonly referred to as loss ratio) (this is more commonly referred to as loss ratio)
skipping to change at page 24, line 16 skipping to change at page 24, line 16
Packet-Loss-Ratio Packet-Loss-Ratio
o Regarding implementation of the loss delay threshold (section o Regarding implementation of the loss delay threshold (section
6.2), the assumption of post-processing is compliant, and the text 6.2), the assumption of post-processing is compliant, and the text
of RFC 2680bis should be revised slightly to include this point. of RFC 2680bis should be revised slightly to include this point.
o The IETF has reached consensus on guidance for reporting metrics o The IETF has reached consensus on guidance for reporting metrics
in [RFC6703], and this memo should be referenced in RFC2680bis to in [RFC6703], and this memo should be referenced in RFC2680bis to
incorporate recent experience where appropriate. incorporate recent experience where appropriate.
We note that there are at least two Eratta on [RFC2680] and these We note that there are at least two Errata on [RFC2680] and these
should be processed as part of the editing process. should be processed as part of the editing process.
We recognize the existence of BCP 170 [RFC6390] providing guidelines
for development of drafts describing new performance metrics.
However, the advancement of [RFC2680] represents fine-tuning of long-
standing specifications based on experience that helped to formulate
BCP 170, and material that satisfies some of the requirements of
[RFC6390] can be found in other RFCs, such as the IPPM Framework
[RFC2330]. Thus, no specific changes to address BCP 170 guidelines
are recommended for RFC 2680bis.
8. Security Considerations 8. Security Considerations
The security considerations that apply to any active measurement of The security considerations that apply to any active measurement of
live networks are relevant here as well. See [RFC4656] and live networks are relevant here as well. See [RFC4656] and
[RFC5357]. [RFC5357].
9. IANA Considerations 9. IANA Considerations
This memo makes no requests of IANA, and the authors hope that IANA This memo makes no requests of IANA, and the authors hope that IANA
personnel will be able to use their valuable time in other worthwhile personnel will be able to use their valuable time in other worthwhile
skipping to change at page 25, line 46 skipping to change at page 26, line 7
May 2008. May 2008.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
RFC 5357, October 2008. RFC 5357, October 2008.
[RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation
and Implementation Reports for Advancement to Draft and Implementation Reports for Advancement to Draft
Standard", BCP 9, RFC 5657, September 2009. Standard", BCP 9, RFC 5657, September 2009.
[RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New
Performance Metric Development", BCP 170, RFC 6390,
October 2011.
[RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP [RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP
Performance Metrics (IPPM) Standard Advancement Testing", Performance Metrics (IPPM) Standard Advancement Testing",
BCP 176, RFC 6576, March 2012. BCP 176, RFC 6576, March 2012.
[RFC6703] Morton, A., Ramachandran, G., and G. Maguluri, "Reporting [RFC6703] Morton, A., Ramachandran, G., and G. Maguluri, "Reporting
IP Network Performance Metrics: Different Points of View", IP Network Performance Metrics: Different Points of View",
RFC 6703, August 2012. RFC 6703, August 2012.
[RFC6808] Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test [RFC6808] Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test
Plan and Results Supporting Advancement of RFC 2679 on the Plan and Results Supporting Advancement of RFC 2679 on the
 End of changes. 18 change blocks. 
19 lines changed or deleted 30 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/