draft-ietf-ippm-testplan-rfc2679-01.txt   draft-ietf-ippm-testplan-rfc2679-02.txt 
Network Working Group L. Ciavattone Network Working Group L. Ciavattone
Internet-Draft AT&T Labs Internet-Draft AT&T Labs
Intended status: Informational R. Geib Intended status: Informational R. Geib
Expires: September 11, 2012 Deutsche Telekom Expires: December 26, 2012 Deutsche Telekom
A. Morton A. Morton
AT&T Labs AT&T Labs
M. Wieser M. Wieser
Technical University Darmstadt Technical University Darmstadt
March 10, 2012 June 24, 2012
Test Plan and Results for Advancing RFC 2679 on the Standards Track Test Plan and Results for Advancing RFC 2679 on the Standards Track
draft-ietf-ippm-testplan-rfc2679-01 draft-ietf-ippm-testplan-rfc2679-02
Abstract Abstract
This memo proposes to advance a performance metric RFC along the This memo proposes to advance a performance metric RFC along the
standards track, specifically RFC 2679 on One-way Delay Metrics. standards track, specifically RFC 2679 on One-way Delay Metrics.
Observing that the metric definitions themselves should be the Observing that the metric definitions themselves should be the
primary focus rather than the implementations of metrics, this memo primary focus rather than the implementations of metrics, this memo
describes the test procedures to evaluate specific metric requirement describes the test procedures to evaluate specific metric requirement
clauses to determine if the requirement has been interpreted and clauses to determine if the requirement has been interpreted and
implemented as intended. Two completely independent implementations implemented as intended. Two completely independent implementations
skipping to change at page 1, line 48 skipping to change at page 1, line 48
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 11, 2012. This Internet-Draft will expire on December 26, 2012.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 9 skipping to change at page 3, line 9
the copyright in such materials, this document may not be modified the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other it for publication as an RFC or to translate it into languages other
than English. than English.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. A Definition-centric metric advancement process . . . . . . . 5 2. A Definition-centric metric advancement process . . . . . . . 5
3. Test configuration . . . . . . . . . . . . . . . . . . . . . . 6 3. Test configuration . . . . . . . . . . . . . . . . . . . . . . 5
4. Error Calibration, RFC 2679 . . . . . . . . . . . . . . . . . 10 4. Error Calibration, RFC 2679 . . . . . . . . . . . . . . . . . 9
4.1. NetProbe Error and Type-P . . . . . . . . . . . . . . . . 11 4.1. NetProbe Error and Type-P . . . . . . . . . . . . . . . . 10
4.2. Perfas Error and Type-P . . . . . . . . . . . . . . . . . 13 4.2. Perfas+ Error and Type-P . . . . . . . . . . . . . . . . . 12
5. Pre-determined Limits on Equivalence . . . . . . . . . . . . . 14 5. Pre-determined Limits on Equivalence . . . . . . . . . . . . . 13
6. Tests to evaluate RFC 2679 Specifications . . . . . . . . . . 14 6. Tests to evaluate RFC 2679 Specifications . . . . . . . . . . 13
6.1. One-way Delay, ADK Sample Comparison - Same & Cross 6.1. One-way Delay, ADK Sample Comparison - Same & Cross
Implementation . . . . . . . . . . . . . . . . . . . . . . 15 Implementation . . . . . . . . . . . . . . . . . . . . . . 14
6.1.1. NetProbe Same-implementation results . . . . . . . . . 16 6.1.1. NetProbe Same-implementation results . . . . . . . . . 15
6.1.2. Perfas Same-implementation results . . . . . . . . . . 17 6.1.2. Perfas+ Same-implementation results . . . . . . . . . 16
6.1.3. One-way Delay, Cross-Implementation ADK Comparison . . 18 6.1.3. One-way Delay, Cross-Implementation ADK Comparison . . 17
6.1.4. Conclusions on the ADK Results for One-way Delay . . . 18 6.1.4. Conclusions on the ADK Results for One-way Delay . . . 17
6.1.5. Additional Investigations . . . . . . . . . . . . . . 19 6.1.5. Additional Investigations . . . . . . . . . . . . . . 18
6.2. One-way Delay, Loss threshold, RFC 2679 . . . . . . . . . 22 6.2. One-way Delay, Loss threshold, RFC 2679 . . . . . . . . . 21
6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 23 6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 22
6.2.2. Perfas Results for Loss Threshold . . . . . . . . . . 23 6.2.2. Perfas+ Results for Loss Threshold . . . . . . . . . . 22
6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 23 6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 22
6.3. One-way Delay, First-bit to Last bit, RFC 2679 . . . . . . 24 6.3. One-way Delay, First-bit to Last bit, RFC 2679 . . . . . . 22
6.3.1. NetProbe and Perfas Results for Serialization . . . . 24 6.3.1. NetProbe and Perfas+ Results for Serialization . . . . 23
6.3.2. Conclusions for Serialization . . . . . . . . . . . . 25 6.3.2. Conclusions for Serialization . . . . . . . . . . . . 24
6.4. One-way Delay, Difference Sample Metric (Lab) . . . . . . 26 6.4. One-way Delay, Difference Sample Metric (Lab) . . . . . . 25
6.4.1. NetProbe results for Differential Delay . . . . . . . 26 6.4.1. NetProbe results for Differential Delay . . . . . . . 25
6.4.2. Perfas results for Differential Delay . . . . . . . . 27 6.4.2. Perfas+ results for Differential Delay . . . . . . . . 26
6.4.3. Conclusions for Differential Delay . . . . . . . . . . 27 6.4.3. Conclusions for Differential Delay . . . . . . . . . . 26
6.5. Implementation of Statistics for One-way Delay . . . . . . 27 6.5. Implementation of Statistics for One-way Delay . . . . . . 26
7. Security Considerations . . . . . . . . . . . . . . . . . . . 28 7. Security Considerations . . . . . . . . . . . . . . . . . . . 27
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 28 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 27
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 29 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 28
10.1. Normative References . . . . . . . . . . . . . . . . . . . 29 10.1. Normative References . . . . . . . . . . . . . . . . . . . 28
10.2. Informative References . . . . . . . . . . . . . . . . . . 30 10.2. Informative References . . . . . . . . . . . . . . . . . . 28
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 30 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 29
1. Introduction 1. Introduction
The IETF (IP Performance Metrics working group, IPPM) has considered The IETF IP Performance Metrics (IPPM) working group has considered
how to advance their metrics along the standards track since 2001, how to advance their metrics along the standards track since 2001,
with the initial publication of Bradner/Paxson/Mankin's memo [ref to with the initial publication of Bradner/Paxson/Mankin's memo
work in progress, draft-bradner-metricstest-]. The original proposal [I-D.bradner-metricstest]. The original proposal was to compare the
was to compare the results of implementations of the metrics, because results of implementations of the metrics, because the usual
the usual procedures for advancing protocols did not appear to apply. procedures for advancing protocols did not appear to apply. It was
It was found to be difficult to achieve consensus on exactly how to found to be difficult to achieve consensus on exactly how to compare
compare implementations, since there were many legitimate sources of implementations, since there were many legitimate sources of
variation that would emerge in the results despite the best attempts variation that would emerge in the results despite the best attempts
to keep the network paths equal, and because considerable variation to keep the network paths equal, and because considerable variation
was allowed in the parameters (and therefore implementation) of each was allowed in the parameters (and therefore implementation) of each
metric. Flexibility in metric definitions, essential for metric. Flexibility in metric definitions, essential for
customization and broad appeal, made the comparison task quite customization and broad appeal, made the comparison task quite
difficult. difficult.
A renewed work effort sought to investigate ways in which the A renewed work effort sought to investigate ways in which the
measurement variability could be reduced and thereby simplify the measurement variability could be reduced and thereby simplify the
problem of comparison for equivalence. problem of comparison for equivalence.
There is consensus represented in [I-D.ietf-ippm-metrictest] that the There is consensus represented in [RFC6576] that the metric
metric definitions should be the primary focus of evaluation rather definitions should be the primary focus of evaluation rather than the
than the implementations of metrics, and equivalent results are implementations of metrics, and equivalent results are deemed to be
deemed to be evidence that the metric specifications are clear and evidence that the metric specifications are clear and unambiguous.
unambiguous. This is the metric specification equivalent of protocol This is the metric specification equivalent of protocol
interoperability. The advancement process either produces confidence interoperability. The advancement process either produces confidence
that the metric definitions and supporting material are clearly that the metric definitions and supporting material are clearly
worded and unambiguous, OR, identifies ways in which the metric worded and unambiguous, OR, identifies ways in which the metric
definitions should be revised to achieve clarity. definitions should be revised to achieve clarity.
The process should also permit identification of options that were The process should also permit identification of options that were
not implemented, so that they can be removed from the advancing not implemented, so that they can be removed from the advancing
specification (this is an aspect more typical of protocol advancement specification (this is an aspect more typical of protocol advancement
along the standards track). along the standards track).
skipping to change at page 5, line 7 skipping to change at page 5, line 7
In particular, consensus is sought on the extent of tolerable errors In particular, consensus is sought on the extent of tolerable errors
when assessing equivalence in the results. In discussions, the IPPM when assessing equivalence in the results. In discussions, the IPPM
working group agreed that test plan and procedures should include the working group agreed that test plan and procedures should include the
threshold for determining equivalence, and this information should be threshold for determining equivalence, and this information should be
available in advance of cross-implementation comparisons. This memo available in advance of cross-implementation comparisons. This memo
includes procedures for same-implementation comparisons to help set includes procedures for same-implementation comparisons to help set
the equivalence threshold. the equivalence threshold.
Another aspect of the metric RFC advancement process is the Another aspect of the metric RFC advancement process is the
requirement to document the work and results. The procedures of requirement to document the work and results. The procedures of
[RFC2026] are expanded in[RFC5657], including sample implementation [RFC2026] are expanded in [RFC5657], including sample implementation
and interoperability reports. This memo follows the template in and interoperability reports. This memo expands on these RFCs and
[I-D.morton-ippm-advance-metrics] for the report that accompanies the the examples in Appendix A of [RFC6576] for the procedure and report
protocol action request submitted to the Area Director, including that accompanies the protocol action request submitted to the Area
description of the test set-up, procedures, results for each Director, including description of the test set-up, results for each
implementation and conclusions. implementation, and conclusions.
2. A Definition-centric metric advancement process 2. A Definition-centric metric advancement process
The process described in Section 3.5 of [I-D.ietf-ippm-metrictest] The process described in Section 3.5 of [RFC6576] takes as a first
takes as a first principle that the metric definitions, embodied in principle that the metric definitions, embodied in the text of the
the text of the RFCs, are the objects that require evaluation and RFCs, are the objects that require evaluation and possible revision
possible revision in order to advance to the next step on the in order to advance to the next step on the standards track. This
standards track. memo follows that process.
IF two implementations do not measure an equivalent singleton or
sample, or produce the an equivalent statistic,
AND sources of measurement error do not adequately explain the lack
of agreement,
THEN the details of each implementation should be audited along with
the exact definition text, to determine if there is a lack of clarity
that has caused the implementations to vary in a way that affects the
correspondence of the results.
IF there was a lack of clarity or multiple legitimate interpretations
of the definition text,
THEN the text should be modified and the resulting memo proposed for
consensus and advancement along the standards track.
Finally, all the findings MUST be documented in a report that can
support advancement on the standards track, similar to those
described in [RFC5657]. The list of measurement devices used in
testing satisfies the implementation requirement, while the test
results provide information on the quality of each specification in
the metric RFC (the surrogate for feature interoperability).
The figure below illustrates this process:
,---.
/ \
( Start )
\ / Implementations
`-+-' +-------+
| /| 1 `.
+---+----+ / +-------+ `.-----------+ ,-------.
| RFC | / |Check for | ,' was RFC `. YES
| | / |Equivalence..... clause x -------+
| |/ +-------+ |under | `. clear? ,' |
| Metric \.....| 2 ....relevant | `---+---' +----+---+
| Metric |\ +-------+ |identical | No | |Report |
| Metric | \ |network | +---+---. |results+|
| ... | \ |conditions | |Modify | |Advance |
| | \ +-------+ | | |Spec +----+ RFC |
+--------+ \| n |.'+-----------+ +-------+ |request?|
+-------+ +--------+
3. Test configuration 3. Test configuration
One metric implementation used was NetProbe version 5.8.5, (an One metric implementation used was NetProbe version 5.8.5, (an
earlier version is used in the WIPM system and deployed world-wide). earlier version is used in the AT&T's IP network performance
NetProbe uses UDP packets of variable size, and can produce test measurement system and deployed world-wide [WIPM]). NetProbe uses
streams with Periodic [RFC3432] or Poisson [RFC2330] sample UDP packets of variable size, and can produce test streams with
distributions. Periodic [RFC3432] or Poisson [RFC2330] sample distributions.
The other metric implementation used was Perfas+ version 3.1, The other metric implementation used was Perfas+ version 3.1,
developed by Deutsche Telekom. Perfas+ uses UDP unicast packets of developed by Deutsche Telekom. Perfas+ uses UDP unicast packets of
variable size (but supports also TCP and multicast). Test streams variable size (but supports also TCP and multicast). Test streams
with periodic, Poisson or uniform sample distributions may be used. with Periodic, Poisson or uniform sample distributions may be used.
Figure 2 shows a view of the test path as each Implementation's test Figure 2 shows a view of the test path as each Implementation's test
flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2), flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2),
based on Figure 1 of [I-D.ietf-ippm-metrictest]. based on Figures 2 and 3 of [RFC6576].
+----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+
|Imp1| |Imp1| ,---. |Imp2| |Imp2| |Imp1| |Imp1| ,---. |Imp2| |Imp2|
+----+ +----+ / \ +-------+ +----+ +----+ +----+ +----+ / \ +-------+ +----+ +----+
| V100 | V200 / \ | Tunnel| | V300 | V400 | V100 | V200 / \ | Tunnel| | V300 | V400
| | ( ) | Head | | | | | ( ) | Head | | |
+--------+ +------+ | |__| Router| +----------+ +--------+ +------+ | |__| Router| +----------+
|Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet | |Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet |
|Switch |--|Head |-| | | |Switch | |Switch |--|Head |-| | | |Switch |
+-+--+---+ |Router| | | +---+---+--+--+--+----+ +-+--+---+ |Router| | | +---+---+--+--+--+----+
|__| +--A---+ ( ) |Network| |__| |__| +--A---+ ( ) |Network| |__|
\ / |Emulat.| \ / |Emulat.|
U-turn \ / |"netem"| U-turn U-turn \ / |"netem"| U-turn
V300 to V400 `-+-' +-------+ V100 to V200 V300 to V400 `-+-' +-------+ V100 to V200
skipping to change at page 8, line 8 skipping to change at page 7, line 8
and L2TPv3 headers are intended to conceal the test equipment and L2TPv3 headers are intended to conceal the test equipment
addresses and ports from hash functions that would tend to spread addresses and ports from hash functions that would tend to spread
different test streams across parallel network resources, with likely different test streams across parallel network resources, with likely
variation in performance as a result. variation in performance as a result.
At each end of the tunnel, one pair of VLANs encapsulated in the At each end of the tunnel, one pair of VLANs encapsulated in the
tunnel are looped-back so that test traffic is returned to each test tunnel are looped-back so that test traffic is returned to each test
site. Thus, test streams traverse the L2TP tunnel twice, but appear site. Thus, test streams traverse the L2TP tunnel twice, but appear
to be one-way tests from the test equipment point of view. to be one-way tests from the test equipment point of view.
The network emulator is a host running Fedora 14 Linux The network emulator is a host running Fedora 14 Linux [Fedora14]
[http://fedoraproject.org/] with IP forwarding enabled and the with IP forwarding enabled and the "netem" Network emulator [netem]
"netem" Network emulator as part of the Fedora Kernel 2.6.35.11 [http loaded and operating as part of the Fedora Kernel 2.6.35.11.
://www.linuxfoundation.org/collaborate/workgroups/networking/netem] Connectivity across the netem/Fedora host was accomplished by
loaded and operating. Connectivity across the netem/Fedora host was bridging Ethernet VLAN interfaces together with "brctl" commands
accomplished by bridging Ethernet VLAN interfaces together with (e.g., eth1.100 <-> eth2.100). The netem emulator was activated on
"brctl" commands (e.g., eth1.100 <-> eth2.100). The netem emulator one interface (eth1) and only operates on test streams traveling in
was activated on one interface (eth1) and only operates on test one direction. In some tests, independent netem instances operated
streams traveling in one direction. In some tests, independent netem separately on each VLAN.
instances operated separately on each VLAN.
The links between the netem emulator host and router and switch were The links between the netem emulator host and router and switch were
found to be 100baseTx-HD (100Mbps half duplex) as reported by "mii- found to be 100baseTx-HD (100Mbps half duplex) when the testing was
tool"when the testing was complete. Use of Half Duplex was not complete. Use of Half Duplex was not intended, but probably added a
intended, but probably added a small amount of delay variation that small amount of delay variation that could have been avoided in full
could have been avoided in full duplex mode. duplex mode.
Each individual test was run with common packet rates (1 pps, 10pps) Each individual test was run with common packet rates (1 pps, 10pps)
Poisson/Periodic distributions, and IP packet sizes of 64, 340, and Poisson/Periodic distributions, and IP packet sizes of 64, 340, and
500 Bytes. 500 Bytes. These sizes cover a reasonable range while avoiding
fragmentation and the complexities it causes, and thus complying with
the notion of "standard formed packets" described in Section 15 of
[RFC2330].
For these tests, a stream of at least 300 packets were sent from For these tests, a stream of at least 300 packets were sent from
Source to Destination in each implementation. Periodic streams (as Source to Destination in each implementation. Periodic streams (as
per [RFC3432]) with 1 second spacing were used, except as noted. per [RFC3432]) with 1 second spacing were used, except as noted.
With the L2TPv3 tunnel in use, the metric name for the testing With the L2TPv3 tunnel in use, the metric name for the testing
configured here (with respect to the IP header exposed to Internet configured here (with respect to the IP header exposed to Internet
processing) is: processing) is:
Type-IP-protocol-115-One-way-Delay-<StreamType>-Stream Type-IP-protocol-115-One-way-Delay-<StreamType>-Stream
skipping to change at page 11, line 25 skipping to change at page 10, line 25
the systematic error and when the median is subtracted from all the systematic error and when the median is subtracted from all
singletons, the remaining variability is the random error. singletons, the remaining variability is the random error.
The test context, or Type-P of the test packets, must also be The test context, or Type-P of the test packets, must also be
reported, as required in Section 3.8 of [RFC2679] and all metrics reported, as required in Section 3.8 of [RFC2679] and all metrics
defined there. Type-P is defined in Section 13 of [RFC2330] (as are defined there. Type-P is defined in Section 13 of [RFC2330] (as are
many terms used below). many terms used below).
4.1. NetProbe Error and Type-P 4.1. NetProbe Error and Type-P
Type-P for this test was IP-UDP with Best Effort DCSP. These headers Type-P for this test was IP-UDP with Best Effort DSCP. These headers
were encapsulated according to the L2TPv3 specifications [RFC3931], were encapsulated according to the L2TPv3 specifications [RFC3931],
and thus may not influence the treatment received as the packets and thus may not influence the treatment received as the packets
traversed the Internet. traversed the Internet.
In general, NetProbe error is dependent on the specific version and In general, NetProbe error is dependent on the specific version and
installation details. installation details.
NetProbe operates using host time above the UDP layer, which is NetProbe operates using host time above the UDP layer, which is
different from the wire-time preferred in [RFC2330], but can be different from the wire-time preferred in [RFC2330], but can be
identified as a source of error according to Section 3.7.2 of identified as a source of error according to Section 3.7.2 of
skipping to change at page 12, line 41 skipping to change at page 11, line 41
NetProbe Calibration with Cross-Connect Cable, one-way delay values NetProbe Calibration with Cross-Connect Cable, one-way delay values
in microseconds (us) in microseconds (us)
The median or systematic error can be as high as 110 us, and the The median or systematic error can be as high as 110 us, and the
range of the random error is also on the order of 116 us for all range of the random error is also on the order of 116 us for all
streams. streams.
Also, anticipating the Anderson-Darling K-sample (ADK) [ADK] Also, anticipating the Anderson-Darling K-sample (ADK) [ADK]
comparisons to follow, we corrected the CAL2 values for the comparisons to follow, we corrected the CAL2 values for the
difference between means between CAL2 and CAL3 (as specified in difference between means between CAL2 and CAL3 (as specified in
[I-D.ietf-ippm-metrictest]), and found strong support for the (Null [RFC6576]), and found strong support for the (Null Hypothesis that)
Hypothesis that) the samples are from the same distribution the samples are from the same distribution (resolution of 1 us and
(resolution of 1 us and alpha equal 0.05 and 0.01) alpha equal 0.05 and 0.01)
> XD4CVCAL2 <- XD4CAL$CAL2 - (mean(XD4CAL$CAL2)-mean(XD4CAL$CAL3)) > XD4CVCAL2 <- XD4CAL$CAL2 - (mean(XD4CAL$CAL2)-mean(XD4CAL$CAL3))
> boxplot(XD4CVCAL2,XD4CAL$CAL3) > boxplot(XD4CVCAL2,XD4CAL$CAL3)
> XD4CV2_ADK <- adk.test(XD4CVCAL2, XD4CAL$CAL3) > XD4CV2_ADK <- adk.test(XD4CVCAL2, XD4CAL$CAL3)
> XD4CV2_ADK > XD4CV2_ADK
Anderson-Darling k-sample test. Anderson-Darling k-sample test.
Number of samples: 2 Number of samples: 2
Sample sizes: 300 300 Sample sizes: 300 300
Total number of values: 600 Total number of values: 600
Number of unique values: 97 Number of unique values: 97
skipping to change at page 13, line 28 skipping to change at page 12, line 28
T = (Anderson Darling Criterion - mean)/sigma T = (Anderson Darling Criterion - mean)/sigma
Null Hypothesis: All samples come from a common population. Null Hypothesis: All samples come from a common population.
t.obs P-value extrapolation t.obs P-value extrapolation
not adj. for ties 0.71734 0.17042 0 not adj. for ties 0.71734 0.17042 0
adj. for ties -0.39553 0.44589 1 adj. for ties -0.39553 0.44589 1
> >
using [Rtool] and [Radk]. using [Rtool] and [Radk].
4.2. Perfas Error and Type-P 4.2. Perfas+ Error and Type-P
Perfas+ is configured to use GPS synchronisation and uses NTP Perfas+ is configured to use GPS synchronisation and uses NTP
synchronization as a fall-back or default. GPS synchronisation synchronization as a fall-back or default. GPS synchronisation
worked throughout this test with the exception of the calibration worked throughout this test with the exception of the calibration
stated here (one implementation was NTP synchronised only). The time stated here (one implementation was NTP synchronised only). The time
stamp accuracy typically is 0.1 ms. stamp accuracy typically is 0.1 ms.
The resolution of the results reported by Perfas+ is 1us (us = The resolution of the results reported by Perfas+ is 1us (us =
microsecond) in the version tested here, which contributes to at microsecond) in the version tested here, which contributes to at
least +/-1us error. least +/-1us error.
Port 5001 5002 5003 Port 5001 5002 5003
Min. -227 -226 294 Min. -227 -226 294
Median -169 -167 323 Median -169 -167 323
Mean -159 -157 335 Mean -159 -157 335
Max. 6 -52 376 Max. 6 -52 376
s 102 102 93 s 102 102 93
Perfas Calibration with Cross-Connect Cable, one-way delay values in Perfas+ Calibration with Cross-Connect Cable, one-way delay values in
microseconds (us) microseconds (us)
The median or systematic error can be as high as 323 us, and the The median or systematic error can be as high as 323 us, and the
range of the random error is also less than 232 us for all streams. range of the random error is also less than 232 us for all streams.
5. Pre-determined Limits on Equivalence 5. Pre-determined Limits on Equivalence
In this section, we provide the numerical limits on comparisons This section provides the numerical limits on comparisons between
between implementations, in order to declare that the results are implementations, in order to declare that the results are equivalent
equivalent and therefore, the tested specification is clear. and therefore, the tested specification is clear. These limits have
their basis in Section 3.1 of [RFC6576] and the Appendix of
[RFC2330], with additional limits representing IPPM consensus prior
to publication of results.
A key point is that the allowable errors, corrections, and confidence A key point is that the allowable errors, corrections, and confidence
levels only need to be sufficient to detect mis-interpretation of the levels only need to be sufficient to detect mis-interpretation of the
tested specification resulting in diverging implementations. tested specification resulting in diverging implementations.
Also, the allowable error must be sufficient to compensate for Also, the allowable error must be sufficient to compensate for
measured path differences. It was simply not possible to measure measured path differences. It was simply not possible to measure
fully identical paths in the VLAN-loopback test configuration used, fully identical paths in the VLAN-loopback test configuration used,
and this practical compromise must be taken into account. and this practical compromise must be taken into account.
skipping to change at page 14, line 51 skipping to change at page 14, line 6
propagation delay error constants given above also apply. propagation delay error constants given above also apply.
6. Tests to evaluate RFC 2679 Specifications 6. Tests to evaluate RFC 2679 Specifications
This section describes some results from real-world (cross-Internet) This section describes some results from real-world (cross-Internet)
tests with measurement devices implementing IPPM metrics and a tests with measurement devices implementing IPPM metrics and a
network emulator to create relevant conditions, to determine whether network emulator to create relevant conditions, to determine whether
the metric definitions were interpreted consistently by implementors. the metric definitions were interpreted consistently by implementors.
The procedures are slightly modified from the original procedures The procedures are slightly modified from the original procedures
contained in Appendix A.1 of [I-D.ietf-ippm-metrictest]. The contained in Appendix A.1 of [RFC6576]. The modifications include
modifications include the use of the mean statistic for comparisons. the use of the mean statistic for comparisons.
Note that there are only five instances of the requirement term Note that there are only five instances of the requirement term
"MUST" in [RFC2679] outside of the boilerplate and [RFC2119] "MUST" in [RFC2679] outside of the boilerplate and [RFC2119]
reference. reference.
6.1. One-way Delay, ADK Sample Comparison - Same & Cross Implementation 6.1. One-way Delay, ADK Sample Comparison - Same & Cross Implementation
This test determines if implementations produce results that appear This test determines if implementations produce results that appear
to come from a common delay distribution, as an overall evaluation of to come from a common delay distribution, as an overall evaluation of
Section 4 of [RFC2679], "A Definition for Samples of One-way Delay". Section 4 of [RFC2679], "A Definition for Samples of One-way Delay".
skipping to change at page 15, line 42 skipping to change at page 14, line 45
2. Measure a sample of one-way delay singletons with 2 or more 2. Measure a sample of one-way delay singletons with 2 or more
implementations, using identical options and network emulator implementations, using identical options and network emulator
settings (if used). settings (if used).
3. Measure a sample of one-way delay singletons with *four* 3. Measure a sample of one-way delay singletons with *four*
instances of the *same* implementations, using identical options, instances of the *same* implementations, using identical options,
noting that connectivity differences SHOULD be the same as for noting that connectivity differences SHOULD be the same as for
the cross implementation testing. the cross implementation testing.
4. Apply the ADK comparison procedures (see Appendix C of 4. Apply the ADK comparison procedures (see Appendix C of [RFC6576])
[I-D.ietf-ippm-metrictest]) and determine the resolution and and determine the resolution and confidence factor for
confidence factor for distribution equivalence of each same- distribution equivalence of each same-implementation comparison
implementation comparison and each cross-implementation and each cross-implementation comparison.
comparison.
5. Take the coarsest resolution and confidence factor for 5. Take the coarsest resolution and confidence factor for
distribution equivalence from the same-implementation pairs, or distribution equivalence from the same-implementation pairs, or
the limit defined in Section 5 above, as a limit on the the limit defined in Section 5 above, as a limit on the
equivalence threshold for these experimental conditions. equivalence threshold for these experimental conditions.
6. Apply constant correction factors to all singletons of the sample 6. Apply constant correction factors to all singletons of the sample
distributions, as described and limited in Section 5 above. distributions, as described and limited in Section 5 above.
7. Compare the cross-implementation ADK performance with the 7. Compare the cross-implementation ADK performance with the
skipping to change at page 17, line 25 skipping to change at page 16, line 25
| sA | 0.60 (0.19) |-0.80 (0.57) | | | sA | 0.60 (0.19) |-0.80 (0.57) | |
| | | | | | | | | |
...........................|.............|.............| ...........................|.............|.............|
| | | | | | | | | |
| sB | 2.64 (0.03) | 0.07 (0.31) |-0.52 (0.48) | | sB | 2.64 (0.03) | 0.07 (0.31) |-0.52 (0.48) |
| | | | | | | | | |
+------------+-------------+-------------+-------------+ +------------+-------------+-------------+-------------+
NetProbe ADK Results for same-implementation NetProbe ADK Results for same-implementation
6.1.2. Perfas Same-implementation results 6.1.2. Perfas+ Same-implementation results
All pair comparisons pass the ADK criterion. All pair comparisons pass the ADK criterion.
+------------------------------------------------------+ +------------------------------------------------------+
| | | | | | | | | |
| ti.obs (P) | p1 | p2 | p3 | | ti.obs (P) | p1 | p2 | p3 |
| | | | | | | | | |
.............|.............|.............|.............| .............|.............|.............|.............|
| | | | | | | | | |
| p2 | 0.06 (0.32) | | | | p2 | 0.06 (0.32) | | |
skipping to change at page 17, line 47 skipping to change at page 16, line 47
.........................................|.............| .........................................|.............|
| | | | | | | | | |
| p3 | 1.09 (0.12) | 0.37 (0.24) | | | p3 | 1.09 (0.12) | 0.37 (0.24) | |
| | | | | | | | | |
...........................|.............|.............| ...........................|.............|.............|
| | | | | | | | | |
| p4 |-0.81 (0.57) |-0.13 (0.37) | 1.36 (0.09) | | p4 |-0.81 (0.57) |-0.13 (0.37) | 1.36 (0.09) |
| | | | | | | | | |
+------------+-------------+-------------+-------------+ +------------+-------------+-------------+-------------+
Perfas ADK Results for same-implementation Perfas+ ADK Results for same-implementation
6.1.3. One-way Delay, Cross-Implementation ADK Comparison 6.1.3. One-way Delay, Cross-Implementation ADK Comparison
The cross-implementation results are compared using a combined ADK The cross-implementation results are compared using a combined ADK
analysis [Radk], where all NetProbe results are compared with all analysis [Radk], where all NetProbe results are compared with all
Perfas results after testing that the combined same-implementation Perfas+ results after testing that the combined same-implementation
results pass the ADK criterion. results pass the ADK criterion.
When 4 (same) samples are compared, the ADK criterion for 0.95 When 4 (same) samples are compared, the ADK criterion for 0.95
confidence is 1.915, and when all 8 (cross) samples are compared it confidence is 1.915, and when all 8 (cross) samples are compared it
is 1.85. is 1.85.
Combination of Anderson-Darling K-Sample Tests. Combination of Anderson-Darling K-Sample Tests.
Sample sizes within each data set: Sample sizes within each data set:
Data set 1 : 299 297 298 300 (NetProbe) Data set 1 : 299 297 298 300 (NetProbe)
Data set 2 : 300 300 298 300 (Perfas) Data set 2 : 300 300 298 300 (Perfas+)
Total sample size per data set: 1194 1198 Total sample size per data set: 1194 1198
Number of unique values per data set: 1188 1192 Number of unique values per data set: 1188 1192
... ...
Null Hypothesis: Null Hypothesis:
All samples within a data set come from a common distribution. All samples within a data set come from a common distribution.
The common distribution may change between data sets. The common distribution may change between data sets.
NetProbe ti.obs P-value extrapolation NetProbe ti.obs P-value extrapolation
not adj. for ties 0.64999 0.21355 0 not adj. for ties 0.64999 0.21355 0
adj. for ties 0.64833 0.21392 0 adj. for ties 0.64833 0.21392 0
Perfas Perfas+
not adj. for ties 0.55968 0.23442 0 not adj. for ties 0.55968 0.23442 0
adj. for ties 0.55840 0.23473 0 adj. for ties 0.55840 0.23473 0
Combined Anderson-Darling Criterion: Combined Anderson-Darling Criterion:
tc.obs P-value extrapolation tc.obs P-value extrapolation
not adj. for ties 0.85537 0.17967 0 not adj. for ties 0.85537 0.17967 0
adj. for ties 0.85329 0.18010 0 adj. for ties 0.85329 0.18010 0
The combined same-implementation samples and the combined cross- The combined same-implementation samples and the combined cross-
implementation comparison all pass the ADK criteria at P>=0.18 and implementation comparison all pass the ADK criteria at P>=0.18 and
skipping to change at page 19, line 4 skipping to change at page 18, line 4
distribution). distribution).
We also see that the paired ADK comparisons are rather critical. We also see that the paired ADK comparisons are rather critical.
Although the NetProbe s1-sB comparison failed, the combined data set Although the NetProbe s1-sB comparison failed, the combined data set
from 4 streams passed the ADK criterion easily. from 4 streams passed the ADK criterion easily.
6.1.4. Conclusions on the ADK Results for One-way Delay 6.1.4. Conclusions on the ADK Results for One-way Delay
Similar testing was repeated many times in the months of March and Similar testing was repeated many times in the months of March and
April 2011. There were many experiments where a single test stream April 2011. There were many experiments where a single test stream
from NetProbe or Perfas proved to be different from the others in from NetProbe or Perfas+ proved to be different from the others in
paired comparisons (even same comparisons). When the out lier stream paired comparisons (even same implementation comparisons). When the
was removed from the comparison, the remaining streams passed outlier stream was removed from the comparison, the remaining streams
combined ADK criterion. Also, the application of correction factors passed combined ADK criterion. Also, the application of correction
resulted in higher comparison success. factors resulted in higher comparison success.
We conclude that the two implementations are capable of producing We conclude that the two implementations are capable of producing
equivalent one-way delay distributions based on their interpretation equivalent one-way delay distributions based on their interpretation
of [RFC2679] . of [RFC2679].
6.1.5. Additional Investigations 6.1.5. Additional Investigations
On the final day of testing, we performed a series of measurements to On the final day of testing, we performed a series of measurements to
evaluate the amount of emulated delay variation necessary to achieve evaluate the amount of emulated delay variation necessary to achieve
successful ADK comparisons. The need for Correction factors (as successful ADK comparisons. The need for Correction factors (as
permitted by Section 5) and the size of the measurement sample permitted by Section 5) and the size of the measurement sample
(obtained as sub-sets of the complete measurement sample) were also (obtained as sub-sets of the complete measurement sample) were also
evaluated. evaluated.
skipping to change at page 20, line 9 skipping to change at page 19, line 9
independently on each VLAN and thus the emulator itself is a independently on each VLAN and thus the emulator itself is a
potential source of error when comparing streams that traverse the potential source of error when comparing streams that traverse the
test path in different directions. test path in different directions.
In the result analysis of this section: In the result analysis of this section:
o All comparisons used 1 microsecond resolution. o All comparisons used 1 microsecond resolution.
o Correction Factors *were* applied as noted (under column heading o Correction Factors *were* applied as noted (under column heading
"mean adj"). The difference between each sample mean and the "mean adj"). The difference between each sample mean and the
lowest mean of the NetProbe or Perfas stream samples was lowest mean of the NetProbe or Perfas+ stream samples was
subtracted from all values in the sample. ("raw" indicates no subtracted from all values in the sample. ("raw" indicates no
correction factors were used.) correction factors were used.) All correction factors applied met
the limits described in Section 5.
o The 0.95 confidence factor (1.960 for paired stream comparison) o The 0.95 confidence factor (1.960 for paired stream comparison)
was used. was used.
When 8 (cross) samples are compared, the ADK criterion for 0.95 When 8 (cross) samples are compared, the ADK criterion for 0.95
confidence is 1.85. The Combined ADK test statistic ("TC observed") confidence is 1.85. The Combined ADK test statistic ("TC observed")
must be less than 1.85 to accept the Null Hypothesis (all samples in must be less than 1.85 to accept the Null Hypothesis (all samples in
the data set are from a common distribution). the data set are from a common distribution).
012345678901234567890123456789012345678901234567890123456789012345678901 Emulated Delay Sub-Sample size
Emulated Delay Sub-Sample size Variation 0ms
Variation 0ms adk.combined (all) 300 values 75 values
adk.combined (all) 300 values 75 values Adj. for ties raw mean adj raw mean adj
Adj. for ties raw mean adj raw mean adj TC observed 226.6563 67.51559 54.01359 21.56513
TC observed 226.6563 67.51559 54.01359 21.56513 P-value 0 0 0 0
P-value 0 0 0 0 Mean std dev (all),us 719 635
Mean std dev (all),us 719 635 Mean diff of means,us 649 0 606 0
Mean diff of means,us 649 0 606 0
Variation +/- 2.5ms Variation +/- 2.5ms
adk.combined (all) 300 values 75 values adk.combined (all) 300 values 75 values
Adj. for ties raw mean adj raw mean adj Adj. for ties raw mean adj raw mean adj
TC observed 14.50436 -1.60196 3.15935 -1.72104 TC observed 14.50436 -1.60196 3.15935 -1.72104
P-value 0 0.873 0.00799 0.89038 P-value 0 0.873 0.00799 0.89038
Mean std dev (all),us 1655 1702 Mean std dev (all),us 1655 1702
Mean diff of means,us 471 0 513 0 Mean diff of means,us 471 0 513 0
Variation +/- 5ms Variation +/- 5ms
adk.combined (all) 300 values 75 values adk.combined (all) 300 values 75 values
Adj. for ties raw mean adj raw mean adj Adj. for ties raw mean adj raw mean adj
TC observed 8.29921 -1.28927 0.37878 -1.81881 TC observed 8.29921 -1.28927 0.37878 -1.81881
P-value 0 0.81601 0.29984 0.90305 P-value 0 0.81601 0.29984 0.90305
Mean std dev (all),us 3023 2991 Mean std dev (all),us 3023 2991
Mean diff of means,us 582 0 513 0 Mean diff of means,us 582 0 513 0
Variation +/- 7.5ms Variation +/- 7.5ms
adk.combined (all) 300 values 75 values adk.combined (all) 300 values 75 values
Adj. for ties raw mean adj raw mean adj Adj. for ties raw mean adj raw mean adj
TC observed 2.53759 -0.72985 0.29241 -1.15840 TC observed 2.53759 -0.72985 0.29241 -1.15840
P-value 0.01950 0.66942 0.32585 0.78686 P-value 0.01950 0.66942 0.32585 0.78686
Mean std dev (all),us 4449 4506 Mean std dev (all),us 4449 4506
Mean diff of means,us 426 0 856 0 Mean diff of means,us 426 0 856 0
From the table above, we conclude the following: From the table above, we conclude the following:
1. None of the raw or mean adjusted results pass the ADK criterion 1. None of the raw or mean adjusted results pass the ADK criterion
with 0 ms emulated delay variation. Use of the 75 value sub- with 0 ms emulated delay variation. Use of the 75 value sub-
sample yielded the same conclusion. (We note the same results sample yielded the same conclusion. (We note the same results
when comparing same implementation samples for both NetProbe and when comparing same implementation samples for both NetProbe and
Perfas.) Perfas+.)
2. When the smallest emulated delay variation was inserted 2. When the smallest emulated delay variation was inserted
(+/-2.5ms), the mean adjusted samples pass the ADK criterion and (+/-2.5ms), the mean adjusted samples pass the ADK criterion and
the high P-value supports the result. The raw results do not the high P-value supports the result. The raw results do not
pass. pass.
3. At higher values of emulated delay variation (+/-5.0ms and 3. At higher values of emulated delay variation (+/-5.0ms and
+/-7.5ms), again the mean adjusted values pass ADK. We also see +/-7.5ms), again the mean adjusted values pass ADK. We also see
that the 75-value sub-sample passed the ADK in both raw and mean that the 75-value sub-sample passed the ADK in both raw and mean
adjusted cases. This indicates that sample size may have played adjusted cases. This indicates that sample size may have played
a role in our results, as noted in the Appendix of [RFC2680] for a role in our results, as noted in the Appendix of [RFC2680] for
Goodness-of-Fit testing. Goodness-of-Fit testing.
We note that 150 value sub-samples were also evaluated, with ADK We note that 150 value sub-samples were also evaluated, with ADK
conclusions that followed the results for 300 values. Also, same- conclusions that followed the results for 300 values. Also, same-
implementation analysis was conducted with results similar to the implementation analysis was conducted with results similar to the
above, except that more of the "raw" or uncorrected samples passed above, except that more of the "raw" or uncorrected samples passed
the ADK criterion. the ADK criterion.
>>>> To be provided:
>>>> Overall statement about Correction Factors w.r.t. section 5
limits.
>>>> Appendix with more details ???
6.2. One-way Delay, Loss threshold, RFC 2679 6.2. One-way Delay, Loss threshold, RFC 2679
This test determines if implementations use the same configured This test determines if implementations use the same configured
maximum waiting time delay from one measurement to another under maximum waiting time delay from one measurement to another under
different delay conditions, and correctly declare packets arriving in different delay conditions, and correctly declare packets arriving in
excess of the waiting time threshold as lost. excess of the waiting time threshold as lost.
See Section 3.5 of [RFC2679], 3rd bullet point and also Section 3.8.2 See Section 3.5 of [RFC2679], 3rd bullet point and also Section 3.8.2
of [RFC2679]. of [RFC2679].
skipping to change at page 23, line 33 skipping to change at page 22, line 24
6.2.1. NetProbe results for Loss Threshold 6.2.1. NetProbe results for Loss Threshold
In NetProbe, the Loss Threshold is implemented uniformly over all In NetProbe, the Loss Threshold is implemented uniformly over all
packets as a post-processing routine. With the Loss Threshold set at packets as a post-processing routine. With the Loss Threshold set at
3 seconds, all packets with one-way delay >3 seconds are marked 3 seconds, all packets with one-way delay >3 seconds are marked
"Lost" and included in the Lost Packet list with their transmission "Lost" and included in the Lost Packet list with their transmission
time (as required in Section 3.3 of [RFC2680]). This resulted in 342 time (as required in Section 3.3 of [RFC2680]). This resulted in 342
packets designated as lost in one of the test streams (with average packets designated as lost in one of the test streams (with average
delay = 3.091 sec). delay = 3.091 sec).
6.2.2. Perfas Results for Loss Threshold 6.2.2. Perfas+ Results for Loss Threshold
Perfas uses a fixed Loss Threshold which was not adjustable during Perfas+ uses a fixed Loss Threshold which was not adjustable during
this study. The Loss Threshold is approximately one minute, and this study. The Loss Threshold is approximately one minute, and
emulation of a delay of this size was not attempted. However, it is emulation of a delay of this size was not attempted. However, it is
possible to implement any delay threshold desired with a post- possible to implement any delay threshold desired with a post-
processing routine and subsequent analysis. Using this method, 195 processing routine and subsequent analysis. Using this method, 195
packets would be declared lost (with average delay = 3.091 sec). packets would be declared lost (with average delay = 3.091 sec).
6.2.3. Conclusions for Loss Threshold 6.2.3. Conclusions for Loss Threshold
Both implementations assume that any constant delay value desired can Both implementations assume that any constant delay value desired can
be used as the Loss Threshold, since all delays are stored as a pair be used as the Loss Threshold, since all delays are stored as a pair
skipping to change at page 24, line 49 skipping to change at page 23, line 39
The common parameters used for tests in this section are: The common parameters used for tests in this section are:
o IP header + payload = 64 octets o IP header + payload = 64 octets
o Periodic sampling at l packet per second o Periodic sampling at l packet per second
o Test duration = 300 seconds total (April 12) o Test duration = 300 seconds total (April 12)
The netem emulator was set to add constant 100ms delay. The netem emulator was set to add constant 100ms delay.
6.3.1. NetProbe and Perfas Results for Serialization 6.3.1. NetProbe and Perfas+ Results for Serialization
When the IP header + payload size was increased from 64 octets to 500 When the IP header + payload size was increased from 64 octets to 500
octets, there was a delay increase observed. octets, there was a delay increase observed.
Mean Delays in us Mean Delays in us
NetProbe NetProbe
Payload s1 s2 sA sB Payload s1 s2 sA sB
500 190893 191179 190892 190971 500 190893 191179 190892 190971
64 189642 189785 189747 189467 64 189642 189785 189747 189467
Diff 1251 1394 1145 1505 Diff 1251 1394 1145 1505
skipping to change at page 25, line 31 skipping to change at page 24, line 31
to 1.5 ms (with one outlier). The typical measurements indicate that to 1.5 ms (with one outlier). The typical measurements indicate that
a link with approximately 3 Mbit/s capacity is present on the path. a link with approximately 3 Mbit/s capacity is present on the path.
Through investigation of the facilities involved, it was determined Through investigation of the facilities involved, it was determined
that the lowest speed link was approximately 45 Mbit/s, and therefore that the lowest speed link was approximately 45 Mbit/s, and therefore
the estimated difference should be about 0.077 ms. The observed the estimated difference should be about 0.077 ms. The observed
differences are much higher. differences are much higher.
The unexpected large delay difference was also the outcome when The unexpected large delay difference was also the outcome when
testing serialization times in a lab environment, using the NIST Net testing serialization times in a lab environment, using the NIST Net
Emulator and NetProbe [ref to earlier lab tests]. Emulator and NetProbe [I-D.morton-ippm-advance-metrics].
6.3.2. Conclusions for Serialization 6.3.2. Conclusions for Serialization
Since it was not possible to confirm the estimated serialization time Since it was not possible to confirm the estimated serialization time
increases in field tests, we resort to examination of the increases in field tests, we resort to examination of the
implementations to determine compliance. implementations to determine compliance.
NetProbe performs all time stamping above the IP-layer, accepting NetProbe performs all time stamping above the IP-layer, accepting
that some compromises must be made to achieve extreme portability and that some compromises must be made to achieve extreme portability and
measurement scale. Therefore, the first-to-last bit convention is measurement scale. Therefore, the first-to-last bit convention is
supported because the serialization time is included in the one-way supported because the serialization time is included in the one-way
delay measurement, enabling comparison with other implementations. delay measurement, enabling comparison with other implementations.
Perfas is optimized for its purpose and performs all time stamping Perfas+ is optimized for its purpose and performs all time stamping
close to the interface hardware. The first-to-last bit convention is close to the interface hardware. The first-to-last bit convention is
supported because the serialization time is included in the one-way supported because the serialization time is included in the one-way
delay measurement, enabling comparison with other implementations. delay measurement, enabling comparison with other implementations.
6.4. One-way Delay, Difference Sample Metric (Lab) 6.4. One-way Delay, Difference Sample Metric (Lab)
This test determines if implementations register the same relative This test determines if implementations register the same relative
increase in delay from one measurement to another under different increase in delay from one measurement to another under different
delay conditions. This test tends to cancel the sources of error delay conditions. This test tends to cancel the sources of error
which may be present in an implementation. which may be present in an implementation.
skipping to change at page 27, line 10 skipping to change at page 26, line 10
Average delays before/after 1 second increase Average delays before/after 1 second increase
The NetProbe implementation observed a 1 second increase with a 182 The NetProbe implementation observed a 1 second increase with a 182
microsecond error (assuming that the netem emulated delay difference microsecond error (assuming that the netem emulated delay difference
is exact). is exact).
We note that this differential delay test has been run under lab We note that this differential delay test has been run under lab
conditions and published in prior work [ref to "advance metrics" conditions and published in prior work [ref to "advance metrics"
draft]. The error was 6 microseconds. draft]. The error was 6 microseconds.
6.4.2. Perfas results for Differential Delay 6.4.2. Perfas+ results for Differential Delay
Average pre-increase delay, microseconds 1089794.0 Average pre-increase delay, microseconds 1089794.0
Average post 1s additional, microseconds 2089801.0 Average post 1s additional, microseconds 2089801.0
Difference (should be ~= Y = 1s) 1000007.0 Difference (should be ~= Y = 1s) 1000007.0
Average delays before/after 1 second increase Average delays before/after 1 second increase
The Perfas implementation observed a 1 second increase with a 7 The Perfas+ implementation observed a 1 second increase with a 7
microsecond error. microsecond error.
6.4.3. Conclusions for Differential Delay 6.4.3. Conclusions for Differential Delay
Again, the live network conditions appear to have influenced the Again, the live network conditions appear to have influenced the
results, but both implementations measured the same delay increase results, but both implementations measured the same delay increase
within their calibration accuracy. within their calibration accuracy.
6.5. Implementation of Statistics for One-way Delay 6.5. Implementation of Statistics for One-way Delay
skipping to change at page 28, line 5 skipping to change at page 27, line 5
delay singletons from two implementations of [RFC2679] appear to be delay singletons from two implementations of [RFC2679] appear to be
from the same overall distribution. By testing this way, we from the same overall distribution. By testing this way, we
economize on the number of comparisons, because comparing a set of economize on the number of comparisons, because comparing a set of
individual summary statistics (as defined in Section 5 of [RFC2679]) individual summary statistics (as defined in Section 5 of [RFC2679])
would require another set of individual evaluations of equivalence. would require another set of individual evaluations of equivalence.
Instead, we can simply check which statistics were implemented, and Instead, we can simply check which statistics were implemented, and
report on those facts, noting that Section 5 of [RFC2679] does not report on those facts, noting that Section 5 of [RFC2679] does not
specify the calculations exactly, and gives only some illustrative specify the calculations exactly, and gives only some illustrative
examples. examples.
NetProbe Perfas NetProbe Perfas+
5.1. Type-P-One-way-Delay-Percentile yes no 5.1. Type-P-One-way-Delay-Percentile yes no
5.2. Type-P-One-way-Delay-Median yes no 5.2. Type-P-One-way-Delay-Median yes no
5.3. Type-P-One-way-Delay-Minimum yes yes 5.3. Type-P-One-way-Delay-Minimum yes yes
5.4. Type-P-One-way-Delay-Inverse-Percentile no no 5.4. Type-P-One-way-Delay-Inverse-Percentile no no
Implementation of Section 5 Statistics Implementation of Section 5 Statistics
skipping to change at page 28, line 28 skipping to change at page 27, line 28
both implementations, so it is a candidate for removal in RFC2679bis. both implementations, so it is a candidate for removal in RFC2679bis.
7. Security Considerations 7. Security Considerations
The security considerations that apply to any active measurement of The security considerations that apply to any active measurement of
live networks are relevant here as well. See [RFC4656] and live networks are relevant here as well. See [RFC4656] and
[RFC5357]. [RFC5357].
8. IANA Considerations 8. IANA Considerations
This memo makes no requests of IANA, and hopes that IANA will be as This memo makes no requests of IANA, and hopes that IANA will welcome
accepting of our new computer overlords as the authors intend to be. our new computer overlords as willingly as the authors.
9. Acknowledgements 9. Acknowledgements
The authors thank Lars Eggert for his continued encouragement to The authors thank Lars Eggert for his continued encouragement to
advance the IPPM metrics during his tenure as AD Advisor. advance the IPPM metrics during his tenure as AD Advisor.
Nicole Kowalski supplied the needed CPE router for the NetProbe side Nicole Kowalski supplied the needed CPE router for the NetProbe side
of the test set-up, and graciously managed her testing in spite of of the test set-up, and graciously managed her testing in spite of
issues caused by dual-use of the router. Thanks Nicole! issues caused by dual-use of the router. Thanks Nicole!
The "NetProbe Team" also acknowledges many useful discussions with The "NetProbe Team" also acknowledges many useful discussions with
Ganga Maguluri. Ganga Maguluri.
10. References 10. References
10.1. Normative References 10.1. Normative References
[I-D.ietf-ippm-metrictest]
Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IPPM
standard advancement testing",
draft-ietf-ippm-metrictest-05 (work in progress),
November 2011.
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision [RFC2026] Bradner, S., "The Internet Standards Process -- Revision
3", BCP 9, RFC 2026, October 1996. 3", BCP 9, RFC 2026, October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
May 1998. May 1998.
skipping to change at page 29, line 36 skipping to change at page 28, line 30
Packet Loss Metric for IPPM", RFC 2680, September 1999. Packet Loss Metric for IPPM", RFC 2680, September 1999.
[RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network
performance measurement with periodic streams", RFC 3432, performance measurement with periodic streams", RFC 3432,
November 2002. November 2002.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
Zekauskas, "A One-way Active Measurement Protocol Zekauskas, "A One-way Active Measurement Protocol
(OWAMP)", RFC 4656, September 2006. (OWAMP)", RFC 4656, September 2006.
[RFC4814] Newman, D. and T. Player, "Hash and Stuffing: Overlooked
Factors in Network Device Benchmarking", RFC 4814,
March 2007.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", BCP 26, RFC 5226,
May 2008.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
RFC 5357, October 2008. RFC 5357, October 2008.
[RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation
and Implementation Reports for Advancement to Draft and Implementation Reports for Advancement to Draft
Standard", BCP 9, RFC 5657, September 2009. Standard", BCP 9, RFC 5657, September 2009.
[RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP
Performance Metrics (IPPM) Standard Advancement Testing",
BCP 176, RFC 6576, March 2012.
10.2. Informative References 10.2. Informative References
[ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling
Tests of fit, for continuous and discrete cases", Tests of fit, for continuous and discrete cases",
University of Washington, Technical Report No. 81, University of Washington, Technical Report No. 81,
May 1986. May 1986.
[Fedora14]
"Fedora Project Home Page", http://fedoraproject.org/,
2012.
[I-D.bradner-metricstest]
Bradner, S. and V. Paxson, "Advancement of metrics
specifications on the IETF Standards Track",
draft-bradner-metricstest-03 (work in progress),
August 2007.
[I-D.morton-ippm-advance-metrics] [I-D.morton-ippm-advance-metrics]
Morton, A., "Lab Test Results for Advancing Metrics on the Morton, A., "Lab Test Results for Advancing Metrics on the
Standards Track", draft-morton-ippm-advance-metrics-02 Standards Track", draft-morton-ippm-advance-metrics-02
(work in progress), October 2010. (work in progress), October 2010.
[RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling
Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005.
[Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and
Combinations of Such Tests. R package version 1.0.", , Combinations of Such Tests. R package version 1.0.", ,
2008. 2008.
[Rtool] R Development Core Team, "R: A language and environment [Rtool] R Development Core Team, "R: A language and environment
for statistical computing. R Foundation for Statistical for statistical computing. R Foundation for Statistical
Computing, Vienna, Austria. ISBN 3-900051-07-0, URL Computing, Vienna, Austria. ISBN 3-900051-07-0, URL
http://www.R-project.org/", , 2011. http://www.R-project.org/", , 2011.
[WIPM] "AT&T Global IP Network",
http://ipnetwork.bgtmo.ip.att.net/pws/index.html, 2012.
[netem] ""netem" Documentation", http://www.linuxfoundation.org/
collaborate/workgroups/networking/netem, 2009.
Authors' Addresses Authors' Addresses
Len Ciavattone Len Ciavattone
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown, NJ 07748 Middletown, NJ 07748
USA USA
Phone: +1 732 420 1239 Phone: +1 732 420 1239
Fax: Fax:
 End of changes. 54 change blocks. 
215 lines changed or deleted 174 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/