Network Working Group                                          A. Morton
Internet-Draft                                           G. Ramachandran
Intended status: Informational                               G. Maguluri
Expires: September 6, December 1, 2010                                      AT&T Labs
                                                           March 5,
                                                            May 30, 2010

              Reporting Metrics: Different Points of View
                  draft-ietf-ippm-reporting-metrics-01
                  draft-ietf-ippm-reporting-metrics-02

Abstract

   Consumers of IP network performance metrics have many different uses
   in mind.  This memo categorizes the different audience points of
   view.  It describes how the categories affect the selection of metric
   parameters and options when seeking info that serves their needs.
   The memo then proceeds to discuss "long-term" reporting
   considerations (e.g, days, weeks or months, as opposed to 10
   seconds).

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts.
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on September 6, December 1, 2010.

Copyright Notice

   Copyright (c) 2010 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

   This document may contain material from IETF Documents or IETF
   Contributions published or made publicly available before November
   10, 2008.  The person(s) controlling the copyright in some of this
   material may not have granted the IETF Trust the right to allow
   modifications of such material outside the IETF Standards Process.
   Without obtaining an adequate license from the person(s) controlling
   the copyright in such materials, this document may not be modified
   outside the IETF Standards Process, and derivative works of it may
   not be created outside the IETF Standards Process, except to format
   it for publication as an RFC or to translate it into languages other
   than English.

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Purpose and Scope  . . . . . . . . . . . . . . . . . . . . . .  4
   3.  Reporting Results  . . . . . . . . . . . . . . . . . . . . . .  5
     3.1.  Overview of Metric Statistics  . . . . . . . . . . . . . .  5
     3.2.  Long-Term Reporting Considerations . . . . . . . . . . . .  6
   4.  Effect of POV on the Loss Metric . . . . . . . . . . . . . . .  5
     3.1.  8
     4.1.  Loss Threshold . . . . . . . . . . . . . . . . . . . . . .  5
       3.1.1.  8
       4.1.1.  Network Characterization . . . . . . . . . . . . . . .  5
       3.1.2.  8
       4.1.2.  Application Performance  . . . . . . . . . . . . . . .  7
     3.2. 10
     4.2.  Errored Packet Designation . . . . . . . . . . . . . . . .  7
     3.3. 10
     4.3.  Causes of Lost Packets . . . . . . . . . . . . . . . . . .  7
     3.4. 10
     4.4.  Summary for Loss . . . . . . . . . . . . . . . . . . . . .  8
   4. 11
   5.  Effect of POV on the Delay Metric  . . . . . . . . . . . . . .  8
     4.1. 11
     5.1.  Treatment of Lost Packets  . . . . . . . . . . . . . . . .  8
       4.1.1. 11
       5.1.1.  Application Performance  . . . . . . . . . . . . . . .  9
       4.1.2. 11
       5.1.2.  Network Characterization . . . . . . . . . . . . . . .  9
       4.1.3. 12
       5.1.3.  Delay Variation  . . . . . . . . . . . . . . . . . . . 10
       4.1.4. 13
       5.1.4.  Reordering . . . . . . . . . . . . . . . . . . . . . . 11
     4.2. 14
     5.2.  Preferred Statistics . . . . . . . . . . . . . . . . . . . 11
     4.3. 14
     5.3.  Summary for Delay  . . . . . . . . . . . . . . . . . . . . 12
   5. 15
   6.  Effect of POV on Raw Capacity Metrics  . . . . . . . . . . . . 12
     5.1. 15
     6.1.  Type-P Parameter . . . . . . . . . . . . . . . . . . . . . 12
     5.2. 15
     6.2.  a priori Factors . . . . . . . . . . . . . . . . . . . . . 13
     5.3. 16
     6.3.  IP-layer Capacity  . . . . . . . . . . . . . . . . . . . . 13
     5.4. 16
     6.4.  IP-layer Utilization . . . . . . . . . . . . . . . . . . . 13
     5.5. 17
     6.5.  IP-layer Available Capacity  . . . . . . . . . . . . . . . 14
     5.6. 17
     6.6.  Variability in Utilization and Avail. Capacity . . . . . . 15
   6. 18
   7.  Test Streams and Sample Size . . . . . . . . . . . . . . . . . 15
     6.1. 18
     7.1.  Test Stream Characteristics  . . . . . . . . . . . . . . . 15
     6.2. 18
     7.2.  Sample Size  . . . . . . . . . . . . . . . . . . . . . . . 15
   7.  Reporting Results  . . . . . . . . . . . . . . . . . . . . . . 16
     7.1.  Overview of Metric Statistics  . . . . . . . . . . . . . . 16
     7.2.  Long-Term Reporting Considerations . . . . . . . . . . . . 17 19
   8.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 18 19
   9.  Security Considerations  . . . . . . . . . . . . . . . . . . . 18 19
   10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 18 19
   11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19 20
     11.1. Normative References . . . . . . . . . . . . . . . . . . . 19 20
     11.2. Informative References . . . . . . . . . . . . . . . . . . 19 21
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 21

1.  Introduction

   When designing measurements of IP networks and presenting the
   results, knowledge of the audience is a key consideration.  To
   present a useful and relevant portrait of network conditions, one
   must answer the following question:

   "How will the results be used?"

   There are two main audience categories:

   1.  Network Characterization - describes conditions in an IP network
       for quality assurance, troubleshooting, modeling, Service Level
       Agreements (SLA), etc.  The point-of-view looks inward, toward
       the network, and the consumer intends their actions there.

   2.  Application Performance Estimation - describes the network
       conditions in a way that facilitates determining affects on user
       applications, and ultimately the users themselves.  This point-
       of-view looks outward, toward the user(s), accepting the network
       as-is.  This consumer intends to estimate a network-dependent
       aspect of performance, or design some aspect of an application's
       accommodation of the network.  (These are *not* application
       metrics, they are defined at the IP layer.)

   This memo considers how these different points-of-view affect both
   the measurement design (parameters and options of the metrics) and
   statistics reported when serving their needs.

   The IPPM framework [RFC2330] and other RFCs describing IPPM metrics
   provide a background for this memo.

2.  Purpose and Scope

   The purpose of this memo is to clearly delineate two points-of-view
   (POV) for using measurements, and describe their effects on the test
   design, including the selection of metric parameters and reporting
   the results.

   The current scope of this memo primarily covers the design and
   reporting of the loss and delay metrics [RFC2680] [RFC2679].  It will
   also discuss the delay variation [RFC3393] and reordering metrics
   [RFC4737] where applicable.

   With capacity metrics growing in relevance to the industry, the memo
   also covers POV and reporting considerations for metrics resulting
   from the Bulk Transfer Capacity Framework [RFC3148] and Network
   Capacity Definitions [RFC5136].  These memos effectively describe two
   different categories of metrics, [RFC3148] with congestion flow-
   control and the notion of unique data bits delivered, and [RFC5136]
   using a definition of raw capacity without the restrictions of data
   uniqueness or congestion-awareness.  It might seem at first glance
   that each of these metrics has an obvious audience (Raw = Network
   Characterization, Restricted = Application Performance), but reality
   is more complex and consistent with the overall topic of capacity
   measurement and reporting.  The Raw and Restricted capacity metrics
   will be  For example, TCP is usually used in
   Restricted capacity measurement methods, while UDP appears in Raw
   capacity measurement.  The Raw and Restricted capacity metrics will
   be treated in separate sections, although they share one common
   reporting issue: representing variability in capacity metric results.

   Sampling, or the design of the active packet stream that is the basis
   for the measurements, is also discussed.

3.  Effect  Reporting Results

   This section gives an overview of POV on recommendations, followed by
   additional considerations for reporting results in the Loss "long-term".

3.1.  Overview of Metric Statistics

   This section describes the ways in which gives an overview of reporting recommendations for the Loss metric can be tuned
   to reflect
   loss, delay, and delay variation metrics based on the preferences discussion and
   conclusions of the two audience categories, or
   different POV. preceding sections.

   The waiting time to declare a packet lost, or minimal report on measurements MUST include both Loss and Delay
   Metrics.

   For Packet Loss, the loss
   threshold ratio defined in [RFC2680] is one area where there would appear to be a difference,
   but the ability to post-process sufficient
   starting point, especially the results may resolve it.

3.1.  Loss Threshold

   RFC 2680 [RFC2680] defines guidance for setting the concept of loss
   threshold waiting time.  We have calculated a waiting time for packets above that
   should be sufficient to arrive, beyond which they differentiate between packets that are declared lost.  The text truly
   lost or have long finite delays under general measurement
   circumstances, 51 seconds.  Knowledge of the RFC
   declines specific conditions can help
   to reduce this threshold, but 51 seconds is considered to recommend a value, instead saying that "good engineering,
   including an understanding of packet lifetimes, will be needed in
   practice."  Later,
   manageable in practice.

   We note that a loss ratio calculated according to [Y.1540] would
   exclude errored packets form the methodology, they give reasons for waiting
   "a reasonable period of time", and leaving numerator.  In practice, the definition of
   "reasonable" intentionally vague.

3.1.1.  Network Characterization

   Practical measurement experience has shown that unusual network
   circumstances can cause long delays.  One such circumstance is when
   routing loops form during IGP re-convergence following a failure or
   drastic link cost change.  Packets will loop
   difference between these two routers
   until new routes are installed, or until the IPv4 Time-to-Live (TTL)
   field (or loss metrics is small if any, depending
   on whether the IPv6 Hop Limit) decrements last link prior to zero.  Very long delays the destination contributes errored
   packets.

   For Packet Delay, we recommend providing both the mean delay and the
   median delay with lost packets designated undefined (as permitted by
   [RFC2679]).  Both statistics are based on a conditional distribution,
   and the order of several seconds have been measured [Casner] [Cia03].

   Therefore, network characterization activities prefer condition is packet arrival prior to a long waiting time in order dT, where
   dT has been set to distinguish these events from other causes of loss
   (such as take maximum packet discard at lifetimes into account, as
   discussed above.  Using a full queue, or tail drop).  This way,
   the metric design long dT helps to distinguish more reliably between packets
   that might yet arrive, and those ensure that delay
   distributions are no longer traversing not truncated.

   For Packet Delay Variation (PDV), the
   network.

   It is possible minimum delay of the
   conditional distribution should be used as the reference delay for
   computing PDV according to calculate a worst-case waiting time, assuming that
   a routing loop [Y.1540] or [RFC3393].  A useful value to
   report is a pseudo range of delay variation based on calculating the cause.  We model the path
   difference between Source and
   Destination as a series high percentile of delays in links (t) delay and queues (q), as
   these two are the dominant contributors to minimum delay.  The normal path
   delay across n hops without encountering a loop, D, is

                                      n
                                     ---
                                     \
                           D = t  +   >   t  + q
                                0    /     i    i
                                     ---
                                    i = 1

                        Figure 1: Normal Path Delay

   and
   For example, the time spent in 99.9%-ile minus the loop minimum will give a value that
   can be compared with L hops, is

                       i + L-1
                        ---
                        \                         (TTL - n)
                 R = C   >   t  + q  where C    = ---------
                        /     i    i        max       L
                        ---
                         i

                Figure 2: Delay due to Rotations objectives in a Loop [Y.1541].

3.2.  Long-Term Reporting Considerations

   [I-D.ietf-ippm-reporting] describes methods to conduct measurements
   and where C is report the number of times results on a packet circles the loop.

   If near-immediate time scale (10 seconds,
   which we take the delays of all links and queues as 100ms each, the
   TTL=255, the number of hops n=5 consider to be "short-term").

   Measurement intervals and reporting intervals need not be the hops in same
   length.  Sometimes, the loop L=4, then

   D = 1.1 sec and R ~= 50 sec, and D + R ~= 51.1 seconds

   We note that user is only concerned with the link delays of 100ms would span most continents, and performance
   levels achieved over a constant queue length relatively long interval of 100ms is also very generous.  When a loop
   occurs, it is almost certain time (e.g, days,
   weeks, or months, as opposed to be resolved in 10 seconds or less.
   The value calculated above is an upper limit for almost any realistic
   circumstance.

   A waiting time threshold parameter, dT, set consistent seconds).  However, there can be
   risks involved with this
   calculation would not truncate the delay distribution (possibly
   causing running a change in its mathematical properties), because measurement continuously over a long
   period without recording intermediate results:

   o  Temporary power failure may cause loss of all the packets
   that might arrive have been given sufficient time results to traverse the
   network.

   It is worth noting that packets that are stored and deliberately
   forwarded at a much later time constitute date.

   o  Measurement system timing synchronization signals may experience a replay attack
      temporary outage, causing sub-sets of measurements to be in error
      or invalid.

   o  Maintenance may be necessary on the measurement system, or its
      connectivity to the network under test.

   For these and are beyond other reasons, such as

   o  the scope constraint to collect measurements on intervals similar to
      user session length, or

   o  the dual-use of normal performance
   reporting.

3.1.2.  Application Performance

   Fortunately, application performance estimation measurements in monitoring activities where
      results are not
   adversely affected by the estimated worst-case transfer time.
   Although the designer's tendency might be to set the Loss Threshold
   at needed on a period of a few minutes,

   there is value equivalent in conducting measurements on intervals that are much
   shorter than the reporting interval.

   There are several approaches for aggregating a series of measurement
   results over time in order to make a particular application's threshold, this
   specific threshold can be applied when post-processing statement about the
   measurements.  A shorter waiting time can be enforced by locating
   packets with delays longer than
   reporting interval.  One approach requires the application's threshold, and re-
   designating such packets as lost.  Thus, storage of all metric
   singletons collected throughout the reporting interval, even though
   the measurement system can
   use a single loss threshold and support both application interval stops and network
   performance POVs simultaneously.

3.2.  Errored Packet Designation

   RFC 2680 designates packets that arrive containing errors starts many times.

   Another approach is described in [RFC5835] as lost
   packets.  Many packets that are corrupted by bit errors are discarded
   within the network and do not reach their intended destination. "temporal aggregation".
   This is consistent with applications that approach would check estimate the payload
   integrity at higher layers, and discard results for the packet.  However, some
   applications prefer to deal with errored payloads reporting interval
   based on their own, and
   even many individual measurement interval statistics (results)
   alone.  The result would ideally appear in the same form as though a corrupted payload is better than no packet at all.

   To address this possibility, and
   continuous measurement was conducted.  A memo to make network characterization
   more complete, it address the details
   of temporal aggregation is recommended yet to distinguish between packets that
   do not arrive (lost) be prepared.

   Yet another approach requires a numerical objective for the metric,
   and errored packets that arrive (conditionally
   lost).

3.3.  Causes the results of Lost Packets

   Although many each measurement systems use a waiting time interval are compared with the
   objective.  Every measurement interval where the results meet the
   objective contribute to determine if
   a packet is lost or not, most the fraction of time with performance as
   specified.  When the waiting reporting interval contains many measurement
   intervals it is in vain.  The packets
   are no-longer traversing the network, and have not reached their
   destination.

   There are many causes of packet loss, including:

   1.  Queue drop, or discard
   2.  Corruption of possible to present the IP header, or other essential header info

   3.  TTL expiration (or use of a TTL value that is too small)

   4.  Link results as "metric A was less
   than or router failure

   After waiting sufficient time, packet loss can probably be attributed equal to one objective X during Y% of these causes.

3.4.  Summary for Loss

   Given time.

   NOTE that measurement post-processing is possible (even encouraged numerical thresholds are not set in IETF performance work
   and are explicitly excluded from the definitions of IPPM metrics), measurements of loss can easily
   serve both points of view:

   o  Use a long waiting time charter.

   In all measurement, it is important to serve avoid unintended
   synchronization with network characterization and
      revise results events.  This topic is treated in
   [RFC2330] for specific application delay thresholds as
      needed.

   o  Distinguish between errored packets and lost packets when possible
      to aid network characterization, Poisson-distributed inter-packet time streams, and combine the results
   [RFC3432] for
      application performance if appropriate.

4.  Effect Periodic streams.  Both avoid synchronization through
   use of POV on the Delay Metric

   This section describes random start times.

   There are network conditions where it is simply more useful to report
   the ways in which connectivity status of the Delay metric Source-Destination path, and to
   distinguish time intervals where connectivity can be
   tuned demonstrated
   from other time intervals (where connectivity does not appear to reflect the preferences
   exist).  [RFC2678] specifies a number of the one-way and two consumer categories, or
   different POV.

4.1.  Treatment of Lost Packets

   The Delay Metric [RFC2679] specifies the treatment connectivity
   metrics of packets increasing complexity.  In this memo, we RECOMMEND that do
   not successfully traverse the network: their delay is undefined.

   " >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined
   (informally, infinite)<< means that Src sent the first bit
   long term reporting of a
   Type-P packet loss, delay, and other metrics be limited to Dst at wire-time T
   time intervals where connectivity can be demonstrated, and that Dst did other
   intervals be summarized as percent of time where connectivity does
   not receive that
   packet."

   It is an accepted, but informal practice to assign infinite delay appear to
   lost packets. exist.  We next look at how these two different treatments
   align with the needs of measurement consumers who wish to
   characterize networks or estimate application performance.  Also, we
   look at the way note that lost packets have this same approach has been treated
   adopted in other metrics:
   delay variation and reordering.

4.1.1.  Application Performance

   Applications need ITU-T Recommendation [Y.1540] where performance parameters
   are only valid during periods of service "availability" (evaluated
   according to perform different functions, dependent a function based on
   whether or not each packet arrives within some finite tolerance.  In
   other words, loss, and sustained periods
   of loss ratio greater than a receivers' packet processing takes one threshold are declared "unavailable").

4.  Effect of two
   directions (or "forks" in POV on the road):

   o  Packets that arrive within expected tolerance are handled by
      processes that remove headers, restore smooth delivery timing (as
      in a de-jitter buffer), restore sending order, check for errors Loss Metric

   This section describes the ways in
      payloads, and many other operations.

   o  Packets that do not arrive when expected spawn other processes
      that attempt recovery from which the apparent loss, such as
      retransmission requests, loss concealment, or forward error
      correction Loss metric can be tuned
   to replace reflect the missing packet.

   So, it is important preferences of the two audience categories, or
   different POV.  The waiting time to maintain declare a distinction between packets that
   actually arrive, and those that do not.  Therefore, it packet lost, or loss
   threshold is preferable one area where there would appear to leave be a difference,
   but the delay ability to post-process the results may resolve it.

4.1.  Loss Threshold

   RFC 2680 [RFC2680] defines the concept of lost a waiting time for packets undefined, and
   to characterize arrive, beyond which they are declared lost.  The text of the
   delay distribution as RFC
   declines to recommend a conditional distribution (conditioned on
   arrival).

4.1.2.  Network Characterization

   In this discussion, we assume value, instead saying that both loss and delay metrics "good engineering,
   including an understanding of packet lifetimes, will be reported needed in
   practice."  Later, in the methodology, they give reasons for network characterization (at least).

   Assume packets waiting
   "a reasonable period of time", and leaving the definition of
   "reasonable" intentionally vague.

4.1.1.  Network Characterization

   Practical measurement experience has shown that do not arrive are reported as Lost, usually as unusual network
   circumstances can cause long delays.  One such circumstance is when
   routing loops form during IGP re-convergence following a
   fraction of all sent packets.  If these lost packets failure or
   drastic link cost change.  Packets will loop between two routers
   until new routes are assigned
   undefined delay, then network's inability to deliver them (in a
   timely way) is captured only in installed, or until the loss metric when we report
   statistics on IPv4 Time-to-Live (TTL)
   field (or the Delay distribution conditioned IPv6 Hop Limit) decrements to zero.  Very long delays
   on the event order of
   packet arrival (within the Loss several seconds have been measured [Casner] [Cia03].

   Therefore, network characterization activities prefer a long waiting
   time threshold).  We can say
   that the Delay and Loss metrics are Orthogonal, in that they convey
   non-overlapping information about the network under test.

   However, if we assign infinite delay order to all lost packets, then:

   o  The delay distinguish these events from other causes of loss
   (such as packet discard at a full queue, or tail drop).  This way,
   the metric results are influenced both by design helps to distinguish more reliably between packets
   that
      arrive might yet arrive, and those that do not.

   o  The delay singleton and are no longer traversing the loss singleton do not appear to be
      orthogonal (Delay is finite when Loss=0, Delay
   network.

   It is infinite when
      Loss=1).

   o  The network possible to calculate a worst-case waiting time, assuming that
   a routing loop is penalized in both the loss and delay metrics,
      effectively double-counting the lost packets.

   As further evidence of overlap, consider cause.  We model the Cumulative Distribution
   Function (CDF) path between Source and
   Destination as a series of Delay when delays in links (t) and queues (q), as
   these two are the value positive infinity is assigned dominant contributors to all lost packets.  Figure 3 shows a CDF where delay.  The normal path
   delay across n hops without encountering a small fraction of
   packets are lost. loop, D, is
                                      n
                                     ---
                                     \
                           D = t  +   >   t  + q
                                0    /     i    i
                                     ---
                                    i = 1 | - - - - - - - - - - - - - - - - - -+
                   |                                    |
                   |          _..----''''''''''''''''''''
                   |      ,-''
                   |    ,'
                   |   /                         Mass at
                   |  /                          +infinity
                   | /                           = fraction
                   ||                            lost
                   |/
                 0 |_____________________________________

                   0               Delay               +o0

                        Figure 3: Cumulative Distribution Function for Delay when Loss =
                                 +Infinity

   We note that a 1: Normal Path Delay CDF that is conditioned on packet arrival would
   not exhibit this apparent overlap with loss.

   Although infinity is a familiar mathematical concept, it is somewhat
   disconcerting to see any time-related metric reported as infinity, in
   the opinion of the authors.  Questions are bound to arise,

   and tend
   to detract from the goal of informing time spent in the consumer loop with a performance
   report.

4.1.3. L hops, is

                       i + L-1
                        ---
                        \                         (TTL - n)
                 R = C   >   t  + q  where C    = ---------
                        /     i    i        max       L
                        ---
                         i

                Figure 2: Delay Variation

   [RFC3393] excludes lost packets from samples, effectively assigning
   an undefined delay due to packets that do not arrive Rotations in a reasonable
   time.  Section 4.1 describes this specification Loop

   and its rationale
   (ipdv = inter-packet delay variation in where C is the quote below).

   "The treatment number of lost packets as having "infinite" or "undefined"
   delay complicates times a packet circles the derivation of statistics for ipdv.
   Specifically, when packets in loop.

   If we take the measurement sequence are lost,
   simple statistics such delays of all links and queues as sample mean cannot be computed.  One
   possible approach to handling this problem is to reduce the event
   space by conditioning.  That is, we consider conditional statistics;
   namely we estimate 100ms each, the mean ipdv (or other derivative statistic)
   conditioned on
   TTL=255, the event that selected packet pairs arrive at the
   destination (within number of hops n=5 and the given timeout).  While this itself is not
   without problems (what happens, for example, when every other packet
   is lost), it offers a way to make some (valid) statements about ipdv,
   at hops in the same time avoiding events with undefined outcomes."

4.1.4.  Reordering

   [RFC4737]defines metrics loop L=4, then

   D = 1.1 sec and R ~= 50 sec, and D + R ~= 51.1 seconds

   We note that are based on evaluation the link delays of packet
   arrival order, 100ms would span most continents, and include a waiting time to declare
   a packet lost
   (to exclude them from further processing).

   If packets are assigned constant queue length of 100ms is also very generous.  When a delay value, then the reordering metric
   would declare any packets with infinite delay loop
   occurs, it is almost certain to be reordered,
   because their sequence numbers will surely be less than the "Next
   Expected" resolved in 10 seconds or less.
   The value calculated above is an upper limit for almost any realistic
   circumstance.

   A waiting time threshold when (or if) they arrive.  But parameter, dT, set consistent with this practice
   calculation would fail to maintain orthogonality between the reordering metric
   and the loss metric.  Confusion can be avoided by designating not truncate the delay of non-arriving packets as undefined, and reserving delay
   values only for distribution (possibly
   causing a change in its mathematical properties), because the packets
   that might arrive within a sufficiently long
   waiting time.

4.2.  Preferred Statistics

   Today in network characterization, have been given sufficient time to traverse the sample mean
   network.

   It is one statistic worth noting that is almost ubiquitously reported.  It is easily computed and
   understood by virtually everyone in this audience category.  Also,
   the sample is usually filtered on packet arrival, so packets that the mean is
   based are stored and deliberately
   forwarded at a conditional distribution.

   The median is another statistic that summarizes much later time constitute a distribution,
   having somewhat different properties from replay attack on the sample mean.  The
   median is stable in distributions with a few outliers or without
   them.  However,
   measurement system, and are beyond the median's stability prevents it from indicating
   when a large fraction scope of normal performance
   reporting.

4.1.2.  Application Performance

   Fortunately, application performance estimation activities are not
   adversely affected by the distribution changes value. 50% or more
   values would need to change for estimated worst-case transfer time.
   Although the median designer's tendency might be to capture set the change.

   Both Loss Threshold
   at a value equivalent to a particular application's threshold, this
   specific threshold can be applied when post-processing the median and sample mean have difficulty
   measurements.  A shorter waiting time can be enforced by locating
   packets with bimodal
   distributions.  The median will reside in only one of the modes, and delays longer than the mean may not lie in either mode range.  For this application's threshold, and other
   reasons, additional statistics re-
   designating such packets as lost.  Thus, the minimum, maximum, and 95%-
   ile have value when summarizing measurement system can
   use a distribution.

   When single loss threshold and support both the sample mean application and median are available, a comparison will
   sometimes be informative, because these two statistics are equal only
   when the delay distribution is perfectly symmetrical.

   Also, these statistics network
   performance POVs simultaneously.

4.2.  Errored Packet Designation

   RFC 2680 designates packets that arrive containing errors as lost
   packets.  Many packets that are generally useful from corrupted by bit errors are discarded
   within the Application
   Performance POV, so there network and do not reach their intended destination.

   This is a common set consistent with applications that should satisfy
   audiences.

4.3.  Summary for Delay

   From would check the perspectives of:

   1.  application/receiver analysis, where subsequent processing
       depends on whether payload
   integrity at higher layers, and discard the packet.  However, some
   applications prefer to deal with errored payloads on their own, and
   even a corrupted payload is better than no packet arrives or times-out,

   2.  straightforward at all.

   To address this possibility, and to make network characterization without double-counting
       defects, and

   3.  consistency with Delay variation and Reordering metric
       definitions,

   the most efficient practice
   more complete, it is recommended to distinguish between truly lost packets that
   do not arrive (lost) and
   delayed errored packets with that arrive (conditionally
   lost).

4.3.  Causes of Lost Packets

   Although many measurement systems use a sufficiently long waiting time, and time to
   designate the delay of non-arriving packets as undefined.

5.  Effect determine if
   a packet is lost or not, most of POV on Raw Capacity Metrics

   This section describes the ways that raw capacity metrics can be
   tuned to reflect waiting is in vain.  The packets
   are no-longer traversing the preferences network, and have not reached their
   destination.

   There are many causes of the two audiences, packet loss, including:

   1.  Queue drop, or different
   Points-of-View (POV).  Raw capacity refers to discard

   2.  Corruption of the metrics defined in
   [RFC5136] which do not include restrictions such as data uniqueness IP header, or flow-control response other essential header info

   3.  TTL expiration (or use of a TTL value that is too small)

   4.  Link or router failure

   After waiting sufficient time, packet loss can probably be attributed
   to congestion.

   In summary, one of these causes.

4.4.  Summary for Loss

   Given that measurement post-processing is possible (even encouraged
   in the metrics considered are IP-layer Capacity, Utilization
   (or used capacity), definitions of IPPM metrics), measurements of loss can easily
   serve both points of view:

   o  Use a long waiting time to serve network characterization and Available Capacity,
      revise results for individual links specific application delay thresholds as
      needed.

   o  Distinguish between errored packets and
   complete paths.  These three metrics form a triad: knowing one metric
   constrains the other two (within their allowed range), lost packets when possible
      to aid network characterization, and knowing
   two determines combine the third.  The link metrics have another key aspect results for
      application performance if appropriate.

5.  Effect of POV on the Delay Metric

   This section describes the ways in common: they are single-measurement-point metrics at which the egress of
   a link.  The path Capacity and Available Capacity are derived by
   examining Delay metric can be
   tuned to reflect the set preferences of single-point link measurements and taking the
   minimum value. two consumer categories, or
   different POV.

5.1.  Type-P Parameter

   The concept of "packets  Treatment of type-P" is defined in [RFC2330]. Lost Packets

   The
   type-P categorization has critical relevance in all forms Delay Metric [RFC2679] specifies the treatment of capacity
   measurement and reporting.  The ability to categorize packets based
   on header fields for assignment to different queues that do
   not successfully traverse the network: their delay is undefined.

   " >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined
   (informally, infinite)<< means that Src sent the first bit of a
   Type-P packet to Dst at wire-time T and scheduling
   mechanisms that Dst did not receive that
   packet."

   It is now common place.  When un-used resources an accepted, but informal practice to assign infinite delay to
   lost packets.  We next look at how these two different treatments
   align with the needs of measurement consumers who wish to
   characterize networks or estimate application performance.  Also, we
   look at the way that lost packets have been treated in other metrics:
   delay variation and reordering.

5.1.1.  Application Performance

   Applications need to perform different functions, dependent on
   whether or not each packet arrives within some finite tolerance.  In
   other words, a receivers' packet processing takes one of two
   directions (or "forks" in the road):

   o  Packets that arrive within expected tolerance are shared
   across queues, handled by
      processes that remove headers, restore smooth delivery timing (as
      in a de-jitter buffer), restore sending order, check for errors in
      payloads, and many other operations.

   o  Packets that do not arrive when expected spawn other processes
      that attempt recovery from the apparent loss, such as
      retransmission requests, loss concealment, or forward error
      correction to replace the missing packet.

   So, it is important to maintain a distinction between packets that
   actually arrive, and those that do not.  Therefore, it is preferable
   to leave the delay of lost packets undefined, and to characterize the
   delay distribution as a conditional distribution (conditioned on
   arrival).

5.1.2.  Network Characterization

   In this discussion, we assume that both loss and delay metrics will
   be reported for network characterization (at least).

   Assume packets that do not arrive are reported as Lost, usually as a
   fraction of all sent packets.  If these lost packets are assigned
   undefined delay, then network's inability to deliver them (in a
   timely way) is captured only in the loss metric when we report
   statistics on the Delay distribution conditioned on the event of
   packet arrival (within the Loss waiting time threshold).  We can say
   that the conditions in all packet categories will affect
   capacity Delay and related measurements.  This is one source of variability Loss metrics are Orthogonal, in that they convey
   non-overlapping information about the network under test.

   However, if we assign infinite delay to all lost packets, then:

   o  The delay metric results are influenced both by packets that all audiences would prefer to see reported in a
   useful
      arrive and easily understood way.

   Type-P in OWAMP those that do not.

   o  The delay singleton and TWAMP is essentially confined to the Diffserv
   Codepoint [ref].  DSCP loss singleton do not appear to be
      orthogonal (Delay is finite when Loss=0, Delay is infinite when
      Loss=1).

   o  The network is penalized in both the most common qualifier for type-P.

   Each audience will have a set of type-P qualifications loss and value
   combinations that are delay metrics,
      effectively double-counting the lost packets.

   As further evidence of interest.  Measurements and reports SHOULD
   have overlap, consider the flexibility Cumulative Distribution
   Function (CDF) of Delay when the value positive infinity is assigned
   to per-type and aggregate performance.

5.2. all lost packets.  Figure 3 shows a priori Factors

   The audience CDF where a small fraction of
   packets are lost.

                 1 | - - - - - - - - - - - - - - - - - -+
                   |                                    |
                   |          _..----''''''''''''''''''''
                   |      ,-''
                   |    ,'
                   |   /                         Mass at
                   |  /                          +infinity
                   | /                           = fraction
                   ||                            lost
                   |/
                 0 |_____________________________________

                   0               Delay               +o0

     Figure 3: Cumulative Distribution Function for Network Characterization may have detailed
   information about each link Delay when Loss =
                                 +Infinity

   We note that comprises a complete path (due Delay CDF that is conditioned on packet arrival would
   not exhibit this apparent overlap with loss.

   Although infinity is a familiar mathematical concept, it is somewhat
   disconcerting to
   ownership, for example), or some of the links see any time-related metric reported as infinity, in
   the path but not
   others, or none opinion of the links.

   There authors.  Questions are cases where the measurement audience only has information
   on one of the links (the local access link), bound to arise, and wishes tend
   to measure
   one or more detract from the goal of informing the raw capacity metrics.  This scenario is quite
   common, and has spawned consumer with a substantial number of experimental
   measurement methods [ref performance
   report.

5.1.3.  Delay Variation

   [RFC3393] excludes lost packets from samples, effectively assigning
   an undefined delay to CAIDA survey page, etc.].  Many of these
   methods respect packets that their users want do not arrive in a result fairly quickly reasonable
   time.  Section 4.1 describes this specification and its rationale
   (ipdv = inter-packet delay variation in the quote below).

   "The treatment of lost packets as having "infinite" or "undefined"
   delay complicates the derivation of statistics for ipdv.
   Specifically, when packets in
   a one-trial.  Thus, the measurement interval is kept short (a few
   seconds sequence are lost,
   simple statistics such as sample mean cannot be computed.  One
   possible approach to a minute).

5.3.  IP-layer Capacity

   For links, handling this metric's theoretical maximum value can be determined
   from problem is to reduce the physical layer bit rate and event
   space by conditioning.  That is, we consider conditional statistics;
   namely we estimate the bit rate reduction due to mean ipdv (or other derivative statistic)
   conditioned on the event that selected packet pairs arrive at the layers between
   destination (within the physical layer and IP.  When measured, given timeout).  While this
   metric takes additional factors into account, such as itself is not
   without problems (what happens, for example, when every other packet
   is lost), it offers a way to make some (valid) statements about ipdv,
   at the ability of same time avoiding events with undefined outcomes."

   We note that the sending device argument above applies to process and forward traffic under various
   conditions.  For example, the arrival all forms of routing updates may spawn
   high priority processes packet delay
   variation that reduce can be constructed using the sending rate temporarily.
   Thus, "selection function"
   concept of [RFC3393].  In recent work the measured capacity two main forms of a link will be variable, delay
   variation metrics have been compared and the
   maximum capacity observed applies to a specific time, time interval,
   and other relevant circumstances.

   For paths composed results are summarized
   in [RFC5481].

5.1.4.  Reordering

   [RFC4737]defines metrics that are based on evaluation of packet
   arrival order, and include a series of links, it is easy waiting time to see how the
   sources of variability for declare a packet lost
   (to exclude them from further processing).

   If packets are assigned a delay value, then the results grow reordering metric
   would declare any packets with each link in the
   path.  Results variability infinite delay to be reordered,
   because their sequence numbers will surely be discussed in more detail below.

5.4.  IP-layer Utilization

   The ideal metric definition of Link Utilization [RFC5136] is based on less than the actual usage (bits successfully received during a time interval)
   and "Next
   Expected" threshold when (or if) they arrive.  But this practice
   would fail to maintain orthogonality between the Maximum Capacity for reordering metric
   and the same interval.

   In practice, Link Utilization loss metric.  Confusion can be calculated by counting the IP-
   layer (or other layer) octets received over a time interval and
   dividing avoided by designating the theoretical maximum
   delay of octets non-arriving packets as undefined, and reserving delay
   values only for packets that could have been
   delivered arrive within a sufficiently long
   waiting time.

5.2.  Preferred Statistics

   Today in network characterization, the same interval.  A commonly used time interval sample mean is 5
   minutes, one statistic
   that is almost ubiquitously reported.  It is easily computed and
   understood by virtually everyone in this interval has been sufficient to support network
   operations and design for some time. 5 minutes is somewhat long
   compared with audience category.  Also,
   the expected download time for web pages, but short
   with respect to large file transfers and TV program viewing.  It sample is
   fair to say usually filtered on packet arrival, so that considerable variability the mean is concealed by reporting
   based a
   single (average) Utilization value for each 5 minute interval.  Some
   performance management systems have begun to make 1 minute averages
   available.

   There conditional distribution.

   The median is also another statistic that summarizes a limit on the smallest useful measurement interval.
   Intervals on the order of distribution,
   having somewhat different properties from the serialization time for a single Maximum
   Transmission Unit (MTU) packet will observe on/off behavior and
   report 100% or 0%.  The smallest interval needs to be some multiple
   of MTU serialization time for averaging to be effective.

5.5.  IP-layer Available Capacity sample mean.  The Available Capacity of
   median is stable in distributions with a link can be calculated using few outliers or without
   them.  However, the Capacity
   and Utilization metrics.

   When Available capacity of median's stability prevents it from indicating
   when a link large fraction of the distribution changes value. 50% or path is estimated through some
   measurement technique, more
   values would need to change for the following parameters SHOULD be reported:

   o  Name and reference median to capture the exact method of measurement

   o  IP packet length, octets (including IP header)

   o  Maximum Capacity that can be assessed in change.

   Both the measurement
      configuration

   o median and sample mean have difficulty with bimodal
   distributions.  The time a duration median will reside in only one of the measurement

   o  All modes, and
   the mean may not lie in either mode range.  For this and other parameters specific to
   reasons, additional statistics such as the measurement method

   Many methods of Available capacity measurement minimum, maximum, and 95%-
   ile have value when summarizing a maximum
   capacity that they can measure, distribution.

   When both the sample mean and this maximum may median are available, a comparison will
   sometimes be less than the
   actual Available capacity of informative, because these two statistics are equal only
   when the link or path.  Therefore, it delay distribution is
   important to know perfectly symmetrical.

   Also, these statistics are generally useful from the capacity value beyond which there will be no
   measured improvement.

   The Application Design audience may have a target capacity value and
   simply wish to assess whether
   Performance POV, so there is sufficient Available Capacity.
   This case simplifies measurement a common set that should satisfy
   audiences.

   Plots of the delay distribution may also be useful when single-value
   statistics indicate that new conditions are present.  An empirically-
   derived probability distribution function will usually describe
   multiple modes more efficiently than any other form of link and path capacity to some
   degree, as long as result.

5.3.  Summary for Delay

   From the measurable maximum exceeds perspectives of:

   1.  application/receiver analysis, where subsequent processing
       depends on whether the target
   capacity.

5.6.  Variability in Utilization packet arrives or times-out,

   2.  straightforward network characterization without double-counting
       defects, and Avail. Capacity

   As

   3.  consistency with most metrics Delay variation and measurements, assessing the consistency or
   variability in Reordering metric
       definitions,

   the results gives most efficient practice is to distinguish between truly lost and
   delayed packets with a sufficiently long waiting time, and to
   designate the user an intuitive feel for the
   degree (or confidence) that any one value is representative delay of other
   results, or non-arriving packets as undefined.

6.  Effect of POV on Raw Capacity Metrics

   This section describes the underlying distribution from which these singleton
   measurements have come.

   Two questions are raised here for further discussion:

   What ways that raw capacity metrics can Utilization be measured and summarized
   tuned to describe reflect the
   potential variability in a useful way?

   How can preferences of the variability in Available Capacity estimates be reported,
   so that two audiences, or different
   Points-of-View (POV).  Raw capacity refers to the confidence metrics defined in
   [RFC5136] which do not include restrictions such as data uniqueness
   or flow-control response to congestion.

   In summary, the results is also conveyed?

6.  Test Streams metrics considered are IP-layer Capacity, Utilization
   (or used capacity), and Sample Size

   This section discusses Available Capacity, for individual links and
   complete paths.  These three metrics form a triad: knowing one metric
   constrains the other two (within their allowed range), and knowing
   two determines the third.  The link metrics have another key aspects of measurement that aspect
   in common: they are
   sometimes omitted from the report: single-measurement-point metrics at the description egress of
   a link.  The path Capacity and Available Capacity are derived by
   examining the test stream
   on which the set of single-point link measurements are based, and taking the sample size.
   minimum value.

6.1.  Test Stream Characteristics

   Network Characterization  Type-P Parameter

   The concept of "packets of type-P" is defined in [RFC2330].  The
   type-P categorization has traditionally used Poisson-distributed
   inter-packet spacing, as this provides an unbiased sample. critical relevance in all forms of capacity
   measurement and reporting.  The
   average inter-packet spacing may be selected ability to allow observation of
   specific network phenomena.  Other test streams are designed categorize packets based
   on header fields for assignment to
   sample some property of the network, such as different queues and scheduling
   mechanisms is now common place.  When un-used resources are shared
   across queues, the presence of
   congestion, link bandwidth, or conditions in all packet reordering.

   If measuring a network categories will affect
   capacity and related measurements.  This is one source of variability
   in order the results that all audiences would prefer to make inferences about applications
   or receiver performance, then there are usually efficiencies derived
   from see reported in a test stream that has similar characteristics
   useful and easily understood way.

   Type-P in OWAMP and TWAMP is essentially confined to the sender.
   In some cases, it Diffserv
   Codepoint [ref].  DSCP is essential to synthesize the sender stream, as
   with Bulk Transfer Capacity estimates.  In other cases, it may be
   sufficient to sample with a "known bias", e.g., most common qualifier for type-P.

   Each audience will have a Periodic stream set of type-P qualifications and value
   combinations that are of interest.  Measurements and reports SHOULD
   have the flexibility to
   estimate real-time application per-type and aggregate performance.

6.2.  Sample Size

   Sample size is directly related  a priori Factors

   The audience for Network Characterization may have detailed
   information about each link that comprises a complete path (due to the accuracy
   ownership, for example), or some of the results, and
   plays a critical role links in the report.  Even if only path but not
   others, or none of the sample size
   (in terms links.

   There are cases where the measurement audience only has information
   on one of number the links (the local access link), and wishes to measure
   one or more of packets) the raw capacity metrics.  This scenario is given for each value or summary
   statistic, it imparts quite
   common, and has spawned a notion substantial number of the confidence experimental
   measurement methods [ref to CAIDA survey page, etc.].  Many of these
   methods respect that their users want a result fairly quickly and in
   a one-trial.  Thus, the result.

   In practice, the sample size will measurement interval is kept short (a few
   seconds to a minute).

6.3.  IP-layer Capacity

   For links, this metric's theoretical maximum value can be selected taking both statistical determined
   from the physical layer bit rate and practical factors into account.  Among these factors are:

   1.  The estimated variability of the quantity being measured

   2.  The desired confidence in bit rate reduction due to
   the result (although layers between the physical layer and IP.  When measured, this may be
       dependent on assumption
   metric takes additional factors into account, such as the ability of
   the underlying distribution sending device to process and forward traffic under various
   conditions.  For example, the arrival of routing updates may spawn
   high priority processes that reduce the sending rate temporarily.
   Thus, the measured quantity).

   3.  The effects capacity of active measurement traffic on user traffic

   4.  etc.

   A sample size may sometimes a link will be referred variable, and the
   maximum capacity observed applies to as "large".  This is a
   relative, specific time, time interval,
   and qualitative term.  It is preferable to describe what
   one is attempting to achieve with their sample. other relevant circumstances.

   For example, stating
   an implication may be helpful: this sample is large enough such that paths composed of a single outlying value at ten times the "typical" sample mean (the
   mean without the outlying value) would influence series of links, it is easy to see how the mean by no more
   than X.

7.  Reporting Results

   This section gives an overview
   sources of recommendations, followed by
   additional considerations variability for reporting the results grow with each link in the "long-term".

7.1.  Overview of Metric Statistics

   This section gives an overview
   path.  Results variability will be discussed in more detail below.

6.4.  IP-layer Utilization

   The ideal metric definition of reporting recommendations for the
   loss, delay, and delay variation metrics Link Utilization [RFC5136] is based on
   the discussion and
   conclusions of the preceding sections.

   The minimal report on measurements MUST include both Loss and Delay
   Metrics.

   For Packet Loss, the loss ratio defined in [RFC2680] is actual usage (bits successfully received during a sufficient
   starting point, especially time interval)
   and the guidance Maximum Capacity for setting the loss
   threshold waiting time.  We have same interval.

   In practice, Link Utilization can be calculated by counting the IP-
   layer (or other layer) octets received over a waiting time above interval and
   dividing by the theoretical maximum of octets that
   should be could have been
   delivered in the same interval.  A commonly used time interval is 5
   minutes, and this interval has been sufficient to differentiate between packets that are truly
   lost or have support network
   operations and design for some time. 5 minutes is somewhat long finite delays under general measurement
   circumstances, 51 seconds.  Knowledge of specific conditions can help
   to reduce this threshold,
   compared with the expected download time for web pages, but 51 seconds short
   with respect to large file transfers and TV program viewing.  It is considered
   fair to be
   manageable in practice.

   We note say that considerable variability is concealed by reporting a loss ratio calculated according
   single (average) Utilization value for each 5 minute interval.  Some
   performance management systems have begun to [Y.1540] would
   exclude errored packets form the numerator.  In practice, the
   difference between these two loss metrics make 1 minute averages
   available.

   There is small if any, depending also a limit on whether the last link prior to the destination contributes errored
   packets.

   For Packet Delay, we recommend providing both smallest useful measurement interval.
   Intervals on the mean delay and order of the
   median delay with lost packets designated undefined (as permitted by
   [RFC2679]).  Both statistics are based on serialization time for a conditional distribution,
   and the condition is single Maximum
   Transmission Unit (MTU) packet arrival prior will observe on/off behavior and
   report 100% or 0%.  The smallest interval needs to a waiting be some multiple
   of MTU serialization time dT, where
   dT has been set for averaging to take maximum packet lifetimes into account, as
   discussed above.  Using be effective.

6.5.  IP-layer Available Capacity

   The Available Capacity of a long dT helps to ensure that delay
   distributions are not truncated.

   For Packet Delay Variation (PDV), link can be calculated using the minimum delay Capacity
   and Utilization metrics.

   When Available capacity of a link or path is estimated through some
   measurement technique, the
   conditional distribution should following parameters SHOULD be used as the reported:

   o  Name and reference delay for
   computing PDV according to [Y.1540] or [RFC3393].  A useful value to
   report is a pseudo range the exact method of delay variation based on calculating measurement

   o  IP packet length, octets (including IP header)

   o  Maximum Capacity that can be assessed in the
   difference between measurement
      configuration

   o  The time a high percentile duration of delay the measurement

   o  All other parameters specific to the measurement method

   Many methods of Available capacity measurement have a maximum
   capacity that they can measure, and this maximum may be less than the minimum delay.
   For example,
   actual Available capacity of the 99.9%-ile minus link or path.  Therefore, it is
   important to know the minimum capacity value beyond which there will give be no
   measured improvement.

   The Application Design audience may have a target capacity value that
   can be compared with objectives in [Y.1541].

7.2.  Long-Term Reporting Considerations

   [I-D.ietf-ippm-reporting] describes methods and
   simply wish to conduct measurements assess whether there is sufficient Available Capacity.
   This case simplifies measurement of link and path capacity to some
   degree, as long as the measurable maximum exceeds the target
   capacity.

6.6.  Variability in Utilization and report Avail. Capacity

   As with most metrics and measurements, assessing the consistency or
   variability in the results on gives a near-immediate time scale (10 seconds,
   which we consider to be "short-term").

   Measurement intervals and reporting intervals need not be the same
   length.  Sometimes, the user is only concerned with an intuitive feel for the performance
   levels achieved over a relatively long interval
   degree (or confidence) that any one value is representative of time (e.g, days,
   weeks, other
   results, or months, as opposed to 10 seconds).  However, there the underlying distribution from which these singleton
   measurements have come.

   Two questions are raised here for further discussion:

   What ways can Utilization be
   risks involved with running a measurement continuously over a long
   period without recording intermediate results:

   o  Temporary power failure may cause loss of all the results measured and summarized to date.

   o  Measurement system timing synchronization signals may experience describe the
   potential variability in a
      temporary outage, causing sub-sets of measurements to be useful way?

   How can the variability in error
      or invalid.

   o  Maintenance may Available Capacity estimates be necessary on reported,
   so that the measurement system, or its
      connectivity to confidence in the network under test.

   For these results is also conveyed?

7.  Test Streams and other reasons, such as

   o Sample Size

   This section discusses two key aspects of measurement that are
   sometimes omitted from the constraint to collect measurements report: the description of the test stream
   on intervals similar which the measurements are based, and the sample size.

7.1.  Test Stream Characteristics

   Network Characterization has traditionally used Poisson-distributed
   inter-packet spacing, as this provides an unbiased sample.  The
   average inter-packet spacing may be selected to
      user session length, or

   o  the dual-use allow observation of measurements in monitoring activities where
      results
   specific network phenomena.  Other test streams are needed on a period designed to
   sample some property of a few minutes,

   there is value in conducting measurements on intervals that are much
   shorter than the reporting interval.

   There are several approaches for aggregating a series network, such as the presence of measurement
   results over time
   congestion, link bandwidth, or packet reordering.

   If measuring a network in order to make a statement inferences about applications
   or receiver performance, then there are usually efficiencies derived
   from a test stream that has similar characteristics to the longer
   reporting interval.  One approach requires the storage of all metric
   singletons collected throughout the reporting interval, even though
   the measurement interval stops and starts many times.

   Another approach sender.
   In some cases, it is described in [I-D.ietf-ippm-framework-compagg] essential to synthesize the sender stream, as
   "temporal aggregation".  This approach would
   with Bulk Transfer Capacity estimates.  In other cases, it may be
   sufficient to sample with a "known bias", e.g., a Periodic stream to
   estimate real-time application performance.

7.2.  Sample Size

   Sample size is directly related to the results for accuracy of the reporting interval based on many individual measurement interval
   statistics (results) alone.  The result would ideally appear results, and
   plays a critical role in the
   same form as though a continuous measurement was conducted.  A memo
   to address report.  Even if only the details sample size
   (in terms of temporal aggregation number of packets) is yet to be prepared.

   Yet another approach requires a numerical objective given for each value or summary
   statistic, it imparts a notion of the metric,
   and confidence in the results of each measurement interval are compared with result.

   In practice, the
   objective.  Every measurement interval where sample size will be selected taking both statistical
   and practical factors into account.  Among these factors are:

   1.  The estimated variability of the results meet quantity being measured

   2.  The desired confidence in the
   objective contribute to result (although this may be
       dependent on assumption of the fraction underlying distribution of time with performance as
   specified.  When the reporting interval contains many
       measured quantity).

   3.  The effects of active measurement
   intervals it is possible traffic on user traffic

   4.  etc.

   A sample size may sometimes be referred to present the results as "metric A was less
   than or equal "large".  This is a
   relative, and qualitative term.  It is preferable to objective X during Y% of time.

   NOTE describe what
   one is attempting to achieve with their sample.  For example, stating
   an implication may be helpful: this sample is large enough such that numerical thresholds are not set in IETF performance work
   and are explicitly excluded from
   a single outlying value at ten times the IPPM charter. "typical" sample mean (the
   mean without the outlying value) would influence the mean by no more
   than X.

8.  IANA Considerations

   This document makes no request of IANA.

   Note to RFC Editor: this section may be removed on publication as an
   RFC.

9.  Security Considerations

   The security considerations that apply to any active measurement of
   live networks are relevant here as well.  See [RFC4656].

10.  Acknowledgements

   The authors would like to thank thank: Phil Chimento for his suggestion to employ
   conditional distributions for Delay, and Steve Konish Jr. for his careful
   review and suggestions. suggestions, Dave Mcdysan and Don McLachlan for useful
   comments based on their long experience with measurement and
   reporting.

11.  References

11.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2330]  Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
              "Framework for IP Performance Metrics", RFC 2330,
              May 1998.

   [RFC2678]  Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring
              Connectivity", RFC 2678, September 1999.

   [RFC2679]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
              Delay Metric for IPPM", RFC 2679, September 1999.

   [RFC2680]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
              Packet Loss Metric for IPPM", RFC 2680, September 1999.

   [RFC3148]  Mathis, M. and M. Allman, "A Framework for Defining
              Empirical Bulk Transfer Capacity Metrics", RFC 3148,
              July 2001.

   [RFC3393]  Demichelis, C. and P. Chimento, "IP Packet Delay Variation
              Metric for IP Performance Metrics (IPPM)", RFC 3393,
              November 2002.

   [RFC3432]  Raisanen, V., Grotefeld, G., and A. Morton, "Network
              performance measurement with periodic streams", RFC 3432,
              November 2002.

   [RFC4656]  Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
              Zekauskas, "A One-way Active Measurement Protocol
              (OWAMP)", RFC 4656, September 2006.

   [RFC4737]  Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
              S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
              November 2006.

   [RFC5136]  Chimento, P. and J. Ishac, "Defining Network Capacity",
              RFC 5136, February 2008.

11.2.  Informative References

   [Casner]   "A Fine-Grained View of High Performance Networking, NANOG
              22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May
              20-22 2001.

   [Cia03]    "Standardized Active Measurements on a Tier 1 IP Backbone,
              IEEE Communications Mag., pp 90-97.", June 2003.

   [I-D.ietf-ippm-framework-compagg]
              Morton, A., "Framework for Metric Composition",
              draft-ietf-ippm-framework-compagg-09 (work in progress),
              December 2009.

   [I-D.ietf-ippm-reporting]
              Shalunov, S. and M. Swany, "Reporting IP Performance
              Metrics to Users", draft-ietf-ippm-reporting-04 (work in
              progress), July 2009.

   [RFC5481]  Morton, A. and B. Claise, "Packet Delay Variation
              Applicability Statement", RFC 5481, March 2009.

   [RFC5835]  Morton, A. and S. Van den Berghe, "Framework for Metric
              Composition", RFC 5835, April 2010.

   [Y.1540]   ITU-T Recommendation Y.1540, "Internet protocol data
              communication service - IP packet transfer and
              availability performance parameters", December  2002.

   [Y.1541]   ITU-T Recommendation Y.1540, "Network Performance
              Objectives for IP-Based Services", February  2006.

Authors' Addresses

   Al Morton
   AT&T Labs
   200 Laurel Avenue South
   Middletown, NJ  07748
   USA

   Phone: +1 732 420 1571
   Fax:   +1 732 368 1192
   Email: acmorton@att.com
   URI:   http://home.comcast.net/~acmacm/
   Gomathi Ramachandran
   AT&T Labs
   200 Laurel Avenue South
   Middletown, New Jersey  07748
   USA

   Phone: +1 732 420 2353
   Fax:
   Email: gomathi@att.com
   URI:

   Ganga Maguluri
   AT&T Labs
   200 Laurel Avenue
   Middletown, New Jersey  07748
   USA

   Phone: 732-420-2486
   Fax:
   Email: gmaguluri@att.com
   URI: