DetNet                                                  J. Korhonen, Ed.
Internet-Draft                                                    Nordic
Intended status: Standards Track                            L. Andersson
Expires: May 3, August 2, 2018                                         Y. Jiang
                                                                 N. Finn
                                                                  Huawei
                                                                B. Varga
                                                               J. Farkas
                                                                Ericsson
                                                           CJ. Bernardos
                                                                    UC3M
                                                              T. Mizrahi
                                                                 Marvell
                                                               L. Berger
                                                                    LabN
                                                        October 30, 2017
                                                        January 29, 2018

                    DetNet Data Plane Encapsulation
                      draft-ietf-detnet-dp-sol-00
                      draft-ietf-detnet-dp-sol-01

Abstract

   This document specifies Deterministic Networking data plane
   encapsulation solutions.  The described data plane solutions can be
   applied over either IP or MPLS Packet Switched Networks.

   Comment #1:  SB> An overarching comment is that the early part of the
     document is really fundamental architecture and perhaps belongs in
     the arch draft, leaving this draft to be more specific about
     solutions.  Indeed if we cannot find a single solution that maps to
     both IP and MPLS underlays I wonder if we should publish two
     specialist RFCs?

   Discussion:  One document at the beginning, split into two if/when
     needed.  Would be post adoption in any case.

   Comment #2:  SB> Whilst I think we should look for a common solution
     to IP and MPLS I do not think that this is where the DT ended up.

   Discussion:  Agree.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on May 3, August 2, 2018.

Copyright Notice

   Copyright (c) 2017 2018 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   5   4
     2.1.  Terms used in this document . . . . . . . . . . . . . . .   5   4
     2.2.  Abbreviations . . . . . . . . . . . . . . . . . . . . . .   7   5
   3.  Requirements language . . . . . . . . . . . . . . . . . . . .   8   6
   4.  DetNet data plane overview  . . . . . . . . . . . . . . . . .   8   6
     4.1.  DetNet data plane encapsulation requirements  . . . . . .   8
     4.2.  Packet replication and elimination considerations . . . .  10
   5.  DetNet data plane solution
     4.3.  Packet reordering considerations  . . . . . . . . . . . .  10
   5.  MPLS-based DetNet data plane solution . . . . . . . . . . .  12 .  10
     5.1.  DetNet specific packet fields . . . . . . . . . . . . . .  12  10
     5.2.  DetNet  Data plane encapsulation  . . . . . . . . . . . . . . . .  11
     5.3.  DetNet control word . .  12
       5.2.1.  PseudoWire-based data plane encapsulation . . . . . .  13
       5.2.2.  Native IPv6-based data plane encapsulation . . . . .  15
     5.3.  DetNet flow identification for duplicate detection . . .  17
       5.3.1.  PseudoWire encapsulation . . . .  12
     5.4.  Flow identification . . . . . . . . . .  17
       5.3.2.  Native IPv6 encapsulation . . . . . . . . .  13
     5.5.  Service layer considerations  . . . . .  18
   6.  PREF specific considerations . . . . . . . . .  13
       5.5.1.  Edge node processing  . . . . . . .  18
     6.1.  PseudoWire-based data plane . . . . . . . . .  13
       5.5.2.  Relay node processing . . . . . .  18
       6.1.1.  Forwarder clarifications . . . . . . . . . .  15
       5.5.3.  End system processing . . . .  18
       6.1.2.  Edge node processing clarifications . . . . . . . . .  19
       6.1.3.  Relay node processing clarifications . . .  17
     5.6.  Transport node considerations . . . . .  21
     6.2.  Native IPv6-based data plane . . . . . . . . .  17
       5.6.1.  Congestion protection . . . . .  23
   7.  Other DetNet data plane considerations . . . . . . . . . . .  23
     7.1.  Class of Service  17
       5.6.2.  Explicit routes . . . . . . . . . . . . . . . . . . .  17
   6.  IPv6-based DetNet data plane solution .  23
     7.2.  Quality of Service . . . . . . . . . . .  17
     6.1.  Data plane encapsulation  . . . . . . . .  24
     7.3.  Cross-DetNet flow resource aggregation . . . . . . . .  17
     6.2.  DetNet destination option .  25
     7.4.  Bidirectional traffic . . . . . . . . . . . . . . .  19
     6.3.  Flow identification . . .  26
     7.5.  Layer 2 addressing and QoS Considerations . . . . . . . .  27
     7.6.  Interworking between PW- and IPv6-based encapsulations .  27
   8.  Time synchronization . . . . . . .  20
     6.4.  Service layer considerations  . . . . . . . . . . . . .  27
   9.  Management and control considerations .  20
       6.4.1.  Edge node processing  . . . . . . . . . . .  29
     9.1.  PW Label and IPv6 Flow Label assignment and distribution   29
     9.2.  Packet replication and elimination . . . . .  21
       6.4.2.  Relay node processing . . . . . .  30
     9.3.  Explicit paths . . . . . . . . . .  23
       6.4.3.  End system processing . . . . . . . . . . .  30
     9.4. . . . . .  23
     6.5.  Transport node processing . . . . . . . . . . . . . . . .  23
       6.5.1.  Congestion protection and latency control . . . . . . . .  30
     9.5.  Flow aggregation control . . . . . . . .  23
       6.5.2.  Explicit routes . . . . . . . .  30
   10. Security . . . . . . . . . . .  24
   7.  Other DetNet data plane considerations  . . . . . . . . . . .  24
     7.1.  Class of Service  . . . . . . . .  30
   11. IANA considerations . . . . . . . . . . . .  24
     7.2.  Quality of Service  . . . . . . . . .  30
   12. Acknowledgements . . . . . . . . . .  24
     7.3.  Cross-DetNet flow resource aggregation  . . . . . . . . .  26
     7.4.  Bidirectional traffic . . .  30
   13. References . . . . . . . . . . . . . . .  27
     7.5.  Layer 2 addressing and QoS Considerations . . . . . . . .  27
     7.6.  Interworking between MPLS- and IPv6-based encapsulations   28
     7.7.  IPv4 considerations . .  31
     13.1.  Normative references . . . . . . . . . . . . . . . . .  28
   8.  Time synchronization  .  31
     13.2.  Informative references . . . . . . . . . . . . . . . . .  33
   Appendix A.  Example of DetNet . .  28
   9.  Management and control considerations . . . . . . . . . . . .  30
     9.1.  MPLS-based data plane operation . . . . . . .  34
   Appendix B.  Example of pinned paths using IPv6 . . . . . . . . .  35
   Authors' Addresses . .  30
       9.1.1.  S-Label assignment and distribution . . . . . . . . .  30
       9.1.2.  Explicit routes . . . . . . . . . . . .  35

1.  Introduction

   Deterministic Networking (DetNet) is a service that can be offered by
   a network to DetNet flows.  DetNet provides these . . . . . . .  30
     9.2.  IPv6-based data plane . . . . . . . . . . . . . . . . . .  30
       9.2.1.  Flow Label assignment and distribution  . . . . . . .  30
       9.2.2.  Explicit routes . . . . . . . . . . . . . . . . . . .  31
     9.3.  Packet replication and elimination  . . . . . . . . . . .  31
     9.4.  Congestion protection and latency control . . . . . . . .  31
     9.5.  Flow aggregation control  . . . . . . . . . . . . . . . .  31
   10. Security considerations . . . . . . . . . . . . . . . . . . .  31
   11. IANA considerations . . . . . . . . . . . . . . . . . . . . .  31
   12. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  31
   13. References  . . . . . . . . . . . . . . . . . . . . . . . . .  32
     13.1.  Normative references . . . . . . . . . . . . . . . . . .  32
     13.2.  Informative references . . . . . . . . . . . . . . . . .  34
   Appendix A.  Example of DetNet data plane operation . . . . . . .  35
   Appendix B.  Example of pinned paths using IPv6 . . . . . . . . .  36
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  36

1.  Introduction

   Deterministic Networking (DetNet) is a service that can be offered by
   a network to DetNet flows.  DetNet provides these flows extremely low
   packet loss rates and assured maximum end-to-end delivery latency.
   General background and concepts of DetNet can be found in
   [I-D.ietf-detnet-architecture].

   This document specifies the DetNet data plane.  It defines how plane and the on-wire
   encapsulation of DetNet
   traffic is encapsulated at flows.  The specified encapsulation provides
   the building blocks to enable the DetNet service layer functions and
   allow flow identification as described in the network layer, and how DetNet-aware
   nodes can identity DetNet flows. Architecture.
   Two data plane definitions are given.

   o  PW-based: One solution is based on

   1.  MPLS-based: The enacapsulation resembles PseudoWires (PW) [RFC3985] and
      [RFC5036] and makes use of multi-segment pseudowires (MS-PW)
      [RFC6073] to map DetNet Relay and Edge Nodes
      [I-D.ietf-detnet-architecture] [I-D.ietf-detnet-dp-alt] to PW
      architecture.  The PW-based data plane can be run over with an
       MPLS
      [RFC4448] [RFC6658] Packet Switched Network (PSN).

   Comment #3:  SB> This is really an MPLS one.  The world of IP PWs is
     a bit scruffy with some work in PWE3 and some in L2TPext which
     really went their own ways.  There is for example no MS-PW IP
     design. (PSN) [RFC3985][RFC4385].

   2.  Native-IP-based: The MS-PW approach needs to be examined more closely by
     the WG and thus at this stage be marked as provisional.

   Discussion:  Agree. "can be" -> "is".

   Comment #3.1  LB> EVPN-based encapsulation encapsulating protocol is also a potential
     candidate.  In general DetNet should look more closely to the
     delevopment around EVPN.

   Discussion  Agree.  EVPN could be a potential solution IPv6 and the on-
     wire encapsuations are likely to be the same as PW-based
     encapsulation would be.  EVPN even recommends using Control Word
     [RFC8214] (in the absence of entropy labels).

   o  Native-IP: The other
       solution is based relies on IP header fields, namely
      on the IPv6 Flow Label existing and a new DetNet Control Word extension
      header option.  It is targeted for native specific
       IPv6 networks.

   Comment #4:  SB> The IP solution has not been properly examined by
     the WG eaxtension header options [RFC8200].

      [Editor's note: MPLS- and needs IPv6-based solutions are likely to be marked as provisional.

   Discussion:  IP vs. MPLS is a deployment choice.
      split into different documents.]

   It is worth noting that while PWs are designed to work over MPLS-based solution can transport IP PSNs
   this document describes
   packets a native-IP solution that operates without
   PWs. is meant for deployments where the
   DetNet service layer functions are provided at the IP-layer rather
   than the underlying transport network.  The primary reason for this
   is the benefit gained by enabling the use of a normal application
   stack, where transport protocols such as TCP or UDP are directly
   encapsulated in IP.

   Comment #5:  SB> We clearly need to look at the implications of
     running this with a new IP header extension.  Firstly we need input
     from 6man, but we also need to understand what happens in middle
     boxes, other components of the host stack etc.

   Discussion:  A WG can develop their own extensions and then get
     approval from 6man.  Sometimes

   The DetNet transport layer functionality that ends up redoing extensions in
     6man but not always.

   This document specifies the encapsulation provides congestion
   protection for DetNet flows, including flows is assumed to be in place in a DetNet Control Word (CW).
   node.

   Furthermore, it this document also describes how DetNet flows are
   identified, how a DetNet Relay and Edge Relay/Edge/Transit nodes work, and how the
   Packet Replication and Elimination function (PREF) is implemented
   with these the two data plane solutions.

   This document does not define the associated control plane functions,
   or Operations, Administration, and Maintenance (OAM).  It also does
   not specify traffic handling capabilities required to deliver
   congestion protection and latency control to for DetNet flows as this is defined to
   be provided by the underlying MPLS or IP network.

   Comment #6:  SB> OK, although I think that this may be a mistake.
     There may well be some coupling needed between the Detnet DP and
     the substrate/transport/underlay needed to make this work.  If this
     is a genuine technical layering we should talk about it.  If this
     is an artificial constraint imposed by the IESG we need to talk to
     them.

   Discussion:  The only interaction needed is that the flow
     identification is possible.  That needs to be available for lower
     layers.

   Comment #6.1:  LA> Even though this document does not specify any OAM
     functions, we will need to verify that at the GACh (Generalized
     Associate Channel) works correctly in a network that has
     replication and elimination.

   Discussion:  --
   DetNet transport layer.

2.  Terminology

2.1.  Terms used in this document

   This document uses the terminology established in the DetNet
   architecture [I-D.ietf-detnet-architecture] and the DetNet Data Plane
   Solution Alternatives [I-D.ietf-detnet-dp-alt].

   The following terms are also used in this document:

   DA-T-PE       MPLS based DetNet edge node: a DetNet-aware PseudoWire
                 Terminating Provider Edge (T-PE).

   DA-S-PE       MPLS based DetNet relay node: a DetNet-aware PseudoWire
                 Switching Provider Edge (S-PE).

   Comment #7  SB> We need to look at whether the S-PE concept is the
      best fit, or whether we should use introduce a Detnet relay to do
      this.  An S-PE just swaps the PW label, but Detnet needs it to do
      more and a better model might be a new construct.  However we
      could also discard the relay concept and run 1+n end to end, in
      which case the S-PEs would retain heir original function.

   Discussion:  Disagree of the dropping comment.  However,

   This document uses the issues
      are most likely terminology related.  The relay concept is part of established in the DetNet
   architecture A DA-S-PE was intended to be a DetNet
      relay, which may do more than just swapping labels (PREF
      functionality).  Current text is quite biased to MS-PW, which was
      the starting point for [I-D.ietf-detnet-architecture] and the DetNet relay in a MPLS PW network. Data Plane
   Solution Alternatives [I-D.ietf-detnet-dp-alt].

   T-Label       A label used to identify the LSP used to transport a
                 DetNet flow across an MPLS PSN, e.g., a hop-by-hop
                 label used between label switching routers (LSR).

   S-Label       A DetNet node to DetNet node "service" label that is used between DA-*-PE devices.

   PW Label      A PseudoWire label DetNet
                 nodes that implment also the DetNet service layer
                 functions.  An S-Label is also used to identify a
                 DetNet flow
                 related PW Instances within a PE node. at DetNet service layer.

   Flow Label    IPv6 header field that is used to identify a DetNet
                 flow (together with the source IP address field).

   Comment #8  SB> If this is the IPv6 Flow label I think caution is
      needed as I don't think the handling of this is well defined or
      consistently implemented in networking equipment.

   Discussion:

   Local-ID      A DetNet specifies the use and discusses possible
      interaction with RFC6347 in this I-D.

   local-ID      An edge Edge and relay Relay node internal construct that
                 uniquely identifies a DetNet flow. flow within a node and
                 never appear on-wire.  It may be used to select proper
                 forwarding and/or DetNet specific service function.

   Comment #9  SB> Is this really an internal construct, or is it an on
      the wire construct?  If it is needed end to end, I don't think it
      is correct to describe it as an internal construct.  When you say
      "select" presumably you mean by potentially any DN aware node on
      the path?

   Discussion:  It is an internal construct, so yes.

   PREF          A Packet Replication and Elimination Function (PREF)
                 does the replication and elimination processing of
                 DetNet flow packets in edge or relay nodes.  The
                 replication function is essentially the existing 1+1
                 protection mechanism.  The elimination function reuses
                 and extends the existing duplicate detection mechanism
                 to operate over multiple (separate) DetNet member flows
                 of a DetNet compound flow.

   Comment #10  SB> I wonder if 1+1 is the right way to go, or whether
      1+n is better.  A bunch of new techniques have emerged over the
      years and we really ought to look at creating paths with MRT.

      With 1+2 on main + the two MRT paths you have a two failure
      resiliency as far as it is possible to construct such paths in the
      underlying topology.

   Discussion:  As observed above, actually 1+n would be closer to what
      is needed. 1+1 was meant to be more an example showing there is
      existing work that can be leveraged. a DetNet compound flow.

   DetNet Control Word  A control word used for sequencing and
                 identifying duplaicate packets at the DetNet service
                 layer.

2.2.  Abbreviations

   The following abbreviations used in this document:

   AC            Attachment Circuit.

   CE            Customer Edge equipment.

   CoS           Class of Service.

   CW            Control Word.

   d-CW          DetNet Control Word.

   DetNet        Deterministic Networking.

   DF            DetNet Flow.

   L2VPN         Layer 2 Virtual Private Network.

   LSR           Label Switching Router.

   MPLS          Multiprotocol Label Switching.

   MPLS-TP       Multiprotocol Label Switching - Transport Profile.

   MS-PW         Multi-Segment PseudoWire (MS-PW).

   NSP           Native Service Processing.

   OAM           Operations, Administration, and Maintenance.

   PE            Provider Edge.

   PREF          Packet Replication and Elimination Function.

   PSN           Packet Switched Network.

   PW            PseudoWire.

   QoS           Quality of Service.

   TSN           Time-Sensitive Network.

3.  Requirements language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

4.  DetNet data plane overview

   Comment #11  I am not sure whether this is a DP overview, or an
      architecture overview and hence whether this needs to be here or
      in the architecture draft.

   Discussion:  Overview is more of an editorial matter and its final
      location can be discussed later on.  Currently it is "no harm" to
      have it here but there are no binding reasons to keep the text in
      either.

   This document describes how to use IP and/or MPLS to support a data
   plane method of flow identification and packet formwarding over
   layer-3.  Two different cases are covered: (i) the inter-connect
   scenario, in which IEEE802.1 TSN is routed over a layer-3 network
   (i.e., to enlarge the layer-2 domain), and (ii) native connectivity
   between DetNet-aware end systems.

   Figure 1 illustrates an exemplary
   scenario.

  TSN              Edge          Transit        Relay        DetNet
  End System       Node            Node         Node         End System

  +---------+    +.........+                                 +---------+
  |  Appl.  |<---:Svc Proxy:-- End to End Service ---------->|  Appl.  |
  +---------+    +---------+                   +---------+   +---------+
  |   TSN   |    |TSN| |Svc|<-- DetNet flow ---: Service :-->| Service |
  +---------+    +---+ +---+    +---------+    +---------+   +---------+
  |Transport|    |Trp| |Trp|    |Transport|    |Trp| |Trp|   |Transport|
  +-------.-+    +-.-+ +-.-+    +--.----.-+    +-.-+ +-.-+   +---.-----+
          :  Link  :    /  ,-----.  \   :  Link  :    /  ,-----.  \
          +........+    +-[  Sub  ]-+   +........+    +-[  Sub  ]-+
                          [Network]                     [Network]
                           `-----'                       `-----'

          Figure 1: A simple DetNet enabled network architecture

   Figure 2 illustrates how DetNet can provide services for IEEE
   802.1TSN end systems over a DetNet enabled network.  The edge nodes
   insert and remove required DetNet data plane encapsulation.  The 'X'
   in the edge and relay nodes represents a potential DetNet flow packet
   replication and elimination point.  This conceptually parallels L2VPN
   services, and could leverage existing related solutions as discussed
   below.

      TSN    |<---------- End to End DetNet Service ------>|  TSN
     Service |           Transit           Transit         | Service
 TSN  (AC)   |        |<-Tunnel->|        |<-Tnl->|        |  (AC)  TSN
 End    |    V        V     1    V        V   2   V        V   |    End
 System |    +--------+          +--------+       +--------+   |  System
 +---+  |    |   E1   |==========|   R1   |=======|   E2   |   |   +---+
 |   |--|----|._X_....|..DetNet..|.._ _...|..DF3..|...._X_.|---|---|   |
 |CE1|  |    |    \   |  Flow 1  |   X    |       |   /    |   |   |CE2|
 |   |       |     \_.|...DF2....|._/ \_..|..DF4..|._/     |       |   |
 +---+       |        |==========|        |=======|        |       +---+
     ^       +--------+          +--------+       +--------+       ^
     |        Edge Node          Relay Node       Edge Node        |
     |                                                             |
     |<----- Emulated Time Sensitive Networking (TSN) Service ---->|

                    Figure 2: 1: IEEE 802.1TSN over DetNet

   Figure 3 2 illustrates how end to end PW-based MPLS-based DetNet service can be
   provided.  In this case, the end systems are able to send and receive
   DetNet flows.  For example, an end system sends data encapsulated in
   PseudoWire (PW) and in
   MPLS.  Like earlier the 'X' in the end
   systems, edge systems, edge and relay nodes
   represents potential DetNet flow packet replication and elimination
   points.  Here the relay nodes may change the underlying transport,
   for example tunneling IP over MPLS, or simply interconnect network
   segments.

         DetNet                                             DetNet
         Service          Transit          Transit          Service
   DetNet  |             |<-Tnl->|        |<-Tnl->|            | DetNet
   End     |             V   1   V        V   2   V            | End
   System  |    +--------+       +--------+       +--------+   | System
   +---+   |    |   R1   |=======|   R2   |=======|   R3   |   |  +---+
   |  X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X |
   |CE1|========|    \   |       |   X    |       |   /    |======|CE2|
   |   |   |    |     \_.|..DF2..|._/ \__.|..DF4..|._/     |   |  |   |
   +---+        |        |=======|        |=======|        |      +---+
       ^        +--------+       +--------+       +--------+      ^
       |        Relay Node       Relay Node       Relay Node      |
       |                                                          |
       |<--------------- End to End DetNet Service -------------->|

                    Figure 2: MPLS-Based Native DetNet

   Figure 3 illustrates how end to end IP-based DetNet service can be
   provided.  In this case, the end systems are able to send and relay nodes represents potential receive
   DetNet flow packet
   replication and elimination points.  Here the relay nodes may change
   the underlying transport, for example tunneling IP over MPLS, or
   simply interconnect network segments. flows.  [Editor's note: TBD]
   NOTE: This figures is TBD

         DetNet                                             DetNet
         Service          Transit          Transit          Service
   DetNet  |             |<-Tnl->|        |<-Tnl->|            | DetNet
   End     |             V   1   V        V   2   V            | End
   System  |    +--------+       +--------+       +--------+   | System
   +---+   |    |   R1   |=======|   R2   |=======|   R3   |   |  +---+
   |  X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X  X...DFa...|        |
   |CE1|========|    \       |        |   X       |        |   /    |======|CE2|     .|.X |
   | H1|========|        |       |        |       |        |======| H2|
   |   |   |    |        |       |        |       |     \_.|..DF2..|._/ \__.|..DF4..|._/        |   |  |   |
   +---+        |        |=======|        |=======|        |      +---+
       ^        +--------+       +--------+       +--------+      ^
       |        Relay Node       Relay Node       Relay Node      |
       |                                                          |
       |<--------------- End to End DetNet Service -------------->|

                     Figure 3: PW-Based IP-Based Native DetNet

   Figure 4 illustrates how end

4.1.  DetNet data plane encapsulation requirements

   Two major groups of scenarios can be distinguished which require flow
   identification during transport:

   1.  DetNet function related scenarios:

       *  Congestion protection and latency control: usage of allocated
          resources (queuing, policing, shaping).

       *  Explicit routes: select/apply the flow specific path.

       *  Service protection: recognize DetNet compound and member flows
          for replication an elimination.

       Comment #12  I am not sure whether the correct architectural
          construct is flow or flow group.  Flow suggests that sharing/
          aggregation is not allowed but whether this is allowed or not
          is an application specific issue.

       Discussion:  Agree that a flow group would be a better
          characterization.

       Comment #13  I think that there needs to be some clarification as
          to whether FG is is understood by the DN system exclusively or
          whether there is an expectation that it is understood by the
          underlay.

       Discussion:  Agree that more detail is needed here.  DetNet aware
          nodes need to understand flow groups.  Underlay needs to be
          aware of flow groups at the resource allocation level.

   2.  OAM function related scenarios:

       *  troubleshooting (e.g., identify misbehaving flows, etc.)

       *  recognize flow(s) for analytics (e.g., increase counters,
          etc.)

       *  correlate events with flows (e.g., volume above threshold,
          etc.)

       *  etc.

   Each DetNet node (edge, relay and transit) use an internal/
   implementation specific local-ID of the DetNet-(compound)-flow in
   order to end IP-based DetNet service can be
   provided.  In this case, accomplish its role during transport.  Recognizing the end systems
   DetNet flow is more relaxed for edge and relay nodes, as they are able to send
   fully aware of both the DetNet service and receive transport layers.  The
   primary DetNet flows.  [Editor's note: TBD]

   NOTE: This figures role of intermediate transport nodes is TBD limited to
   ensuring congestion protection and latency control for the above
   listed DetNet functions.

   The DetNet
         Service          Transit          Transit          Service data plane allows for the aggregation of DetNet  |             |<-Tnl->|        |<-Tnl->|            | flows,
   e.g., via MPLS hierarchical LSPs, to improved scaling.  When DetNet
   End     |             V   1   V        V   2   V            | End
   System  |    +--------+       +--------+       +--------+   | System
   +---+   |    |   R1   |=======|   R2   |=======|   R3   |   |  +---+
   |  X...DFa...|        |       |        |       |        |     .|.X |
   | H1|========|        |       |        |       |        |======| H2|
   |   |   |    |        |       |        |       |        |   |  |   |
   +---+        |        |=======|        |=======|        |      +---+
       ^        +--------+       +--------+       +--------+      ^
       |        Relay Node       Relay Node       Relay Node      |
       |                                                          |
       |<--------------- End
   flows are aggregated, transit nodes may have limited ability to End
   provide service on per-flow DetNet Service -------------->|

                     Figure 4: IP-Based Native identifiers.  Therefore,
   identifying each individual DetNet

4.1. flow on a transit node may not be
   achieved in some network scenarios, but DetNet data plane encapsulation requirements

   Two major groups of scenarios service can still be distinguished which require flow
   identification during transport:

   1.  DetNet function related scenarios:

       *  Congestion protection and latency control: usage of allocated
          resources (queuing, policing, shaping).

       *  Explicit routes: select/apply the flow specific path.

       *  Service protection: recognize DetNet compound
   assured in these scenarios through resource allocation and member flows
          for replication an elimination. control.

   Comment #12  I am not sure whether #14  You could introduce the correct architectural
          construct is concept of a flow or group
      identified into the packet.  You may also include a flow group.  Flow suggests that sharing/
          aggregation is not allowed but whether this is allowed or not
          is an application specific issue. id at a
      lower layer.

   Discussion:  Agree that on the identification properties.  Adding a flow group would be
      specific id into actual on-wire formats is not necessarily needed.

   On each DetNet node dealing with DetNet flows, an internal local-ID
   is assumed to determine what local operation a better
          characterization.

       Comment #13  I think that there needs packet goes through.
   Therefore, local-IDs has to be some clarification as
          to whether FG is unique on each edge and relay nodes.
   Local-ID is understood by unambiguously bound to the DN DetNet flow.

4.2.  Packet replication and elimination considerations

   DetNet service layer introduces packet replication and elimination
   functionality (PREF) for use in DetNet edge and relay node and end
   system exclusively or
          whether there is an expectation that it is understood by packet processing.  PREF MAY be enabled in a DetNet node and
   the
          underlay.

       Discussion:  Agree that more detail required processing is needed here.  DetNet aware
          nodes need to understand flow groups.  Underlay needs only applied to be
          aware of packets with a positive
   flow groups identification at the resource allocation level.

   2.  OAM function related scenarios:

       *  troubleshooting (e.g., identify misbehaving flows, etc.)

       *  recognize flow(s) for analytics (e.g., increase counters,
          etc.)

       *  correlate events with flows (e.g., volume above threshold,
          etc.)

       *  etc.

   Each node (edge, relay and transit) use DetNet service layer.  PREF utilizes a local-ID
   sequence number carried within a DetNet flow packets.

   At a DetNet node level the output of the DetNet-
   (compound)-flow in order to accomplish its role during transport.
   Recognizing PREF elimination function is
   always a single packet.  The output of the PREF replication function
   at a DetNet flow node level is always one or more relaxed packets (i.e., 1:M
   replication).  The replicated packets MUST share the same d-CW i.e.,
   the sequence number is the same for edge and relay nodes,
   as they are fully aware each member flow of both the DetNet service and transport
   layers. compound
   flow.  The primary DetNet role of intermediate transport nodes is
   limited to ensuring congestion protection location and latency control for mechanism on the
   above listed DetNet functions. packet processing pipeline
   used for replication is implementation specific.

   The complex part of the DetNet data plane allows for PREF processing is tracking the aggregation
   history of received packets for multiple DetNet flows,
   e.g., via MPLS hierarchical LSPs, to improved scaling.  When member flows.  These
   ingress DetNet member flows are aggregated, transit nodes may (to a node) MUST have limited ability the same local-ID
   if they belong to
   provide service on per-flow DetNet identifiers.  Therefore,
   identifying each individual the same DetNet (compound) flow on a transit node may not be
   achieved in some network scenarios, but DetNet service can still be
   assured in these scenarios through resource allocation and control.

   Comment #14  You could introduce share the concept same
   sequence number counter and the history information.  The location of a flow group
      identified into
   the packet.  You may also include a flow id at a
      lower layer.

   Discussion:  Agree packet elimination on the identification properties.  Adding a
      specific id into actual on-wire formats packet processing pipeline is not necessarily needed.

   On each node dealing with
   implementation specific.

4.3.  Packet reordering considerations

   DetNet flows, a local-ID is assumed to
   determine what local operation a service layer introduces also packet goes through.  Therefore,
   local-IDs MUST be unique on each reordering functionality
   for use in DetNet edge and relay nodes.  Local-ID MUST
   be unambiguously bound to the DetNet flow.

   Comment #15  I am confused as to what you mean by local ID.  Is this
      an internal construct which node and end system packet parameters map to,
   processing.  The reordering functionality MAY be enabled in which
      case why is it being called up?  IETF practise is to specify a DetNet
   node.  The reordeing functionality relies on
      the wire behaviour but not internal behaviour a presence of equipments.

   Discussion:  Local-id sequence
   numbers in a DetNet (compound) flows.  The reordering processing is an internal construct, which was intended
   only applied to
      clarify the discussion in packets with a positive flow identification at the I-D.  Obviously it did not work as
      intended.
   DetNet service layer.

5.  MPLS-based DetNet data plane solution

5.1.  DetNet specific packet fields

   The DetNet data plane encapsulation should MUST include two DetNet specific
   information element elements in each packet of a DetNet flow: (1) a flow
   identification and (2) a sequence number.

   Comment #16  should, SHOULD, must or MUST?

   Discussion:  SHOULD or MUST is ok.  MUST is probably more
      appropriate.

   The DetNet data plane encapsulation may consists further elements
   used for overlay tunneling, to distinguish between DetNet member
   flows of the same DetNet compound flow or to support OAM functions.

5.2.  DetNet encapsulation

   This document specifies two encapsulations for the DetNet data plane:
   (1) PseudoWire (PW) for MPLS PSN and (2) native IPv6 based
   encapsulation for IP PSN.

5.2.1.  PseudoWire-based data same DetNet compound flow or to support OAM functions.

5.2.  Data plane encapsulation

   Figure 5 4 illustrates a DetNet PW encapsulation over an data plane MPLS PSN. encapsulation.  The
   PW-based
   MPLS-based encapsulation of the DetNet flows fits perfectly is a good fit for the
   Layer-2 interconnect deployment cases (see Figure 2). 1).  Furthermore,
   end to end DetNet service i.e., native DetNet deployment (see
   Figure 3) 2) is also possible if DetNet-aware DetNet end systems are capable of
   initiating and termination MPLS encapsulated PWs.  It is also
   possible use the same encapsulation format with a Packet PW over MPLS
   [RFC6658]. packets.  Transport of
   IP encapsulated DetNet flows, see Section 5.2.2, 6, over MPLS-based DetNet PWs
   data plane is also possible.  Interworking between PW- and IPv6-based
   encapsulations is discussed further in Section 7.6.

   The PW-based MPLS-based DetNet data plane encapsulation consists of:

   o  DetNet control word (d-CW) containing sequencing information for
      packet replication and duplicate elimination purposes.  There is MUST
      a separate sequence number space for each DetNet flow.

   o  PseudoWire  DetNet Label (PW Label) that is a standard PW label
      identifying identifies a DetNet flow and a PW Instance within a (DA-)T-PE DetNet Edge or
      (DA-)S-PE device.
      a Relay node.  The DetNet label MUST be at the bottom of the label
      stack.

   o  An optional S-Label DetNet service lable (S-Label) that represents DetNet
      Service LSP used between (DA-)T-PE or (DA-)S-PE DetNet Egde and/or Relay nodes.  One
      possible use of an S-Label is to identify the different DetNet member flows used
      to provide protection to a DetNet composite compound flow, perhaps even when
      both LSPs appear on the same link for some reason.

   Comment #17  This needs some discussion by the WG.

   Discussion:  Agree, specifically if the I-D becomes WG document.

   o

   One or more MPLS transport LSP label(s) (T-label) which may be a hop-by-hop hop-
   by-hop label used between LSRs.

   Comment #18  Ordinarily this will LSR and MUST appear higher in the label
   stack than S-labels.  A top of course stack T-label may be PHPed before arrival
   arriving at an x-PE.

   Discussion: a DetNet node.  In most cases yes - depends on general T-labels should be considered
   to be part of the underlying transport network
      configuration.  PHP is not mandatory and TP does not even have
      PHP.

    RFC3985 Encapsulation rather the actual
   DetNet PW Encapsulation

   +---------------------+
   |      Payload        | data plane encapsulation.

        DetNet MPLS-based encapsulation

      +---------------------------------+
   /=====================\
      |                                 |
   H Payload Convergence H--.
      |           DetNet Flow           |
   H---------------------H  |
      |         Payload  Packet         |
   H       Timing        H  +-\
      |                                 |
   H---------------------H
      +---------------------------------+ <--\
      |  \    /=================================\
   H     Sequencing      H--'   \-->H       DetNet Control Word       H
   \=====================/          \=================================/       |  PW Demultiplexer   |--------->|            PW Label    |
   +---------------------+
      +---------------------------------+    +--> DetNet data plane
      |  PSN Convergence    |     .--->|      Optional MPLS             S-Label             |
   +---------------------+    |    MPLS encapsulation
      +---------------------------------+ <--/
      |         PSN         |-----+--->|         MPLS           T-Label(s)            |
   +---------------------+
      +---------------------------------+
      |           Data-Link             |
   +---------------------+
      +---------------------------------+
      |           Physical              |
   +---------------------+
      +---------------------------------+

       Figure 5: 4: Encapsulation of a DetNet flow in a PW with MPLS(-TP) PSN in an MPLS(-TP) PSN

5.3.  DetNet control word

   A DetNet control word (d-CW) conforms to the Generic PW MPLS Control
   Word (PWMCW) defined in [RFC4385] and is illustrated in Figure 5.
   The upper nibble of the d-CW MUST be set to zero (0).  The effective
   sequence number bit length is between 0 and 28 bits, and configured
   either by a control plane or manually for each DetNet flow.  The
   sequence number is aligned to the right (least significant bits) and
   unused bits MUST be set to zero (0).  Each DetNet flow MUST have its
   own sequence number counter.  The DetNet control word (d-CW) sequence number is identical to the control word
   defined incremented by
   one for Ethernet over MPLS networks in [RFC4448]. each new packet.

   The DetNet
   control word is illustrated d-CW MUST always be present in Figure 6. a packet.  In a case the sequence
   number is not used (e.g., for DetNet-t-flows) the control plane or
   the manual configuration has to define zero (0) bit length seqeunce
   number and the value of the sequence number MUST be set to zero (0).

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0 0 0 0|  reserved - set to 0  |   16 bit                Sequence Number      |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       Figure 6: DetNet Control Word

   Comment #19  We need to think about whether "identical is the correct
      term.  The Ethernet S/N skips zero (it uses zero to mean no S/N in
      use).  is that the behaviour that we want?

   Discussion:  Good point.  Identical is a wrong statement.  The format
      is the same the behaviour of SN is slightly different as 0 is
      assumed to be a valid SN.

5.2.2.  Native IPv6-based data plane encapsulation

   Comment #20  SB> This part of the design needs to be marked as
      provisional until it has a more thorough WG review.

   Discussion:  Ok, however, this is still an individual I-D.

   Figure 7 illustrates a DetNet native IPv6 encapsulation.  The native
   IPv6 encapsulation is meant for end to end Detnet service use cases,
   where the end stations are DetNet-aware (see Figure 4).  Technically
   it is possible to use the IPv6 encapsulation to tunnel any traffic
   over a DetNet enabled network, which would make native IPv6
   encapsulation also a valid data plane choice for an interconnect use
   case (see Figure 2).

   The native IPv6-based DetNet data plane encapsulation consists of:

   o  IPv6 header as the transport protocol.

   o  IPv6 header Number                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       Figure 5: DetNet Control Word

5.4.  Flow Label that is used to help to identify a identification

   DetNet flow (i.e., roughly an equivalent to the PW Label).  A Flow Label
      together with the IPv6 source address uniquely identifies identification at a DetNet
      flow.

   Comment #21  SB> Have we validated that it service layer is unconditionally safe realized by
   an S-label.  It maps a Detnet flow to
      make this assumption about the use of the FL?

   Discussion:  RFC6437 does not restrict such use and a specific d-CW in a DetNet DP
      solution can always define their own use of
   node.  The S-label used for flow label.  It should identification MUST be noted that bottom label
   of the label stack for a DetNet aware node will always contain new code DetNet-s- or DetNet-st-flow and
      is not a load balancer.

   o  DetNet Control Word IPv6 Destination Option containing sequencing
      information MUST precede
   the d-CW.

   An S-label for packet replication and duplicate elimination
      function (PREF) purposes.  The a single DetNet Destination Option is
      equivalent flow does not need to the be unique DetNet Control Word.

   A DetNet-aware end station (a host)
   domain wide.  As long as two or an intermediate more different DetNet flows do not
   errorneously map to a same d-CW in a DetNet node
   initiating an IPv6 packet is responsible for setting the Flow Label,
   adding the required DetNet Destination Option, and possibly adding labels may vary.

5.5.  Service layer considerations

   [Editor's note: quite a
   routing header such as bit of unfinished and old text in the segment routing option (for pre-defined
   paths [I-D.ietf-6man-segment-routing-header]).
   following sections.]

   The payload edge and relay node internal procedures of the
   native IPv6 encapsulation PREF are
   implementation specific.  The order of a packet elimination or
   replication is any payload protocol that can out of scope in this specification.  However, care
   should be
   identified using taken that the Next Header field either in replication function does not actually
   loopback packets as "replicas".  Looped back packets include
   artificial delay when the node that originally initiated the IPv6 packet
   header or in
   receives it again.  Also, looped back packets may make the last IPv6 extension header. network
   condition to look healthier than it actually is (in some cases link
   failures are not reflected properly because looped back packets make
   the situation appear better than it actually is).

   Comment #22 #29:  SB> We will probably need to agree an option ordering,
      and need There needs to note be some text about preventing a node
      ever receiving its own replicated packets.  Indeed that would
      suggest that the 6man IPv6 solution already operates flow id should be changed and replication should
      only take place on configured flow IDs.  I have a feeling that
      this would all be a lot safer if replication only happened at
      ingress and we managed the edge of the ability diversity of h/w to see that far into the packet. paths.

   Discussion:  RFC8200 describes extension header ordering - there is
      not much to agree there.  Agree on hardening the hardware lookup challenges.
      However, the issues of SR header loopback text considerations.

5.5.1.  Edge node processing

   TBD.

   [Editor's note: Since we are not this I-D to fix.

   Comment #23  SB> I am not sure defining the above is needed since it is by
      definition correct.

   Discussion:  (next header) agree.

   A DetNet-aware end station (a host) or an intermediate node receiving
   an IPv6 packet destined to it inner workings and containing a DetNet Destination
   Option does the appropriate processing
   implementation of the packet.  This may
   involve packet duplication DetNet Egde node - rather only what goes in and elimination (PREF processing),
   terminating a tunnel or delivering
   what comes out, and of course the packet on-wire details, then the figures
   shown in the coming section would not need to detail the upper layers/
   Applications.

                    +---------------------------------+ inner
   architecture of a DetNet Node.]
                 +---------------------------------------+
                 |           DetNet Edge Node            |
                 +---------------------------------------+
                 |           DetNet Flow             |           |             Payload             |
                 |             |
                    /---------------------------------\
                    H DetNet Control Word DstOpt Hdr  H
                    \---------------------------------/           |          IPv6 header             |
     Client AC   |     (with set Flow label)         +---o <-------> o             o<---------->
                 |
                    +---------------------------------+

           Figure 7: Encapsulation of a native IPv6 DetNet flow

   A DetNet flow must carry sequencing information for packet
   replication and elimination function (PREF) purposes.  This document
   specifies a new IPv6 Destination Option: the DetNet Destination
   Option for that purpose.  The format of the option is illustrated in
   Figure 8.

   Comment #24  SB> Can an SR node look at a DO?

   Discussion:  Yes.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   Elim. |     TBD1   |       4           |           Reserved             |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     <---------->o <-------|   |     16 bit Sequence Number           +-------------+
                 |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+         |   |           |             |
                 |         +---o <-------> o             |
                 |             |           |             o<---------->
                 |             |       +-> o             |
                 +-------------+       |   +-------------+
                 |             |       |   |             |
     Client AC   |             | Repl. |   |             |
     <---------->o             o <-----X-> o             o<---------->
                 |             | Elim.     |             |
                 +-------------+           +-------------+
                 |             |           |             |
     Client AC   |             |           |             |
     <---------->o             o <-------> o             o<---------->
                 |             |           |             |
                 +---------------------------------------+

                   Figure 8: 6: DetNet Destination Option

   The Option Type for Edge Node processing

   An edge node participates to the DetNet Destination Option packet replication and duplication
   elimination.  Required processing is set to TBD1.
   [To be removed from done within an extended
   forwarder function.  In the final version of case the document: The Option
   Type MUST have native service processing (NSP)
   is IEEE 802.1CB [IEEE8021CB] capable, the two most significant bits set to 10b]

5.3.  DetNet flow identification for packet replication and
   duplicate detection

   Duplicate elimination depends on flow identification.  Mapping
   between packet fields MAY entirely be done in the NSP and Local-ID may impact bypassing
   the DetNet flow encapsulation and logic entirely, and thus is able to
   operate over unmodified implementation and deployment.  The NSP
   approach works only between edge nodes and cannot make use of
   duplicate elimination. relay
   nodes (see Section 5.5.2).

   Comment #25 #31  SB> So I wonder if the right place This would be a fine way to put operate the FI PW system -
      edge to edge.

   Discussion:  When it comes to use of NSPs, agree.  Also for "island
      interconnect" this is a fine.  However, when there is a need to do
      PREF in a middle, plain edge to edge is not enough.

   The DetNet-aware extended forwarder selects the IPv6 FI, or in the IPv6 address itself?

   Discussion:  Each egress DetNet member
   flow having different address based on the DetNet forwarding rules.  In both "normal AC" and
   "Packet AC" cases there may be no DetNet encapsulation header
   available yet as it is challenging if we
      want to terminate multiple flows into the same node with one
      address or originate multiple flows from a node case with one address
      (note, we are aware of the one /64 per node discussion but cannot
      assume it here, at least not yet).

5.3.1.  PseudoWire encapsulation

   RFC3985 relay nodes (see Section 5.2.1. describes PW sequencing provides a duplicate
   detection service among other things.  This specification clarifies
   this definition as follows:

      DetNet flows that need to undergo PREF processing MUST have 5.5.2).

   It is the responsibility of the
      same PW Label when they arrive at extended forwarder within the DA-*-PE node.

   From edge
   node to push the label stack processing point of view receiving DetNet specific encapsulation (if not already
   present) to the same
   label from multiple sources is analogous packet before forwarding it to Fast Reroute backup
   tunnel behavior [RFC4090].  The PW Label for a the appropriate egress
   DetNet member flow can be
   different on each PW segment. instance(s).

   Comment #26 #32  SB> I am not sure convinced of the utility wisdom of this reference.  In
      FRR you should not receive packets concurrently on two paths.  It
      seems fine to state the the requirement that having a single label mid-
      point node convert a flow into a DN flow, which is
      used for both paths.

   Discussion:  OK with the same label comment.  OK to remove the FRR
      reference what you are
      implying here.

5.3.2.  Native IPv6 encapsulation  This seems like an ingress function.

   Discussion:  OK.  The DetNet flow identification is based on the IPv6 Flow Label text here has issues and
   the source address combination. seems to mix relay and
      edge.

   The two fields uniquelly identify extended forwarder MAY copy the sequencing information from the end to end
   native IPv6 encapsulated DetNet flow.  Obviously, the
   identification fails if any intermediate node modifies either the
   source address or packet into the Flow Label.

   Comment #27  SB> See earlier. DetNet sequence number field and vice
   versa.  If there are enough IPv6 addresses to
      address video fragments, why not DN flows?  Then this problem goes
      away.

   Discussion:  See the earlier comment #25 discussion.  If nodes get
      their addressies via DHCPv6 basically ruins this mechanism.  Also
      the assumption for this to work is that the node has a full /64 to
      use, which is not always no existing sequencing information available in
   the case.  Otherwise native packet or the idea is just
      fine.

6.  PREF specific considerations

   This section applies equally forwarder chose not to copy it from the
   native packet, then the extended forwarder MUST maintain a sequence
   number counter for each DetNet flows transported via IPv6 and
   MPLS.  While flow identification and some header related processing
   will differ between the two, (indexed by the considerations covered in this
   section are common to both.

6.1.  PseudoWire-based data plane

6.1.1.  Forwarder clarifications

   The DetNet specific new functionality in an edge or relay flow
   identification).

5.5.2.  Relay node processing is

   TBD.

   A DetNet Relay node participates to the packet replication and
   duplication elimination
   function (PREF). elimination.  This function processing is a part of the DetNet-aware
   "extended" forwarder.  The PREF done within an extended
   forwarder function.  Whether an ingress DetNet member flow receives
   DetNet specific processing depends on how the forwarding is triggered by
   programmed.  For some DetNet member flows the
   received packet of relay node can act as a
   normal relay node and for some apply the DetNet flow. specific processing
   (i.e., PREF).

   Comment #28 #34  SB> I Again relay node is not a normal term, so am not
      sure what you mean by triggered here.
      Hopefully we are not thinking it does in the absence of dataplane triggered
      configuration? a PREF function.

   Discussion:  "Initiated"  Relay node was a DetNet aware S-PE originally, which is probably more appropriate wording.

   Basically
      not explicitly stated here anymore, thus slightly confusing text
      here.  The text here needs to clarify the forwarding entry has roles of PREF and
      switching functions.  A DetNet relay is described in the
      architecture document.  However, there is definitely room for
      termonilogy and text improvements.

   It is also possible to be extended with treat the relay node as a "PREF
   enabled" boolean configuration switch that transit node, see
   Section 7.3.  Again, this is associated with entirely up to how the
   normal forwarding actions (e.g., in case of MPLS a swap, push, pop,
   ..). has
   been programmed.

   The output of DetNet-aware forwarder selects the PREF elimination function is always a single
   packet. egress DetNet member flow
   segment based on the flow identification.  The output mapping of ingress
   DetNet member flow segment to egress DetNet member flow segment may
   be statically or dynamically configured.  Additionally the PREF DetNet-
   aware forwarder does duplicate frame elimination based on the flow
   identification and the sequence number combination.  The packet
   replication function is always one or
   more packets (i.e., 1:M replication).  The replicated packets MUST
   share also done within the DetNet-aware forwarder.  During
   elimination and the replication process the same DetNet control word sequence number.

   The complex part number of the
   DetNet PREF processing is tracking member flow MUST be preserved and copied to the
   history of received packets for multiple egress DetNet
   member flows.  These
   ingress flow.

                 +---------------------------------------+
                 |          DetNet member flows (to a node) MUST have Relay Node            |
       Ingress   +---------------------------------------+
                 |             |           |             |   Egress
                 |             o---------> o--+  Elim.   |
     ----------->o             |           |  +--------->o----------->
                 |             |   +-----> o--+          |
       Ingress   +-------------+   |       +-------------+
                 |             |   |       |             |   Egress
                 |             |   |       |             |
     ----------->o             o --+   +-> o             o----------->
                 |             |       |   |             |
       Ingress   +-------------+       |   +-------------+
                 |             |       |   |             |   Egress
                 |             | Repl. |   |             |
     ----------->o             o ------+-> o             o----------->
                 |             |           |             |
       Ingress   +-------------+           +-------------+
                 |             |           |             |   Egress
                 |             |           |             |
     ----------->o             o --------> o             o----------->
                 |             |           |             |
                 +---------------------------------------+

                  Figure 7: DetNet Relay Node processing

   Comment #35  SB> Somewhere in the same local-ID
   if they belong dp document there needs to be a
      note of the same DetNet-(compound)-flow and share the same
   sequence number requirement for interfaces to do fast exchange of
      counter state, and a note to those planning the history information.

   The edge network and relay node internal procedures of
      designing the PREF are
   implementation specific.  The order control plane that they need to provide support for
      this.

   Discussion:  We kinf of a packet elimination agree but also think the above exchange or
   replication is out
      synchronization of scope in this specification.  However, care
   should be taken that the replication function does counter states is not actually
   loopback packets as "replicas".  Looped back packets include
   artificial delay when the in our scope to solve.

5.5.3.  End system processing

   TBD.

5.6.  Transport node that originally initiated considerations

5.6.1.  Congestion protection

   TBD.

5.6.2.  Explicit routes

   TBD.

6.  IPv6-based DetNet data plane solution

6.1.  Data plane encapsulation

   Figure 8 illustrates a DetNet native IPv6 encapsulation.  The native
   IPv6 encapsulation is meant for end to end Detnet service use cases,
   where the packet
   receives end stations are DetNet-aware (see Figure 3).  Technically
   it again.  Also, looped back packets may make is possible to use the network
   condition IPv6 encapsulation to look healthier than it actually is (in some cases link
   failures are not reflected properly because looped back packets tunnel any traffic
   over a DetNet enabled network, which would make native IPv6
   encapsulation also a valid data plane choice for an interconnect use
   case (see Figure 1).

   The native IPv6-based DetNet data plane encapsulation consists of:

   o  IPv6 header as the transport protocol.

   o  IPv6 header Flow Label that is used to help to identify a DetNet
      flow (i.e., roughly an equivalent to an S-Label for the situation appear better than it actually is). MPLS
      encapsulation).  A Flow Label together with the IPv6 source
      address uniquely identifies a DetNet flow.

   Comment #29: #21  SB> There needs Have we validated that it is unconditionally safe to be some text
      make this assumption about preventing a node
      ever receiving its own replicated packets.  Indeed that would
      suggest that the use of the FL?

   Discussion:  RFC6437 does not restrict such use and DetNet DP
      solution can always define their own use of flow id label.  It should
      be changed and replication should
      only take place on configured flow IDs.  I have a feeling noted that
      this would all be a lot safer if DetNet aware node will always contain new code and
      is not a load balancer.

   o  Zero, one or two DetNet Destination Options containing sequencing
      information for packet replication only happened at
      ingress and we managed the diversity of duplicate elimination
      function (PREF), and/or packet reordering purposes.  The DetNet
      Destination Option is equivalent to the paths.

   Discussion:  Agree on hardening DetNet Control Word.  If
      PREF or packet reordering is not needed for the loopback text considerations.

6.1.2.  Edge node processing clarifications

   The DetNet data plane solution overloads flow then
      no DetNet Destination Option is inserted into the edge IPv6 header.

   A DetNet-aware end station (a host) or an intermediate Detnet node with
   initiating an (or adding a tunnelling) IPv6 packet is responsible for
   setting the Flow Label, adding the optional DetNet
   Edge Node functions.  Edge nodes are also aware Destination
   Option(s) for DetNet-s- or DetNet-st-flows, and possibly adding a
   routing header such as the segment routing option (e.g., for pre-
   defined paths [I-D.ietf-6man-segment-routing-header]).  If a routing
   header is inserted into the IPv6 packet for DetNet-s- or DetNet-st-
   flows then a second instance of the DetNet flows and
   may need to operate upon those.  Figure 9 illustrates Destination Option MUST be
   added before the overall
   edge device functions.  The figure shows both physical attachment
   circuit (AC) (e.g., Ethernet [RFC4448]) connecting routing header (see Section 4.1 of [RFC8200]).

   A DetNet-aware end station (a host) or an intermediate node receiving
   an IPv6 packet destined to the edge node, it and containing a packet service connecting to DetNet Destination
   Option does the edge node via an embedded
   router function (similarly as described e.g., in [RFC6658]).  Whether
   traffic flow from a client AC appropriate processing of the packet.  This may
   involve packet duplication and PSN elimination (PREF processing),
   terminating a tunnel receives DetNet specific
   treatment is up or delivering the packet to a local configuration and policy.

   Comment #30:  SB> Shouldn't the behaviour simply be a property of a
      given PW?

   Discussion:  Agree in principle.

                +---------------------------------------+
                |           DetNet Edge Device          |
                +---------------------------------------+   Egress/
                |             | Forwarder |             |   Ingress
                |             |           |    Single   | member Inst.
    Client PSN  |   "Packet   o <-X-----> o   Service   o<---------->
    tunnels     |    NSP"     |   | Repl. |   Instance  |
    <---------->o             |   | Elim. +-------------+ Duplicate
                |             |   :       |             |   Egress
                |             |   .       |    Single   | member Inst.
                |             |       +-> o   Service   o<---------->
                |             |       |   |   Instance  |
                +-------------+       |   +-------------+   Egress/ upper layers/
   Applications.

                    +---------------------------------+
                    |                                 |
                    |           DetNet Flow           |
                    |   Ingress
    Client AC             Payload             |    NSP
                    | Repl.                                 |
                    /---------------------------------\
                    H   Optional DetNet DstOpt Hdr    H
                    \---------------------------------/
                    |    Single          IPv6 header            | member Inst.
    <---------->o             o <-----X-> o   Service   o<---------->
                    |     (with set Flow label)       | Elim.
                    +---------------------------------+

   Figure 8: Encapsulation of a native IPv6 DetNet-s- or DetNet-st-flow
                         without a routing header

   Figure 9 illustrates an IPv6 packet for the case where a routing
   header has been added into the packet by a DetNet-aware end system
   (again assuming DetNet-s- or DetNet-st-flows).  Note that the use of
   routing header such as the one with the segment routing option is not
   mandatory for explicit routes.  Similar functionality can be arranged
   using other means as well (e.g., using policy routing or layer-2
   means).

                    +---------------------------------+
                    |   Instance                                 |
                +-------------+           +-------------+   Egress/
                    |           DetNet Flow           |
                    |             Payload             |   Ingress
    Client AC
                    |    NSP                                 |
                    /---------------------------------\
                    H        DetNet DstOpt Hdr        H
                    \---------------------------------/
                    |    Single          Routing header         | member Inst.
    <---------->o             o <-------> o   Service   o<---------->
                    /---------------------------------\
                    H        DetNet DstOpt Hdr        H
                    \---------------------------------/
                    |          IPv6 header            |
                    |   Instance     (with set Flow label)       |
                +---------------------------------------+
                    +---------------------------------+

   Figure 9: DetNet Edge Node processing

   An edge node participates to the packet replication and duplication
   elimination.  Required processing is done within an extended
   forwarder function.  In the case the Encapsulation of a native service processing (NSP)
   is IEEE 802.1CB [IEEE8021CB] capable, the packet replication and
   duplicate elimination MAY entirely be done in the NSP and bypassing
   the DetNet flow encapsulation and logic entirely, and thus is able to
   operate over unmodified implementation and deployment.  The NSP
   approach works IPv6 DetNet-s- or DetNet-st-flows
                            with routing header

   IPv6 extension headers can only between edge nodes and cannot make use of relay
   nodes (see Section 6.1.3).

   Comment #31  SB> This would be inserted by a fine way to operate node that initiated
   the PW system -
      edge to edge.

   Discussion:  When it comes to use of NSPs, agree.  Also IPv6 packet.  IPv6 extension headers, except for "island
      interconnect" this is a fine.  However, when there is a need to do
      PREF in a middle, plain edge to edge is not enough.

   The DetNet-aware extended forwarder selects the egress DetNet member
   flow based on the DetNet forwarding rules.  In both "normal AC" and
   "Packet AC" cases there may Hop-by-Hop
   Option headers, can only be no DetNet encapsulation header
   available yet as it processed by an IPv6 node that is
   identified by the case with relay nodes Destination Address field of the IPv6 header (see
   Section 6.1.3).

   It is the responsibility 0 of [RFC8200].  Therefore, if a DetNet-aware end system only
   inserted the extended forwarder within the edge
   node to push DetNet Destination Option into the IPv6 but e.g., a
   DetNet specific encapsulation (if not already
   present) Edge node is configured to enforce an explicit route for the
   IPv6 packet before forwarding using a source routing header, then it to the appropriate egress
   DetNet member flow instance(s).

   Comment #32  SB> I am not convinced of the wisdom has no other
   possibility than add an outer tunneling IPv6 header with required
   extension headers in it.  The processing of having IPv6 packets in a mid-
      point DetNet
   Edge node convert a is discussed further in Section 6.4.1.

6.2.  DetNet destination option

   A DetNet flow into must carry sequencing information for packet
   replication and elimination function (PREF) purposes.  This document
   specifies a DN flow, which new IPv6 Destination Option: the DetNet Destination
   Option for that purpose.  The format of the option is what you are
      implying here.  This seems like an ingress function.

   Discussion:  OK. illustrated in
   Figure 10.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     TBD1      |       4       |   0   |    28 bit sequence
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        number                       |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                   Figure 10: DetNet Destination Option

   The text here has issues and seems Option Type for the DetNet Destination Option is set to mix relay and
      edge.

   The extended forwarder MAY copy TBD1.
   [To be removed from the final version of the sequencing information from document: The Option
   Type MUST have the
   native DetNet two most significant bits set to 10b]

   If an IPv6 packet into gets dropped due the DetNet sequence number field and vice
   versa.  If there is no existing sequencing information available in Service layer
   processing based on the native DetNet Destination Option an ICMPv6 packet or the forwarder chose not of
   any type MUST NOT be sent back to copy it from the
   native packet, then source of the extended forwarder MUST maintain a sequence
   number counter for each packet.

6.3.  Flow identification

   The DetNet flow (indexed by identification is based on the DetNet flow
   identification).

6.1.3.  Relay node processing clarifications IPv6 Flow Label and
   the source address combination.  The two fields uniquelly identify
   the end to end native IPv6 encapsulated DetNet data plane solution overloads a relay node with DetNet
   Relay node functions.  Relay flow.  Obviously, the
   identification fails if any intermediate node is aware of DetNet flows and may
   operate upon those.  Figure 10 illustrates modifies either the overall DetNet relay
   device functions.
   source address or the Flow Label.

   Comment #33 #27  SB> I don't think See earlier.  If there are enough IPv6 addresses to
      address video fragments, why not DN flows?  Then this problem goes
      away.

   Discussion:  See the earlier comment #25 discussion.  If nodes get
      their addressies via DHCPv6 basically ruins this mechanism.  Also
      the assumption for this to work is that a relay the node in not has a normal
      construct so I am not sure "overload" full /64 to
      use, which is not always the right term here.

   Discussion:  Agree.  There case.  Otherwise the idea is a terminology issue here.

   A DetNet Relay node participates to just
      fine.

6.4.  Service layer considerations

   [Editor's note: this section is TBD.  It will detail the packet replication PREF
   functionality.]

   o  PREF - requires both flow identification and
   duplication elimination.  This sequence numbering.

   o  Packet reordeing - requires both flow identification and sequence
      numbering.

   A DetNet service layer processing is can be done within an extended
   forwarder function.  Whether an ingress at each DetNet member flow receives node
   that matches the IPv6 header's Destination Address.  Then, if the
   DetNet specific processing depends on how flow identification provides a positive match for the forwarding is
   programmed.  For some DetNet member flows
   flow that the relay node can act as has a
   normal relay node and service layer state installed e.g., for some apply the DetNet specific PREF
   or packet reordering purposes, further service layer processing
   (i.e., PREF).

   Comment #34  SB> Again relay node is not takes
   place.  In a normal term, so am not
      sure what it does in the absence case of a PREF function.

   Discussion:  Relay node was a or packet reordering that means processing
   the DetNet aware S-PE originally, which Destination Option for the identified DetNet flow.

6.4.1.  Edge node processing

   [Editor's note: This is
      not explicitly stated here anymore, thus slightly confusing the start of the IPv6 handling text
      here. - there
   are errors and bad language.  The text here needs to clarify founding assumption is the roles use of PREF and
      switching functions.  A DetNet relay
   source routing when intermediate nodes (relays/edges) need to modify
   packets.  This is described due the text in RFC8200 and the
      architecture document.  However, there fact that without
   hph options only routing+dsthdr is definitely room for
      termonilogy usable with intermediates under
   strict RFC8200.. ]

   [Editor's note: Regrading the source routing and the "example" SRv6
   approach.  Current text improvements.

   It is also possible based on the assumption that intermediates
   cannot add/delete extension headers such as the SRv6.  That said
   adding adding a header implies adding a tunneling outer IPv6 header
   and deleting a header implies a tunnel decapsulation.  This is not
   probably desired due to the involved overhead and to be discussed
   whether it is possible/acceptable to treat just "process" the relay node as Application
   flow packets.]

   For a transit node, see
   Section 7.3.  Again, this is entirely up DetNet Edge node there are several scenarios that involve
   modifications to how the forwarding has
   been programmed. DetNet flow IPv6 packets.  The assumption is
   that a DetNet-aware forwarder selects end system has always set the egress DetNet member IPv6 header flow
   segment based on
   label properly for the flow identification.  The mapping of identification purposes.  A DetNet- or
   DetNet-t-flow does not include the DetNet Destination Option.
   Following cases have been identified:

   1.  A DetNet App-flow or a DetNet-t-flow packet arrives at an ingress
       DetNet member Edge node and DetNet service layer functions are done only
       at DetNet Edge nodes.  Possible explicit routes between edge
       nodes are arranged by other than IPv6 specific means.

   2.  A DetNet App-flow or a DetNet-t-flow packet arrives at an ingress
       DetNet Edge node and multiple DetNet Relay nodes may process
       DetNet flow segment to packets before reaching an egress DetNet member flow segment may Edge node.
       Explicit routes between edge nodes has to be statically arranged by IPv6
       specific means.

   3.  A DetNet-s- or dynamically configured.  Additionally the DetNet-
   aware forwarder does duplicate frame elimination based on the flow
   identification and the sequence number combination.  The a DetNet-st-flow packet
   replication is also arrives at an ingress
       DetNet Edge node and DetNet service layer functions are done within the DetNet-aware forwarder.  During
   elimination only
       at DetNet Edge nodes.  Possible explicit routes between edge
       nodes are arranged by other than IPv6 specific means.

   4.  A DetNet-s- or a DetNet-st-flow packet arrives at an ingress
       DetNet Edge node and the replication multiple DetNet Relay nodes may process the sequence number of the
       DetNet member flow MUST be preserved and copied packets before reaching an egress DetNet Edge node.
       Explicit routes between edge nodes has to be arranged by IPv6
       specific means.

   A generic DetNet IPv6 encapsulation for a DetNet flow packet between
   DetNet Edge nodes is shown in Figure 11.  Essentially every time an
   igress DetNet Edge node has to insert something into the egress DetNet
   member flow.

                +---------------------------------------+
                | DetNet Relay Device          |
      Ingress   +---------------------------------------+
      member    |             | Forwarder flow
   packet it has to add an outer tunneling IPv6 header, which then
   contain possible additional extension headers.

        +---------------------------------+
        |                                 |   Egress
      instance
        |   Single           DetNet Flow           |
        |   Single             Payload             | member Inst.
    ----------->o  Service    o --X-----> o  Service    o----------->
        |  Instance                                 |
        +---------------------------------+
        | Elim. Optional DetNet DstOpt Hdr (1)  |  Instance
        +---------------------------------+
        |
      Ingress   +-------------+        Inner IPv6 header        |       +-------------+ Duplicate
      member
        |    (with set Flow label) (1)    |
        +---------------------------------+ <--+
        |   Optional  Routing header      |    |   Egress
      instance
        +---------------------------------+    |   Single
        | Optional DetNet DstOpt Hdr (2)  |    +-- Added/removed by
        +---------------------------------+    |   Single   DetNet Edge node
        | member Inst.
    ----------->o  Service    o --+   +-> o  Service    o----------->        Outer  IPv6 header       |  Instance    |
        |   (with set Flow label) (2)     |  Instance    |
        +---------------------------------+ <--+

    Figure 11: Encapsulation of a DetNet-flow IPv6 packet at the DetNet
                                 Edge node

6.4.1.1.  Ingress   +-------------+       |   +-------------+
      member    |             |       |   |             |   Egress
      instance  |   Single    | Repl. |   |   Single    | member Inst.
    ----------->o  Service    o ------X-> DetNet Edge node processing

   Case 1) MAY require an addition of the DetNet Destination Option if
   packet reordering is requested at the egress DetNet Edge node.
   Otherwise, no modifications except rewriting the IPv6 header flow
   label to the packet is done.  If modifications are required then:

   o  Service    o----------->
                |  Instance   |           |  Instance   |
      Ingress   +-------------+           +-------------+
      member    |             |           |             |   Egress
      instance  |   Single    |           |   Single    | member Inst.
    ----------->o  Service  The outer IPv6 header is added with the Source Address set to the
      ingress DetNet Edge node address and the Destination Address set
      to the egress DetNet Edge node address.

   o -------->  The flow label of the outer IPv6 header SHOULD be set to a value
      maintained by the edge node.

   o  Service    o----------->
                |  Instance   |           |  Instance   |
                +---------------------------------------+

                  Figure 10:  The DetNet Relay Node processing

   Comment #35  SB> Somewhere in Destination Option with the dp document there needs edge node managed per
      DetNet flow sequence number value is inserted into the outer IPv6
      header.

   Case 2) requires an addition of the DetNet Destination Option unless
   neither packet reordeing or PREF is enable at any DetNet Edge/Relay
   node.  A source routing header has to be a
      note added for the explicit route
   purposes.  An example of the requirement for interfaces source routing header is the Segment
   Routing header.  The following modifications to do fast exchange of
      counter state, and a note DetNet flow IPv6
   packets are required:

   o  An outer IPv6 header is added with the Source Address set to those planning the network
      ingress DetNet Edge node address and
      designing the control plane that they need Destination Address set
      to provide support for
      this.

   Discussion:  We kinf the egress DetNet Edge node address.

   o  The flow label of agree but also think the above exchange or
      synchronization outer IPv6 header SHOULD be set to a value
      maintained by the edge node.

   o  The DetNet Destination Option with the edge node managed per
      DetNet flow sequence number value MAY be inserted into the outer
      IPv6 header.

   o  A source routing header with addresses of counter states those DetNet Relay nodes
      that must be traversed is not in our scope to solve.

6.2.  Native IPv6-based data plane

   [Editor's inserted into the outer IPv6 header.

   Case 3) ...[Editor's note: this section is TBD.] it OK if the sequece number added here
   by the edge node has only local significance between the edge nodes
   and not end to end between end systems? ]

   Case 4) ...

6.4.1.2.  Engress DetNet Edge node processing

6.4.2.  Relay node processing

   TBD.

6.4.3.  End system processing

   TBD.

6.5.  Transport node processing

6.5.1.  Congestion protection
6.5.2.  Explicit routes

7.  Other DetNet data plane considerations

7.1.  Class of Service

   Class and quality of service, i.e., CoS and QoS, are terms that are
   often used interchangeably and confused.  In the context of DetNet,
   CoS is used to refer to mechanisms that provide traffic forwarding
   treatment based on aggregate group basis and QoS is used to refer to
   mechanisms that provide traffic forwarding treatment based on a
   specific DetNet flow basis.  Examples of existing network level CoS
   mechanisms include DiffServ which is enabled by IP header
   differentiated services code point (DSCP) field [RFC2474] and MPLS
   label traffic class field [RFC5462], and at Layer-2, by IEEE 802.1p
   priority code point (PCP).

   CoS for DetNet flows carried in PWs and MPLS is provided using the
   existing MPLS Differentiated Services (DiffServ) architecture
   [RFC3270].  Both E-LSP and L-LSP MPLS DiffServ modes MAY be used to
   support DetNet flows.  The Traffic Class field (formerly the EXP
   field) of an MPLS label follows the definition of [RFC5462] and
   [RFC3270].  The Uniform, Pipe, and Short Pipe DiffServ tunneling and
   TTL processing models are described in [RFC3270] and [RFC3443] and
   MAY be used for MPLS LSPs supporting DetNet flows.  MPLS ECN MAY also
   be used as defined in ECN [RFC5129] and updated by [RFC5462].

   CoS for DetNet flows carried in IPv6 is provided using the standard
   differentiated services code point (DSCP) field [RFC2474] and related
   mechanisms.  The 2-bit explicit congestion notification (ECN)
   [RFC3168] field MAY also be used.

   One additional consideration for DetNet nodes which support CoS
   services is that they MUST ensure that the CoS service classes do not
   impact the congestion protection and latency control mechanisms used
   to provide DetNet QoS.  This requirement is similar to requirement
   for MPLS LSRs to that CoS LSPs do not impact the resources allocated
   to TE LSPs via [RFC3473].

7.2.  Quality of Service

   Quality of Service (QoS) mechanisms for flow specific traffic
   treatment typically includes a guarantee/agreement for the service,
   and allocation of resources to support the service.  Example QoS
   mechanisms include discrete resource allocation, admission control,
   flow identification and isolation, and sometimes path control,
   traffic protection, shaping, policing and remarking.  Example
   protocols that support QoS control include Resource ReSerVation
   Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473].
   The existing MPLS mechanisms defined to support CoS [RFC3270] can
   also be used to reserve resources for specific traffic classes.

   In addition to path pinning explicit routes, and packet replication and
   elimination, described in Section 5 above, DetNet provides zero
   congestion loss and bounded latency and jitter.

   Comment #36  SB> I just searched from the beginning of the document
      and this was the the first reference I found to pinning.

   Discussion:  Terminology isssue.  Should use, for example, explicit
      paths which is used in the architecture I-D.  As described in
   [I-D.ietf-detnet-architecture], there are different mechanisms that
   maybe used separately or in combination to deliver a zero congestion
   loss service.  These mechanisms are provided by the either the MPLS
   or IP layers, and may be combined with the mechanisms defined by the
   underlying network layer such as 802.1TSN.

   A baseline set of QoS capabilities for DetNet flows carried in PWs
   and MPLS can provided by MPLS with Traffic Engineering (MPLS-TE)
   [RFC3209] and [RFC3473].  TE LSPs can also support explicit routes
   (path pinning).  Current service definitions for packet TE LSPs can
   be found in "Specification of the Controlled Load Quality of
   Service", [RFC2211], "Specification of Guaranteed Quality of
   Service", [RFC2212], and "Ethernet Traffic Parameters", [RFC6003].
   Additional service definitions are expected in future documents to
   support the full range of DetNet services.  In all cases, the
   existing label-based marking mechanisms defined for TE-LSPs and even
   E-LSPs are use to support the identification of flows requiring
   DetNet QoS.

   QoS for DetNet flows carried in IPv6 MUST be provided locally by the
   DetNet-aware hosts and routers supporting DetNet flows.  Such support
   will leverage the underlying network layer such as 802.1TSN.  The
   traffic control mechanisms used to deliver QoS for IP encapsulated
   DetNet flows are expected to be defined in a future document.  From
   an encapsulation perspective, and as defined in Section 5.2.2, 6, the
   combination of the Flow Label together with the IP source address
   uniquely identifies a DetNet flow.

   Packets that are marked with a DetNet Class of Service value, but
   that have not been the subject of a completed reservation, can
   disrupt the QoS offered to properly reserved DetNet flows by using
   resources allocated to the reserved flows.  Therefore, the network
   nodes of a DetNet network SHOULD:

   Comment #37  SB> Why not MUST?

   Discussion:  OK with MUST. MUST:

   o  Defend the DetNet QoS by discarding or remarking (to a non-DetNet
      CoS) packets received that are not the subject of a completed
      reservation.

   o  Not use a DetNet reserved resource, e.g. a queue or shaper
      reserved for DetNet flows, for any packet that does not carry a
      DetNet Class of Service marker.

7.3.  Cross-DetNet flow resource aggregation

   The ability to aggregate individual flows, and their associated
   resource control, into a larger aggregate is an important technique
   for improving scaling of control in the data, management and control
   planes.  This document identifies the traffic identification related
   aspects of aggregation of DetNet flows.  The resource control and
   management aspects of aggregation (including the queuing/shaping/
   policing implications) will be covered in other documents.  The data
   plane implications of aggregation are independent for PW/MPLS and IP
   encapsulated DetNet flows.

   DetNet flows transported via MPLS can leverage MPLS-TE's existing
   support for hierarchical LSPs (H-LSPs), see [RFC4206].  H-LSPs are
   typically used to aggregate control and resources, they may also be
   used to provide OAM or protection for the aggregated LSPs.  Arbitrary
   levels of aggregation naturally falls out of the definition for
   hierarchy and the MPLS label stack [RFC3032].  DetNet nodes which
   support aggregation (LSP hierarchy) map one or more LSPs (labels)
   into and from an H-LSP.  Both carried LSPs and H-LSPs may or may not
   use the TC field, i.e., L-LSPs or E-LSPs.  Such nodes will need to
   ensure that traffic from aggregated LSPs are placed (shaped/policed/
   enqueued) onto the H-LSPs in a fashion that ensures the required
   DetNet service is preserved.

   DetNet flows transported via IP have more limited aggregation
   options, due to the available traffic flow identification fields of
   the IP solution.  One available approach is to manage the resources
   associated with a DSCP identified traffic class and to map (remark)
   individually controlled DetNet flows onto that traffic class.  This
   approach also requires that nodes support aggregation ensure that
   traffic from aggregated LSPs are placed (shaped/policed/enqueued) in
   a fashion that ensures the required DetNet service is preserved.

   Comment #38  SB> I am sure we can do better than this with SR, or the
      use of routing techniques that map certain addresses to certain
      paths.

   Discussion:  --

   In both the MPLS and IP cases, additional details of the traffic
   control capabilities needed at a DetNet-aware node may be covered in
   the new service descriptions mentioned above or in separate future
   documents.  Management and control plane mechanisms will also need to
   ensure that the service required on the aggregate flow (H-LSP or
   DSCP) are provided, which may include the discarding or remarking
   mentioned in the previous sections.

7.4.  Bidirectional traffic

   Some DetNet applications generate bidirectional traffic.  Using MPLS
   definitions [RFC5654] there are associated bidirectional flows, and
   co-routed bidirectional flows.  MPLS defines a point-to-point
   associated bidirectional LSP as consisting of two unidirectional
   point-to-point LSPs, one from A to B and the other from B to A, which
   are regarded as providing a single logical bidirectional transport
   path.  This would be analogous of standard IP routing, or PWs running
   over two reciprocal unidirection LSPs.  MPLS defines a point-to-point
   co-routed bidirectional LSP as an associated bidirectional LSP which
   satisfies the additional constraint that its two unidirectional
   component LSPs follow the same path (in terms of both nodes and
   links) in both directions.  An important property of co-routed
   bidirectional LSPs is that their unidirectional component LSPs share
   fate.  In both types of bidirectional LSPs, resource allocations may
   differ in each direction.  The concepts of associated bidirectional
   flows and co-routed bidirectional flows can be applied to DetNet
   flows as well whether IPv6 or MPLS is used.

   While the IPv6 and MPLS data planes must support bidirectional DetNet
   flows, there are no special bidirectional features with respect to
   the data plane other than need for the two directions take the same
   paths.  Fate sharing and associated vs co-routed bidirectional flows
   can be managed at the control level.  Note, that there is no stated
   requirement for bidirectional DetNet flows to be supported using the
   same IPv6 Flow Labels or MPLS Labels in each direction.  Control
   mechanisms will need to support such bidirectional flows for both
   IPv6 and MPLS, but such mechanisms are out of scope of this document.
   An example control plane solution for MPLS can be found in [RFC7551].

7.5.  Layer 2 addressing and QoS Considerations

   The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1
   Working Group have defined (and are defining) a number of amendments
   to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and
   bounded latency in bridged networks.  IEEE 802.1CB [IEEE8021CB]
   defines packet replication and elimination functions that should
   prove both compatible with and useful to, DetNet networks.

   As is the case for DetNet, a Layer 2 network node such as a bridge
   may need to identify the specific DetNet flow to which a packet
   belongs in order to provide the TSN/DetNet QoS for that packet.  It
   also will likely need a CoS marking, such as the priority field of an
   IEEE Std 802.1Q VLAN tag, to give the packet proper service.

   Although the flow identification methods described in IEEE 802.1CB
   [IEEE8021CB] are flexible, and in fact, include IP 5-tuple
   identification methods, the baseline TSN standards assume that every
   Ethernet frame belonging to a TSN stream (i.e.  DetNet flow) carries
   a multicast destination MAC address that is unique to that flow
   within the bridged network over which it is carried.  Furthermore,
   IEEE 802.1CB [IEEE8021CB] describes three methods by which a packet
   sequence number can be encoded in an Ethernet frame.

   Ensuring that the proper Ethernet VLAN tag priority and destination
   MAC address are used on a DetNet/TSN packet may require further
   clarification of the customary L2/L3 transformations carried out by
   routers and edge label switches.  Edge nodes may also have to move
   sequence number fields among Layer 2, PW, and IPv6 encapsulations.

7.6.  Interworking between PW- MPLS- and IPv6-based encapsulations

   [Editor's note: add considerations for interworking between PW-based MPLS-
   based and native IPv6-based DetNet encapsuations.]

7.7.  IPv4 considerations

   [Editor's note: The fact is that there are and will be deployments
   using IPv4.  Neglecting it entirely is not feasible.]

8.  Time synchronization

   Comment #39  SB> This section should point the reader to RFC8169
      (residence time in MPLS n/w.  We need to consider if we need to
      introduce the same concept in IP.

   Discussion:  agree.  Agree.  For IP we could reference to PTPv2 or v3 over
      UDP/IP, since it measures residence time among other things.

   [Editor's note: describe a bit of issues and deployment
   considerations related to time-synchronization within DetNet.  Refer
   to DT discussion and the slides that summarize different approaches
   and rough synchronization performance numbers.  Finally, scope time-
   synchronization solution outside data plane.]

   When DetNet is used, there is an underlying assumption that the
   applicaton(s) require clock synchronization such as the Precision
   Time Protocol (PTP) [IEEE1588].  The relay nodes may or may not
   utilize clock synchronization in order to provide zero congestion
   loss and controlled latency delivery.  In either case, there are a
   few possible approaches of how synchronization protocol packets are
   forwarded and handled by the network:

   o  PTP packets can be sent either as DetNet flows or as high-priority
      best effort packets.  Using DetNet for PTP packets requires
      careful consideration to prevent unwanted interactions between
      clock-synchronized network nodes and the packets that synchronize
      the clocks.

   o  PTP packets are sent as a normal DetNet flow through network nodes
      that are not time-synchronized: in this approach PTP traffic is
      forwarded as a DetNet flow, and as such it is forwarded in a way
      that allows a low delay variation.  However, since intermediate
      nodes do not take part in the synchronization protocol, this
      approach provides a relatively low degree of accuracy.

   o  PTP with on-path support: in this approach PTP packets are sent as
      ordinary or as DetNet flows, and intermediate nodes take part in
      the protocol as Transparent Clocks or Boundary Clocks [IEEE1588].
      The on-path PTP support by intermediate nodes provides a higher
      degree of accuracy than the previous approach.  The actual
      accuracy depends on whether all intermediate nodes are PTP-
      capable, or only a subset of them.

   o  Time-as-a-service: in this approach accurate time is provided as-
      a-service to the DetNet source and destination, as well as the
      intermediate nodes.  Since traffic between the source and
      destination is sent over a provider network, if the provider
      supports time-as-a-service, then accurate time can be provided to
      both the source and the destination of DetNet traffic.  This
      approach can potentially provide the highest degree of accuracy.

   It is expected that the latter approach will be the most common one,
   as it provides the highest degree of accuracy, and creates a layer
   separation between the DetNet data and the synchronization service.

   It should be noted that in all four approaches it is not recommended
   to use replication and elimination for synchronization packets; the
   replication/elimination approach may in some cases reduce the
   synchronization accuracy, since the observed path delay will be
   bivalent.

   Comment #40  SB> I am not sure why we sould not use PREP.  We should
      explain to the reader.

   Discussion:  Agree that a this can be opened a bit more in detail.
      The issue is explained briefly in the last sentence but it could
      be more clear.

9.  Management and control considerations

   [Editor's note: This section needs to be different for MPLS and IPv6
   solutions.  Most solutions are technology dependant,]

   While management plane and control planes are traditionally
   considered separately, from the Data Plane perspective there is no
   practical difference based on the origin of flow provisioning
   information.  This document therefore does not distinguish between
   information provided by a control plane protocol, e.g., RSVP-TE
   [RFC3209] and [RFC3473], or by a network management mechanisms, e.g.,
   RestConf [RFC8040] and YANG [RFC7950].

   [Editor's note: This section is a work in progress.  discuss here
   what kind of enhancements are needed for DetNet and specifically for
   PREF and DetNet zero congest loss and latency control.  Need to cover
   both traffic control (queuing) and connection control (control
   plane).]

9.1.  PW Label and IPv6 Flow Label  MPLS-based data plane

9.1.1.  S-Label assignment and distribution

   [Editor's note: Outdated and MPLS specific.. and needs more work.]

   The PW label DetNet S-Label distribution follows the same mechanisms specified
   for
   MS-PW [RFC6073]. XYZ . The details of the control plane protocol solution required
   for the label distribution and the management of the label number
   space are out of scope of this document.

9.1.2.  Explicit routes

   [Editor's note: Outdated.. and needs more work.]

   [TBD: based on MPLS TE and possibly IPv6 SR]

9.2.  IPv6-based data plane

9.2.1.  Flow Label assignment and distribution

   [Editor's note: Outdated and IPv6 Specific.. and needs more work.]

   The IPv6 Flow Label distribution and the label number space are out
   of scope of this document.  However, it should be noted that the
   combination of the IPv6 source address and the IPv6 Flow Label is
   assumed to be unique within the DetNet-enabled network.  Therefore,
   as long as each node is able to assign unique Flow Labels for the
   source address(es) it is using the DetNet-enabled network wide flow
   identification uniqueness is guaranteed.

9.2.

9.2.2.  Explicit routes

   [Editor's note: Outdated.. and needs more work.]

   [TBD: What we have there for IPv6 and explicit routes]

9.3.  Packet replication and elimination

   [Editor's note: Outdated and at the functional level technology
   independent.. but needs more work.]

   The control plane protocol solution required for managing the PREF
   processing is outside the scope of this document.

9.3.  Explicit paths

   [TBD: based on MPLS TE and SR.]

9.4.  Congestion protection and latency control

   [TBD]

9.5.  Flow aggregation control

   [TBD]

10.  Security considerations

   The security considerations of DetNet in general are discussed in
   [I-D.ietf-detnet-architecture] and [I-D.sdt-detnet-security].  Other
   security considerations will be added in a future version of this
   draft.

11.  IANA considerations

   TBD.

12.  Acknowledgements

   The author(s) ACK and NACK.

   The following people were part of the DetNet Data Plane Solution
   Design Team:

      Jouni Korhonen

      Janos Farkas

      Norman Finn
      Balazs Varga

      Loa Andersson

      Tal Mizrahi

      David Mozes

      Yuanlong Jiang

      Carlos J.  Bernardos

   The DetNet chairs serving during the DetNet Data Plane Solution
   Design Team:

      Lou Berger

      Pat Thaler

13.  References

13.1.  Normative references

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC2211]  Wroclawski, J., "Specification of the Controlled-Load
              Network Element Service", RFC 2211, DOI 10.17487/RFC2211,
              September 1997, <https://www.rfc-editor.org/info/rfc2211>.

   [RFC2212]  Shenker, S., Partridge, C., and R. Guerin, "Specification
              of Guaranteed Quality of Service", RFC 2212,
              DOI 10.17487/RFC2212, September 1997,
              <https://www.rfc-editor.org/info/rfc2212>.

   [RFC2474]  Nichols, K., Blake, S., Baker, F., and D. Black,
              "Definition of the Differentiated Services Field (DS
              Field) in the IPv4 and IPv6 Headers", RFC 2474,
              DOI 10.17487/RFC2474, December 1998,
              <https://www.rfc-editor.org/info/rfc2474>.

   [RFC3032]  Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y.,
              Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack
              Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001,
              <https://www.rfc-editor.org/info/rfc3032>.

   [RFC3168]  Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
              of Explicit Congestion Notification (ECN) to IP",
              RFC 3168, DOI 10.17487/RFC3168, September 2001,
              <https://www.rfc-editor.org/info/rfc3168>.

   [RFC3209]  Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
              and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
              Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001,
              <https://www.rfc-editor.org/info/rfc3209>.

   [RFC3270]  Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen,
              P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-
              Protocol Label Switching (MPLS) Support of Differentiated
              Services", RFC 3270, DOI 10.17487/RFC3270, May 2002,
              <https://www.rfc-editor.org/info/rfc3270>.

   [RFC3443]  Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing
              in Multi-Protocol Label Switching (MPLS) Networks",
              RFC 3443, DOI 10.17487/RFC3443, January 2003,
              <https://www.rfc-editor.org/info/rfc3443>.

   [RFC3473]  Berger, L., Ed., "Generalized Multi-Protocol Label
              Switching (GMPLS) Signaling Resource ReserVation Protocol-
              Traffic Engineering (RSVP-TE) Extensions", RFC 3473,
              DOI 10.17487/RFC3473, January 2003,
              <https://www.rfc-editor.org/info/rfc3473>.

   [RFC4206]  Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP)
              Hierarchy with Generalized Multi-Protocol Label Switching
              (GMPLS) Traffic Engineering (TE)", RFC 4206,
              DOI 10.17487/RFC4206, October 2005,
              <https://www.rfc-editor.org/info/rfc4206>.

   [RFC4448]

   [RFC4385]  Bryant, S., Swallow, G., Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron,
              "Encapsulation Methods D. McPherson,
              "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for Transport of Ethernet
              Use over an MPLS
              Networks", PSN", RFC 4448, 4385, DOI 10.17487/RFC4448, April 10.17487/RFC4385,
              February 2006,
              <https://www.rfc-editor.org/info/rfc4448>. <https://www.rfc-editor.org/info/rfc4385>.

   [RFC5129]  Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion
              Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January
              2008, <https://www.rfc-editor.org/info/rfc5129>.

   [RFC5462]  Andersson, L. and R. Asati, "Multiprotocol Label Switching
              (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic
              Class" Field", RFC 5462, DOI 10.17487/RFC5462, February
              2009, <https://www.rfc-editor.org/info/rfc5462>.

   [RFC6003]  Papadimitriou, D., "Ethernet Traffic Parameters",
              RFC 6003, DOI 10.17487/RFC6003, October 2010,
              <https://www.rfc-editor.org/info/rfc6003>.

   [RFC6073]  Martini, L., Metz, C., Nadeau, T., Bocci, M., and M.
              Aissaoui, "Segmented Pseudowire", RFC 6073,
              DOI 10.17487/RFC6073, January 2011,
              <https://www.rfc-editor.org/info/rfc6073>.

   [RFC6658]  Bryant, S., Ed., Martini, L., Swallow, G.,

   [RFC8200]  Deering, S. and A. Malis,
              "Packet Pseudowire Encapsulation over an MPLS PSN", R. Hinden, "Internet Protocol, Version 6
              (IPv6) Specification", STD 86, RFC 6658, 8200,
              DOI 10.17487/RFC6658, 10.17487/RFC8200, July 2012,
              <https://www.rfc-editor.org/info/rfc6658>. 2017,
              <https://www.rfc-editor.org/info/rfc8200>.

13.2.  Informative references

   [I-D.ietf-6man-segment-routing-header]
              Previdi, S., Filsfils, C., Raza, K., Dukes, D., Leddy, J.,
              Field, B., daniel.voyer@bell.ca, d.,
              daniel.bernier@bell.ca, d., Matsushima, S., Leung, I.,
              Linkova, J., Aries, E., Kosugi, T., Vyncke, E., Lebrun,
              D., Steinberg, D., and R. Raszuk, "IPv6 Segment Routing
              Header (SRH)", draft-ietf-6man-
              segment-routing-header-07 draft-ietf-6man-segment-routing-header-08
              (work in progress), July 2017. January 2018.

   [I-D.ietf-detnet-architecture]
              Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", draft-ietf-
              detnet-architecture-03
              detnet-architecture-04 (work in progress), August October 2017.

   [I-D.ietf-detnet-dp-alt]
              Korhonen, J., Farkas, J., Mirsky, G., Thubert, P.,
              Zhuangyan, Z., and L. Berger, "DetNet Data Plane Protocol
              and Solution Alternatives", draft-ietf-detnet-dp-alt-00
              (work in progress), October 2016.

   [I-D.sdt-detnet-security]
              Mizrahi, T., Grossman, E., Hacker, A., Das, S.,
              "Deterministic Networking (DetNet) Security
              Considerations, draft-sdt-detnet-security, work in
              progress", 2017.

   [IEEE1588]
              IEEE, "IEEE 1588 Standard for a Precision Clock
              Synchronization Protocol for Networked Measurement and
              Control Systems Version 2", 2008.

   [IEEE8021CB]
              Finn, N., "Draft Standard for Local and metropolitan area
              networks - Seamless Redundancy", IEEE P802.1CB
              /D2.1 P802.1CB, December 2015,
              <http://www.ieee802.org/1/files/private/cb-drafts/
              d2/802-1CB-d2-1.pdf>.

   [IEEE8021Q]
              IEEE 802.1, "Standard for Local and metropolitan area
              networks--Bridges and Bridged Networks (IEEE Std 802.1Q-
              2014)", 2014, <http://standards.ieee.org/about/get/>.

   [RFC2205]  Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S.
              Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1
              Functional Specification", RFC 2205, DOI 10.17487/RFC2205,
              September 1997, <https://www.rfc-editor.org/info/rfc2205>.

   [RFC3985]  Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
              Edge-to-Edge (PWE3) Architecture", RFC 3985,
              DOI 10.17487/RFC3985, March 2005,
              <https://www.rfc-editor.org/info/rfc3985>.

   [RFC4090]  Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast
              Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090,
              DOI 10.17487/RFC4090, May 2005,
              <https://www.rfc-editor.org/info/rfc4090>.

   [RFC5036]  Andersson, L., Ed., Minei, I., Ed., and B. Thomas, Ed.,
              "LDP Specification", RFC 5036, DOI 10.17487/RFC5036,
              October 2007, <https://www.rfc-editor.org/info/rfc5036>.

   [RFC5654]  Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed.,
              Sprecher, N., and S. Ueno, "Requirements of an MPLS
              Transport Profile", RFC 5654, DOI 10.17487/RFC5654,
              September 2009, <https://www.rfc-editor.org/info/rfc5654>.

   [RFC7551]  Zhang, F., Ed., Jing, R., and R. Gandhi, Ed., "RSVP-TE
              Extensions for Associated Bidirectional Label Switched
              Paths (LSPs)", RFC 7551, DOI 10.17487/RFC7551, May 2015,
              <https://www.rfc-editor.org/info/rfc7551>.

   [RFC7950]  Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language",
              RFC 7950, DOI 10.17487/RFC7950, August 2016,
              <https://www.rfc-editor.org/info/rfc7950>.

   [RFC8040]  Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF
              Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017,
              <https://www.rfc-editor.org/info/rfc8040>.

Appendix A.  Example of DetNet data plane operation

   [Editor's note: Add a simplified example of DetNet data plane and how
   labels etc work in the case of MPLS-based PSN and utilizing PREF.
   The figure is subject to change depending on the further DT decisions
   on the label handling..]

Appendix B.  Example of pinned paths using IPv6

   TBD.

Authors' Addresses

   Jouni Korhonen (editor)
   Nordic Semiconductor

   Email: jouni.nospam@gmail.com

   Loa Andersson
   Huawei

   Email: loa@pi.nu

   Yuanlong Jiang
   Huawei

   Email: jiangyuanlong@huawei.com

   Norman Finn
   Huawei
   3101 Rio Way
   Spring Valley, CA  91977
   USA

   Email: norman.finn@mail01.huawei.com

   Balazs Varga
   Ericsson
   Konyves Kalman krt. 11/B
   Budapest  1097
   Hungary

   Email: balazs.a.varga@ericsson.com
   Janos Farkas
   Ericsson
   Konyves Kalman krt. 11/B
   Budapest  1097
   Hungary

   Email: janos.farkas@ericsson.com

   Carlos J. Bernardos
   Universidad Carlos III de Madrid
   Av. Universidad, 30
   Leganes, Madrid  28911
   Spain

   Phone: +34 91624 6236
   Email: cjbc@it.uc3m.es
   URI:   http://www.it.uc3m.es/cjbc/

   Tal Mizrahi
   Marvell
   6 Hamada st.
   Yokneam
   Israel

   Email: talmi@marvell.com

   Lou Berger
   LabN Consulting, L.L.C.

   Email: lberger@labn.net