DetNet Data Plane solution

DetNet Data Plane solution Broadcom

3151 Zanker Road San Jose 95134 CA USA jouni.nospam@gmail.com

Huawei

loa@pi.nu

Huawei

jiangyuanlong@huawei.com

Ericsson

Konyves Kálmán krt. 11/B Budapest Hungary 1097 balazs.a.varga@ericsson.com

Ericsson

Konyves Kálmán krt. 11/B Budapest Hungary 1097 janos.farkas@ericsson.com

Universidad Carlos III de Madrid

Av. Universidad, 30 Leganes, Madrid 28911 Spain +34 91624 6236 cjbc@it.uc3m.es http://www.it.uc3m.es/cjbc/

Marvell

6 Hamada st. Yokneam Israel talmi@marvell.com

DetNet This document specifies a PseudoWire-based Deterministic Networking data plane solution. The data plane solution can be applied over either IP or MPLS Packet Switched Networks.

This document specifies a Deterministic Networking (DetNet) data plane solution. The solution is based on PseudoWires (PW) and makes use of the multi-segment (MS-PW) to map DetNet Relay and Edge Nodes to PW architecture. The PW-based data plane can be run over either an IP or MPLS Packet Switched Network (PSN). For the purpose of DetNet data plane, this document specifically specifies the PW encapsulation for DetNet flows, a DetNet Control Word (CW), a DetNet label, how MS-PW derived DetNet Relay and Edge nodes work, and as a specific new PW feature how the Packet Replication and Elimination function (PREF) is implemented using PWs. This document does not define the associated control plane functions, or operations and management (OAM).

This document uses the terminology established in the DetNet architecture and the DetNet Data Plane Solution Alternatives . The following terms are also used in this document: A DetNet aware PseudoWire Terminating Provider Edge (T-PE). A DetNet aware PseudoWire Switching Provider Edge (S-PE). A hop-by-hop tunnel label layer between label switching routers (LSR). A DetNet topology overlay label that is used between DA-*-PE devices. A DetNet flow identity that uniquely identifies a DetNet flow in a DetNet network. The flow-ID is part of the PseudoWire Encapsulation header. A DA-T-PE and DA-S-PE node internal construct that uniquely identifies a DetNet flow. The local-ID can be equal to flow-ID or be derived using other means, e.g., programming required label to local-ID mappings directly into the label information base (LFIB). A PseudoWire label that is used to identify DetNet flow related PW Instances within a PE node. A Packet Replication and Elimination Function (PREF) does the replication and elimination processig of DetNet flow packets in DA-T-PE or DA-S-PE nodes. The replication function is essentially the existing 1+1 protection mechanism. The elimination function reuses and extends the existing PseudoWire sequencing provided duplicate detection mechanism to operate over multiple (separate) PseudoWires that are sub-flows of a compound DetNet flow.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL" "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

[Ed. to be written.. describe the scope here fot this document: this document only addresses the inter-connect case i.e., 802.1 over routed network (enlarge the layer-2 domain - EVPAN', and the native DetNet case.]

| Appl. | +---------+ +---------+ +---------+ +---------+ | TSN | |TSN| |Svc|<-- DetNet flow ---: Service :-->| Service | +---------+ +---+ +---+ +---------+ +---------+ +---------+ |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| +-------.-+ +-.-+ +-.-+ +--.----.-+ +-.-+ +-.-+ +---.-----+ : Link : / ,-----. \ : Link : / ,-----. \ +........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ [Network] [Network] `-----' `-----' ]]> illustrates how DetNet can provide services for IEEE 802.1TSN end systems over a DetNet enabled network. The edge nodes insert and remove required DetNet data plane encapsulation. The 'X' in the edge and relay nodes represents a potential DetNet flow packet replication and elimination point. This conceptually parallels L2VPN services, and could leverage existing related solutions as discussed below.

| TSN Service | Transit Transit | Service TSN (AC) | |<-Tunnel->| |<-Tnl->| | (AC) TSN End | V V 1 V V 2 V V | End System | +--------+ +--------+ +--------+ | System +---+ | |DA-T-PE1|==========|DA-S-PE1|=======|DA-T-PE2| | +---+ | |--|----|._X_....|..DetNet..|.._ _...|..DF3..|...._X_.|---|---| | |CE1| | | \ | Flow 1 | X | | / | | |CE2| | | | \_.|...DF2....|._/ \_..|..DF4..|._/ | | | +---+ | |==========| |=======| | +---+ ^ +--------+ +--------+ +--------+ ^ | Edge Node Relay Node Edge Node | | | |<----- Emulated Time Sensitive Networking (TSN) Service ---->| ]]> illustrates how end to end DetNet service can be provided. In this case, the end systems are able to send and receive DetNet flows. For example, put application data in PseudoWire (PW) and encapsulated in IP. Like earlier the 'X' in the end systems, edge and relay nodes represents potential DetNet flow packet replication and elimination points. Here the relay nodes may change the underlying transport, for example replacing IP with MPLS or tunneling IP over MPLS, or simply interconnect network segments.

| |<-Tnl->| | DetNet End | V 1 V V 2 V | End System | +--------+ +--------+ +--------+ | System +---+ | |DA-S-PE1|=======|DA-S-PE2|=======|DA-S-PE3| | +---+ | X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X | |CE1|========| \ | | X | | / |======|CE2| | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | +---+ | |=======| |=======| | +---+ ^ +--------+ +--------+ +--------+ ^ | Relay Node Relay Node Relay Node | | | |<-- End to End Time Sensitive Networking (TSN) Service -->| ]]>

Two major groups of scenarios can be distinguished which require flow identification during transport: DetNet function related scenarios: Congestion protection: usage of allocated resources (queuing, policing, shaping). Explicit routes: select/apply the flow specific path. Service protection: recognize compound / member flows for replication an elimination. OAM function related scenarios: troubleshooting (e.g., identify misbehaving flows, etc.) recognize flow(s) for analytics (e.g, increase counters, etc.) correlate events with flows (e.g., volume above threshold, etc.) etc. Each node (DA-T-PE, DA-S-PE and P) use a local-ID of the DetNet-(compound)-flow in order to accomplish its role during transport. Recognizing the flow-ID is more relaxed for DA-T-PE and DA-S-PE nodes, as they are fully aware of both the DetNet service and transport layers. The DetNet role of intermediate "P" nodes is limited to ensure congestion protection from the above listed DetNet functions. However, P nodes can usually recognize only "T-label" and cannot consider the whole label stack for flow recognition. Therefore, identifying each individual DetNet flow on a P node may not be achieved in some network scenarios. On each node dealing with DetNet flows, a local-ID is assumed to determine what local operation a packet goes through. Therefore, local-IDs MUST be unique on each DA-T-PE and DA-S-PE nodes. Local-ID MUST be unambiguously bound to the Flow-ID encoded in the DetNet packet.

The DetNet control word (d-CW) is identical to the control word defined for Ethernet over MPLS networks in . The DetNet control word is illustrated in .

[Editor's note: Shoudl we care about high speed links, here 16 bits of sequence number wraps fast? For example, in a case of 100Gb/s link, 16 bits of sequence number will wrap in ~6.6ms assuming 1250 octets of packets and ~3.3ms for 625 octets packets. Both numbers mean quite long fiber distances, though.]

The DetNet flow identity word (flow-ID) identifies a DetNet flow uniquely within a DetNet network. The flow-ID is also associated with the sequence number carried in the DetNet control word and used also for PREF purposes.

The management and assignment of the flow-IDs is outside of this specification. It is assumed that each DA-T-PE node have either a preconfigured flow-ID number space or some dynamic control plane protocol is able to coordinate the allocation of the flow-IDs across the DA-T-PE nodes, or a central entity, e.g., a Software Defined Networking controller. The D bit defines the direction of the flow. The value 0 means 'east' and the value 1 means 'west'. The D bit can be used in ring topologies to allow DetNet flows with the same flow-ID cross with or without PREF processing taking place. Whether a DA-*-PE checks for the D bit is based on a local policy setting. The support for D bit is optional. If a DA-*-PE does not support the D bit it MUST be treated as 0. The reserved field in the flow-ID MUST be set to zero and ignored when received. [Editor's note: we need some configuration knob defined for this feature.]

The DetNet data plane follows PW encapsulation. This document specifies a single encapsulation that can be used over both MPLS and IP packet switched Networks (PSN). The DetNet data plane encapsulation consists of a DetNet control word (d-CW): contains sequencing information for packet replication and duplicate elimination purposes. There is a separate sequence number space per each DetNet label. DetNet flow-ID (f-ID): uniquely identifies a DetNet flow within a DetNet network. Multiple DetNet PWs with different PW labels may have the same f-ID, which then implies the PWs are actually subflows of one compound flow. PseudoWire Label (PW Label;): a standard PW label that identifies a PW Instance within a (DA-)T-PE or (DA-)S-PE device. DetNet topology overlay label (L-label): an optional label used between (DA-)T-PE or (DA-)S-PE nodes. The main use of L-labels is to tunnel PWs through a PE node and therefore effectively making a PE node to behave like a P node. In a case of MPLS-based PSN, the tunnel labels between LSRs are referred as T-labels. The DetNet CW and the Detnet flow-ID together constitute the DetNet PseudoWire encapsulation header. [Editor's note: The current design has the DetNet flow-ID as part of the every DetNet flow packet. The flow-ID identifies the flow uniquely within the DetNet network and together with the sequence number information from the DetNet control word is used for PREF purposes. The flow-ID makes is easy for the DA-*-PE node to associate different PWs into one compound flow and perform the elimination of duplicate packets. The flow-ID would point at the node internal construct that holds the received packet history for each DetNet flow of interest. However, it could also be possible to associate multiple PWs into one DetNet flow just using the control plane provided information. In this case different PWs (using any PW label) would be mapped internally within a node to a local-ID (or similar construct), which again points at the internal per DetNet flow received packets history construct. The explicit in-band flow-ID is easy from the processing and control plane point of view. The local-ID approach does not need the in-band information (thus has less overhead) but requires more from the control plane and the mapping information has to be stored into the LFIB. Current design decision is the in-band flow-ID but may be changed to local-ID if there is a strong reason to do the change.] illustrates a DetNet PseudoWire encapsulation using an MPLS PSN. Similarly, illustrates the DetNet PseudoWire encapsulation when IP PSN is used. The encapsulation is uniform above the PSN. Depending on the network topology the "overlay label" (L-label) may be part of the label stack. The L-label tunnels guarantee PW labels remain unchanged between DA-*-PE nodes. Furthermore, L-labels tunnels allow selectively exposing the PW label to DA-*-PE nodes, which means some overlay topologies may just pass through specific DA-S-PEs without any DetNet specific processing.

H---------------------------------H H Sequencing H--' H DetNet Control Word H \=====================/ \=================================/ | PW Demultiplexer |--------->| PW Label | +---------------------+ +---------------------------------+ | PSN Convergence | .--->| Optional Topology overlay Label | +---------------------+ | +---------------------------------+ | PSN |-----+--->| MPLS T-Label(s) | +---------------------+ +---------------------------------+ | Data-Link | +---------------------+ | Physical | +---------------------+ ]]> When IP PSN is used, the label stack it transports is only inspected when the IP packet destination address equals to the IP address of a DA-*-PE or a P node. Essentially there are one more IP tunnels between a number of DA-*-PE and/or P nodes. The LFIB and the forwarding information base (FIB) combination determines whether a PW gets terminated at the node or forwarded to another node within a new IP tunnel.

[Editor's note: The Detnet-aware "extended forwarder" does the heavy lifting on maintaining the sequence numbers associated with the DetNet labels. Extended forwarder is also responsible for packet replication and duplicate elimination. See the excerpt from RFC3985 Section 4.2.1. about forwarder's functions. We extend that to PREF: Some applications have to forward payload elements selectively from one or more ACs to one or more PWs. In such cases, there will also be a need to perform the inverse function on PWE3-PDUs received by a PE from the PSN. This is the function of the forwarder. ] The DetNet specific new functionality in a DA-*-PE PW processing is the packet replication and duplication elimination function (PREF). This functional is a part of the "extended" forwarder. The PREF processing is triggered by the LFIB actions i.e., not all PWs receive DetNet specific processing. Basically the LFIB has to be extended with a "PREF enabled" boolean configuration switch that is associated with the normal label actions (e.g., swap, push, pop, ..). The output of the PREF elimination function is always a single packet. The output of the PREF replication function is always one or more packet (i.e., 1:M replication). The replicated packets MUST share the same DetNet PW control word sequence number and flow identity word flow-id. The complex part of the DetNet PREF processing is tracking the history of received packets for multiple PWs. These PWs do not have the same PW label value while they still share the same PW sequence number counter and the history information. That is where the DetNet encapsulation header flow-ID plays an important role and binds the control word sequence number to the flow specific shared counter and history information within the PREF function. The DetNet flow word contains a D flag bit (see ), which makes the DA-*-PE node aware of the direction the flow-ID arrived from. If the node, based on the local policy, checks for the D bit setting that effectively means the sequence number history has to contain also the D bit information. [Editor's note: draw here an example of LFIB with the elimination action.]

The PW-based DetNet data plane solution overloads the T-PE with a DetNet Edge Node function. Such T-PE is referred as DA-T-PE and implies the T-PE is also aware of DetNet flows and may need to operate upon those. illustrates the overall DA-T-PE device functions. The figure shows both physical attachment circuit (AC) (e.g., Ethernet ) connecting to the PE, and a packet service connecting to the PE via an embedded label switching router (LSR) function . Whether traffic flow from from a client AC and PSN LSP receives DetNet specific treatment is up to a local configuration and policy. A DA-T-PE can also serve as a normal T-PE.

o PW Instance o<----------> LSPs | NSP") | | Repl. | | <---------->o | | Elim. +-------------+ Duplicate | | : | | Egress | | . | Single | PW Instance | | +-> o PW Instance o<----------> | | | | | +-------------+ | +-------------+ Egress/ | | | | | Ingress Client AC | NSP | Repl. | | Single | PW Instance <---------->o o <-----X-> o PW Instance o<----------> | | Elim. | | +-------------+ +-------------+ Egress/ | | | | Ingress Client AC | NSP | | Single | PW Instance <---------->o o <-------> o PW Instance o<----------> | | | | +---------------------------------------+ ]]> A DA-T-PE participates to the packet replication and duplication elimination. Required processing is done within an extended forwarder function. In the case the native service processing (NSP) is IEEE 802.1CB capable, the packet replication and duplicate elimination MAY entirely be done in the NSP and bypassing the DetNet PW encapsulation and logic entirely, and thus is able to operate over unmodified PW implementation and deployment. The NSP approach works only between DA-T-PEs and cannot make use of DA-S-PEs (see ). The DetNet-aware extended forwarder selects the egress segment PW based on the rules described in and . In both "normal AC" and "Packet AC" cases there is no DetNet encapsulation header available yet as it is the case with DA-S-PEs (see ). It is the responsibility of the extended forwarder within the DA-T-PE to push the DetNet encapsulation header (i.e., the DetNet CW and the DetNet flow-ID) to the packet before forwarding it to the appropriate egress PW instance(s). The extended forwarder MAY copy the sequencing information from the native packet into the DetNet CW and vice versa. If there is no existing sequencing information available in the native packet or the forwarder chose not to copy it from the native packet, then the extended forwarder MUST maintain a sequence number counter for each DetNet flow (indexed by the flow-ID).

The PW-based DetNet data plane solution overloads a S-PE with a DetNet Relay function. Such S-PE device is referred as DA-S-PE and implies the S-PE is also aware of DetNet flows and may operate upon those. illustrates the overall DA-S-PE device functions. A DA-S-PE participates to the packet replication and duplication elimination. This processing is done within an extended forwarder function. Whether an ingress PW receives DetNet specific processing depends on how the LFIB is programmed. For some PWs the DA-S-PE can act as a normal S-PE and for some apply the DetNet specific processing. It is also possible to treat the DA-S-PE as a P router using the L-label tunnels. Again, this is entirely up to how the LFIB has been programmed. The DetNet-aware forwarder selects the egress segment PW based on the PW label. The mapping of ingress PW label to egress PW label may be statically or dynamically configured. Additionally the DetNet-aware forwarder does duplicate frame elimination based on the DetNet flow-ID and DetNet Control Word sequence number combination. The packet replication is also done within the DetNet-aware forwarder. During elimination and the replication process both DetNet CW sequence number and DetNet flow-ID MUST be preserved and copied to the egress PW.

[Editor's note: Discuss the CoS.. and how that is archived when using MPLS or IP PSN.]

[Editor's note: Elaborate the QoS issues here..]

[Editor's note: describe a bit of issues and deployment considerations related to time-synchronization within DetNet. Refer to DT discussion and the slides that summarize different approaches and rough synchronization performance numbers. Finally, scope time-synchronization solution outside data plane.] When DetNet is used, there is an underlying assumption that a clock synchronization method is used, such as the Precision Time Protocol (PTP) [IEEE1588]. In this case, there are a few possible approaches of how synchronization protocol packets are forwarded and handled by the network: PTP packets are sent as a normal DetNet flow: in this approach PTP traffic is forwarded as a DetNet flow, and as such it is forwarded in a way that allows a low delay variation. However, since intermediate nodes do not take part in the synchronization protocol, this approach provides a relatively low degree of accuracy. PTP with on-path support: in this approach PTP packets are sent as DetNet flows, and intermediate nodes take part in the protocol as Transparent Clocks or Boundary Clocks [IEEE1588]. The on-path PTP support by intermediate nodes provides a higher degree of accuracy than the previous approach. The actual accuracy depends on whether all intermediate nodes are PTP-capable, or only a subset of them. Time-as-a-service: in this approach accurate time is provided as-a-service to the DetNet source and destination, as well as the intermediate nodes. Since traffic between the source and destination is sent over a provider network, if the provider supports time-as-a-service, then accurate time can be provided to both the source and the destination of DetNet traffic. This approach can potentially provide the highest degree of accuracy. It is expected that the latter approach will be the most common one, as it provides the highest degree of accuracy, and creates a layer separation between the DetNet data and the synchronization service. It should be noted that in all three approaches it is not recommended to use replication and elimination for synchronization packets; the replication/elimination approach may in some cases reduce the synchronization accuracy, since the observed path delay will be bivalent.

Some DetNet applications generate bidirectional traffic and may require symmetric flows. There are already mechanisms that can be used to create bidirectional tunnels at the transport network level, such as MPLS-TP. The data plane solution SHOULD allow establishing bidirectional symmetric flows. Control plane mechanisms would need to also support this, though this is out of scope of this document. [Summary of existing mechanisms to create bidirectional tunnels that can be used.]

[Editor's note: discuss here what kind of enhancements are needed for DetNet and specifically for PREF.]

The PW label distribution follows the same mechanisms specified for MS-PW .

The security considerations of DetNet in general are discussed in and . Other security considerations will be added in a future version of this draft.

TBD.

The author(s) ACK and NACK. The following people were part of the DetNet Data Plane Solution Design Team: Jouni Korhonen János Farkas Norman Finn Balázs Varga Loa Andersson Tal Mizrahi David Mozes Yuanlong Jiang Carlos J. Bernardos The DetNet chairs serving during the DetNet Data Plane Solution Design Team: Lou Berger Pat Thaler

&rfc3985; &rfc6073; &rfc4448; &rfc6658; &rfc2119; &I-D.ietf-detnet-dp-alt; &I-D.ietf-detnet-architecture; Deterministic Networking (DetNet) Security Considerations, draft-sdt-detnet-security, work in progress Mizrahi, T., Grossman, E., Hacker, A., Das, S. IEEE 1588 Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems Version 2 IEEE Draft Standard for Local and metropolitan area networks - Seamless Redundancy IEEE 802.1

[Editor's note: Simplified example of DetNet data plane and how labels etc work in the case of MPLS-based PSN and utilizing PREF. The figure is subject to change depending on the further DT decisions on the label handling..]

R1----->DA-S-PE1---->R2---->DA-S-PE2--->R3 L1 / L1 T2 L2 T3 \ L2 PW1 / PW1 L2 PW1 L2 \ PW1 #seq / #seq PW1 #seq PW1 \ #seq FID1 / FID1 #seq FID1 #seq \ FID1 / FID1 v E1-->DA-T-PE1 DA-T-PE2-->E3 \ T5 T6 ^ T4 \ L2 L3 L4 L5 / L2 \ PW1 PW2 PW1 PW2 / PW1 \ #seq #seq #seq #seq / #seq \ FID1 FID2 FID1 FID2 / FID1 =>R4------------->DA-S-PE2--------------->R5 T11 / ^ \ L5 L3 / \ L9 \ PW2 PW2 / \ PW2 \ #seq #seq / \ #seq \ FID2 FID2 / \ FID2 v E2-->DA-T-PE3 R9<- T10 DA-T-PE4-->E4 \ T8 \ L9 T9 ^ T7 \ L6 L7 L7 \ PW2 L8 / L8 L6 \ PW2 PW2 PW2 \ #seq PW2 / PW2 PW2 \ #seq #seq #seq \FID2 #seq / #seq #seq \ FID2 FID2 FID2 \ FID2 / FID2 FID2 ->R6---->DA-S-PE2---->R7---->DA-S-PE6---->R8 ]]> [Editor's note: the LFIB content to be updated.]