draft-ietf-bess-service-chaining-05.txt | draft-ietf-bess-service-chaining-06.txt | |||
---|---|---|---|---|
Network Working Group R. Fernando | Network Working Group R. Fernando | |||
Internet-Draft Cisco Systems | Internet-Draft Cisco Systems | |||
Intended status: Standards Track S. Mackie | Intended status: Standards Track S. Mackie | |||
Expires: February 7, 2019 Juniper Networks | Expires: June 7, 2019 Juniper Networks | |||
D. Rao | D. Rao | |||
Cisco Systems | Cisco Systems | |||
B. Rijsman | B. Rijsman | |||
M. Napierala | M. Napierala | |||
ATT Labs | ATT Labs | |||
T. Morin | T. Morin | |||
Orange | Orange | |||
August 6, 2018 | December 4, 2018 | |||
Service Function Chaining using Virtual Networks with BGP VPNs | Service Function Chaining using Virtual Networks with BGP VPNs | |||
draft-ietf-bess-service-chaining-05 | draft-ietf-bess-service-chaining-06 | |||
Abstract | Abstract | |||
This document describes how service function chains (SFC) can be | This document describes how service function chains (SFC) can be | |||
applied to traffic flows using routing in a virtual (overlay) network | applied to traffic flows using routing in a virtual (overlay) network | |||
to steer traffic between service nodes. Chains can include services | to steer traffic between service nodes. Chains can include services | |||
running in routers, on physical appliances or in virtual machines. | running in routers, on physical appliances or in virtual machines. | |||
Service chains have applicability at the subscriber edge, business | Service chains have applicability at the subscriber edge, business | |||
edge and in multi-tenant datacenters. The routing function into SFCs | edge and in multi-tenant datacenters. The routing function into SFCs | |||
and between service functions within an SFC can be performed by | and between service functions within an SFC can be performed by | |||
skipping to change at page 2, line 25 ¶ | skipping to change at page 2, line 25 ¶ | |||
Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
Task Force (IETF). Note that other groups may also distribute | Task Force (IETF). Note that other groups may also distribute | |||
working documents as Internet-Drafts. The list of current Internet- | working documents as Internet-Drafts. The list of current Internet- | |||
Drafts is at https://datatracker.ietf.org/drafts/current/. | Drafts is at https://datatracker.ietf.org/drafts/current/. | |||
Internet-Drafts are draft documents valid for a maximum of six months | Internet-Drafts are draft documents valid for a maximum of six months | |||
and may be updated, replaced, or obsoleted by other documents at any | and may be updated, replaced, or obsoleted by other documents at any | |||
time. It is inappropriate to use Internet-Drafts as reference | time. It is inappropriate to use Internet-Drafts as reference | |||
material or to cite them other than as "work in progress." | material or to cite them other than as "work in progress." | |||
This Internet-Draft will expire on February 7, 2019. | This Internet-Draft will expire on June 7, 2019. | |||
Copyright Notice | Copyright Notice | |||
Copyright (c) 2018 IETF Trust and the persons identified as the | Copyright (c) 2018 IETF Trust and the persons identified as the | |||
document authors. All rights reserved. | document authors. All rights reserved. | |||
This document is subject to BCP 78 and the IETF Trust's Legal | This document is subject to BCP 78 and the IETF Trust's Legal | |||
Provisions Relating to IETF Documents | Provisions Relating to IETF Documents | |||
(https://trustee.ietf.org/license-info) in effect on the date of | (https://trustee.ietf.org/license-info) in effect on the date of | |||
publication of this document. Please review these documents | publication of this document. Please review these documents | |||
carefully, as they describe your rights and restrictions with respect | carefully, as they describe your rights and restrictions with respect | |||
to this document. Code Components extracted from this document must | to this document. Code Components extracted from this document must | |||
include Simplified BSD License text as described in Section 4.e of | include Simplified BSD License text as described in Section 4.e of | |||
the Trust Legal Provisions and are provided without warranty as | the Trust Legal Provisions and are provided without warranty as | |||
described in the Simplified BSD License. | described in the Simplified BSD License. | |||
Table of Contents | Table of Contents | |||
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 | |||
1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 | 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 | |||
2. Service Function Chain Architecture Using Virtual Networking 7 | 2. Service Function Chain Architecture Using Virtual Networking 7 | |||
2.1. High Level Architecture . . . . . . . . . . . . . . . . . 8 | 2.1. High Level Architecture . . . . . . . . . . . . . . . . . 8 | |||
2.2. Service Function Chain Logical Model . . . . . . . . . . 10 | 2.2. Service Function Chain Logical Model . . . . . . . . . . 10 | |||
2.3. Service Function Implemented in a Set of SF Instances . . 11 | 2.3. Service Function Implemented in a Set of SF Instances . . 11 | |||
2.4. SF Instance Connections to VRFs . . . . . . . . . . . . . 12 | 2.4. SF Instance Connections to VRFs . . . . . . . . . . . . . 13 | |||
2.4.1. SF Instance in Physical Appliance . . . . . . . . . . 12 | 2.4.1. SF Instance in Physical Appliance . . . . . . . . . . 13 | |||
2.4.2. SF Instance in a Virtualized Environment . . . . . . 13 | 2.4.2. SF Instance in a Virtualized Environment . . . . . . 14 | |||
2.5. Encapsulation Tunneling for Transport . . . . . . . . . . 15 | 2.5. Encapsulation Tunneling for Transport . . . . . . . . . . 15 | |||
2.6. SFC Creation Procedure . . . . . . . . . . . . . . . . . 15 | 2.6. SFC Creation Procedure . . . . . . . . . . . . . . . . . 15 | |||
2.6.1. SFC Provisioning Using Sequential VPNs . . . . . . . 16 | 2.6.1. SFC Provisioning Using Sequential VPNs . . . . . . . 16 | |||
2.6.2. Modified-Route SFC Creation . . . . . . . . . . . . . 17 | 2.6.2. Modified-Route SFC Creation . . . . . . . . . . . . . 17 | |||
2.6.3. Common SFC provisioning considerations . . . . . . . 19 | 2.6.3. Common SFC provisioning considerations . . . . . . . 19 | |||
2.7. Controller Function . . . . . . . . . . . . . . . . . . . 19 | 2.7. Controller Function . . . . . . . . . . . . . . . . . . . 20 | |||
2.8. Variations on Setting Prefixes in an SFC . . . . . . . . 20 | 2.8. Variations on Setting Prefixes in an SFC . . . . . . . . 20 | |||
2.8.1. Using a Default Route . . . . . . . . . . . . . . . . 20 | 2.8.1. Using a Default Route . . . . . . . . . . . . . . . . 20 | |||
2.8.2. Using a Default Route and a Large Prefix . . . . . . 20 | 2.8.2. Using a Default Route and a Large Prefix . . . . . . 21 | |||
2.8.3. Disaggregated Gateway Routers . . . . . . . . . . . . 21 | 2.8.3. Disaggregated Gateway Routers . . . . . . . . . . . . 21 | |||
2.8.4. Optimizing VRF usage . . . . . . . . . . . . . . . . 22 | 2.8.4. Optimizing VRF usage . . . . . . . . . . . . . . . . 22 | |||
2.8.5. Dynamic Entry and Exit Signaling . . . . . . . . . . 22 | 2.8.5. Dynamic Entry and Exit Signaling . . . . . . . . . . 22 | |||
2.8.6. Dynamic Re-Advertisements in Intermediate Systems . . 22 | 2.8.6. Dynamic Re-Advertisements in Intermediate Systems . . 23 | |||
2.9. Layer-2 Virtual Networks and Service Functions . . . . . 23 | 2.9. Layer-2 Virtual Networks and Service Functions . . . . . 23 | |||
2.10. Header Transforming Service Functions . . . . . . . . . . 23 | 2.10. Header Transforming Service Functions . . . . . . . . . . 23 | |||
3. Load Balancing Along a Service Function Chain . . . . . . . . 24 | 3. Load Balancing Along a Service Function Chain . . . . . . . . 24 | |||
3.1. SF Instances Connected to Separate VRFs . . . . . . . . . 24 | 3.1. SF Instances Connected to Separate VRFs . . . . . . . . . 24 | |||
3.2. SF Instances Connected to the Same VRF . . . . . . . . . 25 | 3.2. SF Instances Connected to the Same VRF . . . . . . . . . 25 | |||
3.3. Combination of Egress and Ingress VRF Load Balancing . . 26 | 3.3. Combination of Egress and Ingress VRF Load Balancing . . 26 | |||
3.4. Forward and Reverse Flow Load Balancing . . . . . . . . . 27 | 3.4. Forward and Reverse Flow Load Balancing . . . . . . . . . 28 | |||
3.4.1. Issues with Equal Cost Multi-Path Routing . . . . . . 27 | 3.4.1. Issues with Equal Cost Multi-Path Routing . . . . . . 28 | |||
3.4.2. Modified ECMP with Consistent Hash . . . . . . . . . 28 | 3.4.2. Modified ECMP with Consistent Hash . . . . . . . . . 28 | |||
3.4.3. ECMP with Flow Table . . . . . . . . . . . . . . . . 28 | 3.4.3. ECMP with Flow Table . . . . . . . . . . . . . . . . 29 | |||
3.4.4. Dealing with Different Hash Algorithms in an SFC . . 30 | 3.4.4. Dealing with Different Hash Algorithms in an SFC . . 30 | |||
4. Steering into SFCs Using a Classifier . . . . . . . . . . . . 30 | 4. Sharing Service Functions in Different SFCs . . . . . . . . . 31 | |||
5. External Domain Co-ordination . . . . . . . . . . . . . . . . 32 | 4.1. Shared SFs in L3 SFCs . . . . . . . . . . . . . . . . . . 31 | |||
6. Fine-grained steering using BGP Flow-Spec . . . . . . . . . . 33 | 4.2. Shared SFs in L2 SFCs . . . . . . . . . . . . . . . . . . 31 | |||
7. Controller Federation . . . . . . . . . . . . . . . . . . . . 33 | 5. Steering into SFCs Using a Classifier . . . . . . . . . . . . 31 | |||
8. Coordination Between SF Instances and Controller using BGP . 33 | 6. External Domain Co-ordination . . . . . . . . . . . . . . . . 33 | |||
9. BGP Extended Communities . . . . . . . . . . . . . . . . . . 34 | 7. Fine-grained steering using BGP Flow-Spec . . . . . . . . . . 34 | |||
9.1. Route-Target Record . . . . . . . . . . . . . . . . . . . 34 | 8. Controller Federation . . . . . . . . . . . . . . . . . . . . 34 | |||
9.2. Consistent Hash Sort Order . . . . . . . . . . . . . . . 35 | 9. Coordination Between SF Instances and Controller using BGP . 34 | |||
9.3. Load Balance Settings . . . . . . . . . . . . . . . . . . 35 | 10. BGP Extended Communities . . . . . . . . . . . . . . . . . . 35 | |||
10. Summary and Conclusion . . . . . . . . . . . . . . . . . . . 36 | 10.1. Route-Target Record . . . . . . . . . . . . . . . . . . 35 | |||
11. Security Considerations . . . . . . . . . . . . . . . . . . . 36 | 10.2. Consistent Hash Sort Order . . . . . . . . . . . . . . . 36 | |||
12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 36 | 10.3. Load Balance Settings . . . . . . . . . . . . . . . . . 36 | |||
13. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 37 | 11. Summary and Conclusion . . . . . . . . . . . . . . . . . . . 37 | |||
14. References . . . . . . . . . . . . . . . . . . . . . . . . . 37 | 12. Security Considerations . . . . . . . . . . . . . . . . . . . 37 | |||
14.1. Normative References . . . . . . . . . . . . . . . . . . 37 | 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 38 | |||
14.2. Informational References . . . . . . . . . . . . . . . . 38 | 14. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 38 | |||
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 | 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 38 | |||
15.1. Normative References . . . . . . . . . . . . . . . . . . 38 | ||||
15.2. Informational References . . . . . . . . . . . . . . . . 39 | ||||
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 40 | ||||
1. Introduction | 1. Introduction | |||
The purpose of networks is to allow computing systems to communicate | The purpose of networks is to allow computing systems to communicate | |||
with each other. Requests are usually made from the client or | with each other. Requests are usually made from the client or | |||
customer side of a network, and responses are generated by | customer side of a network, and responses are generated by | |||
applications residing in a datacenter. Over time, the network | applications residing in a datacenter. Over time, the network | |||
between the client and the application has become more complex, and | between the client and the application has become more complex, and | |||
traffic between the client and the application is acted on by | traffic between the client and the application is acted on by | |||
intermediate systems that apply network services. Some of these | intermediate systems that apply network services. Some of these | |||
skipping to change at page 8, line 16 ¶ | skipping to change at page 8, line 21 ¶ | |||
architecture, and outline the processes of route installation and | architecture, and outline the processes of route installation and | |||
subsequent route exchange to create an SFC. | subsequent route exchange to create an SFC. | |||
2.1. High Level Architecture | 2.1. High Level Architecture | |||
Service function chains can be deployed with or without a classifier. | Service function chains can be deployed with or without a classifier. | |||
Use cases where SFCs may be deployed without a classifier include | Use cases where SFCs may be deployed without a classifier include | |||
multi-tenant data centers, private and public cloud and virtual CPE | multi-tenant data centers, private and public cloud and virtual CPE | |||
for business services. Classifiers will primarily be used in mobile | for business services. Classifiers will primarily be used in mobile | |||
and wireline subscriber edge use cases. Use of a classifier is | and wireline subscriber edge use cases. Use of a classifier is | |||
discussed in Section 4. | discussed in Section 5. | |||
A high-level architecture diagram of an SFC without a classifier, | A high-level architecture diagram of an SFC without a classifier, | |||
where traffic is routed into and out of the SFC, is shown in | where traffic is routed into and out of the SFC, is shown in | |||
Figure 1, below. An optional controller is shown that contains a | Figure 1, below. An optional controller is shown that contains a | |||
topological model of the SFC and which configures the network | topological model of the SFC and which configures the network | |||
resources to implement the SFC. | resources to implement the SFC. | |||
+-------------------------+ | +-------------------------+ | |||
|--- Data plane connection| | |--- Data plane connection| | |||
|=== Encapsulation tunnel | | |=== Encapsulation tunnel | | |||
skipping to change at page 10, line 22 ¶ | skipping to change at page 10, line 22 ¶ | |||
routing system, and the service instances can be local or remote from | routing system, and the service instances can be local or remote from | |||
either or both of the routing systems containing the entry and exit | either or both of the routing systems containing the entry and exit | |||
VRFs, and from each other. It is also possible that the ingress and | VRFs, and from each other. It is also possible that the ingress and | |||
egress VRFs are implemented using alternative mechanisms. | egress VRFs are implemented using alternative mechanisms. | |||
The controller is responsible for configuring the VRFs in each | The controller is responsible for configuring the VRFs in each | |||
routing system, installing the routes in each of the VRFs to | routing system, installing the routes in each of the VRFs to | |||
implement the SFC, and, in the case of virtualized services, may | implement the SFC, and, in the case of virtualized services, may | |||
instantiate the service instances. | instantiate the service instances. | |||
The controller is not responsible for configuring the SFs themselves. | ||||
It is assumed that there is a separate system that performs that | ||||
function e.g the VNF Manager functional component in [NFVE2E]. Note | ||||
that coordination may be required between a controller and and VNF | ||||
manager, for instance to ensure that installed firewall filters in a | ||||
SF align with the subnets whose packets will pass through it. At | ||||
this time, there are no standard ways to address this, and custom | ||||
pair-wise integrations between controllers and VNF managers would be | ||||
required. | ||||
2.2. Service Function Chain Logical Model | 2.2. Service Function Chain Logical Model | |||
A service function chain is a set of logically connected service | A service function chain is a set of logically connected service | |||
functions through which traffic can flow. Each egress interface of | functions through which traffic can flow. Each egress interface of | |||
one service function is logically connected to an ingress interface | one service function is logically connected to an ingress interface | |||
of the next service function. | of the next service function. | |||
+------+ +------+ +------+ | +------+ +------+ +------+ | |||
Network-A-->| SF-1 |-->| SF-2 |-->| SF-3 |-->Network-B | Network-A-->| SF-1 |-->| SF-2 |-->| SF-3 |-->Network-B | |||
+------+ +------+ +------+ | +------+ +------+ +------+ | |||
skipping to change at page 15, line 49 ¶ | skipping to change at page 16, line 8 ¶ | |||
connected to SF instances in each VRF, and allocating and configuring | connected to SF instances in each VRF, and allocating and configuring | |||
import and export RTs for each VRF. Additionally, in the second | import and export RTs for each VRF. Additionally, in the second | |||
method, the controller also sends the route-policy containing the | method, the controller also sends the route-policy containing the | |||
service chain logical topology to each routing system. If a | service chain logical topology to each routing system. If a | |||
controller is not used, these procedures will require to be performed | controller is not used, these procedures will require to be performed | |||
manually or through scripting, for instance. | manually or through scripting, for instance. | |||
The source and destination networks' prefixes can be configured in | The source and destination networks' prefixes can be configured in | |||
the controller, or may be automatically learned through peering | the controller, or may be automatically learned through peering | |||
between the controller and each network's gateway. This is further | between the controller and each network's gateway. This is further | |||
described in Section 2.8.5 and Section 5. | described in Section 2.8.5 and Section 6. | |||
The following sub-sections describe how RT configuration, local route | The following sub-sections describe how RT configuration, local route | |||
installation and route distribution occur in each of the methods. | installation and route distribution occur in each of the methods. | |||
It should be noted that depending on the capabilities of the routing | It should be noted that depending on the capabilities of the routing | |||
systems, a controller can use one or more techniques to realize | systems, a controller can use one or more techniques to realize | |||
forwarding along the service chain, ranging from fully centralized to | forwarding along the service chain, ranging from fully centralized to | |||
fully distributed. The goal of describing the following two methods | fully distributed. The goal of describing the following two methods | |||
is to illustrate the broad approaches and as a base for various | is to illustrate the broad approaches and as a base for various | |||
optimization options. | optimization options. | |||
skipping to change at page 19, line 32 ¶ | skipping to change at page 19, line 38 ¶ | |||
configuration of VRFs, interfaces and installation of routes may | configuration of VRFs, interfaces and installation of routes may | |||
instead be performed using a single protocol like XMPP, NC/YANG or an | instead be performed using a single protocol like XMPP, NC/YANG or an | |||
equivalent programmatic interface. | equivalent programmatic interface. | |||
Also in the virtualized case, the actual forwarding table entries to | Also in the virtualized case, the actual forwarding table entries to | |||
be installed in the ingress and egress VRFs may be calculated by the | be installed in the ingress and egress VRFs may be calculated by the | |||
controller based on its internal knowledge of the required SFC | controller based on its internal knowledge of the required SFC | |||
topology and the connectivity of SF instances to routing systems. In | topology and the connectivity of SF instances to routing systems. In | |||
this case, the routes may be directly installed in the forwarders | this case, the routes may be directly installed in the forwarders | |||
using the programmatic interface and no BGP route advertisement is | using the programmatic interface and no BGP route advertisement is | |||
necessary, except when coordination with external domains (Section 5) | necessary, except when coordination with external domains (Section 6) | |||
or federation between controller domains is employed (Section 7). | or federation between controller domains is employed (Section 8). | |||
Note however that this is just one typical model for a virtual | Note however that this is just one typical model for a virtual | |||
forwarding based system. In general, physical and virtual routing | forwarding based system. In general, physical and virtual routing | |||
systems can be treated exactly the same if they have the same | systems can be treated exactly the same if they have the same | |||
capabilities. | capabilities. | |||
In both the methods, the SF instance may also need to be set up | In both the methods, the SF instance may also need to be set up | |||
appropriately to forward traffic between it's input and output | appropriately to forward traffic between it's input and output | |||
interfaces, either via static, dynamic or policy-based routing. If | interfaces, either via static, dynamic or policy-based routing. If | |||
the service function is a transparent L2 service, then the static | the service function is a transparent L2 service, then the static | |||
route installed in the ingress VRF will have a next-hop of the IP | route installed in the ingress VRF will have a next-hop of the IP | |||
skipping to change at page 22, line 44 ¶ | skipping to change at page 22, line 46 ¶ | |||
If dynamic signaling is performed, and a bidirectional SFC set is | If dynamic signaling is performed, and a bidirectional SFC set is | |||
configured, and the gateways to the networks connected via the SFC | configured, and the gateways to the networks connected via the SFC | |||
exchange routes, steps must be taken to ensure that routes to both | exchange routes, steps must be taken to ensure that routes to both | |||
networks do not get advertised from both ends of the SFC set by re- | networks do not get advertised from both ends of the SFC set by re- | |||
origination. This can be achieved if a new BGP Extended Community | origination. This can be achieved if a new BGP Extended Community | |||
[RFC4360] is implemented to control re-origination. When a route is | [RFC4360] is implemented to control re-origination. When a route is | |||
re- originated, the RTs of the re-originated routes are appended to | re- originated, the RTs of the re-originated routes are appended to | |||
the new RT-Record Extended Community, and if the RT for the route | the new RT-Record Extended Community, and if the RT for the route | |||
already exists in the Extended Community, the route is not re- | already exists in the Extended Community, the route is not re- | |||
originated (see Section 9.1). | originated (see Section 10.1). | |||
2.8.6. Dynamic Re-Advertisements in Intermediate Systems | 2.8.6. Dynamic Re-Advertisements in Intermediate Systems | |||
The intermediate routing systems attached to the service instances | The intermediate routing systems attached to the service instances | |||
may also use the dynamic signaling technique from the previous | may also use the dynamic signaling technique from the previous | |||
section to re-advertise received routes up the chain. In this case, | section to re-advertise received routes up the chain. In this case, | |||
the ingress and egress VRFs are combined into one; and a local route- | the ingress and egress VRFs are combined into one; and a local route- | |||
policy ensures the re-advertised routes are associated with labels | policy ensures the re-advertised routes are associated with labels | |||
that direct incoming traffic directly to the attached service | that direct incoming traffic directly to the attached service | |||
instances on that routing system. | instances on that routing system. | |||
2.9. Layer-2 Virtual Networks and Service Functions | 2.9. Layer-2 Virtual Networks and Service Functions | |||
There are SFs that operate at layer-2, in a transparent mode, and | There are SFs that operate at layer-2, in a transparent mode, and | |||
forward traffic based on the MAC DA. When such a SF is present in | forward traffic based on the MAC DA. When such a SF is present in | |||
the SFC, the procedures at the routing system are modified slightly. | the SFC, the procedures at the routing system are modified slightly. | |||
In this case, the IP address associated with the SF instance (and | The L3 routes are the same as the L3 SF case, but now an L2 header | |||
used as the next-hop of routes in the above procedures) is actually | has to be supplied for each packet passing through an SF. A | |||
the one assigned to the routing system interface attached to the | convenient destination MAC address to use is the one that each VRF | |||
other end of the SF instance, or it could be a virtual IP address | uses for the default gateway of its network. This can be the same | |||
logically associated with the service function with a next-hop of the | for all VRFs in routing systems in the domain of a controller. The | |||
other routing system interface. The routing system interface uses | VRF at the egress of the last SF will rewrite the gateway MAC address | |||
distinct interface MAC addresses. This allows the current scheme to | with the MAC address of the actual destination before encapsulating | |||
be supported, while allowing the transparent service function to work | and forwarding. | |||
using its existing behavior. | ||||
A SFC may be also be set up between end systems or network segments | A SFC may be also be set up between end systems or network segments | |||
within the same Layer-2 bridged network. In this case, applying the | within the same Layer-2 bridged network. In this case, applying the | |||
procedures described earlier, the segments or groups of end systems | procedures described earlier, the segments or groups of end systems | |||
are placed in distinct Layer-2 virtual networks, which are then then | are placed in distinct Layer-2 virtual networks, which are then then | |||
inter-connected via a sequence of intermediate Layer-2 virtual | inter-connected via a sequence of intermediate Layer-2 virtual | |||
networks that form the links in the SFC. Each virtual network maps | networks that form the links in the SFC. Each virtual network maps | |||
to a pair of ingress and egress MAC VRFs on the routing systems to | to a pair of ingress and egress MAC VRFs on the routing systems to | |||
which the SF instances are attached. The routing systems at the ends | which the SF instances are attached. The routing systems at the ends | |||
of the SFC will advertise the locally learnt or installed MAC entries | of the SFC will advertise the locally learnt or installed MAC entries | |||
skipping to change at page 28, line 39 ¶ | skipping to change at page 29, line 22 ¶ | |||
the load balancing function in VRFs that connect to each added (or | the load balancing function in VRFs that connect to each added (or | |||
removed) SF instance as part of the same network transaction as route | removed) SF instance as part of the same network transaction as route | |||
updates to ensure that the load balancer configuration is | updates to ensure that the load balancer configuration is | |||
synchronized with the set of SF instances. | synchronized with the set of SF instances. | |||
The consistent ordering among ECMP routes in the routing systems | The consistent ordering among ECMP routes in the routing systems | |||
could be achieved through configuration of the routing systems by the | could be achieved through configuration of the routing systems by the | |||
controller using, for instance, Netconf; or when the routes are | controller using, for instance, Netconf; or when the routes are | |||
signaled using BGP by the controller or a routing system, the order | signaled using BGP by the controller or a routing system, the order | |||
for a given instance can be sent in a new 'Consistent Hash Sort | for a given instance can be sent in a new 'Consistent Hash Sort | |||
Order' BGP Extended Community (defined in Section 9.2). | Order' BGP Extended Community (defined in Section 10.2). | |||
The effect of rehashing when SF instances are added or removed can be | The effect of rehashing when SF instances are added or removed can be | |||
minimized, or even eliminated using variations of the technique of | minimized, or even eliminated using variations of the technique of | |||
consistent hashing [consistent-hash]. Details are outside the scope | consistent hashing [consistent-hash]. Details are outside the scope | |||
of this document. | of this document. | |||
3.4.3. ECMP with Flow Table | 3.4.3. ECMP with Flow Table | |||
A second refinement that can ensure forward/reverse flow consistency, | A second refinement that can ensure forward/reverse flow consistency, | |||
and also provides stability when the number of SF instances changes | and also provides stability when the number of SF instances changes | |||
skipping to change at page 30, line 31 ¶ | skipping to change at page 31, line 15 ¶ | |||
intended if the first SF is stateful. It may be impractical, or | intended if the first SF is stateful. It may be impractical, or | |||
prohibitively expensive to implement the flow table-based methods | prohibitively expensive to implement the flow table-based methods | |||
described above to achieve flow stability and symmetry. This issue | described above to achieve flow stability and symmetry. This issue | |||
can be mitigated by ensuring that the first SF is not stateful, or by | can be mitigated by ensuring that the first SF is not stateful, or by | |||
placing a null SF between the physical router and the first actual SF | placing a null SF between the physical router and the first actual SF | |||
in the SFC. This ensures that the hash method on both sides of | in the SFC. This ensures that the hash method on both sides of | |||
stateful service instances is the same, and the SFC will operate with | stateful service instances is the same, and the SFC will operate with | |||
flow stability and symmetry if the methods described above are | flow stability and symmetry if the methods described above are | |||
employed. | employed. | |||
4. Steering into SFCs Using a Classifier | 4. Sharing Service Functions in Different SFCs | |||
4.1. Shared SFs in L3 SFCs | ||||
Sharing SFs among multiple SFCs requires that packets emerging from | ||||
an SF can be mapped to the correct next SF. This can be achieved by | ||||
configuring an SF to accept packets from multiple subnets on each | ||||
interface, configuring addresses from each of these subnets on SFs | ||||
and by using next-table policies to direct traffic between subnet- | ||||
specific VRFs to and from SF interfaces. However, in mobility and | ||||
wireline applications, which are the most common ones where sharing | ||||
is desired, classification is on the basis of subscriber id and | ||||
traffic type, so discrimination on the basis of subnets is too | ||||
coarse-grained. Using host routes along SFC paths could achieve the | ||||
desired result of SF sharing, but will not scale appropriately. | ||||
4.2. Shared SFs in L2 SFCs | ||||
Layer 2 transparent SFs can be shared among multiple service chains | ||||
by using a different VLAN for each chain as packets pass through each | ||||
SF. Forwarding into different VLANs can be accomplished by using | ||||
different labels in the encapsulation of packets arriving at an | ||||
ingress VRF. Egress VRFs can have next hops into different SFs based | ||||
on the VLAN of each egressing packet. The service chains sharing an | ||||
SF might have different networks from each other at each end, or | ||||
might be selected on the basis of 5-tuple filtering from one network. | ||||
5. Steering into SFCs Using a Classifier | ||||
In many applications of SFCs, a classifier will be used to direct | In many applications of SFCs, a classifier will be used to direct | |||
traffic into SFCs. The classifier inspects the first or first few | traffic into SFCs. The classifier inspects the first or first few | |||
packets in a flow to determine which SFC the flow should be sent | packets in a flow to determine which SFC the flow should be sent | |||
into. The decision criteria can be based on just the IP 5-tuple of | into. The decision criteria can be based on just the IP 5-tuple of | |||
the header (i.e filter-based forwarding), or could involve analysis | the header (i.e filter-based forwarding), or could involve analysis | |||
of the payload of packets using deep packet inspection. Integration | of the payload of packets using deep packet inspection. Integration | |||
with a subscriber management system such as PCRF or AAA may be | with a subscriber management system such as PCRF or AAA may be | |||
required in order to identify which SFC to send traffic to based on | required in order to identify which SFC to send traffic to based on | |||
subscriber policy. | subscriber policy. | |||
skipping to change at page 32, line 4 ¶ | skipping to change at page 33, line 11 ¶ | |||
VRF for each SFC does not peer with a gateway or proxy node in the | VRF for each SFC does not peer with a gateway or proxy node in the | |||
destination network and packets are forwarded using IP lookup in the | destination network and packets are forwarded using IP lookup in the | |||
main routing table or in a VRF that the exit traffic from the SFCs is | main routing table or in a VRF that the exit traffic from the SFCs is | |||
directed into (shown as X in the diagram). A flow table may be | directed into (shown as X in the diagram). A flow table may be | |||
required to ensure that reverse traffic is sent into the correct SFC. | required to ensure that reverse traffic is sent into the correct SFC. | |||
An alternative would be where the classifier is itself a distributed, | An alternative would be where the classifier is itself a distributed, | |||
virtualized service function, but with multiple egress interfaces. | virtualized service function, but with multiple egress interfaces. | |||
In that case, each virtual classifier instance could be attached to a | In that case, each virtual classifier instance could be attached to a | |||
set of VRFs that connect to different SFCs. Each chain | set of VRFs that connect to different SFCs. Each chain | |||
entry VRF would load balance across the first SF instance set in its | entry VRF would load balance across the first SF instance set in its | |||
SFC. The reverse flow table mechanism described in Section 3.4.3 | SFC. The reverse flow table mechanism described in Section 3.4.3 | |||
could be employed to ensure that flows return to the originating | could be employed to ensure that flows return to the originating | |||
classifier instance which may maintain subscriber context and perform | classifier instance which may maintain subscriber context and perform | |||
charging and accounting. | charging and accounting. | |||
5. External Domain Co-ordination | 6. External Domain Co-ordination | |||
It is likely that SFCs will be managed as a separate administrative | It is likely that SFCs will be managed as a separate administrative | |||
domain from the networks that they receive traffic from, and send | domain from the networks that they receive traffic from, and send | |||
traffic to. If the connected networks use BGP for route | traffic to. If the connected networks use BGP for route | |||
distribution, the controller in the SFC domain can join the network | distribution, the controller in the SFC domain can join the network | |||
domains by creating BGP peering sessions with routing systems or | domains by creating BGP peering sessions with routing systems or | |||
route reflectors in those network domains to exchange VPN routes, or | route reflectors in those network domains to exchange VPN routes, or | |||
with local border routers that peer with the external domains. While | with local border routers that peer with the external domains. While | |||
a controller can modify route targets for the VRFs within its SFC | a controller can modify route targets for the VRFs within its SFC | |||
domain, it is likely to not have any control over the external | domain, it is likely to not have any control over the external | |||
skipping to change at page 33, line 5 ¶ | skipping to change at page 34, line 11 ¶ | |||
using non-specific routes inside an SFC, as described in | using non-specific routes inside an SFC, as described in | |||
Section 2.8.1, means that new networks can be attached to a SFC | Section 2.8.1, means that new networks can be attached to a SFC | |||
without needing to configure prefixes inside the chain. | without needing to configure prefixes inside the chain. | |||
The controller will typically remove the destination network's RTs | The controller will typically remove the destination network's RTs | |||
and replace them with the RTs of the source network while advertising | and replace them with the RTs of the source network while advertising | |||
the modified routes. Alternatively, an external domain may be | the modified routes. Alternatively, an external domain may be | |||
provisioned with an additional export-only RT and an import- only RT | provisioned with an additional export-only RT and an import- only RT | |||
that the controller can use. | that the controller can use. | |||
6. Fine-grained steering using BGP Flow-Spec | 7. Fine-grained steering using BGP Flow-Spec | |||
When steering traffic from an external network domain into an SFC | When steering traffic from an external network domain into an SFC | |||
based on attributes of the packet flow, BGP Flow-spec can be used as | based on attributes of the packet flow, BGP Flow-spec can be used as | |||
a signaling option. | a signaling option. | |||
In this case, the controller can advertise one or more flow-spec | In this case, the controller can advertise one or more flow-spec | |||
routes into the entry VRF with the appropriate Service-topology-RT | routes into the entry VRF with the appropriate Service-topology-RT | |||
for the SFC. Alternatively, it can use the procedures described in | for the SFC. Alternatively, it can use the procedures described in | |||
[RFC5575] or [flowspec-redirect-ip] on the gateway router to redirect | [RFC5575] or [flowspec-redirect-ip] on the gateway router to redirect | |||
traffic towards the first SF. | traffic towards the first SF. | |||
If it is desired to steer specific flows from a network domain's | If it is desired to steer specific flows from a network domain's | |||
existing routers, the controller can advertise the above flow-spec | existing routers, the controller can advertise the above flow-spec | |||
routes to the network domain's border routers or route reflectors. | routes to the network domain's border routers or route reflectors. | |||
7. Controller Federation | 8. Controller Federation | |||
When SFCs are distributed geographically, or in very large-scale | When SFCs are distributed geographically, or in very large-scale | |||
environments, there may be multiple SFC controllers present and they | environments, there may be multiple SFC controllers present and they | |||
may variously employ both of the SFC creation methods described in | may variously employ both of the SFC creation methods described in | |||
Section 2.6. If there is a requirement for SFCs to span controller | Section 2.6. If there is a requirement for SFCs to span controller | |||
domains there may be a requirement to exchange information between | domains there may be a requirement to exchange information between | |||
controllers. Again, a BGP session between controllers can be used to | controllers. Again, a BGP session between controllers can be used to | |||
exchange route information as described in the previous sections and | exchange route information as described in the previous sections and | |||
allow such domain spanning SFCs to be created. | allow such domain spanning SFCs to be created. | |||
8. Coordination Between SF Instances and Controller using BGP | 9. Coordination Between SF Instances and Controller using BGP | |||
In many cases, the configuration of SF instance determines its | In many cases, the configuration of SF instance determines its | |||
network behavior. E.g. when NAT pools are set up, or when an SSL | network behavior. E.g. when NAT pools are set up, or when an SSL | |||
gateway is configured with a set of enterprise IP addresses to use. | gateway is configured with a set of enterprise IP addresses to use. | |||
In these cases, the addresses that will be used by the SFs need to be | In these cases, the addresses that will be used by the SFs need to be | |||
known in the networks connecting to them in order that traffic can be | known in the networks connecting to them in order that traffic can be | |||
properly routed. When SFCs are involved, this means that the | properly routed. When SFCs are involved, this means that the | |||
controller has to be notified when such configuration changes are | controller has to be notified when such configuration changes are | |||
made in SF instances. Sometimes, the changes will be made by end- | made in SF instances. Sometimes, the changes will be made by end- | |||
customers and it is desirable the controller adjust the SFC routing | customers and it is desirable the controller adjust the SFC routing | |||
skipping to change at page 34, line 18 ¶ | skipping to change at page 35, line 24 ¶ | |||
reconfigured SF instance. | reconfigured SF instance. | |||
BGP could also be used to signal from the controller to a SF instance | BGP could also be used to signal from the controller to a SF instance | |||
that certain traffic should be sent out from a particular interface. | that certain traffic should be sent out from a particular interface. | |||
This could be used to direct suspect traffic to a security scrubbing | This could be used to direct suspect traffic to a security scrubbing | |||
center,for example. | center,for example. | |||
Note that the SFF need not support a BGP stack itself; it can proxy | Note that the SFF need not support a BGP stack itself; it can proxy | |||
BGP messages to the controller which will support such a stack. | BGP messages to the controller which will support such a stack. | |||
9. BGP Extended Communities | 10. BGP Extended Communities | |||
9.1. Route-Target Record | 10.1. Route-Target Record | |||
Route-Target Record (RT-Record)is defined as a transitive BGP | Route-Target Record (RT-Record)is defined as a transitive BGP | |||
Extended Community, that contains an Route-Target value representing | Extended Community, that contains an Route-Target value representing | |||
one of the RTs that the route has been attached with previously, and | one of the RTs that the route has been attached with previously, and | |||
which may no longer be attached to the route on subsequent re- | which may no longer be attached to the route on subsequent re- | |||
advertisements (see Section 2.8.5). | advertisements (see Section 2.8.5). | |||
A Sub-Type code 0x13 is assigned in the three BGP Extended Community | A Sub-Type code 0x13 is assigned in the three BGP Extended Community | |||
types - Two-Octet AS-Specific 0x00, IPv4-Address-Specific 0x01 and | types - Two-Octet AS-Specific 0x00, IPv4-Address-Specific 0x01 and | |||
Four-Octet AS-Specific 0x02. A Sub-Type code 0x0013 is also assigned | Four-Octet AS-Specific 0x02. A Sub-Type code 0x0013 is also assigned | |||
skipping to change at page 35, line 16 ¶ | skipping to change at page 36, line 20 ¶ | |||
Route-Target value fields are used in the comparison. The sub-type | Route-Target value fields are used in the comparison. The sub-type | |||
field is masked out. | field is masked out. | |||
When a speaker re-originates a route that contains one or more RTs, | When a speaker re-originates a route that contains one or more RTs, | |||
it must add each of these RTs as RT Record extended communities in | it must add each of these RTs as RT Record extended communities in | |||
the re-originated route. | the re-originated route. | |||
A speaker must not re-originate a route with an RT, if this RT is | A speaker must not re-originate a route with an RT, if this RT is | |||
already present as an RT Record extended community. | already present as an RT Record extended community. | |||
9.2. Consistent Hash Sort Order | 10.2. Consistent Hash Sort Order | |||
Consistent Hash Sort Order is an optional transitive Opaque BGP | Consistent Hash Sort Order is an optional transitive Opaque BGP | |||
Extended Community of type 0x14, defined as follows: | Extended Community of type 0x14, defined as follows: | |||
Type Field : The value of the high-order octet is 0x03 (transitive | Type Field : The value of the high-order octet is 0x03 (transitive | |||
opaque). The value of the low-order octet is assigned | opaque). The value of the low-order octet is assigned | |||
as 0x14 by IANA from the Transitive Opaque Extended | as 0x14 by IANA from the Transitive Opaque Extended | |||
Community Sub-Types registry. | Community Sub-Types registry. | |||
Value Field : The value field contains a Sort Order sub-field that | Value Field : The value field contains a Sort Order sub-field that | |||
skipping to change at page 35, line 38 ¶ | skipping to change at page 36, line 42 ¶ | |||
ECMP set for the prefix, to be sorted in increasing | ECMP set for the prefix, to be sorted in increasing | |||
order. It is a 32-bit unsigned integer. The field is | order. It is a 32-bit unsigned integer. The field is | |||
encoded as shown below: | encoded as shown below: | |||
+------------------------------+ | +------------------------------+ | |||
| Sort Order (4 octets) | | | Sort Order (4 octets) | | |||
+------------------------------+ | +------------------------------+ | |||
| Reserved (2 octets) | | | Reserved (2 octets) | | |||
+------------------------------+ | +------------------------------+ | |||
9.3. Load Balance Settings | 10.3. Load Balance Settings | |||
Consistent Hash Sort Order is an optional transitive Opaque BGP | Consistent Hash Sort Order is an optional transitive Opaque BGP | |||
Extended Community of type 0x14, defined as follows: | Extended Community of type 0x14, defined as follows: | |||
Type Field : The value of the high-order octet is 0x03 (transitive | Type Field : The value of the high-order octet is 0x03 (transitive | |||
opaque). The value of the low-order octet is assigned | opaque). The value of the low-order octet is assigned | |||
as 0xaa by IANA from the Transitive Opaque Extended | as 0xaa by IANA from the Transitive Opaque Extended | |||
Community Sub-Types registry. | Community Sub-Types registry. | |||
Value Field : The value field contains flags that indicate which | Value Field : The value field contains flags that indicate which | |||
skipping to change at page 36, line 23 ¶ | skipping to change at page 37, line 28 ¶ | |||
* Type: 0x03 Opaque | * Type: 0x03 Opaque | |||
* SubType: 0xAA LoadBalance attribute information (TBA) | * SubType: 0xAA LoadBalance attribute information (TBA) | |||
* s: Use l3_source_address ECMP Load-balancing | * s: Use l3_source_address ECMP Load-balancing | |||
* d: Use l3_destination_address ECMP Load-balancing | * d: Use l3_destination_address ECMP Load-balancing | |||
* c: Use l4_protocol ECMP Load-balancing | * c: Use l4_protocol ECMP Load-balancing | |||
* p: Use l4_source_port ECMP Load-balancing | * p: Use l4_source_port ECMP Load-balancing | |||
* P: Use l4_destination_port ECMP Load-balancing | * P: Use l4_destination_port ECMP Load-balancing | |||
* B: Use source_bias (instead of ECMP load-balancing) | * B: Use source_bias (instead of ECMP load-balancing) | |||
* R: Reserved | * R: Reserved | |||
10. Summary and Conclusion | 11. Summary and Conclusion | |||
The architecture for service function chains described in this | The architecture for service function chains described in this | |||
document uses virtual networks implemented as overlays in order to | document uses virtual networks implemented as overlays in order to | |||
create service function chains. The virtual networks use standards- | create service function chains. The virtual networks use standards- | |||
based encapsulation tunneling, such as MPLS over GRE/UDP or VXLAN, to | based encapsulation tunneling, such as MPLS over GRE/UDP or VXLAN, to | |||
transport packets into an SFC and between service function instances | transport packets into an SFC and between service function instances | |||
without routing in the user address space. Two methods of installing | without routing in the user address space. Two methods of installing | |||
routes to form service chains are described. | routes to form service chains are described. | |||
In environments with physical routers, a controller may operate in | In environments with physical routers, a controller may operate in | |||
tandem with existing BGP route reflectors, and would contain the SFC | tandem with existing BGP route reflectors, and would contain the SFC | |||
topology model, and the ability to install the local static interface | topology model, and the ability to install the local static interface | |||
routes to SF instances. In a virtualized environment, the controller | routes to SF instances. In a virtualized environment, the controller | |||
can emulate route refection internally and simply install required | can emulate route refection internally and simply install required | |||
routes directly without advertisements occurring. | routes directly without advertisements occurring. | |||
11. Security Considerations | 12. Security Considerations | |||
The security considerations for SFCs are broadly similar to those | The security considerations for SFCs are broadly similar to those | |||
concerning the data, control and management planes of any device | concerning the data, control and management planes of any device | |||
placed in a network. Details are out of scope for this document. | placed in a network. Details are out of scope for this document. | |||
12. IANA Considerations | 13. IANA Considerations | |||
The new BGP Extended Communities in are assigned types as defined | The new BGP Extended Communities in are assigned types as defined | |||
above in the IANA registry for extended communities. | above in the IANA registry for extended communities. | |||
13. Acknowledgments | 14. Acknowledgments | |||
The authors would like to thank D. Daino, D.R. Lopez, D. Bernier, | The authors would like to thank D. Daino, D.R. Lopez, D. Bernier, | |||
W. Haeffner, A. Farrel, L. Fang, and N. So, for their | W. Haeffner, A. Farrel, L. Fang, and N. So, for their | |||
contributions to the earlier drafts. The authors would also like to | contributions to the earlier drafts. The authors would also like to | |||
thank the following individuals for their review and feedback on the | thank the following individuals for their review and feedback on the | |||
original proposals: E. Rosen, J. Guchard, P. Quinn, P. Bosch, D. | original proposals: E. Rosen, J. Guchard, P. Quinn, P. Bosch, D. | |||
Ward, A. Ganesan, N. Seth, G. Pildush and N. Bitar. The authors | Ward, A. Ganesan, N. Seth, G. Pildush and N. Bitar. The authors | |||
also thank Wim Henderickx for his useful suggestions on several | also thank Wim Henderickx for his useful suggestions on several | |||
aspects of the draft. | aspects of the draft. | |||
14. References | 15. References | |||
14.1. Normative References | 15.1. Normative References | |||
[RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol | [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol | |||
Label Switching Architecture", RFC 3031, | Label Switching Architecture", RFC 3031, | |||
DOI 10.17487/RFC3031, January 2001, | DOI 10.17487/RFC3031, January 2001, | |||
<https://www.rfc-editor.org/info/rfc3031>. | <https://www.rfc-editor.org/info/rfc3031>. | |||
[RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., | [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., | |||
"Encapsulating MPLS in IP or Generic Routing Encapsulation | "Encapsulating MPLS in IP or Generic Routing Encapsulation | |||
(GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, | (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, | |||
<https://www.rfc-editor.org/info/rfc4023>. | <https://www.rfc-editor.org/info/rfc4023>. | |||
skipping to change at page 38, line 33 ¶ | skipping to change at page 39, line 38 ¶ | |||
Chaining (SFC) Architecture", RFC 7665, | Chaining (SFC) Architecture", RFC 7665, | |||
DOI 10.17487/RFC7665, October 2015, | DOI 10.17487/RFC7665, October 2015, | |||
<https://www.rfc-editor.org/info/rfc7665>. | <https://www.rfc-editor.org/info/rfc7665>. | |||
[RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., | [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., | |||
Uttaro, J., and W. Henderickx, "A Network Virtualization | Uttaro, J., and W. Henderickx, "A Network Virtualization | |||
Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, | Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, | |||
DOI 10.17487/RFC8365, March 2018, | DOI 10.17487/RFC8365, March 2018, | |||
<https://www.rfc-editor.org/info/rfc8365>. | <https://www.rfc-editor.org/info/rfc8365>. | |||
14.2. Informational References | 15.2. Informational References | |||
[consistent-hash] | [consistent-hash] | |||
Karger, D., Lehman, E., Leighton, T., Panigrahy, R., | Karger, D., Lehman, E., Leighton, T., Panigrahy, R., | |||
Levine, M., and D. Lewin, ""Consistent Hashing and Random | Levine, M., and D. Lewin, ""Consistent Hashing and Random | |||
Trees: Distributed Caching Protocols for Relieving Hot | Trees: Distributed Caching Protocols for Relieving Hot | |||
Spots on the World Wide Web"", 1997, <Proceedings of the | Spots on the World Wide Web"", 1997, <Proceedings of the | |||
Twenty-ninth Annual ACM Symposium on Theory of Computing. | Twenty-ninth Annual ACM Symposium on Theory of Computing. | |||
ACM Press New York, NY, USA. pp. 654--663>. | ACM Press New York, NY, USA. pp. 654--663>. | |||
[draft-ietf-idr-link-bandwidth] | [draft-ietf-idr-link-bandwidth] | |||
End of changes. 36 change blocks. | ||||
62 lines changed or deleted | 102 lines changed or added | |||
This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ |