draft-ietf-rddp-applicability-08.txt   rfc5045.txt 
Remote Direct Data Placement C. Bestler Network Working Group C. Bestler, Ed.
Working group Broadcom Corporation Request for Comments: 5045 Neterion
Internet-Draft L. Coene Category: Informational L. Coene
Expires: December 23, 2006 Siemens Nokia Siemens Networks
June 21, 2006 October 2007
Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct
Data Placement (DDP)
draft-ietf-rddp-applicability-08.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on December 23, 2006. Applicability of Remote Direct Memory Access Protocol (RDMA)
and Direct Data Placement Protocol (DDP)
Copyright Notice Status of This Memo
Copyright (C) The Internet Society (2006). This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Abstract Abstract
This document describes the applicability of Remote Direct Memory This document describes the applicability of Remote Direct Memory
Access Protocol (RDMAP) and the Direct Data Placement Protocol (DDP). Access Protocol (RDMAP) and the Direct Data Placement Protocol (DDP).
It compares and contrasts the different transport options over IP It compares and contrasts the different transport options over IP
that DDP can use, provides guidance to ULP developers on choosing that DDP can use, provides guidance to ULP developers on choosing
between available transports and/or how to be indifferent to the between available transports and/or how to be indifferent to the
specific transport layer used, compares use of DDP with direct use of specific transport layer used, compares use of DDP with direct use of
the supporting transports, and compares DDP over IP transports with the supporting transports, and compares DDP over IP transports with
non-IP transports that support RDMA functionality. non-IP transports that support RDMA functionality.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Direct Placement . . . . . . . . . . . . . . . . . . . . . . . 6 3. Direct Placement . . . . . . . . . . . . . . . . . . . . . . . 5
3.1. Direct Placement using only the LLP . . . . . . . . . . . 6 3.1. Direct Placement Using Only the LLP . . . . . . . . . . . 5
3.2. Fewer Required ULP Interactions . . . . . . . . . . . . . 7 3.2. Fewer Required ULP Interactions . . . . . . . . . . . . . 6
4. Tagged Messages . . . . . . . . . . . . . . . . . . . . . . . 8 4. Tagged Messages . . . . . . . . . . . . . . . . . . . . . . . 6
4.1. Order Independent Reception . . . . . . . . . . . . . . . 8 4.1. Order-Independent Reception . . . . . . . . . . . . . . . 7
4.2. Reduced ULP Notifications . . . . . . . . . . . . . . . . 9 4.2. Reduced ULP Notifications . . . . . . . . . . . . . . . . 7
4.3. Simplified ULP Exchanges . . . . . . . . . . . . . . . . . 9 4.3. Simplified ULP Exchanges . . . . . . . . . . . . . . . . . 8
4.4. Order Independent Sending . . . . . . . . . . . . . . . . 11 4.4. Order-Independent Sending . . . . . . . . . . . . . . . . 9
4.5. Untagged Messages and Tagged Buffers as ULP Credits . . . 12 4.5. Untagged Messages and Tagged Buffers as ULP Credits . . . 10
5. RDMA Read . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5. RDMA Read . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6. LLP Comparisons . . . . . . . . . . . . . . . . . . . . . . . 15 6. LLP Comparisons . . . . . . . . . . . . . . . . . . . . . . . 13
6.1. Multistreaming Implications . . . . . . . . . . . . . . . 15 6.1. Multistreaming Implications . . . . . . . . . . . . . . . 13
6.2. Out of Order Reception Implications . . . . . . . . . . . 15 6.2. Out-of-Order Reception Implications . . . . . . . . . . . 13
6.3. Header and Marker Overhead . . . . . . . . . . . . . . . . 15 6.3. Header and Marker Overhead . . . . . . . . . . . . . . . . 13
6.4. Middlebox Support . . . . . . . . . . . . . . . . . . . . 15 6.4. Middlebox Support . . . . . . . . . . . . . . . . . . . . 14
6.5. Processing Overhead . . . . . . . . . . . . . . . . . . . 16 6.5. Processing Overhead . . . . . . . . . . . . . . . . . . . 14
6.6. Data Integrity Implications . . . . . . . . . . . . . . . 16 6.6. Data Integrity Implications . . . . . . . . . . . . . . . 14
6.6.1. MPA/TCP Specifics . . . . . . . . . . . . . . . . . . 16 6.6.1. MPA/TCP Specifics . . . . . . . . . . . . . . . . . . 15
6.6.2. SCTP Specifics . . . . . . . . . . . . . . . . . . . . 17 6.6.2. SCTP Specifics . . . . . . . . . . . . . . . . . . . . 15
6.7. Non-IP Transports . . . . . . . . . . . . . . . . . . . . 17 6.7. Non-IP Transports . . . . . . . . . . . . . . . . . . . . 15
6.7.1. No RDMA Layer Ack . . . . . . . . . . . . . . . . . . 17 6.7.1. No RDMA-Layer Ack . . . . . . . . . . . . . . . . . . 16
6.8. Other IP Transports . . . . . . . . . . . . . . . . . . . 18 6.8. Other IP Transports . . . . . . . . . . . . . . . . . . . 16
6.9. LLP Independent Session Establishment . . . . . . . . . . 19 6.9. LLP-Independent Session Establishment . . . . . . . . . . 17
6.9.1. RDMA-only Session Establishment . . . . . . . . . . . 19 6.9.1. RDMA-Only Session Establishment . . . . . . . . . . . 17
6.9.2. RDMA-Conditional Session Establishment . . . . . . . . 19 6.9.2. RDMA-Conditional Session Establishment . . . . . . . . 18
7. Local Interface Implications . . . . . . . . . . . . . . . . . 21 7. Local Interface Implications . . . . . . . . . . . . . . . . . 18
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19
9. Security considerations . . . . . . . . . . . . . . . . . . . 23 8.1. Connection/Association Setup . . . . . . . . . . . . . . . 19
9.1. Connection/Association Setup . . . . . . . . . . . . . . . 23 8.2. Tagged Buffer Exposure . . . . . . . . . . . . . . . . . . 19
9.2. Tagged Buffer Exposure . . . . . . . . . . . . . . . . . . 23 8.3. Impact of Encrypted Transports . . . . . . . . . . . . . . 19
9.3. Impact of Encrypted Transports . . . . . . . . . . . . . . 23 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25 9.1. Normative References . . . . . . . . . . . . . . . . . . . 19
10.1. Normative references . . . . . . . . . . . . . . . . . . . 25 9.2. Informative References . . . . . . . . . . . . . . . . . . 19
10.2. Informative References . . . . . . . . . . . . . . . . . . 25
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26
Intellectual Property and Copyright Statements . . . . . . . . . . 27
1. Introduction 1. Introduction
Remote Direct Memory Access Protocol (RDMAP) and Direct Data Remote Direct Memory Access Protocol (RDMAP) [RFC5040] and Direct
Placement (DDP) work together to provide application independent Data Placement (DDP) [RFC5041] work together to provide application-
efficient placement of application payload directly into buffers independent efficient placement of application payload directly into
specified by the Upper Layer Protocol (ULP). buffers specified by the Upper Layer Protocol (ULP).
The DDP protocol is responsible for direct placement of received The DDP protocol is responsible for direct placement of received
payload into ULP specified buffers. The RDMAP protocol provides payload into ULP-specified buffers. The RDMAP protocol provides
completion notifications to the ULP and support for Data Sink completion notifications to the ULP and support for Data-Sink-
initiated fetch of advertised buffers (RDMA Reads). initiated fetch of Advertised Buffers (RDMA Reads).
DDP and RDMAP are both application independent protocols which allow DDP and RDMAP are both application-independent protocols that allow
the ULP to perform remote direct data placement. DDP can use the ULP to perform remote direct data placement. DDP can use
multiple standard IP transports including SCTP and TCP. multiple standard IP transports including SCTP and TCP.
By clarifying the situations where the functionality of these By clarifying the situations where the functionality of these
protocols are applicable, this document can guide implementers, protocols is applicable, this document can guide implementers and
application and protocol designers in selecting which protocols to application and protocol designers in selecting which protocols to
use. use.
The applicability of RDMAP/DDP is driven by their unique The applicability of RDMAP/DDP is driven by their unique
capabilities: capabilities:
o The existence of an application independent protocol allows common o This document will discuss when common data placement procedures
solutions to be implemented in hardware and/or the kernel. This are of more benefit to applications than application-specific
document will discuss when common data placement procedures are of solutions built on top of direct use of the underlying transport.
the greatest benefit to applications as contrasted with
application specific solutions built on top of direct use of the
underlying transport.
o DDP supports both untagged and tagged buffers. Tagged buffers o DDP supports both Untagged and Tagged Buffers. Tagged Buffers
allow the Data Sink ULP to be indifferent to what order (or in allow the Data Sink ULP to be indifferent to what order (or in
what messages) the Data Source sent the data, or what order what messages) the Data Source sent the data, or in what order
packets are received in. Typically tagged data can be used for packets are received. Typically, tagged data can be used for
payload transfer, while untagged is best used for control payload transfer, while untagged is best used for control
messages. However each upper layer protocol can determine the messages. However each upper-layer protocol can determine the
optimal use of tagged and untagged messages for itself. This optimal use of Tagged and Untagged Messages for itself. This
document will discuss when Data Source flexibility is of benefit document will discuss when Data Source flexibility is of benefit
to applications. to applications.
o RDMAP consolidates ULP notifications, thereby minimizing the o RDMAP consolidates ULP notifications, thereby minimizing the
number of required ULP interactions. number of required ULP interactions.
o RDMAP defines RDMA Reads, which allow remote access to advertised o RDMAP defines RDMA Reads, which allow remote access to Advertised
buffers. This document will review the advantages of using RDMA Buffers. This document will review the advantages of using RDMA
Reads as contrasted to alternate solutions. Reads as contrasted to alternate solutions.
A more comprehensive introduction to the RDMAP and DDP protocols
and discussion of their security considerations can be found in
[RFC5042].
Some non-IP transports, such as InfiniBand, directly integrate RDMA Some non-IP transports, such as InfiniBand, directly integrate RDMA
features. This document will review the applicability of providing features. This document will review the applicability of providing
RDMA services over ubiquitous IP transports as opposed to the use of RDMA services over ubiquitous IP transports instead of over
customized transport protocols. Due to the fact that DDP is defined customized transport protocols. Due to the fact that DDP is defined
cleanly as a layer over existing IP transports, DDP has simpler cleanly as a layer over existing IP transports, DDP has simpler
ordering rules than some prior RDMA protocols. This may have some ordering rules than some prior RDMA protocols. This may have some
implications for application designers. implications for application designers.
The full capabilities of DDP and RDMAP can only be fully realized by The full capabilities of DDP and RDMAP can only be fully realized by
applications that are designed to exploit them. The co-existence of applications that are designed to exploit them. The coexistence of
RDMAP/DDP aware local interfaces with traditional socket interfaces RDMAP/DDP-aware local interfaces with traditional socket interfaces
will also be explored. will also be explored.
Finally, DDP support is defined for at least two IP transports: SCTP Finally, DDP support is defined for at least two IP transports: SCTP
and TCP. The rationale for supporting both transports is reviewed, [RFC5043] and TCP [RFC5044]. The rationale for supporting both
as well as when each would be the appropriate selection. transports is reviewed, as well as when each would be the appropriate
selection.
2. Definitions 2. Definitions
Advertisement - the act of informing a Remote Peer that a local RDMA Advertisement - the act of informing a Remote Peer that a local RDMA
Buffer is available to it. A Node makes available an RDMA Buffer Buffer is available to it. A Node makes available an RDMA Buffer
for incoming RDMA Read or RDMA Write access by informing its RDMA/ for incoming RDMA Read or RDMA Write access by informing its RDMA/
DDP peer of the Tagged Buffer identifiers (STag, base address, and DDP peer of the Tagged Buffer identifiers (STag, base address, and
buffer length). This advertisement of Tagged Buffer information buffer length). This Advertisement of Tagged Buffer information
is not defined by RDMA/DDP and is left to the ULP. A typical is not defined by RDMA/DDP and is left to the ULP. A typical
method would be for the Local Peer to embed the Tagged Buffer's method would be for the Local Peer to embed the Tagged Buffer's
Steering Tag, base address, and length in a Send Message destined Steering Tag, base address, and length in a Send Message destined
for the Remote Peer. for the Remote Peer.
Data Sink - The peer receiving a data payload. Note that the Data Data Sink - The peer receiving a data payload. Note that the Data
Sink can be required to both send and receive RDMA/DDP Messages to Sink can be required to both send and receive RDMA/DDP Messages to
transfer a data payload. transfer a data payload.
Data Source - The peer sending a data payload. Note that the Data Data Source - The peer sending a data payload. Note that the Data
Source can be required to both send and receive RDMA/DDP Messages Source can be required to both send and receive RDMA/DDP Messages
to transfer a data payload. to transfer a data payload.
Lower Layer Protocol (LLP) The transport protocol that provides Lower Layer Protocol (LLP) - The transport protocol that provides
services to DDP. This is an IP transport with any required services to DDP. This is an IP transport with any required
adaptation layer. Adaptation layers are defined for SCTP and TCP. adaptation layer. Adaptation layers are defined for SCTP and TCP.
Steering Tag (STag) An identifier of a Tagged Buffer on a Node, valid Steering Tag (STag) - An identifier of a Tagged Buffer on a Node,
as defined within a protocol specification. valid as defined within a protocol specification.
Tagged Message A DDP message that is directed to a ULP specified Tagged Message - A DDP message that is directed to a ULP-specified
buffer based upon imbedded addressing information. In the buffer based upon imbedded addressing information. In the
immediate sense, the destination buffer is specified by the immediate sense, the destination buffer is specified by the
message sender. The message receiver is given no independent message sender. The message receiver is given no independent
indication that a tagged message has been received. indication that a Tagged Message has been received.
Untagged Message A DDP message that is directed to a ULP specified Untagged Message - A DDP message that is directed to a ULP-specified
buffer based upon a Message Sequence Number being matched with a buffer based upon a Message Sequence Number being matched with a
receiver supplied buffer. The destination buffer is specified by receiver-supplied buffer. The destination buffer is specified by
the message receiver. The message receiver is notified by some the message receiver. The message receiver is notified by some
mechanism that an untagged message has been received. mechanism that an Untagged Message has been received.
Upper Layer Protocol (ULP) The direct user of RDMAP/DDP services. In Upper Layer Protocol (ULP) - The direct user of RDMAP/DDP services.
addition to protocols such as iSER [7] and NFSv4 over RDMA [8], In addition to protocols such as iSER [RFC5046] and NFSv4 over
the ULP may be embedded in an application, or a middleware layer RDMA [NFSDIRECT], the ULP may be embedded in an application or a
as is often the case for the Sockets Direct Protocol (SDP) and middleware layer, as is often the case for the Sockets Direct
Remote Procedure Call (RPC) protocols. Protocol (SDP) and Remote Procedure Call (RPC) protocols.
3. Direct Placement 3. Direct Placement
Direct Data Placement optimizes the placement of ULP payload into the Direct Data Placement optimizes the placement of ULP Payload into the
correct destination buffers, typically eliminating intermediate correct destination buffers, typically eliminating intermediate
copying. Placement is enabled without regard to order of arrival, copying. Placement is enabled without regard to order of arrival,
order of transmission or requiring per-placement interaction with the order of transmission, or requirement of per-placement interaction
ULP. with the ULP.
RDMAP minimizes the required ULP interactions . This capability is RDMAP minimizes the required ULP interactions . This capability is
most valuable for applications that require multiple transport layer most valuable for applications that require multiple transport layer
packets for each required ULP interaction. packets for each required ULP interaction.
3.1. Direct Placement using only the LLP 3.1. Direct Placement Using Only the LLP
Direct data placement can be achieved without RDMA. Pre-posting of Direct data placement can be achieved without RDMA. Pre-posting of
receive buffers could allow a non-RDMA network stack to place data receive buffers could allow a non-RDMA network stack to place data
directly to user buffers. directly to user buffers.
The degree to which DDP optimizes depends on which transport is being The degree to which DDP optimizes depends on which transport it is
compared with, and on the nature of the local interface. Without being compared with, and on the nature of the local interface.
RDMAP/DDP pre-posting buffers requires the receiving side to Without RDMAP/DDP, pre-posting buffers require the receiving side to
accurately predict the required buffers and their sizes. This is not accurately predict the required buffers and their sizes. This is not
feasible for all ULPs. By contrast, DDP only requires the ULP to feasible for all ULPs. By contrast, DDP only requires the ULP to
predict the sequence and size of incoming untagged messages. predict the sequence and size of incoming Untagged Messages.
An application that could predict incoming messages and required An application that could predict incoming messages and required
nothing more than direct placement into buffers might be able to do nothing more than direct placement into buffers might be able to do
so with a properly designed local interface to native SCTP or TCP so with a properly designed local interface to native SCTP or TCP
(without RDMA). This is easier using native SCTP because the (without RDMA). This is easier using native SCTP because the
application would only have to predict the sequence of messages and application would only have to predict the sequence of messages and
the maximum size of each message, not the exact size of each message. the maximum size of each message, not the exact size.
The main benefit of DDP for such an application would be that pre- The main benefit of DDP for such an application would be that pre-
posting of receive buffers is a mandated local interface capability, posting of receive buffers is a mandated local interface capability,
and that predictions can always be made on a per-message basis (not and that predictions can always be made on a per-message basis (not
per byte). per byte).
The Lower Layer Protocol, LLP, can also be used directly if ULP The Lower Layer Protocol, LLP, can also be used directly if ULP-
specific knowledge is built into the protocol stack to allow "parse specific knowledge is built into the protocol stack to allow "parse
and place" handling of received packets. Such a solution either and place" handling of received packets. Such a solution either
requires interaction with the ULP, or that the protocol stack have requires interaction with the ULP or the protocol stack's knowledge
knowledge of ULP specific syntax rules. of ULP-specific syntax rules.
DDP achieves the benefits of directly placing incoming payload DDP achieves the benefits of directly placing incoming payload
without requiring tight coupling between the ULP and the protocol without requiring tight coupling between the ULP and the protocol
stack. However, "parse and place" capabilities can certainly provide stack. However, "parse and place" capabilities can certainly provide
equivalent services to a limited number of ULPs. equivalent services to a limited number of ULPs.
3.2. Fewer Required ULP Interactions 3.2. Fewer Required ULP Interactions
While reducing the number of required ULP interactions is in itself While reducing the number of required ULP interactions is in itself
desirable, it is critical for high speed connections. The burst desirable, it is critical for high-speed connections. The burst
packet rate for a high speed interface could easily exceed the host packet rate for a high-speed interface could easily exceed the host
systems ability to switch ULP contexts. system's ability to switch ULP contexts.
Content access applications are primary examples of applications with Content access applications are important examples of applications
both high bandwidth and a high ratio of content transferred per that require high bandwidth and can transfer a significant amount of
required ULP interaction. These applications include file access content between required ULP interactions. These applications
protocols (NAS), storage access (SAN), database access and other include file access protocols (NAS), storage access (SAN), database
application specific forms of content access such as HTTP, XML and access, and other application-specific forms of content access such
email. as HTTP, XML, and email.
4. Tagged Messages 4. Tagged Messages
This section covers the major benefits from the use of Tagged This section covers the major benefits from the use of Tagged
Messages. Messages.
A more critical advantage of DDP is the ability of the Data Source to A more critical advantage of DDP is the ability of the Data Source to
use tagged buffers. Tagging messages allows the Data Source to use Tagged Buffers. Tagging messages allows the Data Source to
choose the ordering and packetization of its payload deliveries. choose the ordering and packetization of its payload deliveries.
With direct data placement based solely upon pre-posted receives, the With direct data placement based solely upon pre-posted receives, the
packetization and delivery of payload must be agreed by the ULP peers packetization and delivery of payload must be agreed by the ULP peers
in advance. in advance.
The Upper Layer Protocol can allocate content between untagged and/or The Upper Layer Protocol can allocate content between Untagged and/or
tagged messages to maximize the potential optimizations. Placing Tagged Messages to maximize the potential optimizations. Placing
content within an untagged message can deliver the content in the content within an Untagged Message can deliver the content in the
same packet that signals completion to the receiver. This can same packet that signals completion to the receiver. This can
improve latency. It can even eliminate round trips. But it requires improve latency. It can even eliminate round trips. But it requires
making larger anonymous buffers to be available. making larger anonymous buffers to be available.
Some examples of data that typically belongs in the untagged message Some examples of data that typically belongs in the Untagged Message
would include: would include:
short fixed-size control data that is inherently part of the short fixed-size control data that is inherently part of the
control message. This is especially true when the data is a control message. This is especially true when the data is a
required part of the control message. required part of the control message.
relatively short payload that is almost always needed, especially relatively short payload that is almost always needed, especially
when its inclusion would eliminate a round-trip to fetch the data. when its inclusion would eliminate a round-trip to fetch the data.
Examples would include the initial data on a write request and Examples would include the initial data on a write request and
advertisements of tagged buffers. Advertisements of Tagged Buffers.
Tagged messages standardizes direct placement of data without per- Tagged Messages standardize direct placement of data without per-
packet interaction with the upper layers. Even if there is an upper packet interaction with the upper layers. Even if there is an upper-
layer protocol encoding of what is being transferred, as is common layer protocol encoding of what is being transferred, as is common
with middleware solutions, this information is not understood at the with middleware solutions, this information is not understood at the
application independent layers. The directions on where to place the application-independent layers. The directions on where to place the
incoming data cannot be accessed without switching to the ULP first. incoming data cannot be accessed without switching to the ULP first.
DDP provides a standardized 'packing list' which can be interpreted DDP provides a standardized 'packing list', which can be interpreted
without requiring ULP interaction. Indeed, it is designed to be without requiring ULP interaction. Indeed, it is designed to be
implementable in hardware. implementable in hardware.
4.1. Order Independent Reception 4.1. Order-Independent Reception
Tagged messages are directed to a buffer based on an included Tagged Messages are directed to a buffer based on an included
Steering Tag. Additionally, no notice is provided to the ULP for each Steering Tag. Additionally, no notice is provided to the ULP for
individual Tagged Message's arrival. Together these allow tagged each individual Tagged Message's arrival. Together these allow
messages received out-of-order to be processed without intermediate Tagged Messages received out of order to be processed without
buffering or additional notifications to the ULP. intermediate buffering or additional notifications to the ULP.
4.2. Reduced ULP Notifications 4.2. Reduced ULP Notifications
RDMAP offers both tagged and untagged messages. No receiving side RDMAP offers both Tagged and Untagged Messages. No receiving-side
ULP interactions are required for tagged messages. By optimally ULP interactions are required for Tagged Messages. By optimally
dividing traffic between tagged and untagged messages the ULP can dividing traffic between Tagged and Untagged Messages, the ULP can
limit the number of events that must be dealt with at the ULP layer. limit the number of events that must be dealt with at the ULP layer.
This typically reduces the number of context switches required and This typically reduces the number of context switches required and
improves performance. improves performance.
RDMAP further reduces required ULP interactions consolidating RDMAP further reduces required ULP interactions, consolidating
completion notifications of tagged messages with the completion completion notifications of Tagged Messages with the completion
notification of a trailing untagged message. For most ULPs this notification of a trailing Untagged Message. For most ULPs, this
radically reduces the number of ULP required interactions even radically reduces the number of ULP required interactions even
further. further.
While RDMAP consolidation of notices is beneficial to most While RDMAP consolidation of notices is beneficial to most
applications, it may be detrimental to some applications that benefit applications, it may be detrimental to some applications that benefit
from streamed delivery to enable ULP processing of received data as from streamed delivery to enable ULP processing of received data as
promptly as possible. A ULP that uses RDMAP cannot begin processing promptly as possible. A ULP that uses RDMAP cannot begin processing
any portion of an exchange until it receives notification that the any portion of an exchange until it receives notification that the
entire exchange has been placed. An "exchange" here is a set of zero entire exchange has been placed. An "exchange" here is a set of zero
or more tagged messages and a single terminating untagged message. or more Tagged Messages and a single terminating Untagged Message.
An application that would prefer to begin work on the received An application that would prefer to begin work on the received
payload, no matter what order it arrived in, as soon as possible payload as soon as possible, no matter what order it arrived in,
might prefer to work directly with the LLP. RDMAP is optimized for might prefer to work directly with the LLP. RDMAP is optimized for
applications that are more concerned when the entire exchange is applications that are more concerned when the entire exchange is
complete. complete.
An application that benefits from being able to begin processing of An application that benefits from being able to begin processing of
each received packet as quickly as possible may find RDMAP interferes each received packet as quickly as possible may find RDMAP interferes
with that goal. with that goal.
Such an application might be able to retain most of the benefits of Such an application might be able to retain most of the benefits of
RDMAP by using the DDP layer directly. However, in addition to RDMAP by using the DDP layer directly. However, in addition to
skipping to change at page 9, line 52 skipping to change at page 8, line 37
DDP, and might not provide an API directly to DDP services. DDP, and might not provide an API directly to DDP services.
These features minimize the required interactions with the ULP. This These features minimize the required interactions with the ULP. This
can be extremely beneficial for applications that use multiple can be extremely beneficial for applications that use multiple
transport layer packets to accomplish what is a single ULP transport layer packets to accomplish what is a single ULP
interaction. interaction.
4.3. Simplified ULP Exchanges 4.3. Simplified ULP Exchanges
The notification rules for Tagged Messages allows ULPs to create The notification rules for Tagged Messages allows ULPs to create
multi-message "exchanges" consisting of zero or more tagged messages multi-message "exchanges" consisting of zero or more Tagged Messages
that represent a single step in the ULP interaction. The receiving that represent a single step in the ULP interaction. The receiving
ULP is notified that the untagged message has arrived, and implicitly ULP is notified that the Untagged Message has arrived, and implicitly
of any associated tagged messages. notified of any associated Tagged Messages.
A ULP where all exchanges would naturally be untagged messages would If a ULP cannot effectively use Tagged Messages, it would derive
derive virtually no benefit from the use of RDMAP/DDP as opposed to little benefit from use of RDMAP/DDP by comparison to direct use of
SCTP directly. But while tagged buffers are the justification for SCTP. But, while Tagged Buffers are the justification for RDMAP/DDP,
RDMAP/DDP, untagged buffers are still necessary. Without untagged Untagged Buffers are still necessary. Without Untagged Buffers, the
buffers the only method to exchange buffer advertisements would only method to exchange buffer Advertisements would require out-of-
require out-of-band communications. Most RDMA-aware ULPs use band communications. Most RDMA-aware ULPs use Untagged Buffers for
untagged buffers for requests and responses. Buffer advertisements requests and responses. Buffer Advertisements are typically done
are typically done within these untagged messages. within these Untagged Messages.
More importantly there would be no reliable method for the upper More importantly, there would be no reliable method for the upper-
layer peers to synchronize. The absence of any guarantees about layer peers to synchronize. The absence of any guarantees about
ordering within or between tagged messages is fundamental to allowing ordering within or between Tagged Messages is fundamental to allowing
the DDP layer to optimize transfer of tagged payload. the DDP layer to optimize transfer of tagged payload.
So no ULP can be defined entirely in terms of tagged messages. Therefore, no ULP can be defined entirely in terms of Tagged
Eventually a notification that confirms delivery must be generated Messages. Eventually, a notification that confirms delivery must be
from the RDMAP/DDP layer. generated from the RDMAP/DDP layer.
Limiting use of untagged buffers to requests and responses by moving Limiting use of Untagged Buffers to requests and responses by moving
all bulk data using tagged transfers can greatly simplify the amount all bulk data using tagged transfers can greatly simplify the amount
of prediction that the Data Sink must perform in pre-posting receive of prediction that the Data Sink must perform in pre-posting receive
buffers. For example, a typical RDMA enabled interaction would buffers. For example, a typical RDMA-enabled interaction would
consist of the following: consist of the following:
Client sends transaction request to server's as an untagged 1. Client sends transaction request to server as an Untagged
message. Message.
This message includes buffer advertisements for the buffers where 2. This message includes buffer Advertisements for the buffers where
the results are to be placed. the results are to be placed.
The Server sends multiple tagged messages to the advertised 3. The server sends multiple Tagged Messages to the Advertised
buffers. buffers.
The Server sends transaction reply as an untagged message to the 4. The server sends transaction reply as an Untagged Message to the
client. client.
Client receives single notification, indicating completion of the 5. Client receives single notification, indicating completion of the
interaction. interaction.
With this type of exchange the pacing and required size of untagged With this type of exchange, the pacing and required size of Untagged
buffers is highly predictable. The variability of response sizes is Buffers are highly predictable. The variability of response sizes is
absorbed by tagged transfers. absorbed by tagged transfers.
4.4. Order Independent Sending 4.4. Order-Independent Sending
Use of tagged messages is especially applicable when the Data Sink Use of Tagged Messages is especially applicable when the Data Sink
does not know the actual size, structure or location of the content does not know the actual size, structure, or location of the content
it is requesting (or updating). it is requesting (or updating).
For example, suppose the Data Sink ULP needs to fetch four related For example, suppose the Data Sink ULP needs to fetch four related
pieces of data into a four separate buffers. With SCTP the Data Sink pieces of data into four separate buffers. With SCTP, the Data Sink
ULP could receive four messages into four separate buffers, only ULP could receive four messages into four separate buffers, only
having to predict the maximum size of each. However it would have to having to predict the maximum size of each. However, it would have
dictate the order in which the Data Source supplied the separate to dictate the order in which the Data Source supplied the separate
pieces. If the Data Source found it advantageous to fetch them in a pieces. If the Data Source found it advantageous to fetch them in a
different order it would have to use intermediate buffering to re- different order, it would have to use intermediate buffering to re-
order the pieces into the expected order even though the application order the pieces into the expected order even though the application
only required that all four be delivered and did not truly have an only required that all four be delivered and did not truly have an
ordering requirement. ordering requirement.
Techniques such as RAID striping and mirroring represent this same Techniques, such as RAID striping and mirroring, represent this same
problem, but one step further. What appears to be a single resource problem, but one step further. What appears to be a single resource
to the Data Sink is actually stored in separate locations by the Data to the Data Sink is actually stored in separate locations by the Data
Source. Non RDMA protocols would either require the Data Source to Source. Non RDMA protocols would either require the Data Source to
fetch the material in the desired order or force the Data Source to fetch the material in the desired order or force the Data Source to
use its own holding buffers to assemble an image of the destination use its own holding buffers to assemble an image of the destination
buffer. buffer.
While sometimes referred to as a "buffer-to-buffer" solution, RDMA While sometimes referred to as a "buffer-to-buffer" solution, RDMA
more fundamentally enables remote buffer access. The ULP is free to more fundamentally enables remote buffer access. The ULP is free to
work with larger remote buffers than it has locally. This reduces work with larger remote buffers than it has locally. This reduces
buffering requirements and the number of times the data must be buffering requirements and the number of times the data must be
copied in an end-to-end transfer. copied in an end-to-end transfer.
There are numerous reasons why the Data Sink would not know the true There are numerous reasons why the Data Sink would not know the true
order or location of the requested data. It could be different for order or location of the requested data. It could be different for
each client, different records selected and/or different sort orders, each client, different records selected and/or different sort orders,
RAID striping, file fragmentation, volume fragmentation, volume as well as RAID striping, file fragmentation, volume fragmentation,
mirroring and server-side dynamic compositing of content (such as volume mirroring, and server-side dynamic compositing of content
server side includes for HTTP). (such as server-side includes for HTTP).
In all of these cases the Data Source is free to assemble the desired In all of these cases, the Data Source is free to assemble the
data in the Data Sink's buffer in whatever order the component data desired data in the Data Sink's buffer in whatever order the
becomes available to it. It is not constrained on ordering. It does component data becomes available to it. It is not constrained on
not have to assemble an image in its own memory before creating it in ordering. It does not have to assemble an image in its own memory
the Data Sink's buffers. before creating it in the Data Sink's buffers.
Note that while DDP enables use of tagged messages for bulk transfer, Note that while DDP enables use of Tagged Messages for bulk transfer,
there are some application scenarios where untagged messages would there are some application scenarios where Untagged Messages would
still be used for bulk transfer. For example, a file server may not still be used for bulk transfer. For example, a file server may not
expose its own memory to its clients. A client wishing to write may expose its own memory to its clients. A client wishing to write may
advertise a buffer which the server will issue RDMA Reads upon. Advertise a buffer upon which the server will issue RDMA Reads.
However, when performing a small write it may be preferable to However, when performing a small write, it may be preferable to
include the data in the untagged message rather than incurring an include the data in the Untagged Message rather than incurring an
additional round trip with the RDMA Read and its response. additional round trip with the RDMA Read and its response.
Generally, the best use of an untagged message is to synchronize and Generally, the best use of an Untagged Message is to synchronize and
to deliver data that is naturally tied to the same message as the to deliver data that is naturally tied to the same message as the
synchronization. For initial data transfers this has the additional synchronization. For initial data transfers, this has the additional
benefit of avoiding the need to advertise specific tagged buffers for benefit of avoiding the need to Advertise specific Tagged Buffers for
indefinite time periods. Instead anonymous buffers can be used for indefinite time periods. Instead, anonymous buffers can be used for
initial data reception. Because anonymous buffers do not need to be initial data reception. Because anonymous buffers do not need to be
tied to specific messages in advance this can be a major benefit. tied to specific messages in advance, this can be a major benefit.
4.5. Untagged Messages and Tagged Buffers as ULP Credits 4.5. Untagged Messages and Tagged Buffers as ULP Credits
The handling of end-to-end buffer credits differs considerably with The handling of end-to-end buffer credits differs considerably with
DDP than when the ULP directly uses either TCP or SCTP. DDP than when the ULP directly uses either TCP or SCTP.
With both TCP and SCTP buffer credits are based upon the receiver With both TCP and SCTP, buffer credits are based upon the receiver
granting transmit permission based on the total number of bytes. granting transmit permission based on the total number of bytes.
These credits reflect system buffering resources and/or simple flow These credits reflect system buffering resources and/or simple flow
control. They do not represent ULP resources. control. They do not represent ULP resources.
DDP defines no standard flow control, but presumes the existince of a DDP defines no standard flow control, but presumes the existence of a
ULP mechanism. The presumed mechanism is that the Data Sink ULP has ULP mechanism. The presumed mechanism is that the Data Sink ULP has
issued credits to the Data Source allowing the Data Source to send a issued credits to the Data Source, allowing the Data Source to send a
specific number of untagged messages. specific number of Untagged Messages.
The ULP peers must ensure that the sender is aware of the maximum The ULP peers must ensure that the sender is aware of the maximum
size that can be sent to any specific target buffer. One method of size that can be sent to any specific target buffer. One method of
doing so is to use a standard size for all untagged buffers within a doing so is to use a standard size for all Untagged Buffers within a
given connection. For example, a ULP may specify an initial untagged given connection. For example, a ULP may specify an initial Untagged
buffer size to be used immediately after session establishment, and Buffer size to be used immediately after session establishment, and
then optionally specify mechanisms for negotiating changes. then optionally specify mechanisms for negotiating changes.
Tagged buffers are ULP resources advertised directly from ULP to ULP. Tagged Buffers are ULP resources Advertised directly from ULP to ULP.
A DDP put to a known tagged buffer is constrained only by transport A DDP put to a known Tagged Buffer is constrained only by transport
level flow control, not by available system buffering. level flow control, not by available system buffering.
Either tagged or untagged buffers allows bypassing of system buffer Either Tagged or Untagged Buffers allows bypassing of system buffer
resources. Use of tagged buffers additionally allows the Data Source resources. Use of Tagged Buffers additionally allows the Data Source
to choose what order to exercise the credits in. to choose in what order to exercise the credits.
To the extent allowed by the ULP, tagged buffers are also divisible To the extent allowed by the ULP, Tagged Buffers are also divisible
resources. The Data Sink can advertise a single 100 KB buffer, and resources. The Data Sink can Advertise a single 100 KB buffer, and
then receive notifications from its peer that it had written 50 KB, then receive notifications from its peer that it had written 50 KB,
20 KB and 30 KB to that buffer in three successive transactions. 20 KB, and 30 KB to that buffer in three successive transactions.
ULP-management of tagged buffer resources, independent of transport ULP management of Tagged Buffer resources, independent of transport
and DDP layer credits, is an additional benefit of RDMA protocols. and DDP layer credits, is an additional benefit of RDMA protocols.
Large bulk transfers cannot be blocked by limited general purpose Large bulk transfers cannot be blocked by limited general-purpose
buffering capacity. Applications can flow control based upon higher buffering capacity. Applications can flow control based upon higher
level abstractions, such as number of outstanding requests, level abstractions, such as number of outstanding requests,
independent of the amount of data that must be transferred. independent of the amount of data that must be transferred.
However, use of system buffering, as offered by direct use of the However, use of system buffering, as offered by direct use of the
underlying transports, can be preferable under certain circumstances. underlying transports, can be preferable under certain circumstances.
One example would be when the number of target ULP buffers is One example would be when the number of target ULP Buffers is
sufficiently large, and the rate at which any writes arrive is sufficiently large, and the rate at which any writes arrive is
sufficiently low, that pinning all the target ULP buffers in memory sufficiently low, that pinning all the target ULP Buffers in memory
would be undesirable. The maximum transfer rate, and hence the would be undesirable. The maximum transfer rate, and hence the
maximum amount of system buffering required, may be more stable and maximum amount of system buffering required, may be more stable and
predictable than the total ULP buffer exposure. predictable than the total ULP Buffer exposure.
Another would be the Data Sink wishes to receive a stream of data at Another example would be when the Data Sink wishes to receive a
a predictable rate, but does not know in advance what the size of stream of data at a predictable rate, but does not know in advance
each data packet will be. This is common from streaming media that what the size of each data packet will be. This is common from
has been encoded with a variable bit rate. With DDP the Data Sink streaming media that has been encoded with a variable bit rate. With
would either have to use untagged buffers large enough for the DDP, the Data Sink would either have to use Untagged Buffers large
largest packet, or advertise a circular buffer. If for security or enough for the largest packet, or Advertise a circular buffer. If,
other reasons the Data Sink did not want the size of its buffer to be for security or other reasons, the Data Sink did not want the size of
publicly known, using the underlying SCTP transport directly may be its buffer to be publicly known, using the underlying SCTP transport
preferable because of their byte-oriented credits. directly may be preferable because of its byte-oriented credits.
5. RDMA Read 5. RDMA Read
RDMA Reads are a further service provided by RDMAP. RDMA Reads allow RDMA Reads are a further service provided by RDMAP. RDMA Reads allow
the Data Sink to fetch exactly the portion of the peer ULP buffer the Data Sink to fetch exactly the portion of the peer ULP Buffer
required on a "just in time" basis. This can be done without required on a "just in time" basis. This can be done without
requiring per-fetch support from the Data Source ULP. requiring per-fetch support from the Data Source ULP.
Storage servers may wish to limit the maximum write buffer allocated Storage servers may wish to limit the maximum write buffer allocated
to any single session. The storage server may be a very minimal to any single session. The storage server may be a very minimal
layer between the client and the disk storage media, or the server layer between the client and the disk storage media, or the server
may merely wish to limit the total resources that would be required may merely wish to limit the total resources that would be required
if all clients could push the entire payload they wished written at if all clients could push the entire payload they wished written at
their own convenience. their own convenience.
In either case, there is little benefit in transferring data from the In either case, there is little benefit in transferring data from the
Data Source far in advance of when it will be written to the Data Source far in advance of when it will be written to the
persistent storage media. RDMA Reads allow the Storage Server to persistent storage media. RDMA Reads allow the Storage Server to
fetch the payload on a "just in time" basis. In this fashion a fetch the payload on a "just in time" basis. In this fashion, a
relatively small number of block sized buffers can be used to execute relatively small number of block-sized buffers can be used to execute
a single transaction that specified writing a large file, or a a single transaction that specified writing a large file, or a
Storage Server with numerous clients can fetch buffers from the Storage Server with numerous clients can fetch buffers from the
individual clients in the order that is most convenient to the individual clients in the order that is most convenient to the
server. server.
This same capability can be used when the desired portion of the This same capability can be used when the desired portion of the
advertised buffer is not known in advance. For example the Advertised Buffer is not known in advance. For example, the
advertised buffer could contain performance statistics. The data Advertised Buffer could contain performance statistics. The Data
sink could request the portions of the data it required, without Sink could request the portions of the data it required, without
requiring an interaction with the Data Source ULP. requiring an interaction with the Data Source ULP.
This is applicable for many applications that publish semi-volatile This is applicable for many applications that publish semi-volatile
data that does not require transactional validity checking (i.e., data that does not require transactional validity checking (i.e.,
authorized users have read access to the entire set of data). It is authorized users have read access to the entire set of data). It is
less applicable when there are ULP consistency checks that must be less applicable when there are ULP consistency checks that must be
performed upon the data. Such applications would be better served by performed upon the data. Such applications would be better served by
having the client send a request, and having the server use RDMA having the client send a request, and having the server use RDMA
Writes to publish the requested data. Neither RDMAP or DDP provide Writes to publish the requested data. Neither RDMAP nor DDP provide
mechanisms for bundling multiple disjoint updates into an atomic mechanisms for bundling multiple disjoint updates into an atomic
operation. Therefore use of an advertised buffer as a data resource operation. Therefore, use of an Advertised Buffer as a data resource
is subject to the same caveats as any randomly updated data resource, is subject to the same caveats as any randomly updated data resource,
such as flat files, that do not enforce their own consistency. such as flat files, that do not enforce their own consistency.
6. LLP Comparisons 6. LLP Comparisons
Normally the choice of underlying IP transport is irrelevant to the Normally, the choice of underlying IP transport is irrelevant to the
ULP. RDMAP and DDP provides the same services over either. There ULP. RDMAP and DDP provides the same services over either. There
may be performance impacts of the choice, however. It is the may be performance impacts of the choice, however. It is the
responsibility of the ULP to determine which IP transport is best responsibility of the ULP to determine which IP transport is best
suited to its needs. suited to its needs.
SCTP provides for preservation of message boundaries. Each DDP SCTP provides for preservation of message boundaries. Each DDP
segment will be delivered within a single SCTP packet. The Segment will be delivered within a single SCTP packet. The
equivalent services are only available with TCP through the use of equivalent services are only available with TCP through the use of
the MPA (Marker PDU Alignment) adaptation layer. the MPA (Marker PDU Alignment) adaptation layer.
6.1. Multistreaming Implications 6.1. Multistreaming Implications
SCTP also provides multi-streaming. When the same pair of hosts have SCTP also provides multi-streaming. When the same pair of hosts have
need for multiple DDP streams this can be a major advantage. A need for multiple DDP streams, this can be a major advantage. A
single SCTP association carries multiple DDP streams, consolidating single SCTP association carries multiple DDP streams, consolidating
connection setup, congestion control and acknowledgements. connection setup, congestion control, and acknowledgements.
Completions are controlled by the DDP Source Sequence Number (DDP- Completions are controlled by the DDP Source Sequence Number (DDP-
SSN) on a per stream basis. Therefore combining multiple DDP Streams SSN) on a per-stream basis. Therefore, combining multiple DDP
into a single SCTP association cannot result in a dropped packet Streams into a single SCTP association cannot result in a dropped
carrying data for one stream delaying completions on others. packet carrying data for one stream delaying completions on others.
6.2. Out of Order Reception Implications 6.2. Out-of-Order Reception Implications
The use of unordered Data Chunks with SCTP guarantees that the DDP The use of unordered Data Chunks with SCTP guarantees that the DDP
layer will be able to perform placements when IP datagrams are layer will be able to perform placements when IP datagrams are
received out of order. received out of order.
Placement of out-of-order DDP Segments carried over MPA/TCP is not Placement of out-of-order DDP Segments carried over MPA/TCP is not
guaranteed, but certainly allowed. The ability of the MPA receiver guaranteed, but certainly allowed. The ability of the MPA receiver
to process out-of-order DDP Segments may be impaired when alignment to process out-of-order DDP Segments may be impaired when alignment
of TCP segments and MPA FPDUs is lost. Using SCTP, each DDP Segment of TCP segments and MPA FPDUs is lost. Using SCTP, each DDP Segment
is encoded in a single Data Chunk and never spread over multiple IP is encoded in a single Data Chunk and never spread over multiple IP
datagrams. datagrams.
6.3. Header and Marker Overhead 6.3. Header and Marker Overhead
MPA and TCP headers together are smaller than the headers used by MPA and TCP headers together are smaller than the headers used by
SCTP and its adaptation layer. However, this advantage can be SCTP and its adaptation layer. However, this advantage can be
reduced by the insertion of MPA markers. The different in ULP reduced by the insertion of MPA markers. The difference in ULP
payload per IP Datagram is not likely to be a signifigant factor. Payload per IP Datagram is not likely to be a significant factor.
6.4. Middlebox Support 6.4. Middlebox Support
Even with the MPA adaptation layer, DDP traffic carried over MPA/TCP Even with the MPA adaptation layer, DDP traffic carried over MPA/TCP
will appear to all network middleboxes as a normal TCP connection. will appear to all network middleboxes as a normal TCP connection.
In many environments there may be a requirement to use only TCP In many environments, there may be a requirement to use only TCP
connections to satisfy existing network elements and/or to facilitate connections to satisfy existing network elements and/or to facilitate
monitoring and control of connections. While SCTP is certainly just monitoring and control of connections. While SCTP is certainly just
as monitorable and controllable as TCP, there is no guarantee that as monitorable and controllable as TCP, there is no guarantee that
the network management infrastructure has the required support for the network management infrastructure has the required support for
both. both.
6.5. Processing Overhead 6.5. Processing Overhead
A DDP stream delivered via MPA/TCP will require more processing A DDP stream delivered via MPA/TCP will require more processing
effort that one delivered over SCTP. However this extra work may be effort than one delivered over SCTP. However, this extra work may be
justified for many deployments where full SCTP support is unavailable justified for many deployments where full SCTP support is unavailable
in the endpoints of the network, or where middleboxes impair the in the endpoints of the network, or where middleboxes impair the
usability of SCTP. usability of SCTP.
6.6. Data Integrity Implications 6.6. Data Integrity Implications
Both the SCTP and MPA/TCP adaptation provide end-to-end CRC32c Both the SCTP [RFC4960] and MPA/TCP [RFC5044] adaptation provide end-
protection against data accidental corruption, or its equivalent. to-end CRC32c protection against data accidental corruption, or its
equivalent.
A ULP that requires a greater degree of protection may add it own. A ULP that requires a greater degree of protection may add its own.
However, DDP and RDMAP headers will only be guaranteed to have the However, DDP and RDMAP headers will only be guaranteed to have the
equivalent of end-to-end CRC32c protection. A ULP that requires data equivalent of end-to-end CRC32c protection. A ULP that requires data
integrity checking more thorough than an end-to-end CRC32c should integrity checking more thorough than an end-to-end CRC32c should
first invalidate all STags that reference a buffer before applying first invalidate all STags that reference a buffer before applying
their own integrity check. its own integrity check.
CRC32c only provides protection against random corruption. To CRC32c only provides protection against random corruption. To
protect against unauthorized alteration or forging of data packets protect against unauthorized alteration or forging of data packets,
security methods must be applied. The security draft [RDMA-Security] security methods must be applied. The RDMA security document
[6] specifies usage of RFC2406 [1] for both adaptation layers. [RFC5042] specifies usage of RFC 2406 [RFC2406] for both adaptation
layers. As stated in [RFC5042], note that the IPsec requirements for
RDDP are based on the version of IPsec specified in RFC 2401
[RFC2401] and related RFCs, as profiled by RFC 3723 [RFC3723],
despite the existence of a newer version of IPsec specified in RFC
4301 [RFC4301] and related RFCs.
6.6.1. MPA/TCP Specifics 6.6.1. MPA/TCP Specifics
It is mandatory for MPA/TCP implementations to implement CRC32c, but It is mandatory for MPA/TCP implementations to implement CRC32c, but
it is NOT mandatory to use the CRC32c during an RDMA connection. The it is not mandatory to use the CRC32c during an RDMA connection. The
activating or deactivating of the CRC in MPA/TCP is an administrative activating or deactivating of the CRC in MPA/TCP is an administrative
configuration operation at the local and remote end. The configuration operation at the local and remote end. The
administration of the CRC(ON/OFF) is invisible to the ULP. administration of the CRC(ON/OFF) is invisible to the ULP.
Applications SHOULD trust that this administrative option will only Applications should assume that disabling CRC32c will only be used
be used when the end-to-end protection is at least as effective as a when the end-to-end protection is at least as effective as a
transport layer CRC32c. Applications SHOULD NOT apply additional transport layer CRC32c. Applications should not use additional
protection as a guard against this administrative option being turned integrity checks based solely on the possibility that CRC32c could be
on inadvertently. disabled without equivalent integrity checks at a lower level.
Administrators MUST NOT enable CRC32c suppression unless the end-to- CRC32c must not be disabled unless equivalent or better end-to-end
end protection is truly equivalent. integrity protection is provided.
If the CRC is active/used for one direction/end , then the use of the If the CRC is active/used for one direction/end , then the use of the
CRC is mandatory in both directions/ends. CRC is mandatory in both directions/ends.
If both ends have been configured NOT to use the CRC, then this is If both ends have been configured not to use the CRC, then this is
allowed as long as an equivalent protection(comparable or better allowed as long as an equivalent protection (comparable to or better
than/to CRC) from undetected errors on the connection is provided. than CRC) from undetected errors on the connection is provided.
6.6.2. SCTP Specifics 6.6.2. SCTP Specifics
SCTP provides CRC32c protection automatically. The adaptation to SCTP provides CRC32c protection automatically. The adaptation to
SCTP provides for no option to suppress SCTP CRC32c protection. SCTP provides for no option to suppress SCTP CRC32c protection.
6.7. Non-IP Transports 6.7. Non-IP Transports
DDP is defined to operate over ubiquitous IP transports such as SCTP DDP is defined to operate over ubiquitous IP transports such as SCTP
and TCP. This enabled a new DDP-enabled node to be added anywhere to and TCP. This enables a new DDP-enabled node to be added anywhere to
an IP network. No DDP-specific support from middle-boxes is an IP network. No DDP-specific support from middleboxes is required.
required.
There are non-IP transport fabric offering RDMA capabilities. There are non-IP transport fabric offering RDMA capabilities.
Because these capabilities are integrated with the transport protocol Because these capabilities are integrated with the transport protocol
they have some technical advantages when compared to RDMA over IP. they have some technical advantages when compared to RDMA over IP.
For example fencing of RDMA operations can be based upon transport For example, fencing of RDMA Operations can be based upon transport
level acks. Because DDP is cleanly layered over an IP transport, any level acks. Because DDP is cleanly layered over an IP transport, any
explicit RDMA layer ack must be separate from the transport layer explicit RDMA layer ack must be separate from the transport layer
ack. ack.
There may be deployments where the benefits of RDMA/transport There may be deployments where the benefits of RDMA/transport
integration outweigh the benefits of being on an IP network. integration outweigh the benefits of being on an IP network.
6.7.1. No RDMA Layer Ack 6.7.1. No RDMA-Layer Ack
DDP does not provide for its own acknowledgements. The only form of DDP does not provide for its own acknowledgements. The only form of
ack provided at the RDMAP layer is an RDMA Read Response. DDP and ack provided at the RDMAP layer is an RDMA Read Response. DDP and
RDMAP rely almost entirely upon other layers for flow control and RDMAP rely almost entirely upon other layers for flow control and
pacing. The LLP is relied upon to guarantee delivery and avoid pacing. The LLP is relied upon to guarantee delivery and avoid
network congestion, and ULP level acking is relied upon for ULP network congestion, and ULP-level acking is relied upon for ULP
pacing and to avoid ULP buffer overruns. pacing and to avoid ULP Buffer overruns.
Previous RDMA protocols, such as InfiniBand, have been able to use Previous RDMA protocols, such as InfiniBand, have been able to use
their integration with the transport layer to provide stronger their integration with the transport layer to provide stronger
ordering guarantees. It is important that application designers that ordering guarantees. It is important that application designers that
require such guarantees to provide them through ULP interaction. require such guarantees provide them through ULP interaction.
Specifically: Specifically:
There is no ability for a local interface to "fence" outbound There is no ability for a local interface to "fence" outbound
messages to guarantee that prior tagged messages have been placed messages to guarantee that prior Tagged Messages have been placed
prior to sending a tagged message. The only guarantees available prior to sending a Tagged Message. The only guarantees available
from the other side would be an RDMA Read Response (coming from from the other side would be an RDMA Read Response (coming from
the RDMAP layer) or a response from the ULP layer. Remember that the RDMAP layer) or a response from the ULP layer. Remember that
the normal ordering rules only guarantee when the Data Sink ULP the normal ordering rules only guarantee when the Data Sink ULP
will be notified of untagged messages, it does not control when will be notified of Untagged Messages; it does not control when
data is placed into receive buffers. data is placed into receive buffers.
Re-use of tagged buffers must be done with extreme care. The fact Re-use of Tagged Buffers must be done with extreme care. The fact
that an untagged message indicates that all prior tagged messages that an Untagged Message indicates that all prior Tagged Messages
have been placed does not guarantee that no later tagged message have been placed does not guarantee that no later Tagged Message
have. The best strategy is to only change the state of any given has. The best strategy is to change only the state of any given
advertised buffers with with untagged messages. Advertised Buffers with Untagged Messages.
As covered elsewhere in this document, flow control of untagged As covered elsewhere in this document, flow control of Untagged
messages MUST be provided by the ULP itself. Messages is the responsibility of the ULP.
6.8. Other IP Transports 6.8. Other IP Transports
Both TCP and SCTP provide DDP with reliable transport with TCP Both TCP and SCTP provide DDP with reliable transport with TCP-
friendly rate control. As currently DDP is defined to work over friendly rate control. Currently, DDP is defined to work over
reliable transports and implicitly relies upon some form of rate reliable transports and implicitly relies upon some form of rate
control. control.
DDP is fully compatible with a non-reliable protocol. Out-of-order DDP is fully compatible with a non-reliable protocol. Out-of-order
placement is obviously not dependent on whether the other DDP placement is obviously not dependent on whether the other DDP
Segments ever actually arrive. Segments ever actually arrive.
However, RDMAP requires the LLP to provide reliable service. An However, RDMAP requires the LLP to provide reliable service. An
alternate completion handling protocol would be required if DDP were alternate completion handling protocol would be required if DDP were
to be deployed over an unreliable IP transport. to be deployed over an unreliable IP transport.
As noted in the prior section on tagged buffers as ULP credits, As noted in the prior section on Tagged Buffers as ULP credits,
neither RDMAP or DDP provide any flow control for tagged messages. neither RDMAP nor DDP provides any flow control for Tagged Messages.
If no transport layer flow control is provided, an RDMAP/DDP If no transport layer flow control is provided, an RDMAP/DDP
application would be only limited by the link layer rate, almost application would be limited only by the link layer rate, almost
inevitably resulting in severe network congestion. inevitably resulting in severe network congestion.
RDMAP encourages applications to be ignorant of the underlying RDMAP encourages applications to be ignorant of the underlying
transport PMTU. The ULP is only notified when all messages ending in transport path MTU. The ULP is only notified when all messages
a single untagged message have completed. The ULP is not aware of ending in a single Untagged Message have completed. The ULP is not
the granularity or ordering of the underlying message. This approach aware of the granularity or ordering of the underlying message. This
assumes that the ULP is only interested in the complete set of approach assumes that the ULP is only interested in the complete set
messages, and has no use for a subset of them. of messages, and has no use for a subset of them.
6.9. LLP Independent Session Establishment 6.9. LLP-Independent Session Establishment
For an RDMAP/DDP application, the transport services provided by a For an RDMAP/DDP application, the transport services provided by a
pair of SCTP Streams and by a TCP connection both provide the same pair of SCTP streams and by a TCP connection both provide the same
service (reliable delivery of DDP Segments between two connected service (reliable delivery of DDP Segments between two connected
RDMAP/DDP endpoints). RDMAP/DDP endpoints).
6.9.1. RDMA-only Session Establishment 6.9.1. RDMA-Only Session Establishment
It is also possible to allow for transport neutral establishment of It is also possible to allow for transport-neutral establishment of
RDMAP/DDP sessions between endpoints. Combined, these two features RDMAP/DDP sessions between endpoints. Combined, these two features
would allow most applications to be unconcerned as to which LLP was would allow most applications to be unconcerned as to which LLP was
actually in use. actually in use.
Specifically, the procedures for DDP Stream Session establishment Specifically, the procedures for DDP Stream Session establishment
discussed in section 3 of the SCTP mapping, and section 13.3 of the discussed in section 3 of the SCTP mapping, and section 13.3 of the
MPA/TCP mapping, both allow for the exchange of ULP specific data MPA/TCP mapping, both allow for the exchange of ULP-specific data
("Private Data") before enabling the exchange of DDP Segments. This ("Private Data") before enabling the exchange of DDP Segments. This
delay can allow for proper selection and/or configuration of the delay can allow for proper selection and/or configuration of the
endpoints based upon the exchanged data. For example, each DDP endpoints based upon the exchanged data. For example, each DDP
Stream Session associated with a single client session might be Stream Session associated with a single client session might be
assigned to the same DDP Protection Domain. assigned to the same DDP Protection Domain.
To be transport neutral, the applications should exchange Private To be transport neutral, the applications should exchange Private
Data as part of session establishment messages to determine how the Data as part of session establishment messages to determine how the
RDMA endpoints are to be configured. One side must be the Initiator, RDMA endpoints are to be configured. One side must be the Initiator,
and the other the Responder. and the other, the Responder.
With SCTP, a pair of SCTP streams can be used for successive sessions With SCTP, a pair of SCTP streams can be used for successive sessions
while the SCTP association remains open. With MPA/TCP each while the SCTP association remains open. With MPA/TCP, each
connection can be used for at most one session. However, the same connection can be used for, at most, one session. However, the same
source/destination pair of ports can be re-used sequentially subject source/destination pair of ports can be re-used for a subsequent TCP
to normal TCP rules. connection, as allowed by TCP.
Both SCTP and MPA limit the private data size to a maximum of 512 Both SCTP and MPA limit the private data size to a maximum of 512
bytes. bytes.
MPA/TCP requires the end of the TCP connection that initiated the MPA/TCP requires the end of the TCP connection that initiated the
conversion to MPA mode to send the first DDP Segment. SCTP does not conversion to MPA mode to send the first DDP Segment. SCTP does not
have this requirement. ULPs which wish to be transport neutral have this requirement. ULPs that wish to be transport neutral should
should require the initiating end to send the first message. A zero- require the initiating end to send the first message. A zero-length
length RDMA Write can be used for this purpose if the ULP logic RDMA Write can be used for this purpose if the ULP logic itself does
itself does naturally support this restriction. naturally support this restriction.
6.9.2. RDMA-Conditional Session Establishment 6.9.2. RDMA-Conditional Session Establishment
It is sometimes desirable for the active side of a session to connect It is sometimes desirable for the active side of a session to connect
with the passive side before knowing whether the passive side with the passive side before knowing whether the passive side
supports RDMA. supports RDMA.
This style of session establishment can be supported with either TCP This style of session establishment can be supported with either TCP
or SCTP, but not as transparently as for RDMA-only sessions. Pre- or SCTP, but not as transparently as for RDMA-only sessions. Pre-
existing non-RDMA servers are also far more likely to be using TCP existing non-RDMA servers are also far more likely to be using TCP
than SCTP. than SCTP.
With TCP. a normal TCP connection is established. It is then used by With TCP, a normal TCP connection is established. It is then used by
the ULP to determine whether or not to convert to MPA mode and use the ULP to determine whether or not to convert to MPA mode and use
RDMA. This will typically be integral with other session RDMA. This will typically be integral with other session-
establishment negotiations. establishment negotiations.
With SCTP, the establishment of an association tests whether RDMA is With SCTP, the establishment of an association tests whether RDMA is
supported. If not supported, the application simply requests the supported. If not supported, the application simply requests the
association without the RDMA adaptation indication. association without the RDMA adaptation indication.
One key difference is that with SCTP the determination as to whether One key difference is that with SCTP the determination as to whether
the peer can support RDMA is made before the transport layer the peer can support RDMA is made before the transport layer
association/connection is established while with TCP the established association/connection is established, while with TCP the established
connection itself is used to determine whether RDMA is supported. connection itself is used to determine whether RDMA is supported.
7. Local Interface Implications 7. Local Interface Implications
Full utilization of DDP and RDMAP capabilities requires a local Full utilization of DDP and RDMAP capabilities requires a local
interface that explicitly requests these services. Protocols such as interface that explicitly requests these services. Protocols such as
Sockets Direct Protocol (SDP) can allow applications to keep their Sockets Direct Protocol (SDP) can allow applications to keep their
traditional byte-stream or message-stream interface and still enjoy traditional byte-stream or message-stream interface and still enjoy
many of the benefits of the optimized wire level protocols. many of the benefits of the optimized wire level protocols.
8. IANA Considerations 8. Security Considerations
There are no IANA considerations in this document.
9. Security considerations
RDMA security considerations are discussed in [RDMA-SEC] [6]. This RDMA security considerations are discussed in the RDMA security
document will only deal with the more usage oriented aspects, and document [RFC5042]. This document will only deal with the more
where there are implications in the choice of underlying transport. usage-oriented aspects, and where there are implications in the
choice of underlying transport.
9.1. Connection/Association Setup 8.1. Connection/Association Setup
Both the SCTP and TCP adaptations allow for existing procedures to be Both the SCTP and TCP adaptations allow for existing procedures to be
followed for the establishment of the SCTP association or TCP followed for the establishment of the SCTP association or TCP
connection. Use of DDP does not impair the use of any security connection. Use of DDP does not impair the use of any security
measures to filter, validate and/or log the remote end of an measures to filter, validate, and/or log the remote end of an
association/connection. association/connection.
9.2. Tagged Buffer Exposure 8.2. Tagged Buffer Exposure
DDP only exposes ULP memory to the extent explicitly allowed by ULP DDP only exposes ULP memory to the extent explicitly allowed by ULP
actions. These include posting of receive operations and enabling of actions. These include posting of receive operations and enabling of
Steering Tags. Steering Tags.
Neither RDMAP or DDP place requirements on how ULP's advertise Neither RDMAP nor DDP places requirements on how ULPs Advertise
buffers. A ULP may use a single Steering Tag for multiple buffer Buffers. A ULP may use a single Steering Tag for multiple buffer
advertisements. However, the ULP should be aware that enforcement on Advertisements. However, the ULP should be aware that enforcement on
STag usage is likely limited to the overall range that is enabled. STag usage is likely limited to the overall range that is enabled.
If the remote peer writes into the 'wrong' advertised buffer, neither If the Remote Peer writes into the 'wrong' Advertised Buffer, neither
the DDP or RDMAP layer will be aware of this. Nor is there any the DDP nor the RDMAP layer will be aware of this. Nor is there any
report to the ULP on how the remote peer specifically used tagged report to the ULP on how the Remote Peer specifically used Tagged
buffers. Buffers.
Unless the ULP peers have an adequate basis for mutual trust, the Unless the ULP peers have an adequate basis for mutual trust, the
receiving ULP might be well advised to use a distinct STag for each receiving ULP might be well advised to use a distinct STag for each
interaction, and to invalidate it after each use or to require its interaction, and to invalidate it after each use, or to require its
peer to use the RDMAP option to invalidate the STag with its peer to use the RDMAP option to invalidate the STag with its
responding untagged message. responding Untagged Message.
9.3. Impact of Encrypted Transports 8.3. Impact of Encrypted Transports
While DDP is cleanly layered over the LLP, its maximum benefit may be While DDP is cleanly layered over the LLP, its maximum benefit may be
limited when the LLP Stream is secured with a streaming cypher, such limited when the LLP Stream is secured with a streaming cypher, such
as Transport Layer Security (TLS) RFC4346 [9]. If the LLP must as Transport Layer Security (TLS) [RFC4346]. If the LLP must decrypt
decrypt in order, it cannot provide out-of-order DDP Segments to the in order, it cannot provide out-of-order DDP Segments to the DDP
DDP layer for placement purposes. IPsec RFC2406 [1]. tunnel mode layer for placement purposes. IPsec [RFC2401] tunnel mode encrypts
encrypts entire IP Datagrams. IPsec transport mode encrypts TCP entire IP Datagrams. IPsec transport mode encrypts TCP Segments or
Segments or SCTP packets, as does use of DTLS RFC4347 [10] over UDP SCTP packets, as does use of Datagram TLS (DTLS) [RFC4347] over UDP
beneath TCP or SCTP. Neither IPsec nor this use of DTLS preclude beneath TCP or SCTP. Neither IPsec nor this use of DTLS precludes
providing out-of-order DDP Segments to the DDP layer for placement. providing out-of-order DDP Segments to the DDP layer for placement.
Note that end-to-end use of cryptographic integrity protection may Note that end-to-end use of cryptographic integrity protection may
allow suppression of MPA CRC generation and checking under certain allow suppression of MPA CRC generation and checking under certain
circumstances. This is one example where the LLP may be judged to circumstances. This is one example where the LLP may be judged to
have "or equivalent" protection to an end-to-end CRC32c. have "or equivalent" protection to an end-to-end CRC32c.
10. References 9. References
10.1. Normative references 9.1. Normative References
[1] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload [RFC2401] Kent, S. and R. Atkinson, "Security Architecture for the
(ESP)", RFC 2406, November 1998. Internet Protocol", RFC 2401, November 1998.
[2] Recio, R., "An RDMA Protocol Specification", [RFC2406] Kent, S. and R. Atkinson, "IP Encapsulating Security
draft-ietf-rddp-rdmap-05 (work in progress), July 2005. Payload (ESP)", RFC 2406, November 1998.
[3] Shah, H., "Direct Data Placement over Reliable Transports", [RFC4960] Stewart, R., "Stream Control Transmission Protocol",
draft-ietf-rddp-ddp-05 (work in progress), July 2005. RFC 4960, September 2007.
[4] Stewart, R., "Stream Control Transmission Protocol (SCTP) Remote [RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
Direct Memory Access (RDMA) Direct Data Placement (DDP) Garcia, "A Remote Direct Memory Access Protocol
Adaptation", draft-ietf-rddp-sctp-03 (work in progress), Specification", RFC 5040, October 2007.
June 2006.
[5] Culley, P., "Marker PDU Aligned Framing for TCP Specification", [RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley,
draft-ietf-rddp-mpa-04 (work in progress), May 2006. "Direct Data Placement over Reliable Transports",
RFC 5041, October 2007.
[6] Pinkerton, J., "DDP/RDMAP Security", draft-ietf-rddp-security-09 [RFC5042] Pinkerton, J. and E. Deleganes, "DDP/RDMAP Security",
(work in progress), May 2006. RFC 5042, October 2007.
10.2. Informative References [RFC5043] Bestler, C. and R. Stewart, "Stream Control Transmission
Protocol (SCTP) Direct Data Placement (DDP) Adaptation",
RFC 5043, October 2007.
[7] Ko, M., "iSCSI Extensions for RDMA Specification", [RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
draft-ietf-ips-iser-05 (work in progress), October 2005. Carrier, "Marker PDU Aligned Framing for TCP
Specification", RFC 5044, October 2007.
[8] Callaghan, B. and T. Talpey, "NFS Direct Data Placement", 9.2. Informative References
draft-ietf-nfsv4-nfsdirect-02 (work in progress), October 2005.
[9] Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) [NFSDIRECT] Talpey, T., Callaghan, B., and I. Property, "NFS Direct
Protocol Version 1.1", RFC 4346, April 2006. Data Placement", Work in Progress, June 2007.
[10] Rescorla, E. and N. Modadugu, "Datagram Transport Layer [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
Travostino, "Securing Block Storage Protocols over IP",
RFC 3723, April 2004.
[RFC4301] Kent, S. and K. Seo, "Security Architecture for the
Internet Protocol", RFC 4301, December 2005.
[RFC4346] Dierks, T. and E. Rescorla, "The Transport Layer
Security (TLS) Protocol Version 1.1", RFC 4346,
April 2006.
[RFC4347] Rescorla, E. and N. Modadugu, "Datagram Transport Layer
Security", RFC 4347, April 2006. Security", RFC 4347, April 2006.
[RFC5046] Ko, M., Chadalapaka, M., Elzur, U., Shah, H., and P.
Thaler, "Internet Small Computer System Interface
(iSCSI) Extensions for Remote Direct Memory Access
(RDMA)", RFC 5046, October 2007.
Authors' Addresses Authors' Addresses
Caitlin Bestler Caitlin Bestler (editor)
Broadcom Corporation Neterion
16215 Alton Parkway 20230 Stevens Creek Blvd.
P.O. Box 57013 Suite C
Irvine, CA 92619-7013 Cupertino, CA 95014
USA USA
Phone: 949-926-6383 Phone: 408-366-4639
Email: caitlinb@broadcom.com EMail: caitlin.bestler@neterion.com
Lode Coene Lode Coene
Siemens Nokia Siemens Networks
Atealaan 26 Atealaan 26
Herentals, 2200 Herentals 2200
Belgium Belgium
Phone: +32-14-252081 Phone: +32-14-252081
Email: lode.coene@siemens.com EMail: lode.coene@nsn.com
Intellectual Property Statement Full Copyright Statement
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79. found in BCP 78 and BCP 79.
skipping to change at page 27, line 28 skipping to change at line 990
attempt made to obtain a general license or permission for the use of attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr. http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at this standard. Please address the information to the IETF at
ietf-ipr@ietf.org. ietf-ipr@ietf.org.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2006). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
 End of changes. 169 change blocks. 
384 lines changed or deleted 395 lines changed or added

This html diff was produced by rfcdiff 1.34. The latest version is available from http://tools.ietf.org/tools/rfcdiff/