draft-ietf-rddp-applicability-02.txt   draft-ietf-rddp-applicability-03.txt 
Remote Direct Data Placement C. Bestler Remote Direct Data Placement C. Bestler
Working group L. Coene Working group Broadcom
Internet-Draft L. Coene
Expires: March 11, 2005 Expires: March 30, 2006 Siemens
September 26, 2005
Applicability of Remote Direct Memory Access Protocol (RDMA) and Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct
Direct Data Placement (DDP) Data Placement (DDP)
draft-ietf-rddp-applicability-02.txt draft-ietf-rddp-applicability-03.txt
Status of this Memo Status of this Memo
This document is an Internet-Draft and is subject to all provisions By submitting this Internet-Draft, each author represents that any
of section 3 of RFC 3667. By submitting this Internet-Draft, each applicable patent or other IPR claims of which he or she is aware
author represents that any applicable patent or other IPR claims of have been or will be disclosed, and any of which he or she becomes
which he or she is aware have been or will be disclosed, and any of aware will be disclosed, in accordance with Section 6 of BCP 79.
which he or she become aware will be disclosed, in accordance with
RFC 3668.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as other groups may also distribute working documents as Internet-
Internet-Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on March 11, 2005. This Internet-Draft will expire on March 30, 2006.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2004). Copyright (C) The Internet Society (2005).
Abstract Abstract
This document describes the applicability of Remote Direct Memory This document describes the applicability of Remote Direct Memory
Access Protocol (RDMAP) and the Direct Data Placement Protocol Access Protocol (RDMAP) and the Direct Data Placement Protocol (DDP).
(DDP). It contrasts the different transport options over IP that DDP It comparese and contrasts the different transport options over IP
can use, compares use of DDP with direct use of the supporting that DDP can use, provides guidance to ULP developers on choosing
transports, and compares DDP over IP transports with non-IP between available transports and/or how to be indifferent to the
transports that support RDMA functionality. specific transport layer used, compares use of DDP with direct use of
the supporting transports, and compares DDP over IP transports with
non-IP transports that support RDMA functionality.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Direct Placement . . . . . . . . . . . . . . . . . . . . . . . 6 3. Direct Placement . . . . . . . . . . . . . . . . . . . . . . . 6
3.1 Fewer Required ULP Interactions . . . . . . . . . . . . . 6 3.1. Fewer Required ULP Interactions . . . . . . . . . . . . . 6
3.2 Direct Placement using only the LLP . . . . . . . . . . . 6 3.2. Direct Placement using only the LLP . . . . . . . . . . . 6
4. Tagged Messages . . . . . . . . . . . . . . . . . . . . . . . 8 4. Tagged Messages . . . . . . . . . . . . . . . . . . . . . . . 8
4.1 Order Independent Reception . . . . . . . . . . . . . . . 8 4.1. Order Independent Reception . . . . . . . . . . . . . . . 8
4.2 Reduced ULP Notifications . . . . . . . . . . . . . . . . 8 4.2. Reduced ULP Notifications . . . . . . . . . . . . . . . . 8
4.3 Simplified ULP Exchanges . . . . . . . . . . . . . . . . . 9 4.3. Simplified ULP Exchanges . . . . . . . . . . . . . . . . . 9
4.4 Order Independent Sending . . . . . . . . . . . . . . . . 10 4.4. Order Independent Sending . . . . . . . . . . . . . . . . 10
4.5 Tagged Buffers as ULP Credits . . . . . . . . . . . . . . 11 4.5. Tagged Buffers as ULP Credits . . . . . . . . . . . . . . 11
5. RDMA Read . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5. RDMA Read . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6. LLP Comparisons . . . . . . . . . . . . . . . . . . . . . . . 14 6. LLP Comparisons . . . . . . . . . . . . . . . . . . . . . . . 14
6.1 Multistreaming Implications . . . . . . . . . . . . . . . 14 6.1. Multistreaming Implications . . . . . . . . . . . . . . . 14
6.2 Out of Order Reception Implications . . . . . . . . . . . 14 6.2. Out of Order Reception Implications . . . . . . . . . . . 14
6.3 Header and Marker Overhead . . . . . . . . . . . . . . . . 14 6.3. Header and Marker Overhead . . . . . . . . . . . . . . . . 14
6.4 Middlebox Support . . . . . . . . . . . . . . . . . . . . 14 6.4. Middlebox Support . . . . . . . . . . . . . . . . . . . . 15
6.5 Processing Overhead . . . . . . . . . . . . . . . . . . . 15 6.5. Processing Overhead . . . . . . . . . . . . . . . . . . . 15
6.6 Data Integrity Implications . . . . . . . . . . . . . . . 15 6.6. Data Integrity Implications . . . . . . . . . . . . . . . 15
6.6.1 MPA/TCP Specifics . . . . . . . . . . . . . . . . . . 15 6.6.1. MPA/TCP Specifics . . . . . . . . . . . . . . . . . . 15
6.6.2 SCTP Specifics . . . . . . . . . . . . . . . . . . . . 16 6.6.2. SCTP Specifics . . . . . . . . . . . . . . . . . . . . 16
6.7 Non-IP Transports . . . . . . . . . . . . . . . . . . . . 16 6.7. Non-IP Transports . . . . . . . . . . . . . . . . . . . . 16
6.7.1 No RDMA Layer Ack . . . . . . . . . . . . . . . . . . 16 6.7.1. No RDMA Layer Ack . . . . . . . . . . . . . . . . . . 16
6.8 Other IP Transports . . . . . . . . . . . . . . . . . . . 17 6.8. Other IP Transports . . . . . . . . . . . . . . . . . . . 17
6.9 LLP Independent Session Establishment . . . . . . . . . . 17 6.9. LLP Independent Session Establishment . . . . . . . . . . 17
6.9.1 RDMA-only Session Establishment . . . . . . . . . . . 17 6.9.1. RDMA-only Session Establishment . . . . . . . . . . . 18
6.9.2 RDMA-Conditional Session Establishment . . . . . . . . 18 6.9.2. RDMA-Conditional Session Establishment . . . . . . . . 18
7. Local Interface Implications . . . . . . . . . . . . . . . . . 20 7. Local Interface Implications . . . . . . . . . . . . . . . . . 20
8. Security considerations . . . . . . . . . . . . . . . . . . . 21 8. Security considerations . . . . . . . . . . . . . . . . . . . 21
8.1 Connection/Association Setup . . . . . . . . . . . . . . . 21 8.1. Connection/Association Setup . . . . . . . . . . . . . . . 21
8.2 Tagged Buffer Exposure . . . . . . . . . . . . . . . . . . 21 8.2. Tagged Buffer Exposure . . . . . . . . . . . . . . . . . . 21
8.3 Impact of Encrypted Transports . . . . . . . . . . . . . . 21 8.3. Impact of Encrypted Transports . . . . . . . . . . . . . . 21
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 22 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23
Intellectual Property and Copyright Statements . . . . . . . . 24 Intellectual Property and Copyright Statements . . . . . . . . . . 24
1. Introduction 1. Introduction
Remote Direct Memory Access Protocol (RDMAP) and Direct Data Remote Direct Memory Access Protocol (RDMAP) and Direct Data
Placement (DDP) work together to provide application independent Placement (DDP) work together to provide application independent
efficient placemenet of application payload directly into buffers efficient placement of application payload directly into buffers
specified by the Upper Layer Protocol (ULP). specified by the Upper Layer Protocol (ULP).
The DDP protocol is responsible for direct placement of received The DDP protocol is responsible for direct placement of received
payload into ULP specified buffers. The RDMAP protocol provides payload into ULP specified buffers. The RDMAP protocol provides
completion notifications to the ULP and support for Data Sink completion notifications to the ULP and support for Data Sink
initiated fetch of advertised buffers (RDMA Reads). initiated fetch of advertised buffers (RDMA Reads).
DDP and RDMAP are both application independent protocols which allow DDP and RDMAP are both application independent protocols which allow
the ULP to perform remote direct data placement. DDP can use the ULP to perform remote direct data placement. DDP can use
multiple standard IP transports including SCTP and TCP. multiple standard IP transports including SCTP and TCP.
skipping to change at page 6, line 17 skipping to change at page 6, line 17
Direct Data Placement optimizes the placement of ULP payload into the Direct Data Placement optimizes the placement of ULP payload into the
correct destination buffers, typically eliminating intermediate correct destination buffers, typically eliminating intermediate
copying. Placement is enabled without regard to order of arrival, copying. Placement is enabled without regard to order of arrival,
order of transmission or requiring per-placement interaction with the order of transmission or requiring per-placement interaction with the
ULP. ULP.
RDMAP minimizes the required ULP interactions . This capability is RDMAP minimizes the required ULP interactions . This capability is
most valuable for applications that require multiple transport layer most valuable for applications that require multiple transport layer
packets for each required ULP interaction. packets for each required ULP interaction.
3.1 Fewer Required ULP Interactions 3.1. Fewer Required ULP Interactions
While reducing the number of required ULP interactions is in itself While reducing the number of required ULP interactions is in itself
desirable, it is critical for high speed connections. The burst desirable, it is critical for high speed connections. The burst
packet rate for a high speed interface could easily exceed the host packet rate for a high speed interface could easily exceed the host
systems ability to switch ULP contexts. systems ability to switch ULP contexts.
Content access applications are primary examples of applications with Content access applications are primary examples of applications with
both high bandwidth and high content to required ULP interaction both high bandwidth and high content to required ULP interaction
ratios. These applications include file access protocols (NAS), ratios. These applications include file access protocols (NAS),
storage access (SAN), database access and other application specific storage access (SAN), database access and other application specific
forms of content access such as HTTP, XML and email. forms of content access such as HTTP, XML and email.
3.2 Direct Placement using only the LLP 3.2. Direct Placement using only the LLP
Direct data placement can be achieved without RDMA. Pre-posting of Direct data placement can be achieved without RDMA. Pre-posting of
receive buffers could allow a non-RDMA network stack to place data receive buffers could allow a non-RDMA network stack to place data
directly to user buffers. directly to user buffers.
The degree to which DDP optimizes depends on which transport is being The degree to which DDP optimizes depends on which transport is being
compared with, and on the nature of the local interface. Without compared with, and on the nature of the local interface. Without
RDMAP/DDP pre-posting buffers requires the receiving side to RDMAP/DDP pre-posting buffers requires the receiving side to
accurately predict the required buffers and their sizes. This is not accurately predict the required buffers and their sizes. This is not
feasible for all ULPs. By contrast, DDP only requires the ULP to feasible for all ULPs. By contrast, DDP only requires the ULP to
predict the sequence and size of incoming untagged messages. predict the sequence and size of incoming untagged messages.
An application that could predict incoming messages and required An application that could predict incoming messages and required
nothing more than direct placement into buffers might be able to do nothing more than direct placement into buffers might be able to do
so with a properly designed local interface to SCTP or TCP. Doing so so with a properly designed local interface to SCTP or TCP. Doing so
for TCP requires making predictions at a byte level rather than a for TCP requires making predictions at a byte level rather than a
message level. message level.
The main benefit of DDP for such an application would be that The main benefit of DDP for such an application would be that pre-
pre-posting of receive buffers is a mandated local interface posting of receive buffers is a mandated local interface capability,
capability, and that predictions can be made on a per-message basis and that predictions can be made on a per-message basis (not per
(not per byte). byte).
The LLP can also be used directly if ULP specific knowledge is built The LLP can also be used directly if ULP specific knowledge is built
into the protocol stack to allow "parse and place" handling of into the protocol stack to allow "parse and place" handling of
received packets. Such a solution either requires interaction with received packets. Such a solution either requires interaction with
the ULP, or that the protocol stack have knowledge of ULP specific the ULP, or that the protocol stack have knowledge of ULP specific
syntax rules. syntax rules.
DDP achieves the benefits of directly placing incoming payload DDP achieves the benefits of directly placing incoming payload
without requiring tight coupling between the ULP and the protocol without requiring tight coupling between the ULP and the protocol
stack. However, "parse and place" capabilities can certainly provide stack. However, "parse and place" capabilities can certainly provide
skipping to change at page 8, line 23 skipping to change at page 8, line 23
With direct data placement based solely upon pre-posted receives, the With direct data placement based solely upon pre-posted receives, the
packetization and delivery of payload must be agreed by the ULP peers packetization and delivery of payload must be agreed by the ULP peers
in advance. Even if there is an encoding of what is being in advance. Even if there is an encoding of what is being
transferred, as is common with middleware solutions, this information transferred, as is common with middleware solutions, this information
is not understood at the application independent layers. The is not understood at the application independent layers. The
directions on where to place the incoming data cannot be accessed directions on where to place the incoming data cannot be accessed
without switching to the ULP first. DDP provides a standardized without switching to the ULP first. DDP provides a standardized
'packing list' which can be interpreted without requiring ULP 'packing list' which can be interpreted without requiring ULP
interaction. Indeed, it is designed to be implementable in hardware. interaction. Indeed, it is designed to be implementable in hardware.
4.1 Order Independent Reception 4.1. Order Independent Reception
Tagged messages are directed to a buffer based on an included Tagged messages are directed to a buffer based on an included
Steering Tag. Additionally, no notice is provided to the ULP for Steering Tag. Additionally, no notice is provided to the ULP for each
each individual Tagged Message's arrival. Together these allow individual Tagged Message's arrival. Together these allow tagged
tagged messages received out-of-order to be processed without messages received out-of-order to be processed without intermediate
intermediate buffering or additional notifications to the ULP. buffering or additional notifications to the ULP.
4.2 Reduced ULP Notifications 4.2. Reduced ULP Notifications
RDMAP further reduces required ULP interactions consolidating RDMAP further reduces required ULP interactions consolidating
completion notifications of tagged messages with the completion completion notifications of tagged messages with the completion
notification of a trailing untagged message. For most ULPs this notification of a trailing untagged message. For most ULPs this
radically reduces the number of ULP required interactions even radically reduces the number of ULP required interactions even
further. further.
While RDMAP consolidation of notices is beneficial to most While RDMAP consolidation of notices is beneficial to most
applications. It may be detrimental to some applications that applications, it may be detrimental to some applications that benefit
benefit from streamed delivery to enable ULP processing of received from streamed delivery to enable ULP processing of received data as
data as promptly as possible. A ULP that uses RDMAP cannot begin promptly as possible. A ULP that uses RDMAP cannot begin processing
processing any portion of an exchange until it receives notification any portion of an exchange until it receives notification that the
that the entire exchange has been placed. An "exchange" here is a entire exchange has been placed. An "exchange" here is a set of zero
set of zero or more tagged messages and a single terminating untagged or more tagged messages and a single terminating untagged message.
message. An application that would prefer to begin work on the An application that would prefer to begin work on the received
received payload, no matter what order it arrived in, as soon as payload, no matter what order it arrived in, as soon as possible
possible might prefer to work directly with the LLP. RDMAP is might prefer to work directly with the LLP. RDMAP is optimized for
optimized for applications that are more concerned when the entire applications that are more concerned when the entire exchange is
exchange is complete. complete.
An application that benefits from being able to begin processing of An application that benefits from being able to begin processing of
each received packet as quickly as possible may find RDMAP interferes each received packet as quickly as possible may find RDMAP interferes
with that goal. with that goal.
Such an application might be able to retain most of the benefits of Such an application might be able to retain most of the benefits of
RDMAP by using the DDP layer directly. However, in addition to RDMAP by using the DDP layer directly. However, in addition to
taking on the responsibilities of the RDMAP layer, the application taking on the responsibilities of the RDMAP layer, the application
would likely have more difficulty finding support for a DDP-only API. would likely have more difficulty finding support for a DDP-only API.
Many hardware implementations may choose to tightly couple RDMAP and Many hardware implementations may choose to tightly couple RDMAP and
DDP, and might not provide an API directly to DDP services. DDP, and might not provide an API directly to DDP services.
These features minimize the required interactions with the ULP. This These features minimize the required interactions with the ULP. This
can be extremely beneficial for applications that use multiple can be extremely beneficial for applications that use multiple
transport layer packets to accomplish what is a single ULP transport layer packets to accomplish what is a single ULP
interaction. interaction.
4.3 Simplified ULP Exchanges 4.3. Simplified ULP Exchanges
The notification rules for Tagged Messages allows ULPs to create The notification rules for Tagged Messages allows ULPs to create
multi-message "exchanges" consisting of zero or more tagged messages multi-message "exchanges" consisting of zero or more tagged messages
that represent a single step in the ULP interaction. The receiving that represent a single step in the ULP interaction. The receiving
ULP is notified that the untagged message has arrived, and implicitly ULP is notified that the untagged message has arrived, and implicitly
of any associated tagged messages. of any associated tagged messages.
A ULP where all exchanges would naturally be only the untagged A ULP where all exchanges would naturally be only the untagged
message would derive virtually no benefit from the use of RDMAP/DDP message would derive virtually no benefit from the use of RDMAP/DDP
as opposed to SCTP. But while tagged buffers are the justification as opposed to SCTP. But while tagged buffers are the justification
skipping to change at page 10, line 12 skipping to change at page 10, line 15
The Server sends transaction reply as an untagged message to the The Server sends transaction reply as an untagged message to the
client. client.
Client receives single notification, indicating completion of the Client receives single notification, indicating completion of the
interaction. interaction.
With this type of exchange the pacing and required size of untagged With this type of exchange the pacing and required size of untagged
buffers is highly predictable. The variability of response sizes is buffers is highly predictable. The variability of response sizes is
absorbed by tagged transfers. absorbed by tagged transfers.
4.4 Order Independent Sending 4.4. Order Independent Sending
Use of tagged messages is especially applicable when the Data Sink Use of tagged messages is especially applicable when the Data Sink
does not know the actual size, structure or location of the content does not know the actual size, structure or location of the content
it is requesting (or updating). it is requesting (or updating).
For example, suppose the Data Sink ULP needs to fetch four related For example, suppose the Data Sink ULP needs to fetch four related
pieces of data into a four separate buffers. With SCTP the Data Sink pieces of data into a four separate buffers. With SCTP the Data Sink
ULP could receive four messages into four separate buffers, only ULP could receive four messages into four separate buffers, only
having to predict the maximum size of each. However it would have to having to predict the maximum size of each. However it would have to
dictate the order in which the Data Source supplied the separate dictate the order in which the Data Source supplied the separate
pieces. If the Data Source found it advantageous to fetch them in a pieces. If the Data Source found it advantageous to fetch them in a
different order it would have to use intermediate buffering to different order it would have to use intermediate buffering to re-
re-order the pieces into the expected order even though the order the pieces into the expected order even though the application
application only required that all four be delivered and did not only required that all four be delivered and did not truly have an
truly have an ordering requirement. ordering requirement.
Techniques such as RAID striping and mirroring represent this same Techniques such as RAID striping and mirroring represent this same
problem, but one step further. What appears to be a single resource problem, but one step further. What appears to be a single resource
to the Data Sink is actually stored in separate locations by the Data to the Data Sink is actually stored in separate locations by the Data
Source. Non RDMA protocols would either require the Data Source to Source. Non RDMA protocols would either require the Data Source to
fetch the material in the desired order or force the Data Source to fetch the material in the desired order or force the Data Source to
use its own holding buffers to assemble an image of the destination use its own holding buffers to assemble an image of the destination
buffer. buffer.
While sometimes referred to as a "buffer-to-buffer" solution, RDMA While sometimes referred to as a "buffer-to-buffer" solution, RDMA
skipping to change at page 11, line 17 skipping to change at page 11, line 21
Note that while DDP enables use of tagged messages for bulk transfer, Note that while DDP enables use of tagged messages for bulk transfer,
there are some application scenarios where untagged messages would there are some application scenarios where untagged messages would
still be used for bulk transfer. For example, under the Direct still be used for bulk transfer. For example, under the Direct
Access File Server (DAFS) protocol the file server does not expose Access File Server (DAFS) protocol the file server does not expose
its own memory to its clients. A client wishing to write may its own memory to its clients. A client wishing to write may
advertise a buffer which the server will issue RDMA Reads upon. advertise a buffer which the server will issue RDMA Reads upon.
However, when performing a small write it may be preferable to However, when performing a small write it may be preferable to
include the data in the untagged message rather than incurring an include the data in the untagged message rather than incurring an
additional round trip with the RDMA Read and its response. additional round trip with the RDMA Read and its response.
4.5 Tagged Buffers as ULP Credits 4.5. Tagged Buffers as ULP Credits
The handling of end-to-end buffer credits differs considerably with The handling of end-to-end buffer credits differs considerably with
DDP than when the ULP directly uses either TCP or SCTP. DDP than when the ULP directly uses either TCP or SCTP.
With both TCP and SCTP buffer credits are based upon the receiver With both TCP and SCTP buffer credits are based upon the receiver
granting transmit permission based on the total number of bytes. granting transmit permission based on the total number of bytes.
These credits reflect system buffering resources and/or simple flow These credits reflect system buffering resources and/or simple flow
control. They do not represent ULP resources. control. They do not represent ULP resources.
DDP defines no standard flow control, but presumes the existince of a DDP defines no standard flow control, but presumes the existince of a
skipping to change at page 14, line 18 skipping to change at page 14, line 18
ULP. RDMAP and DDP provides the same services over either. There ULP. RDMAP and DDP provides the same services over either. There
may be performance impacts of the choice, however. It is the may be performance impacts of the choice, however. It is the
responsibility of the ULP to determine which IP transport is best responsibility of the ULP to determine which IP transport is best
suited to its needs. suited to its needs.
SCTP provides for preservation of message boundaries. Each DDP SCTP provides for preservation of message boundaries. Each DDP
segment will be delivered within a single SCTP packet. The segment will be delivered within a single SCTP packet. The
equivalent services are only available with TCP through the use of equivalent services are only available with TCP through the use of
the MPA adaptation layer. the MPA adaptation layer.
6.1 Multistreaming Implications 6.1. Multistreaming Implications
SCTP also provides multi-streaming. When the same pair of hosts have SCTP also provides multi-streaming. When the same pair of hosts have
need for multiple DDP streams this can be a major advantage. A need for multiple DDP streams this can be a major advantage. A
single SCTP association carries multiple DDP streams, consolidating single SCTP association carries multiple DDP streams, consolidating
connection setup, congestion control and acknowledgements. connection setup, congestion control and acknowledgements.
Completions are controlled by the DDP Source Sequence Number Completions are controlled by the DDP Source Sequence Number (DDP-
(DDP-SSN) on a per stream basis. Therefore combining multiple DDP SSN) on a per stream basis. Therefore combining multiple DDP Streams
Streams into a single SCTP association cannot result in a dropped into a single SCTP association cannot result in a dropped packet
packet carrying data for one stream delaying completions on others. carrying data for one stream delaying completions on others.
6.2 Out of Order Reception Implications 6.2. Out of Order Reception Implications
The use of unordered Data Chunks with SCTP guarantees that the DDP The use of unordered Data Chunks with SCTP guarantees that the DDP
layer will be able to perform placements when IP datagrams are layer will be able to perform placements when IP datagrams are
received out of order. received out of order.
Placement of out-of-order DDP Segments carried over MPA/TCP is not Placement of out-of-order DDP Segments carried over MPA/TCP is not
guaranteed, but certainly allowed. The ability of the MPA receiver guaranteed, but certainly allowed. The ability of the MPA receiver
to process out-of-order DDP Segments may be impaired when TCP to process out-of-order DDP Segments may be impaired when alignment
alignment is lost. Using SCTP, each DDP Segment is encoded in a of TCP segments and MPA FPDUs is lost. Using SCTP, each DDP Segment
single Data Chunk and never spread over multiple IP datagrams. is encoded in a single Data Chunk and never spread over multiple IP
datagrams.
6.3 Header and Marker Overhead 6.3. Header and Marker Overhead
MPA and TCP headers together are smaller than the headers used by MPA and TCP headers together are smaller than the headers used by
SCTP and its adaptation layer. However, this advantage can be SCTP and its adaptation layer. However, this advantage can be
considerably reduced by the insertion of MPA markers. In any event considerably reduced by the insertion of MPA markers. In any event
the different in ULP payload per IP Datagram is not likely to be a the different in ULP payload per IP Datagram is not likely to be a
signifigant factor. signifigant factor.
6.4 Middlebox Support 6.4. Middlebox Support
Even with the MPA adaptation layer, DDP traffic carried over MPA/TCP Even with the MPA adaptation layer, DDP traffic carried over MPA/TCP
will appear to all network middleboxes as a normal TCP connection. will appear to all network middleboxes as a normal TCP connection.
In many environmenets there may be a requirement to use only TCP In many environments there may be a requirement to use only TCP
connections to satisfy existing network elements and/or to facilitate connections to satisfy existing network elements and/or to facilitate
monitoring and control of connections. While SCTP is certainly just monitoring and control of connections. While SCTP is certainly just
as monitorable and controllable as TCP, there is no guarantee that as monitorable and controllable as TCP, there is no guarantee that
the network management infrastructure has the required support for the network management infrastructure has the required support for
both. both.
6.5 Processing Overhead 6.5. Processing Overhead
A DDP stream delivered via MPA/TCP will require more processing A DDP stream delivered via MPA/TCP will required more processing
effort than one delivered over SCTP. However this extra work may be effort that one delivered over SCTP. However this extra work may be
justified for many deployments where full SCTP support is unavailable justified for many deployments where full SCTP support is unavailable
in the intermediate network. in the endpoints of the network, or where middleboxes impair the
usability of SCTP.
6.6 Data Integrity Implications 6.6. Data Integrity Implications
Both the SCTP and MPA/TCP adaptation provide end-to-end CRC32c Both the SCTP and MPA/TCP adaptation provide end-to-end CRC32c
protection against data corruption, or its equivalent. protection against data corruption, or its equivalent.
A ULP that requires a greater degree of protection may add it own. A ULP that requires a greater degree of protection may add it own.
However, DDP and RDMAP headers will only be guaranteed to have the However, DDP and RDMAP headers will only be guaranteed to have the
equivalent of end-to-end CRC32c protection. A ULP that requires data equivalent of end-to-end CRC32c protection. A ULP that requires data
integrity checking more thorough than an end-to-end CRC32c should integrity checking more thorough than an end-to-end CRC32c should
first invalidate all STags that reference a buffer before applying first invalidate all STags that reference a buffer before applying
their own integrity check. their own integrity check.
6.6.1 MPA/TCP Specifics 6.6.1. MPA/TCP Specifics
It is mandatory for MPA/TCP implementations to implement CRC32c, but It is mandatory for MPA/TCP implementations to implement CRC32c, but
it is NOT mandatory to use the CRC32c during an RDMA connection. The it is NOT mandatory to use the CRC32c during an RDMA connection. The
activating or deactivating of the CRC in MPA/TCP is an administrative activating or deactivating of the CRC in MPA/TCP is an administrative
configuration operation at the local and remote end. The configuration operation at the local and remote end. The
administration of the CRC(ON/OFF) is invisible to the ULP. administration of the CRC(ON/OFF) is invisible to the ULP.
Applications SHOULD trust that this administrative option will only Applications SHOULD trust that this administrative option will only
be used when the end-to-end protection is at least as effective as a be used when the end-to-end protection is at least as effective as a
transport layer CRC32c. Applications SHOULD NOT apply additional transport layer CRC32c. Applications SHOULD NOT apply additional
protection as a guard against this administrative option being turned protection as a guard against this administrative option being turned
on inadvertently. on inadvertently.
Administrators MUST NOT enable CRC32c suppression unless the Administrators MUST NOT enable CRC32c suppression unless the end-to-
end-to-end protection is truly equivalent. end protection is truly equivalent.
If the CRC is active/used for one direction/end , then the use of the If the CRC is active/used for one direction/end , then the use of the
CRC is mandatory in both directions/ends. CRC is mandatory in both directions/ends.
If both ends have been configured NOT to use the CRC, then this is If both ends have been configured NOT to use the CRC, then this is
allowed as long as an equivalent protection(comparable or better allowed as long as an equivalent protection(comparable or better
than/to CRC) from undetected errors on the connection is provided. than/to CRC) from undetected errors on the connection is provided.
6.6.2 SCTP Specifics 6.6.2. SCTP Specifics
SCTP provides CRC32c protection automatically. The adaptation to SCTP provides CRC32c protection automatically. The adaptation to
SCTP provides for no option to suppress SCTP CRC32c protection. SCTP provides for no option to suppress SCTP CRC32c protection.
6.7 Non-IP Transports 6.7. Non-IP Transports
DDP is defined to operate over ubiquitous IP transports such as SCTP DDP is defined to operate over ubiquitous IP transports such as SCTP
and TCP. This enabled a new DDP-enabled node to be added anywhere to and TCP. This enabled a new DDP-enabled node to be added anywhere to
an IP network. No DDP-specific support from middle-boxes is an IP network. No DDP-specific support from middle-boxes is
required. required.
There are non-IP transport fabric offering RDMA capabilities. There are non-IP transport fabric offering RDMA capabilities.
Because these capabilities are integrated with the transport protocol Because these capabilities are integrated with the transport protocol
they have some technical advantages when compared to RDMA over IP. they have some technical advantages when compared to RDMA over IP.
For example fencing of RDMA operations can be based upon transport For example fencing of RDMA operations can be based upon transport
level acks. Because DDP is cleanly layered over an IP transport, any level acks. Because DDP is cleanly layered over an IP transport, any
explicit RDMA layer ack must be separate from the transport layer explicit RDMA layer ack must be separate from the transport layer
ack. ack.
There may be deployments where the benefits of RDMA/transport There may be deployments where the benefits of RDMA/transport
integration outweigh the benefits of being on an IP network. integration outweigh the benefits of being on an IP network.
6.7.1 No RDMA Layer Ack 6.7.1. No RDMA Layer Ack
DDP does not provide for its own acknowledgements. The only form of DDP does not provide for its own acknowledgements. The only form of
ack provided at the RDMAP layer is an RDMA Read Response. DDP and ack provided at the RDMAP layer is an RDMA Read Response. DDP and
RDMAP rely almost entirely upon other layers for flow control and RDMAP rely almost entirely upon other layers for flow control and
pacing. The LLP is relied upon to guarantee delivery and avoid pacing. The LLP is relied upon to guarantee delivery and avoid
network congestion, and ULP level acking is relied upon for ULP network congestion, and ULP level acking is relied upon for ULP
pacing and to avoid ULP buffer overruns. pacing and to avoid ULP buffer overruns.
Previous RDMA protocols, such as InfiniBand, have been able to use Previous RDMA protocols, such as InfiniBand, have been able to use
their integration with the transport layer to provide stronger their integration with the transport layer to provide stronger
skipping to change at page 16, line 45 skipping to change at page 16, line 50
pacing. The LLP is relied upon to guarantee delivery and avoid pacing. The LLP is relied upon to guarantee delivery and avoid
network congestion, and ULP level acking is relied upon for ULP network congestion, and ULP level acking is relied upon for ULP
pacing and to avoid ULP buffer overruns. pacing and to avoid ULP buffer overruns.
Previous RDMA protocols, such as InfiniBand, have been able to use Previous RDMA protocols, such as InfiniBand, have been able to use
their integration with the transport layer to provide stronger their integration with the transport layer to provide stronger
ordering guarantees. It is important that application designers that ordering guarantees. It is important that application designers that
require such guarantees to provide them through ULP interaction. require such guarantees to provide them through ULP interaction.
Specifically: Specifically:
There is no ability for a local interface to "fence" outbound There is no ability for a local interface to "fence" outbound
messages to guarantee that prior tagged messages have been placed messages to guarantee that prior tagged messages have been placed
prior to sending a tagged message. The only guarantees available prior to sending a tagged message. The only guarantees available
from the other side would be an RDMA Read Response (coming from from the other side would be an RDMA Read Response (coming from
the RDMAP layer) or a response from the ULP layer. Remember that the RDMAP layer) or a response from the ULP layer. Remember that
the normal ordering rules only guarantee when the Data Sink ULP the normal ordering rules only guarantee when the Data Sink ULP
will be notified of untagged messages, it does not control when will be notified of untagged messages, it does not control when
data is placed into receive buffers. data is placed into receive buffers.
Re-use of tagged buffers must be done with extreme care. The fact Re-use of tagged buffers must be done with extreme care. The fact
that an untagged message indicates that all prior tagged messages that an untagged message indicates that all prior tagged messages
have been placed does not guarantee that no later tagged message have been placed does not guarantee that no later tagged message
have. The best strategy is to only change the state of any given have. The best strategy is to only change the state of any given
advertised buffers with with untagged messages. advertised buffers with with untagged messages.
As covered elsewhere in this document, flow control of untagged As covered elsewhere in this document, flow control of untagged
messages MUST be provided by the ULP itself. messages MUST be provided by the ULP itself.
6.8 Other IP Transports 6.8. Other IP Transports
Both TCP and SCTP provide DDP with reliable transport with TCP Both TCP and SCTP provide DDP with reliable transport with TCP
friendly rate control. As currently DDP is defined to work over friendly rate control. As currently DDP is defined to work over
reliable transports and implicitly relies upon some form of rate reliable transports and implicitly relies upon some form of rate
control. control.
DDP is fully compatible with a non-reliable protocol. Out-of-order DDP is fully compatible with a non-reliable protocol. Out-of-order
placement is obviously not dependent on whether the other DDP placement is obviously not dependent on whether the other DDP
Segments ever actually arrive. Segments ever actually arrive.
skipping to change at page 17, line 41 skipping to change at page 17, line 48
application would be only limited by the link layer rate, almost application would be only limited by the link layer rate, almost
inevitably resulting in severe network congestion. inevitably resulting in severe network congestion.
RDMAP encourages applications to be ignorant of the underlying RDMAP encourages applications to be ignorant of the underlying
transport PMTU. The ULP is only notified when all messages ending in transport PMTU. The ULP is only notified when all messages ending in
a single untagged message have completed. The ULP is not aware of a single untagged message have completed. The ULP is not aware of
the granularity or ordering of the underlying message. This approach the granularity or ordering of the underlying message. This approach
assumes that the ULP is only interested in the complete set of assumes that the ULP is only interested in the complete set of
messages, and has no use for a subset of them. messages, and has no use for a subset of them.
6.9 LLP Independent Session Establishment 6.9. LLP Independent Session Establishment
For an RDMAP/DDP application, the transport services provided by a For an RDMAP/DDP application, the transport services provided by a
pair of SCTP Streams and by a TCP connection both provide the same pair of SCTP Streams and by a TCP connection both provide the same
service (reliable delivery of DDP Segments between two connected service (reliable delivery of DDP Segments between two connected
RDMAP/DDP endpoints). RDMAP/DDP endpoints).
6.9.1 RDMA-only Session Establishment 6.9.1. RDMA-only Session Establishment
It is also possible to allow for transport neutral establishment of It is also possible to allow for transport neutral establishment of
RDMAP/DDP sessions between endpoints. Combined, these two features RDMAP/DDP sessions between endpoints. Combined, these two features
would allow most applications to be unconcerned as to which LLP was would allow most applications to be unconcerned as to which LLP was
actually in use. actually in use.
Specifically, the procedures for DDP Stream Session establishment Specifically, the procedures for DDP Stream Session establishment
discussed in section 3 of the SCTP mapping, and section 13.3 of the discussed in section 3 of the SCTP mapping, and section 13.3 of the
MPA/TCP mapping, both allow for the exchange of ULP specific data MPA/TCP mapping, both allow for the exchange of ULP specific data
("Private Data") before enabling the exchange of DDP Segments. This ("Private Data") before enabling the exchange of DDP Segments. This
skipping to change at page 18, line 26 skipping to change at page 18, line 33
To be transport neutral, the applications should exchange Private To be transport neutral, the applications should exchange Private
Data as part of session establishment messages to determine how the Data as part of session establishment messages to determine how the
RDMA endpoints are to be configured. One side must be the Initiator, RDMA endpoints are to be configured. One side must be the Initiator,
and the other the Responder. and the other the Responder.
With SCTP, a pair of SCTP streams can be used for sequential With SCTP, a pair of SCTP streams can be used for sequential
sessions. With MPA/TCP each connection can be used for at most one sessions. With MPA/TCP each connection can be used for at most one
session. However, the same source/destination pair of ports can be session. However, the same source/destination pair of ports can be
re-used sequentially subject to normal TCP rules. re-used sequentially subject to normal TCP rules.
SCTP limits the Stream Session Initiate control message to a single Both SCTP and MPA limit the private data size to a maximum of 512
SCTP Data Chunk, or 516 bytes whichever is greater. MPA only limits bytes.
the private data in a MPA Request or Response frame to 65,535 bytes.
An IP transport neutral application SHOULD limit itself to 512 bytes
or less of Private Data in the connection establishment phase. An
application that wanted to be InfiniBand compatible as well would
need to limit the size even further.
MPA/TCP requires the end of the TCP connection that initiated the MPA/TCP requires the end of the TCP connection that initiated the
conversion to MPA mode to send the first DDP Segment. SCTP does not conversion to MPA mode to send the first DDP Segment. SCTP does not
have this requirement. ULPs which wish to be transport neutral have this requirement. ULPs which wish to be transport neutral
should require the initiating end to send the first message. A should require the initiating end to send the first message. A zero-
zero-length RDMA Write can be used for this purpose if the ULP logic length RDMA Write can be used for this purpose if the ULP logic
itself does naturally support this restriction. itself does naturally support this restriction.
6.9.2 RDMA-Conditional Session Establishment 6.9.2. RDMA-Conditional Session Establishment
It is sometimes desirable for the active side of a session to connect It is sometimes desirable for the active side of a session to connect
with the passive side before knowing whether the passive side with the passive side before knowing whether the passive side
supports RDMA. supports RDMA.
This style of session establishment can be supported with either TCP This style of session establishment can be supported with either TCP
or SCTP, but not as transparently as for RDMA-only sessions. or SCTP, but not as transparently as for RDMA-only sessions. Pre-
Pre-existing non-RDMA servers are also far more likely to be using existing non-RDMA servers are also far more likely to be using TCP
TCP than SCTP. than SCTP.
With TCP. a normal TCP connection is established. It is then used With TCP. a normal TCP connection is established. It is then used by
by the ULP to determine whether or not to convert to MPA mode and use the ULP to determine whether or not to convert to MPA mode and use
RDMA. This will typically be integral with other session RDMA. This will typically be integral with other session
establishment negotiations. establishment negotiations.
With SCTP, the establishment of an association tests whether RDMA is With SCTP, the establishment of an association tests whether RDMA is
supported. If not supported, the application simply requests the supported. If not supported, the application simply requests the
association without the RDMA adaptation indication. association without the RDMA adaptation indication.
In key difference is that with SCTP the determination as to whether In key difference is that with SCTP the determination as to whether
the peer can support RDMA is made before the transport layer the peer can support RDMA is made before the transport layer
association/connection is established while with TCP the established association/connection is established while with TCP the established
skipping to change at page 21, line 7 skipping to change at page 21, line 7
7. Local Interface Implications 7. Local Interface Implications
Full utilization of DDP and RDMAP capabilities requires a local Full utilization of DDP and RDMAP capabilities requires a local
interface that explicitly requests these services. Protocols such as interface that explicitly requests these services. Protocols such as
Sockets Direct Protocol (SDP) can allow applications to keep their Sockets Direct Protocol (SDP) can allow applications to keep their
traditional byte-stream or message-stream interface and still enjoy traditional byte-stream or message-stream interface and still enjoy
many of the benefits of the optimized wire level protocols. many of the benefits of the optimized wire level protocols.
8. Security considerations 8. Security considerations
8.1 Connection/Association Setup 8.1. Connection/Association Setup
Both the SCTP and TCP adaptations allow for existing procedures to be Both the SCTP and TCP adaptations allow for existing procedures to be
followed for the establishment of the SCTP association or TCP followed for the establishment of the SCTP association or TCP
connection. Use of DDP does not impair the use of any security connection. Use of DDP does not impair the use of any security
measures to filter, validate and/or log the remote end of an measures to filter, validate and/or log the remote end of an
association/connection. association/connection.
8.2 Tagged Buffer Exposure 8.2. Tagged Buffer Exposure
DDP only exposes ULP memory to the extent explicitly allowed by ULP DDP only exposes ULP memory to the extent explicitly allowed by ULP
actions. These include posting of receive operations and enabling of actions. These include posting of receive operations and enabling of
Steering Tags. Steering Tags.
Neither RDMAP or DDP place requirements on how ULP's advertise Neither RDMAP or DDP place requirements on how ULP's advertise
buffers. A ULP may use a single Steering Tag for multiple buffer buffers. A ULP may use a single Steering Tag for multiple buffer
advertisements. However, the ULP should be aware that enforcement on advertisements. However, the ULP should be aware that enforcement on
STag usage is likely limited to the overall range that is enabled. STag usage is likely limited to the overall range that is enabled.
If the remote peer writes into the 'wrong' advertised buffer, neither If the remote peer writes into the 'wrong' advertised buffer, neither
the DDP or RDMAP layer will be aware of this. Nor is there any the DDP or RDMAP layer will be aware of this. Nor is there any
report to the ULP on how the remote peer specifically used tagged report to the ULP on how the remote peer specifically used tagged
buffers. buffers.
Unless the ULP peers have an adequate basis for mutual trust, the Unless the ULP peers have an adequate basis for mutual trust, the
receiving ULP might be well advised to use a distinct STag for each receiving ULP might be well advised to use a distinct STag for each
interaction, and to invalidate it after each use or to require its interaction, and to invalidate it after each use or to require its
peer to use the RDMAP option to invalidate the STag with its peer to use the RDMAP option to invalidate the STag with its
responding untagged message. responding untagged message.
8.3 Impact of Encrypted Transports 8.3. Impact of Encrypted Transports
While DDP is cleanly layered over the LLP, its maximum benefit may be While DDP is cleanly layered over the LLP, its maximum benefit may be
limited when the LLP Stream is secured with a streaming cypher, such limited when the LLP Stream is secured with a streaming cypher, such
as Transport Layer Security (TLS). If the LLP must decrypt in order, as Transport Layer Security (TLS). If the LLP must decrypt in order,
it cannot provide out-of-order DDP Segments to the DDP layer for it cannot provide out-of-order DDP Segments to the DDP layer for
placement purposes. IPsec tunnel mode encrypts entire IP Datagrams. placement purposes. IPsec tunnel mode encrypts entire IP Datagrams.
IPsec transport mode encrypts TCP Segments or SCTP packets. In IPsec transport mode encrypts TCP Segments or SCTP packets. In
neither case should IPsec preclude providing out-of-order DDP neither case should IPsec preclude providing out-of-order DDP
Segments to the DDP layer for placement. Segments to the DDP layer for placement.
Note that end-to-end use of IPsec cryptographic integrity protection Note that end-to-end use of IPsec cryptographic integrity protection
may allow suppression of MPA CRC generation and checking under may allow suppression of MPA CRC generation and checking under
certain circumstances. This is one example where the LLP may be certain circumstances. This is one example where the LLP may be
judged to have "or equivalent" protection to an end-to-end CRC32c. judged to have "or equivalent" protection to an end-to-end CRC32c.
9 References 9. References
[1] Bradner, S., "Key words for use in RFCs to Indicate Requirement [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", BCP 14, RFC 2119, March 1997. Levels", BCP 14, RFC 2119, March 1997.
[2] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0", RFC [2] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0",
2246, January 1999. RFC 2246, January 1999.
[3] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload [3] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload
(ESP)", RFC 2406, November 1998. (ESP)", RFC 2406, November 1998.
[4] Stewart, R., Xie, Q., Morneault, K., Sharp, C., Schwarzbauer, [4] Stewart, R., Xie, Q., Morneault, K., Sharp, C., Schwarzbauer,
H., Taylor, T., Rytina, I., Kalla, M., Zhang, L. and V. Paxson, H., Taylor, T., Rytina, I., Kalla, M., Zhang, L., and V.
"Stream Control Transmission Protocol", RFC 2960, October 2000. Paxson, "Stream Control Transmission Protocol", RFC 2960,
October 2000.
[5] Coene, L., "Stream Control Transmission Protocol Applicability [5] Coene, L., "Stream Control Transmission Protocol Applicability
Statement", RFC 3257, April 2002. Statement", RFC 3257, April 2002.
[6] Recio, R., "An RDMA Protocol Specification", [6] Recio, R., "An RDMA Protocol Specification",
draft-ietf-rddp-rdmap-02 (work in progress), September 2004. draft-ietf-rddp-rdmap-05 (work in progress), July 2005.
[7] Shah, H., "Direct Data Placement over Reliable Transports", [7] Shah, H., "Direct Data Placement over Reliable Transports",
draft-ietf-rddp-ddp-03 (work in progress), August 2004. draft-ietf-rddp-ddp-05 (work in progress), July 2005.
[8] Stewart, R., "Stream Control Transmission Protocol (SCTP) Remote [8] Stewart, R., "Stream Control Transmission Protocol (SCTP)
Direct Memory Access (RDMA) Direct Data Placement (DDP) Remote Direct Memory Access (RDMA) Direct Data Placement (DDP)
Adaptationn", draft-ietf-rddp-sctp-02 (work in progress), August Adaptationn", draft-ietf-rddp-sctp-02 (work in progress),
2004. August 2005.
[9] Culley, P., "Marker PDU Aligned Framing for TCP Specification", [9] Culley, P., "Marker PDU Aligned Framing for TCP Specification",
draft-ietf-rddp-mpa-01 (work in progress), July 2004. draft-ietf-rddp-mpa-02 (work in progress), February 2005.
[10] "Direct Access File System versino 1.0", September 2001.
[11] Pinkerton, J., "Sockets Direct Protocol (SDP) for iWARP over
TCP 1.0", October 2003.
Authors' Addresses Authors' Addresses
Caitlin Bestler Caitlin Bestler
1241 W. North Shore Broadcom
# 2G 49 Discovery
Chicago, IL 60626 Irvine, CA 92618
USA USA
Phone: +1-773-743-1594 Phone: 949-926-6383
EMail: cait@asomi.com Email: caitlinb@broadcom.com
Lode Coene Lode Coene
Siemens
Atealaan 26 Atealaan 26
Herentals, 2200 Herentals, 2200
Belgium Belgium
Phone: +32-14-252081 Phone: +32-14-252081
EMail: lode.coene@siemens.com Email: lode.coene@siemens.com
Intellectual Property Statement Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be on the procedures with respect to rights in RFC documents can be
skipping to change at page 24, line 41 skipping to change at page 24, line 41
This document and the information contained herein are provided on an This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement Copyright Statement
Copyright (C) The Internet Society (2004). This document is subject Copyright (C) The Internet Society (2005). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights. except as set forth therein, the authors retain all their rights.
Acknowledgment Acknowledgment
Funding for the RFC Editor function is currently provided by the Funding for the RFC Editor function is currently provided by the
Internet Society. Internet Society.
 End of changes. 66 change blocks. 
141 lines changed or deleted 150 lines changed or added

This html diff was produced by rfcdiff 1.27, available from http://www.levkowetz.com/ietf/tools/rfcdiff/