draft-ietf-rddp-applicability-06.txt   draft-ietf-rddp-applicability-07.txt 
Remote Direct Data Placement C. Bestler Remote Direct Data Placement C. Bestler
Working group Broadcom Corporation Working group Broadcom Corporation
Internet-Draft L. Coene Internet-Draft L. Coene
Expires: October 26, 2006 Siemens Expires: December 1, 2006 Siemens
April 24, 2006 May 30, 2006
Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct
Data Placement (DDP) Data Placement (DDP)
draft-ietf-rddp-applicability-06.txt draft-ietf-rddp-applicability-07.txt
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 36 skipping to change at page 1, line 36
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on October 26, 2006. This Internet-Draft will expire on December 1, 2006.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2006). Copyright (C) The Internet Society (2006).
Abstract Abstract
This document describes the applicability of Remote Direct Memory This document describes the applicability of Remote Direct Memory
Access Protocol (RDMAP) and the Direct Data Placement Protocol (DDP). Access Protocol (RDMAP) and the Direct Data Placement Protocol (DDP).
It compares and contrasts the different transport options over IP It compares and contrasts the different transport options over IP
skipping to change at page 2, line 12 skipping to change at page 2, line 12
between available transports and/or how to be indifferent to the between available transports and/or how to be indifferent to the
specific transport layer used, compares use of DDP with direct use of specific transport layer used, compares use of DDP with direct use of
the supporting transports, and compares DDP over IP transports with the supporting transports, and compares DDP over IP transports with
non-IP transports that support RDMA functionality. non-IP transports that support RDMA functionality.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Direct Placement . . . . . . . . . . . . . . . . . . . . . . . 6 3. Direct Placement . . . . . . . . . . . . . . . . . . . . . . . 6
3.1. Fewer Required ULP Interactions . . . . . . . . . . . . . 6 3.1. Direct Placement using only the LLP . . . . . . . . . . . 6
3.2. Direct Placement using only the LLP . . . . . . . . . . . 6 3.2. Fewer Required ULP Interactions . . . . . . . . . . . . . 7
4. Tagged Messages . . . . . . . . . . . . . . . . . . . . . . . 8 4. Tagged Messages . . . . . . . . . . . . . . . . . . . . . . . 8
4.1. Order Independent Reception . . . . . . . . . . . . . . . 8 4.1. Order Independent Reception . . . . . . . . . . . . . . . 8
4.2. Reduced ULP Notifications . . . . . . . . . . . . . . . . 9 4.2. Reduced ULP Notifications . . . . . . . . . . . . . . . . 9
4.3. Simplified ULP Exchanges . . . . . . . . . . . . . . . . . 9 4.3. Simplified ULP Exchanges . . . . . . . . . . . . . . . . . 9
4.4. Order Independent Sending . . . . . . . . . . . . . . . . 11 4.4. Order Independent Sending . . . . . . . . . . . . . . . . 11
4.5. Untagged Messages and Tagged Buffers as ULP Credits . . . 12 4.5. Untagged Messages and Tagged Buffers as ULP Credits . . . 12
5. RDMA Read . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5. RDMA Read . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6. LLP Comparisons . . . . . . . . . . . . . . . . . . . . . . . 15 6. LLP Comparisons . . . . . . . . . . . . . . . . . . . . . . . 15
6.1. Multistreaming Implications . . . . . . . . . . . . . . . 15 6.1. Multistreaming Implications . . . . . . . . . . . . . . . 15
6.2. Out of Order Reception Implications . . . . . . . . . . . 15 6.2. Out of Order Reception Implications . . . . . . . . . . . 15
skipping to change at page 2, line 41 skipping to change at page 2, line 41
6.7.1. No RDMA Layer Ack . . . . . . . . . . . . . . . . . . 17 6.7.1. No RDMA Layer Ack . . . . . . . . . . . . . . . . . . 17
6.8. Other IP Transports . . . . . . . . . . . . . . . . . . . 18 6.8. Other IP Transports . . . . . . . . . . . . . . . . . . . 18
6.9. LLP Independent Session Establishment . . . . . . . . . . 19 6.9. LLP Independent Session Establishment . . . . . . . . . . 19
6.9.1. RDMA-only Session Establishment . . . . . . . . . . . 19 6.9.1. RDMA-only Session Establishment . . . . . . . . . . . 19
6.9.2. RDMA-Conditional Session Establishment . . . . . . . . 19 6.9.2. RDMA-Conditional Session Establishment . . . . . . . . 19
7. Local Interface Implications . . . . . . . . . . . . . . . . . 21 7. Local Interface Implications . . . . . . . . . . . . . . . . . 21
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22
9. Security considerations . . . . . . . . . . . . . . . . . . . 23 9. Security considerations . . . . . . . . . . . . . . . . . . . 23
9.1. Connection/Association Setup . . . . . . . . . . . . . . . 23 9.1. Connection/Association Setup . . . . . . . . . . . . . . . 23
9.2. Tagged Buffer Exposure . . . . . . . . . . . . . . . . . . 23 9.2. Tagged Buffer Exposure . . . . . . . . . . . . . . . . . . 23
9.3. Impact of Encrypted Transports . . . . . . . . . . . . . . 24 9.3. Impact of Encrypted Transports . . . . . . . . . . . . . . 23
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25
10.1. Normative references . . . . . . . . . . . . . . . . . . . 25 10.1. Normative references . . . . . . . . . . . . . . . . . . . 25
10.2. Informative References . . . . . . . . . . . . . . . . . . 25 10.2. Informative References . . . . . . . . . . . . . . . . . . 25
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26
Intellectual Property and Copyright Statements . . . . . . . . . . 27 Intellectual Property and Copyright Statements . . . . . . . . . . 27
1. Introduction 1. Introduction
Remote Direct Memory Access Protocol [6] and Direct Data Placement Remote Direct Memory Access Protocol (RDMAP) and Direct Data
[7] work together to provide application independent efficient Placement (DDP) work together to provide application independent
placement of application payload directly into buffers specified by efficient placement of application payload directly into buffers
the Upper Layer Protocol (ULP). specified by the Upper Layer Protocol (ULP).
The DDP protocol is responsible for direct placement of received The DDP protocol is responsible for direct placement of received
payload into ULP specified buffers. The RDMAP protocol provides payload into ULP specified buffers. The RDMAP protocol provides
completion notifications to the ULP and support for Data Sink completion notifications to the ULP and support for Data Sink
initiated fetch of advertised buffers (RDMA Reads). initiated fetch of advertised buffers (RDMA Reads).
DDP and RDMAP are both application independent protocols which allow DDP and RDMAP are both application independent protocols which allow
the ULP to perform remote direct data placement. DDP can use the ULP to perform remote direct data placement. DDP can use
multiple standard IP transports including SCTP and TCP. multiple standard IP transports including SCTP and TCP.
skipping to change at page 4, line 19 skipping to change at page 4, line 19
cleanly as a layer over existing IP transports, DDP has simpler cleanly as a layer over existing IP transports, DDP has simpler
ordering rules than some prior RDMA protocols. This may have some ordering rules than some prior RDMA protocols. This may have some
implications for application designers. implications for application designers.
The full capabilities of DDP and RDMAP can only be fully realized by The full capabilities of DDP and RDMAP can only be fully realized by
applications that are designed to exploit them. The co-existence of applications that are designed to exploit them. The co-existence of
RDMAP/DDP aware local interfaces with traditional socket interfaces RDMAP/DDP aware local interfaces with traditional socket interfaces
will also be explored. will also be explored.
Finally, DDP support is defined for at least two IP transports: SCTP Finally, DDP support is defined for at least two IP transports: SCTP
[8] and MPA over TCP [10]. The rationale for supporting both and TCP. The rationale for supporting both transports is reviewed,
transports is reviewed, as well as when each would be the appropriate as well as when each would be the appropriate selection.
selection.
2. Definitions 2. Definitions
Advertisement - the act of informing a Remote Peer that a local RDMA Advertisement - the act of informing a Remote Peer that a local RDMA
Buffer is available to it. A Node makes available an RDMA Buffer Buffer is available to it. A Node makes available an RDMA Buffer
for incoming RDMA Read or RDMA Write access by informing its RDMA/ for incoming RDMA Read or RDMA Write access by informing its RDMA/
DDP peer of the Tagged Buffer identifiers (STag, base address, and DDP peer of the Tagged Buffer identifiers (STag, base address, and
buffer length). This advertisement of Tagged Buffer information buffer length). This advertisement of Tagged Buffer information
is not defined by RDMA/DDP and is left to the ULP. A typical is not defined by RDMA/DDP and is left to the ULP. A typical
method would be for the Local Peer to embed the Tagged Buffer's method would be for the Local Peer to embed the Tagged Buffer's
skipping to change at page 5, line 45 skipping to change at page 5, line 45
message sender. The message receiver is given no independent message sender. The message receiver is given no independent
indication that a tagged message has been received. indication that a tagged message has been received.
Untagged Message A DDP message that is directed to a ULP specified Untagged Message A DDP message that is directed to a ULP specified
buffer based upon a Message Sequence Number being matched with a buffer based upon a Message Sequence Number being matched with a
receiver supplied buffer. The destination buffer is specified by receiver supplied buffer. The destination buffer is specified by
the message receiver. The message receiver is notified by some the message receiver. The message receiver is notified by some
mechanism that an untagged message has been received. mechanism that an untagged message has been received.
Upper Layer Protocol (ULP) The direct user of RDMAP/DDP services. In Upper Layer Protocol (ULP) The direct user of RDMAP/DDP services. In
addition to protocols such as iSER [11] and NFSv4 over RDMA [12], addition to protocols such as iSER [8] and NFSv4 over RDMA [9],
the ULP may be embedded in an application, or a middleware layer the ULP may be embedded in an application, or a middleware layer
as is often the case for the Sockets Direct Protocol (SDP) and as is often the case for the Sockets Direct Protocol (SDP) and
Remote Procedure Call (RPC) protocols. Remote Procedure Call (RPC) protocols.
3. Direct Placement 3. Direct Placement
Direct Data Placement optimizes the placement of ULP payload into the Direct Data Placement optimizes the placement of ULP payload into the
correct destination buffers, typically eliminating intermediate correct destination buffers, typically eliminating intermediate
copying. Placement is enabled without regard to order of arrival, copying. Placement is enabled without regard to order of arrival,
order of transmission or requiring per-placement interaction with the order of transmission or requiring per-placement interaction with the
ULP. ULP.
RDMAP minimizes the required ULP interactions . This capability is RDMAP minimizes the required ULP interactions . This capability is
most valuable for applications that require multiple transport layer most valuable for applications that require multiple transport layer
packets for each required ULP interaction. packets for each required ULP interaction.
3.1. Fewer Required ULP Interactions 3.1. Direct Placement using only the LLP
While reducing the number of required ULP interactions is in itself
desirable, it is critical for high speed connections. The burst
packet rate for a high speed interface could easily exceed the host
systems ability to switch ULP contexts.
Content access applications are primary examples of applications with
both high bandwidth and high content to required ULP interaction
ratios. These applications include file access protocols (NAS),
storage access (SAN), database access and other application specific
forms of content access such as HTTP, XML and email.
3.2. Direct Placement using only the LLP
Direct data placement can be achieved without RDMA. Pre-posting of Direct data placement can be achieved without RDMA. Pre-posting of
receive buffers could allow a non-RDMA network stack to place data receive buffers could allow a non-RDMA network stack to place data
directly to user buffers. directly to user buffers.
The degree to which DDP optimizes depends on which transport is being The degree to which DDP optimizes depends on which transport is being
compared with, and on the nature of the local interface. Without compared with, and on the nature of the local interface. Without
RDMAP/DDP pre-posting buffers requires the receiving side to RDMAP/DDP pre-posting buffers requires the receiving side to
accurately predict the required buffers and their sizes. This is not accurately predict the required buffers and their sizes. This is not
feasible for all ULPs. By contrast, DDP only requires the ULP to feasible for all ULPs. By contrast, DDP only requires the ULP to
predict the sequence and size of incoming untagged messages. predict the sequence and size of incoming untagged messages.
An application that could predict incoming messages and required An application that could predict incoming messages and required
nothing more than direct placement into buffers might be able to do nothing more than direct placement into buffers might be able to do
so with a properly designed local interface to SCTP or TCP. Doing so so with a properly designed local interface to native SCTP or TCP
for TCP requires making predictions at a byte level rather than a (without RDMA). This is easier using native SCTP because the
message level. application would only have to predict the sequence of messages and
the maximum size of each message, not the exact size of each message.
The main benefit of DDP for such an application would be that pre- The main benefit of DDP for such an application would be that pre-
posting of receive buffers is a mandated local interface capability, posting of receive buffers is a mandated local interface capability,
and that predictions can be made on a per-message basis (not per and that predictions can always be made on a per-message basis (not
byte). per byte).
The Lower Layer Protocol, LLP, can also be used directly if ULP The Lower Layer Protocol, LLP, can also be used directly if ULP
specific knowledge is built into the protocol stack to allow "parse specific knowledge is built into the protocol stack to allow "parse
and place" handling of received packets. Such a solution either and place" handling of received packets. Such a solution either
requires interaction with the ULP, or that the protocol stack have requires interaction with the ULP, or that the protocol stack have
knowledge of ULP specific syntax rules. knowledge of ULP specific syntax rules.
DDP achieves the benefits of directly placing incoming payload DDP achieves the benefits of directly placing incoming payload
without requiring tight coupling between the ULP and the protocol without requiring tight coupling between the ULP and the protocol
stack. However, "parse and place" capabilities can certainly provide stack. However, "parse and place" capabilities can certainly provide
equivalent services to a limited number of ULPs. equivalent services to a limited number of ULPs.
3.2. Fewer Required ULP Interactions
While reducing the number of required ULP interactions is in itself
desirable, it is critical for high speed connections. The burst
packet rate for a high speed interface could easily exceed the host
systems ability to switch ULP contexts.
Content access applications are primary examples of applications with
both high bandwidth and a high ratio of content transferred per
required ULP interaction. These applications include file access
protocols (NAS), storage access (SAN), database access and other
application specific forms of content access such as HTTP, XML and
email.
4. Tagged Messages 4. Tagged Messages
This section covers the major benefits from the use of Tagged This section covers the major benefits from the use of Tagged
Messages. Messages.
A more critical advantage of DDP is the ability of the Data Source to A more critical advantage of DDP is the ability of the Data Source to
use tagged buffers. Tagging messages allows the Data Source to use tagged buffers. Tagging messages allows the Data Source to
choose the ordering and packetization of its payload deliveries. choose the ordering and packetization of its payload deliveries.
With direct data placement based solely upon pre-posted receives, the With direct data placement based solely upon pre-posted receives, the
packetization and delivery of payload must be agreed by the ULP peers packetization and delivery of payload must be agreed by the ULP peers
in advance. in advance.
The Upper Layer Protocol can allocate content between untagged and/or The Upper Layer Protocol can allocate content between untagged and/or
tagged messages to maximize the potential optimizations. Placing tagged messages to maximize the potential optimizations. Placing
content within an untagged message can deliver the content in the content within an untagged message can deliver the content in the
same packet that signals completion to the receiver. This can same packet that signals completion to the receiver. This can
improve latency. It can even eliminate round trips. But it requires improve latency. It can even eliminate round trips. But it requires
making larger anonymous buffers to be available. making larger anonymous buffers to be available.
Some examples of data that typically belongs in the untagged message Some examples of data that typically belongs in the untagged message
would include short fixed size control data that is inherently part would include:
of the control message almost always should be included in the
untagged message, relatively short payload that is almost always short fixed-size control data that is inherently part of the
needed (especially when it would eliminate a round-trip to fetch the control message. This is especially true when the data is a
data. For example, the initial data on a write request, and of required part of the control message.
course advertising tagged buffers that specify the location of data
not in the untagged message. relatively short payload that is almost always needed, especially
when its inclusion would eliminate a round-trip to fetch the data.
Examples would include the initial data on a write request and
advertisements of tagged buffers.
Tagged messages standardizes direct placement of data without per- Tagged messages standardizes direct placement of data without per-
packet interaction with the upper layers. Even if there is an upper packet interaction with the upper layers. Even if there is an upper
layer protocol encoding of what is being transferred, as is common layer protocol encoding of what is being transferred, as is common
with middleware solutions, this information is not understood at the with middleware solutions, this information is not understood at the
application independent layers. The directions on where to place the application independent layers. The directions on where to place the
incoming data cannot be accessed without switching to the ULP first. incoming data cannot be accessed without switching to the ULP first.
DDP provides a standardized 'packing list' which can be interpreted DDP provides a standardized 'packing list' which can be interpreted
without requiring ULP interaction. Indeed, it is designed to be without requiring ULP interaction. Indeed, it is designed to be
implementable in hardware. implementable in hardware.
skipping to change at page 10, line 8 skipping to change at page 10, line 8
interaction. interaction.
4.3. Simplified ULP Exchanges 4.3. Simplified ULP Exchanges
The notification rules for Tagged Messages allows ULPs to create The notification rules for Tagged Messages allows ULPs to create
multi-message "exchanges" consisting of zero or more tagged messages multi-message "exchanges" consisting of zero or more tagged messages
that represent a single step in the ULP interaction. The receiving that represent a single step in the ULP interaction. The receiving
ULP is notified that the untagged message has arrived, and implicitly ULP is notified that the untagged message has arrived, and implicitly
of any associated tagged messages. of any associated tagged messages.
A ULP where all exchanges would naturally be only the untagged A ULP where all exchanges would naturally be untagged messages would
message would derive virtually no benefit from the use of RDMAP/DDP derive virtually no benefit from the use of RDMAP/DDP as opposed to
as opposed to SCTP. But while tagged buffers are the justification SCTP directly. But while tagged buffers are the justification for
for RDMAP/DDP, untagged buffers are still necessary. Without RDMAP/DDP, untagged buffers are still necessary. Without untagged
untagged buffers the only method to exchange buffer advertisements buffers the only method to exchange buffer advertisements would
would involve out-of-band communications and/or sharing of compile require out-of-band communications. Most RDMA-aware ULPs use
time constants. Most RDMA-aware ULPs use untagged buffers for untagged buffers for requests and responses. Buffer advertisements
requests and responses. Buffer advertisements are typically done are typically done within these untagged messages.
within these untagged messages.
More importantly there would be no reliable method for the upper More importantly there would be no reliable method for the upper
layer peers to synchronize. The absence of any guarantees about layer peers to synchronize. The absence of any guarantees about
ordering within or between tagged messages is fundamental to allowing ordering within or between tagged messages is fundamental to allowing
the DDP layer to optimize transfer of tagged payload. the DDP layer to optimize transfer of tagged payload.
So no ULP can be defined entirely in terms of tagged messages. So no ULP can be defined entirely in terms of tagged messages.
Eventually a notification that confirms delivery must be generated Eventually a notification that confirms delivery must be generated
from the RDMAP/DDP layer. from the RDMAP/DDP layer.
skipping to change at page 16, line 33 skipping to change at page 16, line 33
protection against data accidental corruption, or its equivalent. protection against data accidental corruption, or its equivalent.
A ULP that requires a greater degree of protection may add it own. A ULP that requires a greater degree of protection may add it own.
However, DDP and RDMAP headers will only be guaranteed to have the However, DDP and RDMAP headers will only be guaranteed to have the
equivalent of end-to-end CRC32c protection. A ULP that requires data equivalent of end-to-end CRC32c protection. A ULP that requires data
integrity checking more thorough than an end-to-end CRC32c should integrity checking more thorough than an end-to-end CRC32c should
first invalidate all STags that reference a buffer before applying first invalidate all STags that reference a buffer before applying
their own integrity check. their own integrity check.
CRC32c only provides protection against random corruption. To CRC32c only provides protection against random corruption. To
protect against unauthorized alteration or forging of data packets, protect against unauthorized alteration or forging of data packets
security methods must be applied. IPsec is supported for both SCTP security methods must be applied. The security draft [RDMA-Security]
and MPA/TCP. [7] specifies usage of RFC2406 [2] for both adaptation layers.
6.6.1. MPA/TCP Specifics 6.6.1. MPA/TCP Specifics
It is mandatory for MPA/TCP implementations to implement CRC32c, but It is mandatory for MPA/TCP implementations to implement CRC32c, but
it is NOT mandatory to use the CRC32c during an RDMA connection. The it is NOT mandatory to use the CRC32c during an RDMA connection. The
activating or deactivating of the CRC in MPA/TCP is an administrative activating or deactivating of the CRC in MPA/TCP is an administrative
configuration operation at the local and remote end. The configuration operation at the local and remote end. The
administration of the CRC(ON/OFF) is invisible to the ULP. administration of the CRC(ON/OFF) is invisible to the ULP.
Applications SHOULD trust that this administrative option will only Applications SHOULD trust that this administrative option will only
skipping to change at page 19, line 33 skipping to change at page 19, line 33
delay can allow for proper selection and/or configuration of the delay can allow for proper selection and/or configuration of the
endpoints based upon the exchanged data. For example, each DDP endpoints based upon the exchanged data. For example, each DDP
Stream Session associated with a single client session might be Stream Session associated with a single client session might be
assigned to the same DDP Protection Domain. assigned to the same DDP Protection Domain.
To be transport neutral, the applications should exchange Private To be transport neutral, the applications should exchange Private
Data as part of session establishment messages to determine how the Data as part of session establishment messages to determine how the
RDMA endpoints are to be configured. One side must be the Initiator, RDMA endpoints are to be configured. One side must be the Initiator,
and the other the Responder. and the other the Responder.
With SCTP, a pair of SCTP streams can be used for sequential With SCTP, a pair of SCTP streams can be used for successive sessions
sessions. With MPA/TCP each connection can be used for at most one while the SCTP association remains open. With MPA/TCP each
session. However, the same source/destination pair of ports can be connection can be used for at most one session. However, the same
re-used sequentially subject to normal TCP rules. source/destination pair of ports can be re-used sequentially subject
to normal TCP rules.
Both SCTP and MPA limit the private data size to a maximum of 512 Both SCTP and MPA limit the private data size to a maximum of 512
bytes. bytes.
MPA/TCP requires the end of the TCP connection that initiated the MPA/TCP requires the end of the TCP connection that initiated the
conversion to MPA mode to send the first DDP Segment. SCTP does not conversion to MPA mode to send the first DDP Segment. SCTP does not
have this requirement. ULPs which wish to be transport neutral have this requirement. ULPs which wish to be transport neutral
should require the initiating end to send the first message. A zero- should require the initiating end to send the first message. A zero-
length RDMA Write can be used for this purpose if the ULP logic length RDMA Write can be used for this purpose if the ULP logic
itself does naturally support this restriction. itself does naturally support this restriction.
skipping to change at page 23, line 7 skipping to change at page 23, line 7
Sockets Direct Protocol (SDP) can allow applications to keep their Sockets Direct Protocol (SDP) can allow applications to keep their
traditional byte-stream or message-stream interface and still enjoy traditional byte-stream or message-stream interface and still enjoy
many of the benefits of the optimized wire level protocols. many of the benefits of the optimized wire level protocols.
8. IANA Considerations 8. IANA Considerations
There are no IANA considerations in this document. There are no IANA considerations in this document.
9. Security considerations 9. Security considerations
RDMA security considerations are discussed in [RDMA-SEC] [7]. This
document will only deal with the more usage oriented aspects, and
where there are implications in the choice of underlying transport.
9.1. Connection/Association Setup 9.1. Connection/Association Setup
Both the SCTP and TCP adaptations allow for existing procedures to be Both the SCTP and TCP adaptations allow for existing procedures to be
followed for the establishment of the SCTP association or TCP followed for the establishment of the SCTP association or TCP
connection. Use of DDP does not impair the use of any security connection. Use of DDP does not impair the use of any security
measures to filter, validate and/or log the remote end of an measures to filter, validate and/or log the remote end of an
association/connection. association/connection.
Authentication of peers and approval of connections is outside of the
scope of DDP. Connection authentication is the responsibility of the
ULP, which may be based upon information from the LLP. IPSEC is
usable for both TCP and SCTP.
9.2. Tagged Buffer Exposure 9.2. Tagged Buffer Exposure
DDP only exposes ULP memory to the extent explicitly allowed by ULP DDP only exposes ULP memory to the extent explicitly allowed by ULP
actions. These include posting of receive operations and enabling of actions. These include posting of receive operations and enabling of
Steering Tags. Steering Tags.
DDP validates that STags are only used by the remote peer to the
extent authorized by the ULP. The STag selects from a pool of
buffers previously authorized by the ULP; an STag by itself does not
authorize access.
Use of randomization in generating STag values may be useful in
preventing 'off by one' and other programmatic errors, but is of
limited value in countering generation and misuse of STag values by
an active attacker. IPsec provides countermeasures that can prevent
such an unauthorized attacker from gaining access to buffers used by
DDP and RDMAP.
Neither RDMAP or DDP place requirements on how ULP's advertise Neither RDMAP or DDP place requirements on how ULP's advertise
buffers. A ULP may use a single Steering Tag for multiple buffer buffers. A ULP may use a single Steering Tag for multiple buffer
advertisements. However, the ULP should be aware that enforcement on advertisements. However, the ULP should be aware that enforcement on
STag usage is likely limited to the overall range that is enabled. STag usage is likely limited to the overall range that is enabled.
If the remote peer writes into the 'wrong' advertised buffer, neither If the remote peer writes into the 'wrong' advertised buffer, neither
the DDP or RDMAP layer will be aware of this. Nor is there any the DDP or RDMAP layer will be aware of this. Nor is there any
report to the ULP on how the remote peer specifically used tagged report to the ULP on how the remote peer specifically used tagged
buffers. buffers.
Unless the ULP peers have an adequate basis for mutual trust, the Unless the ULP peers have an adequate basis for mutual trust, the
receiving ULP might be well advised to use a distinct STag for each receiving ULP might be well advised to use a distinct STag for each
interaction, and to invalidate it after each use or to require its interaction, and to invalidate it after each use or to require its
peer to use the RDMAP option to invalidate the STag with its peer to use the RDMAP option to invalidate the STag with its
responding untagged message. responding untagged message.
9.3. Impact of Encrypted Transports 9.3. Impact of Encrypted Transports
While DDP is cleanly layered over the LLP, its maximum benefit may be While DDP is cleanly layered over the LLP, its maximum benefit may be
limited when the LLP Stream is secured with a streaming cypher, such limited when the LLP Stream is secured with a streaming cypher, such
as Transport Layer Security (TLS). If the LLP must decrypt in order, as Transport Layer Security (TLS) RFC2246 [1]. If the LLP must
it cannot provide out-of-order DDP Segments to the DDP layer for decrypt in order, it cannot provide out-of-order DDP Segments to the
placement purposes. IPsec tunnel mode encrypts entire IP Datagrams. DDP layer for placement purposes. IPsec RFC2406 [2]. tunnel mode
IPsec transport mode encrypts TCP Segments or SCTP packets. In encrypts entire IP Datagrams. IPsec transport mode encrypts TCP
neither case should IPsec preclude providing out-of-order DDP Segments or SCTP packets. In neither case should IPsec preclude
Segments to the DDP layer for placement. providing out-of-order DDP Segments to the DDP layer for placement.
Note that end-to-end use of IPsec cryptographic integrity protection Note that end-to-end use of IPsec cryptographic integrity protection
may allow suppression of MPA CRC generation and checking under may allow suppression of MPA CRC generation and checking under
certain circumstances. This is one example where the LLP may be certain circumstances. This is one example where the LLP may be
judged to have "or equivalent" protection to an end-to-end CRC32c. judged to have "or equivalent" protection to an end-to-end CRC32c.
10. References 10. References
10.1. Normative references 10.1. Normative references
[1] Bradner, S., "Key words for use in RFCs to Indicate Requirement [1] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0",
Levels", BCP 14, RFC 2119, March 1997.
[2] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0",
RFC 2246, January 1999. RFC 2246, January 1999.
[3] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload [2] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload
(ESP)", RFC 2406, November 1998. (ESP)", RFC 2406, November 1998.
[4] Stewart, R., Xie, Q., Morneault, K., Sharp, C., Schwarzbauer, [3] Recio, R., "An RDMA Protocol Specification",
H., Taylor, T., Rytina, I., Kalla, M., Zhang, L., and V.
Paxson, "Stream Control Transmission Protocol", RFC 2960,
October 2000.
[5] Coene, L., "Stream Control Transmission Protocol Applicability
Statement", RFC 3257, April 2002.
[6] Recio, R., "An RDMA Protocol Specification",
draft-ietf-rddp-rdmap-05 (work in progress), July 2005. draft-ietf-rddp-rdmap-05 (work in progress), July 2005.
[7] Shah, H., "Direct Data Placement over Reliable Transports", [4] Shah, H., "Direct Data Placement over Reliable Transports",
draft-ietf-rddp-ddp-05 (work in progress), July 2005. draft-ietf-rddp-ddp-05 (work in progress), July 2005.
[8] Stewart, R., "Stream Control Transmission Protocol (SCTP) [5] Stewart, R., "Stream Control Transmission Protocol (SCTP) Remote
Remote Direct Memory Access (RDMA) Direct Data Placement (DDP) Direct Memory Access (RDMA) Direct Data Placement (DDP)
Adaptation", draft-ietf-rddp-sctp-02 (work in progress), Adaptation", draft-ietf-rddp-sctp-02 (work in progress),
August 2005. August 2005.
[9] Pinkerton, J., "DDP/RDMAP Security", [6] Culley, P., "Marker PDU Aligned Framing for TCP Specification",
draft-ietf-rddp-security-08 (work in progress), March 2006. draft-ietf-rddp-mpa-03 (work in progress), October 2005.
[10] Culley, P., "Marker PDU Aligned Framing for TCP Specification", [7] Pinkerton, J., "DDP/RDMAP Security", draft-ietf-rddp-security-09
draft-ietf-rddp-mpa-02 (work in progress), February 2005. (work in progress), May 2006.
10.2. Informative References 10.2. Informative References
[11] Ko, M., "iSCSI Extensions for RDMA Specification", [8] Ko, M., "iSCSI Extensions for RDMA Specification", October 2005.
October 2005.
[12] Callaghan, B. and T. Talpey, "NFS Direct Data Placement", [9] Callaghan, B. and T. Talpey, "NFS Direct Data Placemetn",
draft-ietf-nfsv4-nfsdirect-02 (work in progress), October 2005. draft-ietf-nfsv4-nfsdirect-02 (work in progress), October 2005.
Authors' Addresses Authors' Addresses
Caitlin Bestler Caitlin Bestler
Broadcom Corporation Broadcom Corporation
16215 Alton Parkway 16215 Alton Parkway
P.O. Box 57013 P.O. Box 57013
Irvine, CA 92619-7013 Irvine, CA 92619-7013
USA USA
 End of changes. 29 change blocks. 
104 lines changed or deleted 83 lines changed or added

This html diff was produced by rfcdiff 1.31. The latest version is available from http://www.levkowetz.com/ietf/tools/rfcdiff/