draft-ietf-rddp-arch-07.txt   rfc4296.txt 
Internet-Draft Stephen Bailey (Sandburst) Network Working Group S. Bailey
Expires: August 2005 Tom Talpey (NetApp) Request for Comments: 4296 Sandburst
Category: Informational T. Talpey
NetApp
December 2005
The Architecture of Direct Data Placement (DDP) The Architecture of Direct Data Placement (DDP)
and Remote Direct Memory Access (RDMA) and Remote Direct Memory Access (RDMA) on Internet Protocols
on Internet Protocols
draft-ietf-rddp-arch-07
Status of this Memo
By submitting this Internet-Draft, I certify that any applicable
patent or other IPR claims of which I am aware have been disclosed,
or will be disclosed, and any of which I become aware will be
disclosed, in accordance with RFC 3668.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in
progress."
The list of current Internet-Drafts can be accessed at Status of This Memo
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at This memo provides information for the Internet community. It does
http://www.ietf.org/shadow.html not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2005). All Rights Reserved. Copyright (C) The Internet Society (2005).
Abstract Abstract
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols, not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding but is, rather, a descriptive tool for defining and understanding the
the protocols. DDP allows the efficient placement of data into protocols. DDP allows the efficient placement of data into buffers
buffers designated by Upper Layer Protocols (e.g. RDMA). RDMA designated by Upper Layer Protocols (e.g., RDMA). RDMA provides the
provides the semantics to enable Remote Direct Memory Access semantics to enable Remote Direct Memory Access between peers in a
between peers in a way consistent with application requirements. way consistent with application requirements.
Table Of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction ....................................................2
1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Terminology ................................................2
1.2. DDP and RDMA Protocols . . . . . . . . . . . . . . . . . 3 1.2. DDP and RDMA Protocols .....................................3
2. Architecture . . . . . . . . . . . . . . . . . . . . . . 4 2. Architecture ....................................................4
2.1. Direct Data Placement (DDP) Protocol Architecture . . . 4 2.1. Direct Data Placement (DDP) Protocol Architecture ..........4
2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 6 2.1.1. Transport Operations ................................6
2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 7 2.1.2. DDP Operations ......................................7
2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10 2.1.3. Transport Characteristics in DDP ...................10
2.2. Remote Direct Memory Access Protocol Architecture . . . 12 2.2. Remote Direct Memory Access (RDMA) Protocol Architecture ..12
2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 14 2.2.1. RDMA Operations ....................................14
2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 16 2.2.2. Transport Characteristics in RDMA ..................16
3. Security Considerations . . . . . . . . . . . . . . . . 17 3. Security Considerations ........................................17
3.1. Security Services . . . . . . . . . . . . . . . . . . . 18 3.1. Security Services .........................................18
3.2. Error Considerations . . . . . . . . . . . . . . . . . . 19 3.2. Error Considerations ......................................19
4. IANA Considerations . . . . . . . . . . . . . . . . . . 19 4. Acknowledgements ...............................................19
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 20 5. Informative References .........................................20
Informative References . . . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . 21
Full Copyright Statement . . . . . . . . . . . . . . . . 21
1. Introduction 1. Introduction
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols, not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding but is, rather, a descriptive tool for defining and understanding the
the protocols. This document uses C language notation as a protocols. This document uses C language notation as a shorthand to
shorthand to describe the architectural elements of DDP and RDMA describe the architectural elements of DDP and RDMA protocols. The
protocols. The choice of C notation is not intended to describe choice of C notation is not intended to describe concrete protocols
concrete protocols or programming interfaces. or programming interfaces.
The first part of the document describes the architecture of DDP The first part of the document describes the architecture of DDP
protocols, including what assumptions are made about the transports protocols, including what assumptions are made about the transports
on which DDP is built. The second part describes the architecture on which DDP is built. The second part describes the architecture of
of RDMA protocols layered on top of DDP. RDMA protocols layered on top of DDP.
1.1. Terminology 1.1. Terminology
Before introducing the protocols, certain definitions will be Before introducing the protocols, certain definitions will be useful
useful to guide discussion: to guide discussion:
o Placement - writing to a data buffer. o Placement - writing to a data buffer.
o Operation - a protocol message, or sequence of messages, which o Operation - a protocol message, or sequence of messages, which
provide a architectural semantic, such as reading or writing provide an architectural semantic, such as reading or writing of
of a data buffer. a data buffer.
o Delivery - informing any Upper Layer or application that a o Delivery - informing any Upper Layer or application that a
particular message is available for use. Delivery therefore particular message is available for use. Therefore, delivery
may be viewed as the "control" signal associated with a unit may be viewed as the "control" signal associated with a unit of
of data. Note that the order of delivery is defined more data. Note that the order of delivery is defined more strictly
strictly than it is for placement. than it is for placement.
o Completion - informing any Upper Layer or application that a o Completion - informing any Upper Layer or application that a
particular operation has finished. A completion, for particular operation has finished. A completion, for instance,
instance, may require the delivery of several messages, or it may require the delivery of several messages, or it may also
may also reflect that some local processing has finished. reflect that some local processing has finished.
o Data Sink - the peer on which any placement occurs. o Data Sink - the peer on which any placement occurs.
o Data Source - the peer from which the placed data originates. o Data Source - the peer from which the placed data originates.
o Steering Tag - a "handle" used to identify the buffer which is o Steering Tag - a "handle" used to identify the buffer that is
the target of placement. A "tagged" message is one which the target of placement. A "tagged" message is one that
references such a handle. references such a handle.
o RDMA Write - an Operation which places data from a local data o RDMA Write - an Operation that places data from a local data
buffer to a remote data buffer specified by a Steering Tag. buffer to a remote data buffer specified by a Steering Tag.
o RDMA Read - an Operation which places data to a local data o RDMA Read - an Operation that places data to a local data buffer
buffer specified by a Steering Tag from a remote data buffer specified by a Steering Tag from a remote data buffer specified
specified by another Steering Tag. by another Steering Tag.
o Send - an Operation which places data from a local data buffer o Send - an Operation that places data from a local data buffer to
to a remote data buffer of the data sink's choice. Sends are a remote data buffer of the data sink's choice. Therefore,
therefore "untagged". sends are "untagged".
1.2. DDP and RDMA Protocols 1.2. DDP and RDMA Protocols
The goal of the DDP protocol is to allow the efficient placement of The goal of the DDP protocol is to allow the efficient placement of
data into buffers designated by protocols layered above DDP (e.g. data into buffers designated by protocols layered above DDP (e.g.,
RDMA). This is described in detail in [ROM]. Efficiency may be RDMA). This is described in detail in [ROM]. Efficiency may be
characterized by the minimization of the number of transfers of the characterized by the minimization of the number of transfers of the
data over the receiver's system buses. data over the receiver's system buses.
The goal of the RDMA protocol is to provide the semantics to enable The goal of the RDMA protocol is to provide the semantics to enable
Remote Direct Memory Access between peers in a way consistent with Remote Direct Memory Access between peers in a way consistent with
application requirements. The RDMA protocol provides facilities application requirements. The RDMA protocol provides facilities
immediately useful to existing and future networking, storage, and immediately useful to existing and future networking, storage, and
other application protocols. [DAFS, FCVI, IB, MYR, SDP, SRVNET, other application protocols. [FCVI, IB, MYR, SDP, SRVNET, VI]
VI]
The DDP and RDMA protocols work together to achieve their
respective goals. DDP provides facilities to safely steer payloads
to specific buffers at the Data Sink. RDMA provides facilities to
Upper Layers for identifying these buffers, controlling the
transfer of data between peers' buffers, supporting authorized
bidirectional transfer between buffers, and signalling completion.
Upper Layer Protocols that do not require the features of RDMA may
be layered directly on top of DDP.
The DDP and RDMA protocols are transport independent. The The DDP and RDMA protocols work together to achieve their respective
following figure shows the relationship between RDMA, DDP, Upper goals. DDP provides facilities to safely steer payloads to specific
Layer Protocols and Transport. buffers at the Data Sink. RDMA provides facilities to Upper Layers
for identifying these buffers, controlling the transfer of data
between peers' buffers, supporting authorized bidirectional transfer
between buffers, and signalling completion. Upper Layer Protocols
that do not require the features of RDMA may be layered directly on
top of DDP.
The DDP and RDMA protocols are transport independent. The following
figure shows the relationship between RDMA, DDP, Upper Layer
Protocols, and Transport.
+--------------------------------------------------+ +--------------------------------------------------+
| Upper Layer Protocol | | Upper Layer Protocol |
+---------+------------+---------------------------+ +---------+------------+---------------------------+
| | | RDMA | | | | RDMA |
| | +---------------------------+ | | +---------------------------+
| | DDP | | | DDP |
| +----------------------------------------+ | +----------------------------------------+
| Transport | | Transport |
+--------------------------------------------------+ +--------------------------------------------------+
skipping to change at page 4, line 40 skipping to change at page 4, line 36
Placement Protocol architecture and Remote Direct Memory Access Placement Protocol architecture and Remote Direct Memory Access
Protocol architecture. Protocol architecture.
2.1. Direct Data Placement (DDP) Protocol Architecture 2.1. Direct Data Placement (DDP) Protocol Architecture
The central idea of general-purpose DDP is that a data sender will The central idea of general-purpose DDP is that a data sender will
supplement the data it sends with placement information that allows supplement the data it sends with placement information that allows
the receiver's network interface to place the data directly at its the receiver's network interface to place the data directly at its
final destination without any copying. DDP can be used to steer final destination without any copying. DDP can be used to steer
received data to its final destination, without requiring layer- received data to its final destination, without requiring layer-
specific behavior for each different layer. Data sent with such specific behavior for each different layer. Data sent with such DDP
DDP information is said to be `tagged'. information is said to be `tagged'.
The central component of the DDP architecture is the `buffer', The central components of the DDP architecture are the `buffer',
which is an object with beginning and ending addresses, and a which is an object with beginning and ending addresses, and a method
method (set()) to set the value of an octet at an address. In many (set()), which sets the value of an octet at an address. In many
cases, a buffer corresponds directly to a portion of host user cases, a buffer corresponds directly to a portion of host user
memory. However, DDP does not depend on this---a buffer could be a memory. However, DDP does not depend on this; a buffer could be a
disk file, or anything else that can be viewed as an addressable disk file, or anything else that can be viewed as an addressable
collection of octets. Abstractly, a buffer provides the interface: collection of octets. Abstractly, a buffer provides the interface:
typedef struct { typedef struct {
const address_t start; const address_t start;
const address_t end; const address_t end;
void set(address_t a, data_t v); void set(address_t a, data_t v);
} ddp_buffer_t; } ddp_buffer_t;
address_t address_t
a reference to local memory a reference to local memory
data_t data_t
an octet data value. an octet data value.
The protocol layering and in-line data flow of DDP is: The protocol layering and in-line data flow of DDP is:
skipping to change at page 5, line 22 skipping to change at page 5, line 15
a reference to local memory a reference to local memory
data_t data_t
an octet data value. an octet data value.
The protocol layering and in-line data flow of DDP is: The protocol layering and in-line data flow of DDP is:
DDP Client Protocol DDP Client Protocol
(e.g. RDMA or Upper Layer Protocol) (e.g., RDMA or Upper Layer Protocol)
| ^ | ^
untagged messages | | untagged message delivery untagged messages | | untagged message delivery
tagged messages | | tagged message delivery tagged messages | | tagged message delivery
v | v |
DDP+---> data placement DDP+---> data placement
^ ^
| transport messages | transport messages
v v
Transport Transport
(e.g. SCTP, DCCP, framed TCP) (e.g., SCTP, DCCP, framed TCP)
^ ^
| IP datagrams | IP datagrams
v v
. . . . . .
In addition to in-line data flow, the client protocol registers In addition to in-line data flow, the client protocol registers
buffers with DDP, and DDP performs buffer update (set()) operations buffers with DDP, and DDP performs buffer update (set()) operations
as a result of receiving tagged messages. as a result of receiving tagged messages.
DDP messages may be split into multiple, smaller DDP messages, each DDP messages may be split into multiple, smaller DDP messages, each
in a separate transport message. However, if the transport is in a separate transport message. However, if the transport is
unreliable or unordered, messages split across transport messages unreliable or unordered, messages split across transport messages may
may or may not provide useful behavior, in the same way as or may not provide useful behavior, in the same way as splitting
splitting arbitrary Upper Layer messages across unreliable or arbitrary Upper Layer messages across unreliable or unordered
unordered transport messages may or may not provide useful transport messages may or may not provide useful behavior. In other
behavior. In other words, the same considerations apply to words, the same considerations apply to building client protocols on
building client protocols on different types of transports with or different types of transports with or without the use of DDP.
without the use of DDP.
A DDP message split across transport messages looks like: A DDP message split across transport messages looks like:
DDP message: Transport messages: DDP message: Transport messages:
stag=s, offset=o, message 1: stag=s, offset=o, message 1:
notify=y, id=i |type=ddp | notify=y, id=i |type=ddp |
message= |stag=s | message= |stag=s |
|aabbccddee|-------. |offset=o | |aabbccddee|-------. |offset=o |
~ ... ~----. \ |notify=n | ~ ... ~----. \ |notify=n |
skipping to change at page 6, line 30 skipping to change at page 6, line 30
\ | |type=ddp | \ | |type=ddp |
\ | |stag=s | \ | |stag=s |
\ + |offset=o+n| \ + |offset=o+n|
\ \ |notify=y | \ \ |notify=y |
\ \ |id=i | \ \ |id=i |
\ `-->|nnooppqqrr| \ `-->|nnooppqqrr|
\ ~ ... ~ \ ~ ... ~
`---->|vvwwxxyyzz| `---->|vvwwxxyyzz|
Although this picture suggests that DDP information is carried in- Although this picture suggests that DDP information is carried in-
line with the message payload, components of the DDP information line with the message payload, components of the DDP information may
may also be in transport-specific fields, or derived from also be in transport-specific fields, or derived from transport-
transport-specific control information if the transport permits. specific control information if the transport permits.
2.1.1. Transport Operations 2.1.1. Transport Operations
For the purposes of this architecture, the transport provides: For the purposes of this architecture, the transport provides:
void xpt_send(socket_t s, message_t m); void xpt_send(socket_t s, message_t m);
message_t xpt_recv(socket_t s); message_t xpt_recv(socket_t s);
msize_t xpt_max_msize(socket_t s); msize_t xpt_max_msize(socket_t s);
socket_t socket_t
skipping to change at page 8, line 4 skipping to change at page 7, line 49
ddp_addr_t ddp_addr_t
the buffer address portion of a tagged message: the buffer address portion of a tagged message:
typedef struct { typedef struct {
stag_t stag; stag_t stag;
address_t offset; address_t offset;
} ddp_addr_t; } ddp_addr_t;
stag_t (scalar) stag_t (scalar)
a Steering Tag. A stag_t identifies the destination buffer
for tagged messages. stag_ts are generated when the buffer is a Steering Tag. A stag_t identifies the destination buffer for
tagged messages. stag_ts are generated when the buffer is
registered, communicated to the sender by some client protocol registered, communicated to the sender by some client protocol
convention and inserted in DDP messages. stag_t values in convention and inserted in DDP messages. stag_t values in this
this DDP architecture are assumed to be completely opaque to DDP architecture are assumed to be completely opaque to the
the client protocol, and implementation-dependent. However, client protocol, and implementation-dependent. However,
particular implementations, such as DDP on a multicast particular implementations, such as DDP on a multicast transport
transport (see below), may provide the buffer holder some (see below), may provide the buffer holder some control in
control in selecting stag_ts. selecting stag_ts.
ddp_notify_t ddp_notify_t
the notification portion of a DDP message, used to signal that the notification portion of a DDP message, used to signal
the message represents the final fragment of a multi-segmented that the message represents the final fragment of a
DDP message: multi-segmented DDP message:
typedef struct { typedef struct {
boolean_t notify; boolean_t notify;
ddp_msg_id_t i; ddp_msg_id_t i;
} ddp_notify_t; } ddp_notify_t;
ddp_msg_id_t (scalar) ddp_msg_id_t (scalar)
a DDP message identifier. msg_id_ts are chosen by the DDP a DDP message identifier. msg_id_ts are chosen by the DDP
message receiver (buffer holder), communicated to the sender message receiver (buffer holder), communicated to the sender by
by some client protocol convention and inserted in DDP some client protocol convention and inserted in DDP messages.
messages. Whether a message reception indication is requested Whether a message reception indication is requested for a DDP
for a DDP message is a matter of client protocol convention. message is a matter of client protocol convention. Unlike
Unlike stag_ts, the structure of msg_id_ts is opaque to DDP, stag_ts, the structure of msg_id_ts is opaque to DDP, and
and therefore, completely in the hands of the client protocol. therefore, it is completely in the hands of the client protocol.
bdesc_t bdesc_t
a description of a registered buffer: a description of a registered buffer:
typedef struct { typedef struct {
bhand_t bh; bhand_t bh;
ddp_addr_t a; ddp_addr_t a;
} bdesc_t; } bdesc_t;
`a.offset' is the starting offset of the registered buffer, `a.offset' is the starting offset of the registered buffer,
which may have no relationship to the `start' or `end' which may have no relationship to the `start' or `end' addresses
addresses of that buffer. However, particular of that buffer. However, particular implementations, such as
implementations, such as DDP on a multicast transport (see DDP on a multicast transport (see below), may allow some client
below), may allow some client protocol control over the protocol control over the starting offset.
starting offset.
bhand_t bhand_t
an opaque buffer handle used to deregister a buffer. an opaque buffer handle used to deregister a buffer.
recv_message_t recv_message_t
a description of a completed untagged receive buffer: a description of a completed untagged receive buffer:
typedef struct { typedef struct {
bdesc_t b; bdesc_t b;
length_t l; length_t l;
} recv_message_t; } recv_message_t;
ddp_ind_t ddp_ind_t
an untagged message, a tagged message reception indication, or an untagged message, a tagged message reception indication, or a
a tagged message reception error: tagged message reception error:
typedef union { typedef union {
recv_message_t m; recv_message_t m;
ddp_msg_id_t i; ddp_msg_id_t i;
ddp_err_t e; ddp_err_t e;
} ddp_ind_t; } ddp_ind_t;
ddp_err_t ddp_err_t
indicates an error while receiving a tagged message, typically indicates an error while receiving a tagged message, typically
skipping to change at page 10, line 4 skipping to change at page 9, line 46
typedef struct { typedef struct {
msize_t max_untagged; msize_t max_untagged;
msize_t max_tagged; msize_t max_tagged;
} msizes_t; } msizes_t;
ddp_send(socket_t s, message_t m) ddp_send(socket_t s, message_t m)
send an untagged message. send an untagged message.
ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n) ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n)
send a tagged message to remote buffer address d. send a tagged message to remote buffer address d.
ddp_post_recv(socket_t s, bdesc_t b) ddp_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single received untagged post a registered buffer to accept a single received untagged
message. Each buffer is returned to the caller in a message. Each buffer is returned to the caller in a ddp_recv()
ddp_recv() untagged message reception indication, in the order untagged message reception indication, in the order in which it
in which it was posted. The same buffer may be enabled on was posted. The same buffer may be enabled on multiple sockets;
multiple sockets, receipt of an untagged message into the receipt of an untagged message into the buffer from any of these
buffer from any of these sockets unposts the buffer from all sockets unposts the buffer from all sockets.
sockets.
ddp_recv(socket_t s) ddp_recv(socket_t s)
get the next received untagged message, tagged message get the next received untagged message, tagged message reception
reception indication, or tagged message error. indication, or tagged message error.
ddp_register(socket_t s, ddp_buffer_t b) ddp_register(socket_t s, ddp_buffer_t b)
register a buffer for DDP on a socket. The same buffer may be register a buffer for DDP on a socket. The same buffer may be
registered multiple times on the same or different sockets. registered multiple times on the same or different sockets. The
The same buffer registered on different sockets may result in same buffer registered on different sockets may result in a
a common registration. Different buffers may also refer to common registration. Different buffers may also refer to
portions of the same underlying addressable object (buffer portions of the same underlying addressable object (buffer
aliasing). aliasing).
ddp_deregister(bhand_t bh) ddp_deregister(bhand_t bh)
remove a registration from a buffer. remove a registration from a buffer.
ddp_max_msizes(socket_t s) ddp_max_msizes(socket_t s)
get the current maximum untagged and tagged message sizes that get the current maximum untagged and tagged message sizes that
will fit in a single transport message. will fit in a single transport message.
2.1.3. Transport Characteristics In DDP 2.1.3. Transport Characteristics in DDP
Certain characteristics of the transport on which DDP is mapped Certain characteristics of the transport on which DDP is mapped
determine the nature of the service provided to client protocols. determine the nature of the service provided to client protocols.
Fundamentally, the characteristics of the transport will not be Fundamentally, the characteristics of the transport will not be
changed by the presence of DDP. The choice of transport is changed by the presence of DDP. The choice of transport is therefore
therefore driven not by DDP, but by the requirements of the Upper driven not by DDP, but by the requirements of the Upper Layer, and
Layer, and employing the DDP service. employing the DDP service.
Specifically, transports are: Specifically, transports are:
o reliable or unreliable, o reliable or unreliable,
o ordered or unordered, o ordered or unordered,
o single source or multisource, o single source or multisource,
o single destination or multidestination (multicast or anycast). o single destination or multidestination (multicast or anycast).
Some transports support several combinations of these Some transports support several combinations of these
characteristics. For example, SCTP [SCTP] is reliable, single characteristics. For example, SCTP [SCTP] is reliable, single
source, single destination (point-to-point) and supports both source, single destination (point-to-point) and supports both ordered
ordered and unordered modes. and unordered modes.
DDP messages carried by transport are framed for processing by the DDP messages carried by transport are framed for processing by the
receiver, and may be further protected for integrity or privacy in receiver, and may be further protected for integrity or privacy in
accordance with the transport capabilities. DDP does not provide accordance with the transport capabilities. DDP does not provide
such functions. such functions.
In general, transport characteristics equally affect transport and In general, transport characteristics equally affect transport and
DDP message delivery. However, there are several issues specific DDP message delivery. However, there are several issues specific to
to DDP messages. DDP messages.
A key component of DDP is how the following operations on the A key component of DDP is how the following operations on the
receiving side are ordered among themselves, and how they relate to receiving side are ordered among themselves, and how they relate to
corresponding operations on the sending side: corresponding operations on the sending side:
o set()s, o set()s,
o untagged message reception indications, and o untagged message reception indications, and
o tagged message reception indications. o tagged message reception indications.
These relationships depend upon the characteristics of the These relationships depend upon the characteristics of the underlying
underlying transport in a way which is defined by the DDP protocol. transport in a way that is defined by the DDP protocol. For example,
For example, if the transport is unreliable and unordered, the DDP if the transport is unreliable and unordered, the DDP protocol might
protocol might specify that the client protocol is subject to the specify that the client protocol is subject to the consequences of
consequences of transport messages being lost or duplicated, rather transport messages being lost or duplicated, rather than requiring
than requiring different characteristics be presented to the client that different characteristics be presented to the client protocol.
protocol.
Buffer access must be implemented consistently across endpoint IP Buffer access must be implemented consistently across endpoint IP
addresses on transports allowing multiple IP addresses per addresses on transports allowing multiple IP addresses per endpoint,
endpoint, for example, SCTP. In particular, the Steering Tag must for example, SCTP. In particular, the Steering Tag must be
be consistently scoped and must address the same buffer across all consistently scoped and must address the same buffer across all IP
IP address associations belonging to the endpoint. Additionally, address associations belonging to the endpoint. Additionally,
operation ordering relationships across IP addresses within an operation ordering relationships across IP addresses within an
association (set(), get(), etc.) depend on the underlying association (set(), get(), etc.) depend on the underlying transport.
transport. If the above consistency relationships cannot be If the above consistency relationships cannot be maintained by a
maintained by a transport endpoint, then the endpoint is unsuitable transport endpoint, then the endpoint is unsuitable for a DDP
for a DDP connection. connection.
Multidestination data delivery is a transport characteristic which Multidestination data delivery is a transport characteristic that may
may require specific consideration in a DDP protocol. As mentioned require specific consideration in a DDP protocol. As mentioned
above, the basic DDP model assumes that buffer address values above, the basic DDP model assumes that buffer address values
returned by ddp_register() are opaque to the client protocol, and returned by ddp_register() are opaque to the client protocol, and can
can be implementation dependent. The most natural way to map DDP be implementation dependent. The most natural way to map DDP to a
to a multidestination transport is to require all receivers produce multidestination transport is to require that all receivers produce
the same buffer address when registering a multidestination the same buffer address when registering a multidestination
destination buffer. Restriction of the DDP model to accommodate destination buffer. Restriction of the DDP model to accommodate
multiple destinations involves engineering tradeoffs comparable to multiple destinations involves engineering tradeoffs comparable to
those of providing non-DDP multidestination transport capability. those of providing non-DDP multidestination transport capability.
A registered buffer is identified within DDP by its stag_t, which A registered buffer is identified within DDP by its stag_t, which in
in turn is associated with a socket. This registration therefore turn is associated with a socket. Therefore, this registration
grants a capability to the DDP peer, and the socket (using the grants a capability to the DDP peer, and the socket (using the
underlying properties of its chosen transport and possible underlying properties of its chosen transport and possible security)
security) identifies the peer and authenticates the stag_t. identifies the peer and authenticates the stag_t.
The same buffer may be enabled by ddp_post_recv() on multiple The same buffer may be enabled by ddp_post_recv() on multiple
sockets. In this case any ddp_recv() untagged message reception sockets. In this case any ddp_recv() untagged message reception
indication may be provided on a different socket from that on which indication may be provided on a different socket from that on which
the buffer was posted. Such indications are not ordered among the buffer was posted. Such indications are not ordered among
multiple DDP sockets. multiple DDP sockets.
When multiple sockets reference an untagged message reception When multiple sockets reference an untagged message reception buffer,
buffer, local interfaces are responsible for managing the local interfaces are responsible for managing the mechanisms of
mechanisms of allocating posted buffers to received untagged allocating posted buffers to received untagged messages, the handling
messages, the handling of received untagged messages when no buffer of received untagged messages when no buffer is available, and of
is available, and of resource management among multiple sockets. resource management among multiple sockets. Where underprovisioning
Where underprovisioning of buffers on multiple sockets is allowed, of buffers on multiple sockets is allowed, mechanisms should be
mechanisms should be provided to manage buffer consumption on a provided to manage buffer consumption on a per-socket or group of
per-socket or group of related sockets basis. related sockets basis.
Architecturally, therefore, DDP is a flexible and general paradigm Architecturally, therefore, DDP is a flexible and general paradigm
which may be applied to any variety of transports. Implementations that may be applied to any variety of transports. Implementations of
of DDP may, however, adapt themselves to these differences in ways DDP may, however, adapt themselves to these differences in ways
appropriate to each transport. In all cases the layering of DDP appropriate to each transport. In all cases, the layering of DDP
must continue to express the transport's underlying must continue to express the transport's underlying characteristics.
characteristics.
2.2. Remote Direct Memory Access (RDMA) Protocol Architecture 2.2. Remote Direct Memory Access (RDMA) Protocol Architecture
Remote Direct Memory Access (RDMA) extends the capabilities of DDP Remote Direct Memory Access (RDMA) extends the capabilities of DDP
with two primary functions. with two primary functions.
First, it adds the ability to read from buffers registered to a First, it adds the ability to read from buffers registered to a
socket (RDMA Read). This allows a client protocol to perform socket (RDMA Read). This allows a client protocol to perform
arbitrary, bidirectional data movement without involving the remote arbitrary, bidirectional data movement without involving the remote
client. When RDMA is implemented in hardware, arbitrary data client. When RDMA is implemented in hardware, arbitrary data
movement can be performed without involving the remote host CPU at movement can be performed without involving the remote host CPU at
all. all.
In addition, RDMA specifies a transport-independent untagged In addition, RDMA specifies a transport-independent untagged message
message service (Send) with characteristics which are both very service (Send) with characteristics that are both very efficient to
efficient to implement in hardware, and convenient for client implement in hardware, and convenient for client protocols.
protocols.
The RDMA architecture is patterned after the traditional model for The RDMA architecture is patterned after the traditional model for
device programming, where the client requests an operation using device programming, where the client requests an operation using
Send-like actions (programmed I/O), the server performs the Send-like actions (programmed I/O), the server performs the necessary
necessary data transfers for the operation (DMA reads and writes), data transfers for the operation (DMA reads and writes), and notifies
and notifies the client of completion. The programmed I/O+DMA the client of completion. The programmed I/O+DMA model efficiently
model efficiently supports a high degree of concurrency and supports a high degree of concurrency and flexibility for both the
flexibility for both the client and server, even when operations client and server, even when operations have a wide range of
have a wide range of intrinsic latencies. intrinsic latencies.
RDMA is layered as a client protocol on top of DDP: RDMA is layered as a client protocol on top of DDP:
Client Protocol Client Protocol
| ^ | ^
Sends | | Send reception indications Sends | | Send reception indications
RDMA Read Requests | | RDMA Read Completion indications RDMA Read Requests | | RDMA Read Completion indications
RDMA Writes | | RDMA Write Completion indications RDMA Writes | | RDMA Write Completion indications
v | v |
RDMA RDMA
skipping to change at page 13, line 41 skipping to change at page 13, line 38
untagged messages | | untagged message delivery untagged messages | | untagged message delivery
tagged messages | | tagged message delivery tagged messages | | tagged message delivery
v | v |
DDP+---> data placement DDP+---> data placement
^ ^
| transport messages | transport messages
v v
. . . . . .
In addition to in-line data flow, read (get()) and update (set()) In addition to in-line data flow, read (get()) and update (set())
operations are performed on buffers registered with RDMA as a operations are performed on buffers registered with RDMA as a result
result of RDMA Read Requests and RDMA Writes, respectively. of RDMA Read Requests and RDMA Writes, respectively.
An RDMA `buffer' extends a DDP buffer with a get() operation that An RDMA `buffer' extends a DDP buffer with a get() operation that
retrieves the value of the octet at address `a': retrieves the value of the octet at address `a':
typedef struct { typedef struct {
const address_t start; const address_t start;
const address_t end; const address_t end;
void set(address_t a, data_t v); void set(address_t a, data_t v);
data_t get(address_t a); data_t get(address_t a);
} rdma_buffer_t; } rdma_buffer_t;
skipping to change at page 14, line 28 skipping to change at page 14, line 21
rdma_notify_t n); rdma_notify_t n);
void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d); void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d);
void rdma_post_recv(socket_t s, bdesc_t b); void rdma_post_recv(socket_t s, bdesc_t b);
rdma_ind_t rdma_recv(socket_t s); rdma_ind_t rdma_recv(socket_t s);
bdesc_t rdma_register(socket_t s, rdma_buffer_t b, bdesc_t rdma_register(socket_t s, rdma_buffer_t b,
bmode_t mode); bmode_t mode);
void rdma_deregister(bhand_t bh); void rdma_deregister(bhand_t bh);
msizes_t rdma_max_msizes(socket_t s); msizes_t rdma_max_msizes(socket_t s);
Although, for clarity, these data transfer interfaces are Although, for clarity, these data transfer interfaces are
synchronous, rdma_read() and possibly rdma_send() (in the presence synchronous, rdma_read() and possibly rdma_send() (in the presence of
of Send flow control), can require an arbitrary amount of time to Send flow control) can require an arbitrary amount of time to
complete. To express the full concurrency and interleaving of RDMA complete. To express the full concurrency and interleaving of RDMA
data transfer, these interfaces should also be reentrant. For data transfer, these interfaces should also be reentrant. For
example, a client protocol may perform an rdma_send(), while an example, a client protocol may perform an rdma_send(), while an
rdma_read() operation is in progress. rdma_read() operation is in progress.
rdma_notify_t rdma_notify_t
RDMA Write notification information, used to signal that the RDMA Write notification information, used to signal that the
message represents the final fragment of a multi-segmented message represents the final fragment of a multi-segmented RDMA
RDMA message: message:
typedef struct { typedef struct {
boolean_t notify; boolean_t notify;
rdma_write_id_t i; rdma_write_id_t i;
} rdma_notify_t; } rdma_notify_t;
identical in function to ddp_notify_t, except that the type identical in function to ddp_notify_t, except that the type
rdma_write_id_t may not be equivalent to ddp_msg_id_t. rdma_write_id_t may not be equivalent to ddp_msg_id_t.
rdma_write_id_t (scalar) rdma_write_id_t (scalar)
skipping to change at page 15, line 19 skipping to change at page 15, line 18
typedef union { typedef union {
recv_message_t m; recv_message_t m;
rdma_err_t e; rdma_err_t e;
} rdma_ind_t; } rdma_ind_t;
rdma_err_t rdma_err_t
an RDMA protocol error indication. RDMA errors include buffer an RDMA protocol error indication. RDMA errors include buffer
addressing errors corresponding to ddp_err_ts, and buffer addressing errors corresponding to ddp_err_ts, and buffer
protection violations (e.g. RDMA Writing a buffer only protection violations (e.g., RDMA Writing a buffer only
registered for reading). registered for reading).
bmode_t bmode_t
buffer registration mode (permissions). Any combination of buffer registration mode (permissions). Any combination of
permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE) permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE)
operations. operations.
rdma_send(socket_t s, message_t m) rdma_send(socket_t s, message_t m)
send a message, delivering it to the next untagged RDMA buffer send a message, delivering it to the next untagged RDMA buffer
at the remote peer. at the remote peer.
rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n) rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n)
RDMA Write to remote buffer address d. RDMA Write to remote buffer address d.
rdma_read(socket_t s, ddp_addr_t s, length_t l, ddp_addr_t d) rdma_read(socket_t s, ddp_addr_t s, length_t l, ddp_addr_t d)
RDMA Read l octets from remote buffer address s to local RDMA Read l octets from remote buffer address s to local buffer
buffer address d. address d.
rdma_post_recv(socket_t s, bdesc_t b) rdma_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single Send message, to post a registered buffer to accept a single Send message, to be
be filled and returned in-order to a subsequent caller of filled and returned in-order to a subsequent caller of
rdma_recv(). As with DDP, buffers may be enabled on multiple rdma_recv(). As with DDP, buffers may be enabled on multiple
sockets, in which case ordering guarantees are relaxed. Also sockets, in which case ordering guarantees are relaxed. Also as
as with DDP, local interfaces must manage the mechanisms of with DDP, local interfaces must manage the mechanisms of
allocation and management of buffers posted to multiple allocation and management of buffers posted to multiple sockets.
sockets.
rdma_recv(socket_t s); rdma_recv(socket_t s);
get the next received Send message, RDMA Write completion get the next received Send message, RDMA Write completion
identifier, or RDMA error. identifier, or RDMA error.
rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode) rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode)
register a buffer for RDMA on a socket (for read access, write register a buffer for RDMA on a socket (for read access, write
access or both). As with DDP, the same buffer may be access or both). As with DDP, the same buffer may be registered
registered multiple times on the same or different sockets, multiple times on the same or different sockets, and different
and different buffers may refer to portions of the same buffers may refer to portions of the same underlying addressable
underlying addressable object. object.
rdma_deregister(bhand_t bh) rdma_deregister(bhand_t bh)
remove a registration from a buffer. remove a registration from a buffer.
rdma_max_msizes(socket_t s) rdma_max_msizes(socket_t s)
get the current maximum Send (max_untagged) and RDMA Read or get the current maximum Send (max_untagged) and RDMA Read or
Write (max_tagged) operations that will fit in a single Write (max_tagged) operations that will fit in a single
transport message. The values returned by rdma_max_msizes() transport message. The values returned by rdma_max_msizes() are
are closely related to the values returned by closely related to the values returned by ddp_max_msizes(), but
ddp_max_msizes(), but may not be equal. may not be equal.
2.2.2. Transport Characteristics In RDMA 2.2.2. Transport Characteristics in RDMA
As with DDP, RDMA can be used on transports with a variety of As with DDP, RDMA can be used on transports with a variety of
different characteristics that manifest themselves directly in the different characteristics that manifest themselves directly in the
service provided by RDMA. Also as with DDP, the fundamental service provided by RDMA. Also, as with DDP, the fundamental
characteristics of the transport will not be changed by the characteristics of the transport will not be changed by the presence
presence of RDMA. of RDMA.
Like DDP, an RDMA protocol must specify how: Like DDP, an RDMA protocol must specify how:
o set()s, o set()s,
o get()s, o get()s,
o Send messages, and o Send messages, and
o RDMA Read completions o RDMA Read completions
are ordered among themselves and how they relate to corresponding are ordered among themselves and how they relate to corresponding
operations on the remote peer(s). These relationships are likely operations on the remote peer(s). These relationships are likely to
to be a function of the underlying transport characteristics. be a function of the underlying transport characteristics.
There are some additional characteristics of RDMA which may There are some additional characteristics of RDMA that may translate
translate poorly to unreliable or multipoint transports due to poorly to unreliable or multipoint transports due to attendant
attendant complexities in managing endpoint state: complexities in managing endpoint state:
o Send flow control o Send flow control
o RDMA Read o RDMA Read
These difficulties can be overcome by placing restrictions on the These difficulties can be overcome by placing restrictions on the
service provided by RDMA. However, many RDMA clients, especially service provided by RDMA. However, many RDMA clients, especially
those that separate data transfer and application logic concerns, those that separate data transfer and application logic concerns, are
are likely to depend upon capabilities only provided by RDMA on a likely to depend upon capabilities only provided by RDMA on a point-
point-to-point, reliable transport. In other words, many potential to-point, reliable transport. In other words, many potential Upper
Upper Layers which might avail themselves of RDMA services are Layers, which might avail themselves of RDMA services, are naturally
naturally already biased toward these transport classes. already biased toward these transport classes.
3. Security Considerations 3. Security Considerations
Fundamentally, the DDP and RDMA protocols themselves should not Fundamentally, the DDP and RDMA protocols themselves should not
introduce additional vulnerabilities. They are intermediate introduce additional vulnerabilities. They are intermediate
protocols and so should not perform or require functions such as protocols and so should not perform or require functions such as
authorization, which are the domain of Upper Layers. However, the authorization, which are the domain of Upper Layers. However, the
DDP and RDMA protocols should allow mapping by strict Upper Layers DDP and RDMA protocols should allow mapping by strict Upper Layers
which are not permissive of new vulnerabilities -- DDP and RDMAP that are not permissive of new vulnerabilities; DDP and RDMAP
implementations should be prohibited from `cutting corners' that implementations should be prohibited from `cutting corners' that
create new vulnerabilities. Implementations must ensure that only create new vulnerabilities. Implementations must ensure that only
`supplied' resources (i.e. buffers) can be manipulated by DDP or `supplied' resources (i.e., buffers) can be manipulated by DDP or
RDMAP messages. RDMAP messages.
System integrity must be maintained in any RDMA solution. System integrity must be maintained in any RDMA solution. Mechanisms
Mechanisms must be specified to prevent RDMA or DDP operations from must be specified to prevent RDMA or DDP operations from impairing
impairing system integrity. For example, threats can include system integrity. For example, threats can include potential buffer
potential buffer reuse or buffer overflow, and are not merely a reuse or buffer overflow, and are not merely a security issue. Even
security issue. Even trusted peers must not be allowed to damage trusted peers must not be allowed to damage local integrity. Any DDP
local integrity. Any DDP and RDMA protocol must address the issue and RDMA protocol must address the issue of giving end-systems and
of giving end-systems and applications the capabilities to offer applications the capabilities to offer protection from such
protection from such compromises. compromises.
Because a Steering Tag exports access to a buffer, one critical Because a Steering Tag exports access to a buffer, one critical
aspect of security is the scope of this access. It must be aspect of security is the scope of this access. It must be possible
possible to individually control specific attributes of the access to individually control specific attributes of the access provided by
provided by a Steering Tag on the endpoint (socket) on which it was a Steering Tag on the endpoint (socket) on which it was registered,
registered, including remote read access, remote write access, and including remote read access, remote write access, and others that
others that might be identified. DDP and RDMA specifications must might be identified. DDP and RDMA specifications must provide both
provide both implementation requirements relevant to this issue, implementation requirements relevant to this issue, and guidelines to
and guidelines to assist implementors in making the appropriate assist implementors in making the appropriate design decisions.
design decisions.
For example, it must not be possible for DDP to enable evasion of For example, it must not be possible for DDP to enable evasion of
buffer consistency checks at the recipient. The DDP and RDMA buffer consistency checks at the recipient. The DDP and RDMA
specifications must allow the recipient to rely on its consistent specifications must allow the recipient to rely on its consistent
buffer contents by explicitly controlling peer access to buffer buffer contents by explicitly controlling peer access to buffer
regions at appropriate times. regions at appropriate times.
The use of DDP and RDMA on a transport connection may interact with The use of DDP and RDMA on a transport connection may interact with
any security mechanism, and vice-versa. For example, if the any security mechanism, and vice-versa. For example, if the security
security mechanism is implemented above the transport layer, the mechanism is implemented above the transport layer, the DDP and RDMA
DDP and RDMA headers may not be protected. Such a layering may headers may not be protected. Therefore, such a layering may be
therefore be inappropriate, depending on requirements. inappropriate, depending on requirements.
3.1. Security Services 3.1. Security Services
The following end-to-end security services protect DDP and RDMAP The following end-to-end security services protect DDP and RDMAP
operation streams: operation streams:
o Authentication of the data source, to protect against peer o Authentication of the data source, to protect against peer
impersonation, stream hijacking, and man-in-the-middle attacks impersonation, stream hijacking, and man-in-the-middle attacks
exploiting capabilities offered by the RDMA implementation. exploiting capabilities offered by the RDMA implementation.
Peer connections which do not pass authentication and Peer connections that do not pass authentication and
authorization checks must not be permitted to begin processing authorization checks must not be permitted to begin processing
in RDMA mode with an inappropriate endpoint. Once associated, in RDMA mode with an inappropriate endpoint. Once associated,
peer accesses to buffer regions must be authenticated and made peer accesses to buffer regions must be authenticated and made
subject to authorization checks in the context of the subject to authorization checks in the context of the
association and endpoint (socket) on which they are to be association and endpoint (socket) on which they are to be
performed, prior to any transfer operation or data being performed, prior to any transfer operation or data being
accessed. The RDMA protocols must ensure that these region accessed. The RDMA protocols must ensure that these region
protections be under strict application control. protections be under strict application control.
o Integrity, to protect against modification of the control o Integrity, to protect against modification of the control
skipping to change at page 18, line 48 skipping to change at page 18, line 50
important for the DDP and RDMAP protocols that the RDMA important for the DDP and RDMAP protocols that the RDMA
control information carried in each operation be protected, in control information carried in each operation be protected, in
order to direct the payloads appropriately. order to direct the payloads appropriately.
o Sequencing, to protect against replay attacks (a special case o Sequencing, to protect against replay attacks (a special case
of the above modifications). of the above modifications).
o Confidentiality, to protect the stream from eavesdropping. o Confidentiality, to protect the stream from eavesdropping.
IPsec, operating to secure the connection on a packet-by-packet IPsec, operating to secure the connection on a packet-by-packet
basis, is a natural fit to securing RDMA placement, which operates basis, is a natural fit to securing RDMA placement, which operates in
in conjunction with transport. Because RDMA enables an conjunction with transport. Because RDMA enables an implementation
implementation to avoid buffering, it is preferable to perform all to avoid buffering, it is preferable to perform all applicable
applicable security protection prior to processing of each segment security protection prior to processing of each segment by the
by the transport and RDMA layers. Such a layering enables the most transport and RDMA layers. Such a layering enables the most
efficient secure RDMA implementation. efficient secure RDMA implementation.
The TLS record protocol, on the other hand, is layered on top of The TLS record protocol, on the other hand, is layered on top of
reliable transports and cannot provide such security assurance reliable transports and cannot provide such security assurance until
until an entire record is available, which may require the an entire record is available, which may require the buffering and/or
buffering and/or assembly of several distinct messages prior to TLS assembly of several distinct messages prior to TLS processing. This
processing. This defers RDMA processing and introduces overheads defers RDMA processing and introduces overheads that RDMA is designed
that RDMA is designed to avoid. In addition, TLS length to avoid. In addition, TLS length restrictions on records themselves
restrictions on records themselves impose additional buffering and impose additional buffering and processing for long operations that
processing, for long operations which must span multiple records. must span multiple records. TLS therefore is viewed as potentially a
TLS therefore is viewed as potentially a less natural fit for less natural fit for protecting the RDMA protocols.
protecting the RDMA protocols.
Any DDP and RDMAP specification must provide the means to satisfy Any DDP and RDMAP specification must provide the means to satisfy the
the above security service requirements. above security service requirements.
IPsec is sufficient to provide the required security services to IPsec is sufficient to provide the required security services to the
the DDP and RDMAP protocols, while enabling efficient DDP and RDMAP protocols, while enabling efficient implementations.
implementations.
3.2. Error Considerations 3.2. Error Considerations
Resource issues leading to denial-of-service attacks, overwrites Resource issues leading to denial-of-service attacks, overwrites and
and other concurrent operations, the ordering of completions as other concurrent operations, the ordering of completions as required
required by the RDMA protocol, and the granularity of transfer are by the RDMA protocol, and the granularity of transfer are all within
all within the required scope of any security analysis of RDMA and the required scope of any security analysis of RDMA and DDP.
DDP.
The RDMA operations require checking of what is essentially user The RDMA operations require checking of what is essentially user
information, explicitly including addressing information and information, explicitly including addressing information and
operation type (read or write), and implicitly including protection operation type (read or write), and implicitly including protection
and attributes. The semantics associated with each class of error and attributes. The semantics associated with each class of error
resulting from possible failure of such checks must be clearly resulting from possible failure of such checks must be clearly
defined, and the expected action to be taken by the protocols in defined, and the expected action to be taken by the protocols in each
each case must be specified. case must be specified.
In some cases, this will result in a catastrophic error on the RDMA In some cases, this will result in a catastrophic error on the RDMA
association, however in others, a local or remote error may be association; however, in others, a local or remote error may be
signalled. Certain of these errors may require consideration of signalled. Certain of these errors may require consideration of
abstract local semantics. The result of the error on the RDMA abstract local semantics. The result of the error on the RDMA
association must be carefully specified so as to provide useful association must be carefully specified so as to provide useful
behavior, while not constraining the implementation. behavior, while not constraining the implementation.
4. IANA Considerations 4. Acknowledgements
IANA considerations are not addressed in by this document. Any
IANA considerations resulting from the use of DDP or RDMA must be
addressed in the relevant standards.
5. Acknowledgements
The authors wish to acknowledge the valuable contributions of
Caitlin Bestler, David Black, Jeff Mogul and Allyn Romanow.
6. Informative References The authors wish to acknowledge the valuable contributions of Caitlin
Bestler, David Black, Jeff Mogul, and Allyn Romanow.
[DAFS] 5. Informative References
DAFS Collaborative, "Direct Access File System Specification
v1.0", September 2001, available from
http://www.dafscollaborative.org
[FCVI] [FCVI] ANSI Technical Committee T11, "Fibre Channel Standard
ANSI Technical Committee T11, "Fibre Channel Standard Virtual Virtual Interface Architecture Mapping", ANSI/NCITS 357-
Interface Architecture Mapping", ANSI/NCITS 357-2001, March 2001, March 2001, available from
2001, available from http://www.t11.org/t11/stat.nsf/fcproj http://www.t11.org/t11/stat.nsf/fcproj.
[IB] InfiniBand Trade Association, "InfiniBand Architecture [IB] InfiniBand Trade Association, "InfiniBand Architecture
Specification Volumes 1 and 2", Release 1.1, November 2002, Specification Volumes 1 and 2", Release 1.1, November 2002,
available from http://www.infinibandta.org/specs available from http://www.infinibandta.org/specs.
[MYR] [MYR] VMEbus International Trade Association, "Myrinet on VME
VMEbus International Trade Association, "Myrinet on VME
Protocol Specification", ANSI/VITA 26-1998, August 1998, Protocol Specification", ANSI/VITA 26-1998, August 1998,
available from http://www.myri.com/open-specs available from http://www.myri.com/open-specs.
[ROM] [ROM] Romanow, A., Mogul, J., Talpey, T., and S. Bailey, "Remote
A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP Direct Memory Access (RDMA) over IP Problem Statement", RFC
Problem Statement", draft-ietf-rddp-problem-statement, 4297, December 2005.
Internet Draft Work in Progress
[SCTP] [SCTP] Stewart, R., Xie, Q., Morneault, K., Sharp, C.,
R. Stewart et al., "Stream Transmission Control Protocol", RFC Schwarzbauer, H., Taylor, T., Rytina, I., Kalla, M., Zhang,
2960, Standards Track L., and V. Paxson, "Stream Control Transmission Protocol",
RFC 2960, October 2000.
[SDP] [SDP] InfiniBand Trade Association, "Sockets Direct Protocol
InfiniBand Trade Association, "Sockets Direct Protocol v1.0", v1.0", Annex A of InfiniBand Architecture Specification
Annex A of InfiniBand Architecture Specification Volume 1, Volume 1, Release 1.1, November 2002, available from
Release 1.1, November 2002, available from http://www.infinibandta.org/specs.
http://www.infinibandta.org/specs
[SRVNET] [SRVNET] R. Horst, "TNet: A reliable system area network", IEEE
R. Horst, "TNet: A reliable system area network", IEEE Micro, Micro, pp. 37-45, February 1995.
pp. 37-45, February 1995
[VI] Compaq Computer Corp., Intel Corporation and Microsoft [VI] D. Cameron and G. Regnier, "The Virtual Interface
Corporation, "Virtual Interface Architecture Specification Architecture", ISBN 0971288704, Intel Press, April 2002,
Version 1.0", December 1997, available from more info at http://www.intel.com/intelpress/via/.
http://www.vidf.org/info/04standards.html
Authors' Addresses Authors' Addresses
Stephen Bailey Stephen Bailey
Sandburst Corporation Sandburst Corporation
600 Federal Street 600 Federal Street
Andover, MA 01810 USA Andover, MA 01810 USA
USA USA
Phone: +1 978 689 1614 Phone: +1 978 689 1614
Email: steph@sandburst.com EMail: steph@sandburst.com
Tom Talpey Tom Talpey
Network Appliance Network Appliance
375 Totten Pond Road 1601 Trapelo Road
Waltham, MA 02451 USA Waltham, MA 02451 USA
Phone: +1 781 768 5329 Phone: +1 781 768 5329
Email: thomas.talpey@netapp.com EMail: thomas.talpey@netapp.com
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2005). This document is Copyright (C) The Internet Society (2005).
subject to the rights, licenses and restrictions contained in BCP
78 and except as set forth therein, the authors retain all their
rights.
This document and the information contained herein are provided on This document is subject to the rights, licenses and restrictions
an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE contained in BCP 78, and except as set forth therein, the authors
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND retain all their rights.
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT This document and the information contained herein are provided on an
THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
PARTICULAR PURPOSE. ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property Intellectual Property
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed Intellectual Property Rights or other rights that might be claimed to
to pertain to the implementation or use of the technology described pertain to the implementation or use of the technology described in
in this document or the extent to which any license under such this document or the extent to which any license under such rights
rights might or might not be available; nor does it represent that might or might not be available; nor does it represent that it has
it has made any independent effort to identify any such rights. made any independent effort to identify any such rights. Information
Information on the procedures with respect to rights in RFC on the procedures with respect to rights in RFC documents can be
documents can be found in BCP 78 and BCP 79. found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use attempt made to obtain a general license or permission for the use of
of such proprietary rights by implementers or users of this such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository specification can be obtained from the IETF on-line IPR repository at
at http://www.ietf.org/ipr. http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf- this standard. Please address the information to the IETF at ietf-
ipr@ietf.org. ipr@ietf.org.
Acknowledgement Acknowledgement
Funding for the RFC Editor function is currently provided by the Funding for the RFC Editor function is currently provided by the
Internet Society. Internet Society.
 End of changes. 103 change blocks. 
358 lines changed or deleted 316 lines changed or added

This html diff was produced by rfcdiff 1.28, available from http://www.levkowetz.com/ietf/tools/rfcdiff/