draft-ietf-rddp-arch-03.txt   draft-ietf-rddp-arch-04.txt 
Internet-Draft Stephen Bailey (Sandburst) Internet-Draft Stephen Bailey (Sandburst)
Expires: March 2004 Tom Talpey (NetApp) Expires: July 2004 Tom Talpey (NetApp)
The Architecture of Direct Data Placement (DDP) The Architecture of Direct Data Placement (DDP)
and Remote Direct Memory Access (RDMA) and Remote Direct Memory Access (RDMA)
on Internet Protocols on Internet Protocols
draft-ietf-rddp-arch-03 draft-ietf-rddp-arch-04
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 35 skipping to change at page 1, line 35
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved. Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract Abstract
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols, not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding but is, rather, a descriptive tool for defining and understanding
the protocols. DDP allows the efficient placement of data into the protocols. DDP allows the efficient placement of data into
buffers designated by Upper Layer Protocols (e.g. RDMA). RDMA buffers designated by Upper Layer Protocols (e.g. RDMA). RDMA
provides the semantics to enable Remote Direct Memory Access provides the semantics to enable Remote Direct Memory Access
between peers in a way consistent with application requirements. between peers in a way consistent with application requirements.
Table Of Contents Table Of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2
2. Architecture . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . 2
2.1. Direct Data Placement (DDP) Protocol Architecture . . . 3 1.2. DDP and RDMA Protocols . . . . . . . . . . . . . . . . . 3
2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 5 2. Architecture . . . . . . . . . . . . . . . . . . . . . . 4
2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 6 2.1. Direct Data Placement (DDP) Protocol Architecture . . . 4
2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 6
2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 7
2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10 2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10
2.2. Remote Direct Memory Access Protocol Architecture . . . 11 2.2. Remote Direct Memory Access Protocol Architecture . . . 12
2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 12 2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 13
2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 15 2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 16
3. Security Considerations . . . . . . . . . . . . . . . . 16 3. Security Considerations . . . . . . . . . . . . . . . . 17
4. IANA Considerations . . . . . . . . . . . . . . . . . . 16 4. IANA Considerations . . . . . . . . . . . . . . . . . . 18
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 16 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 18
Informative References . . . . . . . . . . . . . . . . . 16 Informative References . . . . . . . . . . . . . . . . . 18
Authors' Addresses . . . . . . . . . . . . . . . . . . . 17 Authors' Addresses . . . . . . . . . . . . . . . . . . . 19
Full Copyright Statement . . . . . . . . . . . . . . . . 18 Full Copyright Statement . . . . . . . . . . . . . . . . 19
1. Introduction 1. Introduction
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols, not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding but is, rather, a descriptive tool for defining and understanding
the protocols. the protocols. This document uses C language notation as a
shorthand to describe the architectural elements of DDP and RDMA
protocols. The choice of C notation is not intended to describe
concrete protocols or programming interfaces.
The first part of the document describes the architecture of DDP The first part of the document describes the architecture of DDP
protocols, including what assumptions are made about the transports protocols, including what assumptions are made about the transports
on which DDP is built. The second part describes the architecture on which DDP is built. The second part describes the architecture
of RDMA protocols layered on top of DDP. of RDMA protocols layered on top of DDP.
Before introducing the protocols, three definitions will be useful 1.1. Terminology
to guide discussion:
Before introducing the protocols, certain definitions will be
useful to guide discussion:
o Placement - writing to a data buffer. o Placement - writing to a data buffer.
o Delivery - informing the Upper Layer Protocol (ULP) (e.g. o Operation - a protocol message, or sequence of messages, which
RDMA) that a particular message is available for use. provide a architectural semantic, such as reading or writing
Delivery therefore may be viewed as the "control" signal of a data buffer.
associated with a unit of data. Note that the order of
delivery is defined more strictly than it is for placement.
o Completion - informing the ULP or application that a o Delivery - informing any Upper Layer or application that a
particular RDMA operation has finished. A completion, for particular message is available for use. Delivery therefore
may be viewed as the "control" signal associated with a unit
of data. Note that the order of delivery is defined more
strictly than it is for placement.
o Completion - informing any Upper Layer or application that a
particular operation has finished. A completion, for
instance, may require the delivery of several messages, or it instance, may require the delivery of several messages, or it
may also reflect that some local processing has finished. may also reflect that some local processing has finished.
o Data Sink - the peer on which any placement occurs.
o Data Source - the peer from which the placed data originates.
o Steering Tag - a "handle" used to identify memory which is the
target of placement. A "tagged" message is one which
references such a handle.
o RDMA Write - an Operation which places data from a local data
buffer to a remote data buffer specified by a Steering Tag.
o RDMA Read - an Operation which places data to a local data
buffer specified by a Steering Tag from a remote data buffer
specified by another Steering Tag.
o Send - an Operation which places data from a local data buffer
to a remote data buffer of the data sink's choice. Sends are
therefore "untagged".
1.2. DDP and RDMA Protocols
The goal of the DDP protocol is to allow the efficient placement of The goal of the DDP protocol is to allow the efficient placement of
data into buffers designated by Upper Layer Protocols (e.g. RDMA). data into buffers designated by protocols layered above DDP (e.g.
This is described in detail in [ROM]. Efficiency may be RDMA). This is described in detail in [ROM]. Efficiency may be
characterized by the minimization of the number of transfers of the characterized by the minimization of the number of transfers of the
data over the receiver's system buses. data over the receiver's system buses.
The goal of the RDMA protocol is to provide the semantics to enable The goal of the RDMA protocol is to provide the semantics to enable
Remote Direct Memory Access between peers in a way consistent with Remote Direct Memory Access between peers in a way consistent with
application requirements. The RDMA protocol provides facilities application requirements. The RDMA protocol provides facilities
immediately useful to existing and future networking, storage, and immediately useful to existing and future networking, storage, and
other application protocols. [DAFS, FCVI, IB, MYR, SDP, SRVNET, other application protocols. [DAFS, FCVI, IB, MYR, SDP, SRVNET,
VI] VI]
The DDP and RDMA protocols work together to achieve their The DDP and RDMA protocols work together to achieve their
respective goals. DDP provides facilities to safely steer payloads respective goals. DDP provides facilities to safely steer payloads
to specific buffers at the Data Sink. RDMA provides facilities to to specific buffers at the Data Sink. RDMA provides facilities to
a ULP for identifying these buffers, controlling the transfer of Upper Layers for identifying these buffers, controlling the
data between ULP peers, and signalling completion to the ULP. ULPs transfer of data between peers' buffers, supporting authorized
that do not require the features of RDMA may be layered directly on bidirectional transfer between buffers, and signalling completion.
top of DDP. Upper Layer Protocols that do not require the features of RDMA may
be layered directly on top of DDP.
The DDP and RDMA protocols are transport independent. The The DDP and RDMA protocols are transport independent. The
following figure shows the relationship between RDMA, DDP, Upper following figure shows the relationship between RDMA, DDP, Upper
Layer Protocols and Transport. Layer Protocols and Transport.
+---------------------------------------------------+ +--------------------------------------------------+
| ULP | | Upper Layer Protocol |
+---------+------------+----------------------------+ +---------+------------+---------------------------+
| | | RDMA | | | | RDMA |
| | +----------------------------+ | | +---------------------------+
| | DDP | | | DDP |
| +-----------------------------------------+ | +----------------------------------------+
| Transport | | Transport |
+---------------------------------------------------+ +--------------------------------------------------+
2. Architecture 2. Architecture
The Architecture section is presented in two parts: Direct Data The Architecture section is presented in two parts: Direct Data
Placement Protocol architecture and Remote Direct Memory Access Placement Protocol architecture and Remote Direct Memory Access
Protocol architecture. Protocol architecture.
2.1. Direct Data Placement (DDP) Protocol Architecture 2.1. Direct Data Placement (DDP) Protocol Architecture
The central idea of general-purpose DDP is that a data sender will The central idea of general-purpose DDP is that a data sender will
skipping to change at page 4, line 31 skipping to change at page 5, line 21
address_t address_t
a reference to local memory a reference to local memory
data_t data_t
an octet data value. an octet data value.
The protocol layering and in-line data flow of DDP is: The protocol layering and in-line data flow of DDP is:
Client Protocol DDP Client Protocol
(e.g. ULP or RDMA) (e.g. RDMA or Upper Layer Protocol)
| ^ | ^
untagged messages | | untagged message delivery untagged messages | | untagged message delivery
tagged messages | | tagged message delivery tagged messages | | tagged message delivery
v | v |
DDP+---> data placement DDP+---> data placement
^ ^
| transport messages | transport messages
v v
Transport Transport
(e.g. SCTP, DCP, framed TCP) (e.g. SCTP, DCCP, framed TCP)
^ ^
| IP datagrams | IP datagrams
v v
. . . . . .
In addition to in-line data flow, the client protocol registers In addition to in-line data flow, the client protocol registers
buffers with DDP, and DDP performs buffer update (set()) operations buffers with DDP, and DDP performs buffer update (set()) operations
as a result of receiving tagged messages. as a result of receiving tagged messages.
DDP messages may be split into multiple, smaller DDP messages, each DDP messages may be split into multiple, smaller DDP messages, each
in a separate transport message. However, if the transport is in a separate transport message. However, if the transport is
unreliable or unordered, messages split across transport messages unreliable or unordered, messages split across transport messages
may or may not provide useful behavior, in the same way as may or may not provide useful behavior, in the same way as
splitting arbitrary upper layer messages across unreliable or splitting arbitrary Upper Layer messages across unreliable or
unordered transport messages may or may not provide useful unordered transport messages may or may not provide useful
behavior. In other words, the same considerations apply to behavior. In other words, the same considerations apply to
building client protocols on different types of transports with or building client protocols on different types of transports with or
without the use of DDP. without the use of DDP.
A DDP message split across transport messages looks like: A DDP message split across transport messages looks like:
DDP message: Transport messages: DDP message: Transport messages:
stag=s, offset=o, message 1: stag=s, offset=o, message 1:
skipping to change at page 8, line 27 skipping to change at page 9, line 15
bhand_t bhand_t
an opaque buffer handle used to deregister a buffer. an opaque buffer handle used to deregister a buffer.
recv_message_t recv_message_t
a description of a completed untagged receive buffer: a description of a completed untagged receive buffer:
typedef struct { typedef struct {
bdesc_t b; bdesc_t b;
length l; length_t l;
} recv_message_t; } recv_message_t;
ddp_ind_t ddp_ind_t
an untagged message, a tagged message reception indication, or an untagged message, a tagged message reception indication, or
a tagged message reception error: a tagged message reception error:
typedef union { typedef union {
recv_message_t m; recv_message_t m;
ddp_msg_id_t i; ddp_msg_id_t i;
skipping to change at page 10, line 4 skipping to change at page 10, line 35
The same buffer registered on different sockets may result in The same buffer registered on different sockets may result in
a common registration. Different buffers may also refer to a common registration. Different buffers may also refer to
portions of the same underlying addressable object (buffer portions of the same underlying addressable object (buffer
aliasing). aliasing).
ddp_deregister(bhand_t bh) ddp_deregister(bhand_t bh)
remove a registration from a buffer. remove a registration from a buffer.
ddp_max_msizes(socket_t s) ddp_max_msizes(socket_t s)
get the current maximum untagged and tagged message sizes that get the current maximum untagged and tagged message sizes that
will fit in a single transport message. will fit in a single transport message.
2.1.3. Transport Characteristics In DDP 2.1.3. Transport Characteristics In DDP
Certain characteristics of the transport on which DDP is mapped Certain characteristics of the transport on which DDP is mapped
determine the nature of the service provided to client protocols. determine the nature of the service provided to client protocols.
Fundamentally, the characteristics of the transport will not be
changed by the presence of DDP. The choice of transport is
therefore driven not by DDP, but by the requirements of the Upper
Layer, and employing the DDP service.
Specifically, transports are: Specifically, transports are:
o reliable or unreliable, o reliable or unreliable,
o ordered or unordered, o ordered or unordered,
o single source or multisource, o single source or multisource,
o single destination or multidestination (multicast or anycast). o single destination or multidestination (multicast or anycast).
Some transports support several combinations of these Some transports support several combinations of these
characteristics. For example, SCTP [SCTP] is reliable, single characteristics. For example, SCTP [SCTP] is reliable, single
source, single destination (point-to-point) and supports both source, single destination (point-to-point) and supports both
ordered and unordered modes. ordered and unordered modes.
skipping to change at page 11, line 32 skipping to change at page 12, line 21
When multiple sockets reference an untagged message reception When multiple sockets reference an untagged message reception
buffer, local interfaces are responsible for managing the buffer, local interfaces are responsible for managing the
mechanisms of allocating posted buffers to received untagged mechanisms of allocating posted buffers to received untagged
messages, the handling of received untagged messages when no buffer messages, the handling of received untagged messages when no buffer
is available, and of resource management among multiple sockets. is available, and of resource management among multiple sockets.
Where underprovisioning of buffers on multiple sockets is allowed, Where underprovisioning of buffers on multiple sockets is allowed,
mechanisms should be provided to manage buffer consumption on a mechanisms should be provided to manage buffer consumption on a
per-socket or group of related sockets basis. per-socket or group of related sockets basis.
Architecturally, therefore, DDP is a flexible and general paradigm
which may be applied to any variety of transports. Implementations
of DDP may, however, adapt themselves to these differences in ways
appropriate to each transport. In all cases the layering of DDP
must continue to express the transport's underlying
characteristics.
2.2. Remote Direct Memory Access (RDMA) Protocol Architecture 2.2. Remote Direct Memory Access (RDMA) Protocol Architecture
Remote Direct Memory Access (RDMA) extends the capabilities of DDP Remote Direct Memory Access (RDMA) extends the capabilities of DDP
with the ability to read from buffers registered to a socket (RDMA with two primary functions.
Read). This allows a client protocol to perform arbitrary,
bidirectional data movement without involving the remote client.
When RDMA is implemented in hardware, arbitrary data movement can
be performed without involving the remote host CPU at all.
In addition, RDMA protocols usually specify a transport-independent First, it adds the ability to read from buffers registered to a
untagged message service (Send) with characteristics which are both socket (RDMA Read). This allows a client protocol to perform
very efficient to implement in hardware, and convenient for client arbitrary, bidirectional data movement without involving the remote
client. When RDMA is implemented in hardware, arbitrary data
movement can be performed without involving the remote host CPU at
all.
In addition, RDMA specifies a transport-independent untagged
message service (Send) with characteristics which are both very
efficient to implement in hardware, and convenient for client
protocols. protocols.
The RDMA architecture is patterned after the traditional model for The RDMA architecture is patterned after the traditional model for
device programming, where the client requests an operation using device programming, where the client requests an operation using
Send-like actions (programmed I/O), the server performs the Send-like actions (programmed I/O), the server performs the
necessary data transfers for the operation (DMA reads and writes), necessary data transfers for the operation (DMA reads and writes),
and notifies the client of completion. The programmed I/O+DMA and notifies the client of completion. The programmed I/O+DMA
model efficiently supports a high degree of concurrency and model efficiently supports a high degree of concurrency and
flexibility for both the client and server, even when operations flexibility for both the client and server, even when operations
have a wide range of intrinsic latencies. have a wide range of intrinsic latencies.
skipping to change at page 14, line 26 skipping to change at page 15, line 26
rdma_send(socket_t s, message_t m) rdma_send(socket_t s, message_t m)
send a message, delivering it to the next untagged RDMA buffer send a message, delivering it to the next untagged RDMA buffer
at the remote peer. at the remote peer.
rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n) rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n)
RDMA Write to remote buffer address d. RDMA Write to remote buffer address d.
rdma_read(socket_t s, ddp_addr_t s, length l, ddp_addr_t d) rdma_read(socket_t s, ddp_addr_t s, length_t l, ddp_addr_t d)
RDMA Read l octets from remote buffer address s to local RDMA Read l octets from remote buffer address s to local
buffer address d. buffer address d.
rdma_post_recv(socket_t s, bdesc_t b) rdma_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single Send message, to post a registered buffer to accept a single Send message, to
be filled and returned in-order to a subsequent caller of be filled and returned in-order to a subsequent caller of
rdma_recv(). As with DDP, buffers may be enabled on multiple rdma_recv(). As with DDP, buffers may be enabled on multiple
sockets, in which case ordering guarantees are relaxed. Also sockets, in which case ordering guarantees are relaxed. Also
skipping to change at page 15, line 22 skipping to change at page 16, line 22
get the current maximum Send (max_untagged) and RDMA Read or get the current maximum Send (max_untagged) and RDMA Read or
Write (max_tagged) operations that will fit in a single Write (max_tagged) operations that will fit in a single
transport message. The values returned by rdma_max_msizes() transport message. The values returned by rdma_max_msizes()
are closely related to the values returned by are closely related to the values returned by
ddp_max_msizes(), but may not be equal. ddp_max_msizes(), but may not be equal.
2.2.2. Transport Characteristics In RDMA 2.2.2. Transport Characteristics In RDMA
As with DDP, RDMA can be used on transports with a variety of As with DDP, RDMA can be used on transports with a variety of
different characteristics that manifest themselves directly in the different characteristics that manifest themselves directly in the
service provided by RDMA. service provided by RDMA. Also as with DDP, the fundamental
characteristics of the transport will not be changed by the
presence of RDMA.
Like DDP, an RDMA protocol must specify how: Like DDP, an RDMA protocol must specify how:
o set()s, o set()s,
o get()s, o get()s,
o Send messages, and o Send messages, and
o RDMA Read completions o RDMA Read completions
skipping to change at page 15, line 50 skipping to change at page 17, line 4
attendant complexities in managing endpoint state: attendant complexities in managing endpoint state:
o Send flow control o Send flow control
o RDMA Read o RDMA Read
These difficulties can be overcome by placing restrictions on the These difficulties can be overcome by placing restrictions on the
service provided by RDMA. However, many RDMA clients, especially service provided by RDMA. However, many RDMA clients, especially
those that separate data transfer and application logic concerns, those that separate data transfer and application logic concerns,
are likely to depend upon capabilities only provided by RDMA on a are likely to depend upon capabilities only provided by RDMA on a
point-to-point, reliable transport. point-to-point, reliable transport. In other words, many potential
Upper Layers which might avail themselves of RDMA services are
naturally already biased toward these transport classes.
3. Security Considerations 3. Security Considerations
Fundamentally, the DDP and RDMA protocols should not introduce
additional vulnerabilities. They are intermediate protocols and so
should not perform or require functions such as authorization,
which are the domain of Upper Layers. However, the DDP and RDMA
protocols should allow mapping by strict Upper Layers which are not
permissive of new vulnerabilities -- DDP and RDMAP implementations
should be prohibited from `cutting corners' that create new
vulnerabilities. Implementations must ensure that only `supplied'
resources (i.e. buffers) can be manipulated by DDP or RDMAP
messages.
System integrity must be maintained in any RDMA solution. System integrity must be maintained in any RDMA solution.
Mechanisms must be specified to prevent RDMA or DDP operations from Mechanisms must be specified to prevent RDMA or DDP operations from
impairing system integrity. For example, the threat caused by impairing system integrity. For example, threats can include
potential buffer overflow needs full examination, and prevention potential buffer reuse or buffer overflow, and are not merely a
mechanisms must be spelled out. security issue. Even trusted peers must not be allowed to damage
local integrity. Any DDP and RDMA protocol must address the issue
of giving end-systems and applications the capabilities to offer
protection from such compromises.
Because a Steering Tag exports access to a memory region, one Because a Steering Tag exports access to a memory region, one
critical aspect of security is the scope of this access. It must critical aspect of security is the scope of this access. It must
be possible to individually control specific attributes of the be possible to individually control specific attributes of the
access provided by a Steering Tag, including remote read access, access provided by a Steering Tag, including remote read access,
remote write access, and others that might be identified. DDP and remote write access, and others that might be identified. DDP and
RDMA specifications must provide both implementation requirements RDMA specifications must provide both implementation requirements
relevant to this issue, and guidelines to assist implementors in relevant to this issue, and guidelines to assist implementors in
making the appropriate design decisions. making the appropriate design decisions.
The use of DDP and RDMA on a transport connection may interact with
any security mechanism, and vice-versa. For example, if the
security mechanism is implemented above the transport layer, the
DDP and RDMA headers may not be protected. Such a layering may
therefore be inappropriate, depending on requirements. Or, when
TLS is employed, it may not be possible for DDP and RDMA to process
segments out of order, due to the in-order requirement of TLS.
These interactions should be well explored.
Resource issues leading to denial-of-service attacks, overwrites Resource issues leading to denial-of-service attacks, overwrites
and other concurrent operations, the ordering of completions as and other concurrent operations, the ordering of completions as
required by the RDMA protocol, and the granularity of transfer are required by the RDMA protocol, and the granularity of transfer are
all within the required scope of any security analysis of RDMA and all within the required scope of any security analysis of RDMA and
DDP. DDP.
4. IANA Considerations 4. IANA Considerations
IANA considerations are not addressed in by this document. Any IANA considerations are not addressed in by this document. Any
IANA considerations resulting from the use of DDP or RDMA must be IANA considerations resulting from the use of DDP or RDMA must be
skipping to change at page 17, line 13 skipping to change at page 18, line 40
Specification Volumes 1 and 2", Release 1.1, November 2002, Specification Volumes 1 and 2", Release 1.1, November 2002,
available from http://www.infinibandta.org/specs available from http://www.infinibandta.org/specs
[MYR] [MYR]
VMEbus International Trade Association, "Myrinet on VME VMEbus International Trade Association, "Myrinet on VME
Protocol Specification", ANSI/VITA 26-1998, August 1998, Protocol Specification", ANSI/VITA 26-1998, August 1998,
available from http://www.myri.com/open-specs available from http://www.myri.com/open-specs
[ROM] [ROM]
A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP
Problem Statement", draft-ietf-rddp-problem-statement-02, Work Problem Statement", draft-ietf-rddp-problem-statement-03, Work
in Progress, June 2003 in Progress, January 2004
RFC Editor note: Replace problem statement draft-ietf- name, status and RFC Editor note: Replace problem statement draft-ietf- name, status and
date with appropriate reference when assigned. date with appropriate reference when assigned.
[SCTP] [SCTP]
R. Stewart et al., "Stream Transmission Control Protocol", RFC R. Stewart et al., "Stream Transmission Control Protocol", RFC
2960, Standards Track 2960, Standards Track
[SDP] [SDP]
InfiniBand Trade Association, "Sockets Direct Protocol v1.0", InfiniBand Trade Association, "Sockets Direct Protocol v1.0",
Annex A of InfiniBand Architecture Specification Volume 1, Annex A of InfiniBand Architecture Specification Volume 1,
skipping to change at page 18, line 14 skipping to change at page 19, line 37
Tom Talpey Tom Talpey
Network Appliance Network Appliance
375 Totten Pond Road 375 Totten Pond Road
Waltham, MA 02451 USA Waltham, MA 02451 USA
Phone: +1 781 768 5329 Phone: +1 781 768 5329
Email: thomas.talpey@netapp.com Email: thomas.talpey@netapp.com
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2003). All Rights Reserved. Copyright (C) The Internet Society (2004). All Rights Reserved.
This document and translations of it may be copied and furnished to This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain others, and derivative works that comment on or otherwise explain
it or assist in its implementation may be prepared, copied, it or assist in its implementation may be prepared, copied,
published and distributed, in whole or in part, without restriction published and distributed, in whole or in part, without restriction
of any kind, provided that the above copyright notice and this of any kind, provided that the above copyright notice and this
paragraph are included on all such copies and derivative works. paragraph are included on all such copies and derivative works.
However, this document itself may not be modified in any way, such However, this document itself may not be modified in any way, such
as by removing the copyright notice or references to the Internet as by removing the copyright notice or references to the Internet
Society or other Internet organizations, except as needed for the Society or other Internet organizations, except as needed for the
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/