draft-ietf-rddp-arch-01.txt   draft-ietf-rddp-arch-02.txt 
Internet-Draft Stephen Bailey (Sandburst) Internet-Draft Stephen Bailey (Sandburst)
Expires: August 2003 Tom Talpey (NetApp) Expires: December 2003 Tom Talpey (NetApp)
The Architecture of Direct Data Placement (DDP) The Architecture of Direct Data Placement (DDP)
And Remote Direct Memory Access (RDMA) and Remote Direct Memory Access (RDMA)
On Internet Protocols on Internet Protocols
draft-ietf-rddp-arch-01 draft-ietf-rddp-arch-02
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 44 skipping to change at page 1, line 44
Copyright (C) The Internet Society (2003). All Rights Reserved. Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract Abstract
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols, not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding but is, rather, a descriptive tool for defining and understanding
the protocols. the protocols. DDP allows the efficient placement of data into
buffers designated by Upper Layer Protocols (e.g. RDMA). RDMA
provides the semantics to enable Remote Direct Memory Access
between peers in a way consistent with application requirements.
Table Of Contents Table Of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2
2. Architecture . . . . . . . . . . . . . . . . . . . . . . 3 2. Architecture . . . . . . . . . . . . . . . . . . . . . . 3
2.1. Direct Data Placement (DDP) Protocol Architecture . . . 3 2.1. Direct Data Placement (DDP) Protocol Architecture . . . 3
2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 5 2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 5
2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 6 2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 6
2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 9 2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 9
2.2. Remote Direct Memory Access Protocol Architecture . . . 11 2.2. Remote Direct Memory Access Protocol Architecture . . . 10
2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 12 2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 12
2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 14 2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 14
3. Security Considerations . . . . . . . . . . . . . . . . 15 3. Security Considerations . . . . . . . . . . . . . . . . 14
4. IANA Considerations . . . . . . . . . . . . . . . . . . 15 4. IANA Considerations . . . . . . . . . . . . . . . . . . 15
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 15 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 15
References . . . . . . . . . . . . . . . . . . . . . . . 16 Informative References . . . . . . . . . . . . . . . . . 15
Authors' Addresses . . . . . . . . . . . . . . . . . . . 16 Authors' Addresses . . . . . . . . . . . . . . . . . . . 16
Full Copyright Statement . . . . . . . . . . . . . . . . 17 Full Copyright Statement . . . . . . . . . . . . . . . . 17
1. Introduction 1. Introduction
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports [RDDP, ROM]. This run on Internet Protocol-suite transports. This architecture does
architecture does not necessarily reflect the proper way to not necessarily reflect the proper way to implement such protocols,
implement such protocols, but is, rather, a descriptive tool for but is, rather, a descriptive tool for defining and understanding
defining and understanding the protocols. the protocols.
The first part of the document describes the architecture of DDP The first part of the document describes the architecture of DDP
protocols, including what assumptions are made about the transports protocols, including what assumptions are made about the transports
on which DDP is built. The second part describes the architecture on which DDP is built. The second part describes the architecture
of RDMA protocols layered on top of DDP. of RDMA protocols layered on top of DDP.
Before introducing the protocols, three definitions will be useful Before introducing the protocols, three definitions will be useful
to guide discussion: to guide discussion:
o Placement - writing to a data buffer. o Placement - writing to a data buffer.
skipping to change at page 3, line 15 skipping to change at page 3, line 15
The goal of the DDP protocol is to allow the efficient placement of The goal of the DDP protocol is to allow the efficient placement of
data into buffers designated by Upper Layer Protocols (e.g. RDMA). data into buffers designated by Upper Layer Protocols (e.g. RDMA).
This is described in detail in [ROM]. Efficiency may be This is described in detail in [ROM]. Efficiency may be
characterized by the minimization of the number of transfers of the characterized by the minimization of the number of transfers of the
data over the receiver's system buses. data over the receiver's system buses.
The goal of the RDMA protocol is to provide the semantics to enable The goal of the RDMA protocol is to provide the semantics to enable
Remote Direct Memory Access between peers in a way consistent with Remote Direct Memory Access between peers in a way consistent with
application requirements. The RDMA protocol provides facilities application requirements. The RDMA protocol provides facilities
immediately useful to existing and future networking, storage, and immediately useful to existing and future networking, storage, and
other application protocols. [DAFS, FIBRE, IB, MYR, SDP, SRVNET, other application protocols. [DAFS, FCVI, IB, MYR, SDP, SRVNET,
VI] VI]
The DDP and RDMA protocols work together to achieve their The DDP and RDMA protocols work together to achieve their
respective goals. RDMA provides facilities to a ULP for respective goals. DDP provides facilities to safely steer payloads
identifying buffers, controlling the transfer of data between ULP to specific buffers at the Data Sink. RDMA provides facilities to
peers, and providing completion notifications to the ULP. RDMA a ULP for identifying these buffers, controlling the transfer of
uses the features of DDP to steer payloads to specific buffers at data between ULP peers, and signalling completion to the ULP. ULPs
the Data Sink. ULPs that do not require the features of RDMA may that do not require the features of RDMA may be layered directly on
be layered directly on top of DDP. top of DDP.
The DDP and RDMA protocols are transport independent. The The DDP and RDMA protocols are transport independent. The
following figure shows the relationship between RDMA, DDP, Upper following figure shows the relationship between RDMA, DDP, Upper
Layer Protocols and Transport. Layer Protocols and Transport.
+---------------------------------------------------+ +---------------------------------------------------+
| ULP | | ULP |
+---------+------------+----------------------------+ +---------+------------+----------------------------+
| | | RDMA | | | | RDMA |
| | +----------------------------+ | | +----------------------------+
skipping to change at page 6, line 46 skipping to change at page 6, line 46
Real implementations of xpt_send() and xpt_recv() typically return Real implementations of xpt_send() and xpt_recv() typically return
error indications, but that is not relevant to this architecture. error indications, but that is not relevant to this architecture.
2.1.2. DDP Operations 2.1.2. DDP Operations
The DDP layer provides: The DDP layer provides:
void ddp_send(socket_t s, message_t m); void ddp_send(socket_t s, message_t m);
void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d,
ddp_notify_t n); ddp_notify_t n);
void ddp_post_recv(socket_t s, bdesc_t b);
ddp_ind_t ddp_recv(socket_t s); ddp_ind_t ddp_recv(socket_t s);
bdesc_t ddp_register(socket_t s, ddp_buffer_t b); bdesc_t ddp_register(socket_t s, ddp_buffer_t b);
void ddp_deregister(bhand_t bh); void ddp_deregister(bhand_t bh);
msizes_t ddp_max_msizes(socket_t s); msizes_t ddp_max_msizes(socket_t s);
ddp_addr_t ddp_addr_t
the buffer address portion of a tagged message: the buffer address portion of a tagged message:
typedef struct { typedef struct {
stag_t stag; stag_t stag;
skipping to change at page 8, line 21 skipping to change at page 8, line 21
which may have no relationship to the `start' or `end' which may have no relationship to the `start' or `end'
addresses of that buffer. However, particular addresses of that buffer. However, particular
implementations, such as DDP on a multicast transport (see implementations, such as DDP on a multicast transport (see
below), may allow some client protocol control over the below), may allow some client protocol control over the
starting offset. starting offset.
bhand_t bhand_t
an opaque buffer handle used to deregister a buffer. an opaque buffer handle used to deregister a buffer.
recv_message_t
a description of a completed untagged receive buffer:
typedef struct {
bdesc_t b;
length l;
} recv_message_t;
ddp_ind_t ddp_ind_t
an untagged message, a tagged message reception indication, or an untagged message, a tagged message reception indication, or
a tagged message reception error: a tagged message reception error:
typedef union { typedef union {
recv_message_t m; message_t m;
ddp_msg_id_t i; ddp_msg_id_t i;
ddp_err_t e; ddp_err_t e;
} ddp_ind_t; } ddp_ind_t;
ddp_err_t ddp_err_t
indicates an error while receiving a tagged message, typically indicates an error while receiving a tagged message, typically
`offset' out of bounds, or `stag' is not registered to the `offset' out of bounds, or `stag' is not registered to the
socket. socket.
skipping to change at page 9, line 4 skipping to change at page 8, line 39
ddp_err_t e; ddp_err_t e;
} ddp_ind_t; } ddp_ind_t;
ddp_err_t ddp_err_t
indicates an error while receiving a tagged message, typically indicates an error while receiving a tagged message, typically
`offset' out of bounds, or `stag' is not registered to the `offset' out of bounds, or `stag' is not registered to the
socket. socket.
msizes_t msizes_t
The maximum untagged and tagged messages that fit in a single The maximum untagged and tagged messages that fit in a single
transport message: transport message:
typedef struct { typedef struct {
msize_t max_untagged; msize_t max_untagged;
msize_t max_tagged; msize_t max_tagged;
} msizes_t; } msizes_t;
ddp_send(socket_t s, message_t m) ddp_send(socket_t s, message_t m)
send an untagged message. send an untagged message.
ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n) ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n)
send a tagged message to remote buffer address d. send a tagged message to remote buffer address d.
ddp_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept received untagged messages.
ddp_recv(socket_t s) ddp_recv(socket_t s)
get the next received untagged message, tagged message get the next received untagged message, tagged message
reception indication, or tagged message error. reception indication, or tagged message error.
ddp_register(socket_t s, ddp_buffer_t b) ddp_register(socket_t s, ddp_buffer_t b)
register a buffer for DDP on a socket. The same buffer may be register a buffer for DDP on a socket. The same buffer may be
registered multiple times on the same or different sockets. registered multiple times on the same or different sockets.
The same buffer registered on different sockets may result in The same buffer registered on different sockets may result in
skipping to change at page 10, line 20 skipping to change at page 10, line 5
o single source or multisource, o single source or multisource,
o single destination or multidestination (multicast or anycast). o single destination or multidestination (multicast or anycast).
Some transports support several combinations of these Some transports support several combinations of these
characteristics. For example, SCTP [SCTP] is reliable, single characteristics. For example, SCTP [SCTP] is reliable, single
source, single destination (point-to-point) and supports both source, single destination (point-to-point) and supports both
ordered and unordered modes. ordered and unordered modes.
In general, these transport characteristics equally affect
transport and DDP message delivery. However, there are several
issues specific to DDP messages.
DDP messages carried by transport are framed for processing by the DDP messages carried by transport are framed for processing by the
receiver, and may be further protected for integrity or privacy in receiver, and may be further protected for integrity or privacy in
accordance with the transport capabilities. DDP does not provide accordance with the transport capabilities. DDP does not provide
such functions. such functions.
In general, transport characteristics equally affect transport and
DDP message delivery. However, there are several issues specific
to DDP messages.
A key component of DDP is how the following operations on the A key component of DDP is how the following operations on the
receiving side are ordered among themselves, and how they relate to receiving side are ordered among themselves, and how they relate to
corresponding operations on the sending side: corresponding operations on the sending side:
o set()s, o set()s,
o untagged message reception indications, and o untagged message reception indications, and
o tagged message reception indications. o tagged message reception indications.
These relationships depend upon the characteristics of the These relationships depend upon the characteristics of the
underlying transport in a way which is defined by the DDP protocol. underlying transport in a way which is defined by the DDP protocol.
For example, if the transport is unreliable and unordered, the DDP For example, if the transport is unreliable and unordered, the DDP
protocol might specify that the client protocol is subject to the protocol might specify that the client protocol is subject to the
consequences of transport messages being lost or duplicated, rather consequences of transport messages being lost or duplicated, rather
requiring different characteristics be presented to the client than requiring different characteristics be presented to the client
protocol. protocol.
Multidestination data delivery is the other transport Multidestination data delivery is the other transport
characteristic which may require specific consideration in a DDP characteristic which may require specific consideration in a DDP
protocol. As mentioned above, the basic DDP model assumes that protocol. As mentioned above, the basic DDP model assumes that
buffer address values returned by ddp_register() are opaque to the buffer address values returned by ddp_register() are opaque to the
client protocol, and can be implementation dependent. The most client protocol, and can be implementation dependent. The most
natural way to map DDP to a multidestination transport is to natural way to map DDP to a multidestination transport is to
require all receivers produce the same buffer address when require all receivers produce the same buffer address when
registering a multidestination destination buffer. Restriction of registering a multidestination destination buffer. Restriction of
skipping to change at page 12, line 27 skipping to change at page 12, line 13
} rdma_buffer_t; } rdma_buffer_t;
2.2.1. RDMA Operations 2.2.1. RDMA Operations
The RDMA layer provides: The RDMA layer provides:
void rdma_send(socket_t s, message_t m); void rdma_send(socket_t s, message_t m);
void rdma_write(socket_t s, message_t m, ddp_addr_t d, void rdma_write(socket_t s, message_t m, ddp_addr_t d,
rdma_notify_t n); rdma_notify_t n);
void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d); void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d);
void rdma_post_recv(socket_t s, bdesc_t b);
rdma_ind_t rdma_recv(socket_t s); rdma_ind_t rdma_recv(socket_t s);
bdesc_t rdma_register(socket_t s, rdma_buffer_t b, bdesc_t rdma_register(socket_t s, rdma_buffer_t b,
bmode_t mode); bmode_t mode);
void rdma_deregister(bhand_t bh); void rdma_deregister(bhand_t bh);
msizes_t rdma_max_msizes(socket_t s); msizes_t rdma_max_msizes(socket_t s);
Although, for clarity, these data transfer interfaces are Although, for clarity, these data transfer interfaces are
synchronous, rdma_read() and possibly rdma_send() (in the presence synchronous, rdma_read() and possibly rdma_send() (in the presence
of Send flow control), can require an arbitrary amount of time to of Send flow control), can require an arbitrary amount of time to
complete. To express the full concurrency and interleaving of RDMA complete. To express the full concurrency and interleaving of RDMA
skipping to change at page 13, line 22 skipping to change at page 13, line 6
rdma_write_id_t (scalar) rdma_write_id_t (scalar)
an RDMA Write identifier. an RDMA Write identifier.
rdma_ind_t rdma_ind_t
a Send message, or an RDMA error: a Send message, or an RDMA error:
typedef union { typedef union {
recv_message_t m; message_t m;
rdma_err_t e; rdma_err_t e;
} rdma_ind_t; } rdma_ind_t;
rdma_err_t rdma_err_t
an RDMA protocol error indication. RDMA errors include buffer an RDMA protocol error indication. RDMA errors include buffer
addressing errors corresponding to ddp_err_ts, and buffer addressing errors corresponding to ddp_err_ts, and buffer
protection violations (e.g. RDMA Writing a buffer only protection violations (e.g. RDMA Writing a buffer only
registered for reading). registered for reading).
skipping to change at page 14, line 5 skipping to change at page 13, line 37
rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n) rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n)
RDMA Write to remote buffer address d. RDMA Write to remote buffer address d.
rdma_read(socket_t s, ddp_addr_t s, length l, ddp_addr_t d) rdma_read(socket_t s, ddp_addr_t s, length l, ddp_addr_t d)
RDMA Read l octets from remote buffer address s to local RDMA Read l octets from remote buffer address s to local
buffer address d. buffer address d.
rdma_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept received Send messages.
rdma_recv(socket_t s); rdma_recv(socket_t s);
get the next received Send message, RDMA Write completion get the next received Send message, RDMA Write completion
identifier, or RDMA error. identifier, or RDMA error.
rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode) rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode)
register a buffer for RDMA on a socket (for read access, write register a buffer for RDMA on a socket (for read access, write
access or both). As with DDP, the same buffer may be access or both). As with DDP, the same buffer may be
registered multiple times on the same or different sockets, registered multiple times on the same or different sockets,
skipping to change at page 15, line 31 skipping to change at page 15, line 12
System integrity must be maintained in any RDMA solution. System integrity must be maintained in any RDMA solution.
Mechanisms must be specified to prevent RDMA or DDP operations from Mechanisms must be specified to prevent RDMA or DDP operations from
impairing system integrity. For example, the threat caused by impairing system integrity. For example, the threat caused by
potential buffer overflow needs full examination, and prevention potential buffer overflow needs full examination, and prevention
mechanisms must be spelled out. mechanisms must be spelled out.
Because a Steering Tag exports access to a memory region, one Because a Steering Tag exports access to a memory region, one
critical aspect of security is the scope of this access. It must critical aspect of security is the scope of this access. It must
be possible to individually control specific attributes of the be possible to individually control specific attributes of the
access provided by a Steering Tag, including remote read access, access provided by a Steering Tag, including remote read access,
remote write access, and others that might be identified. A remote write access, and others that might be identified. DDP and
specification must provide both implementation requirements RDMA specifications must provide both implementation requirements
relevant to this issue, and guidelines to assist implementors in relevant to this issue, and guidelines to assist implementors in
making the appropriate design decisions. making the appropriate design decisions.
Resource issues leading to denial-of-service attacks, overwrites Resource issues leading to denial-of-service attacks, overwrites
and other concurrent operations, the ordering of completions as and other concurrent operations, the ordering of completions as
required by the RDMA protocol, and the granularity of transfer are required by the RDMA protocol, and the granularity of transfer are
all within the required scope of any security analysis of RDMA and all within the required scope of any security analysis of RDMA and
DDP. DDP.
4. IANA Considerations 4. IANA Considerations
IANA considerations are not addressed in by this document. Any IANA considerations are not addressed in by this document. Any
IANA considerations resulting from the use of DDP or RDMA must be IANA considerations resulting from the use of DDP or RDMA must be
addressed in the relevant standards. addressed in the relevant standards.
5. Acknowledgements 5. Acknowledgements
The authors wish to acknowledge the valuable contributions of David The authors wish to acknowledge the valuable contributions of David
Black, Jeff Mogul and Allyn Romanow. Black, Jeff Mogul and Allyn Romanow.
6. References 6. Informative References
[DAFS] [DAFS]
Direct Access File System http://www.dafscollaborative.org DAFS Collaborative, "Direct Access File System Specification
http://www.ietf.org/internet-drafts/draft-wittle-dafs-00.txt v1.0", September 2001, available from
http://www.dafscollaborative.org
[FIBRE] [FCVI]
Fibre Channel Standard ANSI Technical Committee T11, "Fibre Channel Standard Virtual
http://www.fibrechannel.com/technology/index.master.html Interface Architecture Mapping", ANSI/NCITS 357-2001, March
2001, available from http://www.t11.org/t11/stat.nsf/fcproj
[IB] InfiniBand Architecture Specification, Volumes 1 and 2, [IB] InfiniBand Trade Association, "InfiniBand Architecture
Release 1.0.a. http://www.infinibandta.org Specification Volumes 1 and 2", Release 1.1, November 2002,
available from http://www.infinibandta.org/specs
[MYR] [MYR]
Myrinet, http://www.myricom.com VMEbus International Trade Association, "Myrinet on VME
Protocol Specification", ANSI/VITA 26-1998, August 1998,
available from http://www.myri.com/open-specs
[RDDP] RFC Editor note:
Remote Direct Data Placement Working Group charter, Replace following problem statement draft-ietf- name, status and
http://www.ietf.org/html.charters/rddp-charter.html date with appropriate reference when assigned.
[ROM] [ROM]
A. Romanow, J. Mogul, T. Talpey, S. Bailey, "RDMA over IP A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP
Problem Statement", http://www.ietf.org/internet-drafts/draft- Problem Statement", draft-ietf-rddp-problem-statement-02, Work
ietf-rddp-problem-statement-01.txt, Work in Progress, February in Progress, June 2003
2003
[SCTP] [SCTP]
R. Stewart et al., "Stream Transmission Control Protocol", R. Stewart et al., "Stream Transmission Control Protocol", RFC
Standards Track RFC, http://www.ietf.org/rfc/rfc2960.txt 2960, Standards Track
[SDP] [SDP]
Sockets Direct Protocol v1.0 InfiniBand Trade Association, "Sockets Direct Protocol v1.0",
Annex A of InfiniBand Architecture Specification Volume 1,
Release 1.1, November 2002, available from
http://www.infinibandta.org/specs
[SRVNET] [SRVNET]
Compaq Servernet, R. Horst, "TNet: A reliable system area network", IEEE Micro,
http://nonstop.compaq.com/view.asp?PAGE=ServerNet pp. 37-45, February 1995
[VI] Virtual Interface Architecture Specification Version 1.0. [VI] Compaq Computer Corp., Intel Corporation and Microsoft
Corporation, "Virtual Interface Architecture Specification
Version 1.0", December 1997, available from
http://www.vidf.org/info/04standards.html http://www.vidf.org/info/04standards.html
Authors' Addresses Authors' Addresses
Stephen Bailey Stephen Bailey
Sandburst Corporation Sandburst Corporation
600 Federal Street 600 Federal Street
Andover, MA 01810 USA Andover, MA 01810 USA
USA USA
Phone: +1 978 689 1614 Phone: +1 978 689 1614
Email: steph@sandburst.com Email: steph@sandburst.com
Tom Talpey Tom Talpey
Network Appliance Network Appliance
375 Totten Pond Road 375 Totten Pond Road
Waltham, MA 02451 USA Waltham, MA 02451 USA
Phone: +1 781 768 5329 Phone: +1 781 768 5329
Email: thomas.talpey@netapp.com Email: thomas.talpey@netapp.com
Full Copyright Statement Full Copyright Statement
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/