draft-ietf-rddp-arch-02.txt   draft-ietf-rddp-arch-03.txt 
Internet-Draft Stephen Bailey (Sandburst) Internet-Draft Stephen Bailey (Sandburst)
Expires: December 2003 Tom Talpey (NetApp) Expires: March 2004 Tom Talpey (NetApp)
The Architecture of Direct Data Placement (DDP) The Architecture of Direct Data Placement (DDP)
and Remote Direct Memory Access (RDMA) and Remote Direct Memory Access (RDMA)
on Internet Protocols on Internet Protocols
draft-ietf-rddp-arch-02 draft-ietf-rddp-arch-03
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 2, line 12 skipping to change at page 2, line 12
provides the semantics to enable Remote Direct Memory Access provides the semantics to enable Remote Direct Memory Access
between peers in a way consistent with application requirements. between peers in a way consistent with application requirements.
Table Of Contents Table Of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2
2. Architecture . . . . . . . . . . . . . . . . . . . . . . 3 2. Architecture . . . . . . . . . . . . . . . . . . . . . . 3
2.1. Direct Data Placement (DDP) Protocol Architecture . . . 3 2.1. Direct Data Placement (DDP) Protocol Architecture . . . 3
2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 5 2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 5
2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 6 2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 6
2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 9 2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10
2.2. Remote Direct Memory Access Protocol Architecture . . . 10 2.2. Remote Direct Memory Access Protocol Architecture . . . 11
2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 12 2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 12
2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 14 2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 15
3. Security Considerations . . . . . . . . . . . . . . . . 14 3. Security Considerations . . . . . . . . . . . . . . . . 16
4. IANA Considerations . . . . . . . . . . . . . . . . . . 15 4. IANA Considerations . . . . . . . . . . . . . . . . . . 16
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 15 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 16
Informative References . . . . . . . . . . . . . . . . . 15 Informative References . . . . . . . . . . . . . . . . . 16
Authors' Addresses . . . . . . . . . . . . . . . . . . . 16 Authors' Addresses . . . . . . . . . . . . . . . . . . . 17
Full Copyright Statement . . . . . . . . . . . . . . . . 17 Full Copyright Statement . . . . . . . . . . . . . . . . 18
1. Introduction 1. Introduction
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols, not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding but is, rather, a descriptive tool for defining and understanding
the protocols. the protocols.
skipping to change at page 6, line 46 skipping to change at page 6, line 46
Real implementations of xpt_send() and xpt_recv() typically return Real implementations of xpt_send() and xpt_recv() typically return
error indications, but that is not relevant to this architecture. error indications, but that is not relevant to this architecture.
2.1.2. DDP Operations 2.1.2. DDP Operations
The DDP layer provides: The DDP layer provides:
void ddp_send(socket_t s, message_t m); void ddp_send(socket_t s, message_t m);
void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d,
ddp_notify_t n); ddp_notify_t n);
void ddp_post_recv(socket_t s, bdesc_t b);
ddp_ind_t ddp_recv(socket_t s); ddp_ind_t ddp_recv(socket_t s);
bdesc_t ddp_register(socket_t s, ddp_buffer_t b); bdesc_t ddp_register(socket_t s, ddp_buffer_t b);
void ddp_deregister(bhand_t bh); void ddp_deregister(bhand_t bh);
msizes_t ddp_max_msizes(socket_t s); msizes_t ddp_max_msizes(socket_t s);
ddp_addr_t ddp_addr_t
the buffer address portion of a tagged message: the buffer address portion of a tagged message:
typedef struct { typedef struct {
stag_t stag; stag_t stag;
skipping to change at page 8, line 21 skipping to change at page 8, line 21
which may have no relationship to the `start' or `end' which may have no relationship to the `start' or `end'
addresses of that buffer. However, particular addresses of that buffer. However, particular
implementations, such as DDP on a multicast transport (see implementations, such as DDP on a multicast transport (see
below), may allow some client protocol control over the below), may allow some client protocol control over the
starting offset. starting offset.
bhand_t bhand_t
an opaque buffer handle used to deregister a buffer. an opaque buffer handle used to deregister a buffer.
recv_message_t
a description of a completed untagged receive buffer:
typedef struct {
bdesc_t b;
length l;
} recv_message_t;
ddp_ind_t ddp_ind_t
an untagged message, a tagged message reception indication, or an untagged message, a tagged message reception indication, or
a tagged message reception error: a tagged message reception error:
typedef union { typedef union {
message_t m; recv_message_t m;
ddp_msg_id_t i; ddp_msg_id_t i;
ddp_err_t e; ddp_err_t e;
} ddp_ind_t; } ddp_ind_t;
ddp_err_t ddp_err_t
indicates an error while receiving a tagged message, typically indicates an error while receiving a tagged message, typically
`offset' out of bounds, or `stag' is not registered to the `offset' out of bounds, or `stag' is not registered to the
socket. socket.
skipping to change at page 8, line 39 skipping to change at page 9, line 4
ddp_err_t e; ddp_err_t e;
} ddp_ind_t; } ddp_ind_t;
ddp_err_t ddp_err_t
indicates an error while receiving a tagged message, typically indicates an error while receiving a tagged message, typically
`offset' out of bounds, or `stag' is not registered to the `offset' out of bounds, or `stag' is not registered to the
socket. socket.
msizes_t msizes_t
The maximum untagged and tagged messages that fit in a single The maximum untagged and tagged messages that fit in a single
transport message: transport message:
typedef struct { typedef struct {
msize_t max_untagged; msize_t max_untagged;
msize_t max_tagged; msize_t max_tagged;
} msizes_t; } msizes_t;
ddp_send(socket_t s, message_t m) ddp_send(socket_t s, message_t m)
send an untagged message. send an untagged message.
ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n) ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n)
send a tagged message to remote buffer address d. send a tagged message to remote buffer address d.
ddp_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single received untagged
message. Each buffer is returned to the caller in a
ddp_recv() untagged message reception indication, in the order
in which it was posted. The same buffer may be enabled on
multiple sockets, receipt of an untagged message into the
buffer from any of these sockets unposts the buffer from all
sockets.
ddp_recv(socket_t s) ddp_recv(socket_t s)
get the next received untagged message, tagged message get the next received untagged message, tagged message
reception indication, or tagged message error. reception indication, or tagged message error.
ddp_register(socket_t s, ddp_buffer_t b) ddp_register(socket_t s, ddp_buffer_t b)
register a buffer for DDP on a socket. The same buffer may be register a buffer for DDP on a socket. The same buffer may be
registered multiple times on the same or different sockets. registered multiple times on the same or different sockets.
The same buffer registered on different sockets may result in The same buffer registered on different sockets may result in
skipping to change at page 10, line 44 skipping to change at page 11, line 17
protocol. As mentioned above, the basic DDP model assumes that protocol. As mentioned above, the basic DDP model assumes that
buffer address values returned by ddp_register() are opaque to the buffer address values returned by ddp_register() are opaque to the
client protocol, and can be implementation dependent. The most client protocol, and can be implementation dependent. The most
natural way to map DDP to a multidestination transport is to natural way to map DDP to a multidestination transport is to
require all receivers produce the same buffer address when require all receivers produce the same buffer address when
registering a multidestination destination buffer. Restriction of registering a multidestination destination buffer. Restriction of
the DDP model to accommodate multiple destinations involves the DDP model to accommodate multiple destinations involves
engineering tradeoffs comparable to those of providing non-DDP engineering tradeoffs comparable to those of providing non-DDP
multidestination transport capability. multidestination transport capability.
The same buffer may be enabled by ddp_post_recv() on multiple
sockets. In this case the ddp_recv() untagged message reception
indication may be provided on a different socket from that on which
the buffer was posted. Such indications are not ordered among
multiple DDP sockets.
When multiple sockets reference an untagged message reception
buffer, local interfaces are responsible for managing the
mechanisms of allocating posted buffers to received untagged
messages, the handling of received untagged messages when no buffer
is available, and of resource management among multiple sockets.
Where underprovisioning of buffers on multiple sockets is allowed,
mechanisms should be provided to manage buffer consumption on a
per-socket or group of related sockets basis.
2.2. Remote Direct Memory Access (RDMA) Protocol Architecture 2.2. Remote Direct Memory Access (RDMA) Protocol Architecture
Remote Direct Memory Access (RDMA) extends the capabilities of DDP Remote Direct Memory Access (RDMA) extends the capabilities of DDP
with the ability to read from buffers registered to a socket (RDMA with the ability to read from buffers registered to a socket (RDMA
Read). This allows a client protocol to perform arbitrary, Read). This allows a client protocol to perform arbitrary,
bidirectional data movement without involving the remote client. bidirectional data movement without involving the remote client.
When RDMA is implemented in hardware, arbitrary data movement can When RDMA is implemented in hardware, arbitrary data movement can
be performed without involving the remote host CPU at all. be performed without involving the remote host CPU at all.
In addition, RDMA protocols usually specify a transport-independent In addition, RDMA protocols usually specify a transport-independent
skipping to change at page 12, line 13 skipping to change at page 13, line 9
} rdma_buffer_t; } rdma_buffer_t;
2.2.1. RDMA Operations 2.2.1. RDMA Operations
The RDMA layer provides: The RDMA layer provides:
void rdma_send(socket_t s, message_t m); void rdma_send(socket_t s, message_t m);
void rdma_write(socket_t s, message_t m, ddp_addr_t d, void rdma_write(socket_t s, message_t m, ddp_addr_t d,
rdma_notify_t n); rdma_notify_t n);
void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d); void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d);
void rdma_post_recv(socket_t s, bdesc_t b);
rdma_ind_t rdma_recv(socket_t s); rdma_ind_t rdma_recv(socket_t s);
bdesc_t rdma_register(socket_t s, rdma_buffer_t b, bdesc_t rdma_register(socket_t s, rdma_buffer_t b,
bmode_t mode); bmode_t mode);
void rdma_deregister(bhand_t bh); void rdma_deregister(bhand_t bh);
msizes_t rdma_max_msizes(socket_t s); msizes_t rdma_max_msizes(socket_t s);
Although, for clarity, these data transfer interfaces are Although, for clarity, these data transfer interfaces are
synchronous, rdma_read() and possibly rdma_send() (in the presence synchronous, rdma_read() and possibly rdma_send() (in the presence
of Send flow control), can require an arbitrary amount of time to of Send flow control), can require an arbitrary amount of time to
complete. To express the full concurrency and interleaving of RDMA complete. To express the full concurrency and interleaving of RDMA
skipping to change at page 13, line 6 skipping to change at page 13, line 47
rdma_write_id_t (scalar) rdma_write_id_t (scalar)
an RDMA Write identifier. an RDMA Write identifier.
rdma_ind_t rdma_ind_t
a Send message, or an RDMA error: a Send message, or an RDMA error:
typedef union { typedef union {
message_t m; recv_message_t m;
rdma_err_t e; rdma_err_t e;
} rdma_ind_t; } rdma_ind_t;
rdma_err_t rdma_err_t
an RDMA protocol error indication. RDMA errors include buffer an RDMA protocol error indication. RDMA errors include buffer
addressing errors corresponding to ddp_err_ts, and buffer addressing errors corresponding to ddp_err_ts, and buffer
protection violations (e.g. RDMA Writing a buffer only protection violations (e.g. RDMA Writing a buffer only
registered for reading). registered for reading).
bmode_t bmode_t
buffer registration mode (permissions). Any combination of buffer registration mode (permissions). Any combination of
skipping to change at page 13, line 37 skipping to change at page 14, line 31
rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n) rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n)
RDMA Write to remote buffer address d. RDMA Write to remote buffer address d.
rdma_read(socket_t s, ddp_addr_t s, length l, ddp_addr_t d) rdma_read(socket_t s, ddp_addr_t s, length l, ddp_addr_t d)
RDMA Read l octets from remote buffer address s to local RDMA Read l octets from remote buffer address s to local
buffer address d. buffer address d.
rdma_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single Send message, to
be filled and returned in-order to a subsequent caller of
rdma_recv(). As with DDP, buffers may be enabled on multiple
sockets, in which case ordering guarantees are relaxed. Also
as with DDP, local interfaces must manage the mechanisms of
allocation and management of buffers posted to multiple
sockets.
rdma_recv(socket_t s); rdma_recv(socket_t s);
get the next received Send message, RDMA Write completion get the next received Send message, RDMA Write completion
identifier, or RDMA error. identifier, or RDMA error.
rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode) rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode)
register a buffer for RDMA on a socket (for read access, write register a buffer for RDMA on a socket (for read access, write
access or both). As with DDP, the same buffer may be access or both). As with DDP, the same buffer may be
registered multiple times on the same or different sockets, registered multiple times on the same or different sockets,
skipping to change at page 15, line 31 skipping to change at page 16, line 36
DDP. DDP.
4. IANA Considerations 4. IANA Considerations
IANA considerations are not addressed in by this document. Any IANA considerations are not addressed in by this document. Any
IANA considerations resulting from the use of DDP or RDMA must be IANA considerations resulting from the use of DDP or RDMA must be
addressed in the relevant standards. addressed in the relevant standards.
5. Acknowledgements 5. Acknowledgements
The authors wish to acknowledge the valuable contributions of David The authors wish to acknowledge the valuable contributions of
Black, Jeff Mogul and Allyn Romanow. Caitlin Bestler, David Black, Jeff Mogul and Allyn Romanow.
6. Informative References 6. Informative References
[DAFS] [DAFS]
DAFS Collaborative, "Direct Access File System Specification DAFS Collaborative, "Direct Access File System Specification
v1.0", September 2001, available from v1.0", September 2001, available from
http://www.dafscollaborative.org http://www.dafscollaborative.org
[FCVI] [FCVI]
ANSI Technical Committee T11, "Fibre Channel Standard Virtual ANSI Technical Committee T11, "Fibre Channel Standard Virtual
skipping to change at page 16, line 7 skipping to change at page 17, line 11
[IB] InfiniBand Trade Association, "InfiniBand Architecture [IB] InfiniBand Trade Association, "InfiniBand Architecture
Specification Volumes 1 and 2", Release 1.1, November 2002, Specification Volumes 1 and 2", Release 1.1, November 2002,
available from http://www.infinibandta.org/specs available from http://www.infinibandta.org/specs
[MYR] [MYR]
VMEbus International Trade Association, "Myrinet on VME VMEbus International Trade Association, "Myrinet on VME
Protocol Specification", ANSI/VITA 26-1998, August 1998, Protocol Specification", ANSI/VITA 26-1998, August 1998,
available from http://www.myri.com/open-specs available from http://www.myri.com/open-specs
RFC Editor note:
Replace following problem statement draft-ietf- name, status and
date with appropriate reference when assigned.
[ROM] [ROM]
A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP
Problem Statement", draft-ietf-rddp-problem-statement-02, Work Problem Statement", draft-ietf-rddp-problem-statement-02, Work
in Progress, June 2003 in Progress, June 2003
RFC Editor note: Replace problem statement draft-ietf- name, status and
date with appropriate reference when assigned.
[SCTP] [SCTP]
R. Stewart et al., "Stream Transmission Control Protocol", RFC R. Stewart et al., "Stream Transmission Control Protocol", RFC
2960, Standards Track 2960, Standards Track
[SDP] [SDP]
InfiniBand Trade Association, "Sockets Direct Protocol v1.0", InfiniBand Trade Association, "Sockets Direct Protocol v1.0",
Annex A of InfiniBand Architecture Specification Volume 1, Annex A of InfiniBand Architecture Specification Volume 1,
Release 1.1, November 2002, available from Release 1.1, November 2002, available from
http://www.infinibandta.org/specs http://www.infinibandta.org/specs
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/