draft-ietf-rddp-arch-05.txt   draft-ietf-rddp-arch-06.txt 
Internet-Draft Stephen Bailey (Sandburst) Internet-Draft Stephen Bailey (Sandburst)
Expires: January 2005 Tom Talpey (NetApp) Expires: April 2005 Tom Talpey (NetApp)
The Architecture of Direct Data Placement (DDP) The Architecture of Direct Data Placement (DDP)
and Remote Direct Memory Access (RDMA) and Remote Direct Memory Access (RDMA)
on Internet Protocols on Internet Protocols
draft-ietf-rddp-arch-05 draft-ietf-rddp-arch-06
Status of this Memo Status of this Memo
By submitting this Internet-Draft, I certify that any applicable By submitting this Internet-Draft, I certify that any applicable
patent or other IPR claims of which I am aware have been disclosed, patent or other IPR claims of which I am aware have been disclosed,
or will be disclosed, and any of which I become aware will be or will be disclosed, and any of which I become aware will be
disclosed, in accordance with RFC 3668. disclosed, in accordance with RFC 3668.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in as reference material or to cite them other than as "work in
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt The list of http://www.ietf.org/ietf/1id-abstracts.txt
Internet-Draft Shadow Directories can be accessed at
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html http://www.ietf.org/shadow.html
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2004). All Rights Reserved. Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract Abstract
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
skipping to change at page 2, line 19 skipping to change at page 2, line 19
1.2. DDP and RDMA Protocols . . . . . . . . . . . . . . . . . 3 1.2. DDP and RDMA Protocols . . . . . . . . . . . . . . . . . 3
2. Architecture . . . . . . . . . . . . . . . . . . . . . . 4 2. Architecture . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Direct Data Placement (DDP) Protocol Architecture . . . 4 2.1. Direct Data Placement (DDP) Protocol Architecture . . . 4
2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 6 2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 6
2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 7 2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 7
2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10 2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10
2.2. Remote Direct Memory Access Protocol Architecture . . . 12 2.2. Remote Direct Memory Access Protocol Architecture . . . 12
2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 13 2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 13
2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 16 2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 16
3. Security Considerations . . . . . . . . . . . . . . . . 17 3. Security Considerations . . . . . . . . . . . . . . . . 17
4. IANA Considerations . . . . . . . . . . . . . . . . . . 18 4. IANA Considerations . . . . . . . . . . . . . . . . . . 19
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 18 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 19
Informative References . . . . . . . . . . . . . . . . . 18 Informative References . . . . . . . . . . . . . . . . . 19
Authors' Addresses . . . . . . . . . . . . . . . . . . . 19 Authors' Addresses . . . . . . . . . . . . . . . . . . . 20
Full Copyright Statement . . . . . . . . . . . . . . . . 19 Full Copyright Statement . . . . . . . . . . . . . . . . 20
1. Introduction 1. Introduction
This document defines an abstract architecture for Direct Data This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols, not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding but is, rather, a descriptive tool for defining and understanding
the protocols. This document uses C language notation as a the protocols. This document uses C language notation as a
shorthand to describe the architectural elements of DDP and RDMA shorthand to describe the architectural elements of DDP and RDMA
skipping to change at page 12, line 6 skipping to change at page 12, line 6
protocol. As mentioned above, the basic DDP model assumes that protocol. As mentioned above, the basic DDP model assumes that
buffer address values returned by ddp_register() are opaque to the buffer address values returned by ddp_register() are opaque to the
client protocol, and can be implementation dependent. The most client protocol, and can be implementation dependent. The most
natural way to map DDP to a multidestination transport is to natural way to map DDP to a multidestination transport is to
require all receivers produce the same buffer address when require all receivers produce the same buffer address when
registering a multidestination destination buffer. Restriction of registering a multidestination destination buffer. Restriction of
the DDP model to accommodate multiple destinations involves the DDP model to accommodate multiple destinations involves
engineering tradeoffs comparable to those of providing non-DDP engineering tradeoffs comparable to those of providing non-DDP
multidestination transport capability. multidestination transport capability.
A registered buffer is identified within DDP by its stag_t, which
in turn is associated with a socket. This registration therefore
grants a capability to the DDP peer, and the socket (using the
underlying properties of its chosen transport and possible
security) identifies the peer and authenticates the stag_t.
The same buffer may be enabled by ddp_post_recv() on multiple The same buffer may be enabled by ddp_post_recv() on multiple
sockets. In this case the ddp_recv() untagged message reception sockets. In this case any ddp_recv() untagged message reception
indication may be provided on a different socket from that on which indication may be provided on a different socket from that on which
the buffer was posted. Such indications are not ordered among the buffer was posted. Such indications are not ordered among
multiple DDP sockets. multiple DDP sockets.
When multiple sockets reference an untagged message reception When multiple sockets reference an untagged message reception
buffer, local interfaces are responsible for managing the buffer, local interfaces are responsible for managing the
mechanisms of allocating posted buffers to received untagged mechanisms of allocating posted buffers to received untagged
messages, the handling of received untagged messages when no buffer messages, the handling of received untagged messages when no buffer
is available, and of resource management among multiple sockets. is available, and of resource management among multiple sockets.
Where underprovisioning of buffers on multiple sockets is allowed, Where underprovisioning of buffers on multiple sockets is allowed,
skipping to change at page 17, line 33 skipping to change at page 17, line 33
impairing system integrity. For example, threats can include impairing system integrity. For example, threats can include
potential buffer reuse or buffer overflow, and are not merely a potential buffer reuse or buffer overflow, and are not merely a
security issue. Even trusted peers must not be allowed to damage security issue. Even trusted peers must not be allowed to damage
local integrity. Any DDP and RDMA protocol must address the issue local integrity. Any DDP and RDMA protocol must address the issue
of giving end-systems and applications the capabilities to offer of giving end-systems and applications the capabilities to offer
protection from such compromises. protection from such compromises.
Because a Steering Tag exports access to a memory region, one Because a Steering Tag exports access to a memory region, one
critical aspect of security is the scope of this access. It must critical aspect of security is the scope of this access. It must
be possible to individually control specific attributes of the be possible to individually control specific attributes of the
access provided by a Steering Tag, including remote read access, access provided by a Steering Tag on the endpoint (socket) on which
remote write access, and others that might be identified. DDP and it was registered, including remote read access, remote write
RDMA specifications must provide both implementation requirements access, and others that might be identified. DDP and RDMA
specifications must provide both implementation requirements
relevant to this issue, and guidelines to assist implementors in relevant to this issue, and guidelines to assist implementors in
making the appropriate design decisions. making the appropriate design decisions.
For example, it must not be possible for DDP to enable evasion of
memory consistency checks at the recipient. The DDP and RDMA
specifications must allow the recipient to rely on its consistent
memory contents by explicitly controlling peer access to memory
regions at appropriate times.
Peer connections which do not pass authentication and authorization
checks by upper layers must not be permitted to begin processing in
RDMA mode with an inappropriate endpoint. Once associated, peer
accesses to memory regions must be authenticated and made subject
to authorization checks in the context of the association and
endpoint (socket) on which they are to be performed, prior to any
transfer operation or data being accessed. The RDMA protocols must
ensure that these region protections be under strict application
control.
The use of DDP and RDMA on a transport connection may interact with The use of DDP and RDMA on a transport connection may interact with
any security mechanism, and vice-versa. For example, if the any security mechanism, and vice-versa. For example, if the
security mechanism is implemented above the transport layer, the security mechanism is implemented above the transport layer, the
DDP and RDMA headers may not be protected. Such a layering may DDP and RDMA headers may not be protected. Such a layering may
therefore be inappropriate, depending on requirements. Or, when therefore be inappropriate, depending on requirements.
TLS is employed, it may not be possible for DDP and RDMA to process
segments out of order, due to the in-order requirement of TLS. IPsec, operating to secure the connection on a packet-by-packet
These interactions should be well explored. basis, seems to be a natural fit to securing RDMA placement, which
operates in conjunction with transport. Because RDMA enables an
implementation to avoid buffering, it is preferable to perform all
applicable security protection prior to processing of each segment
by the transport and RDMA layers. Such a layering enables the most
efficient secure RDMA implementation.
The TLS record protocol, on the other hand, is layered on top of
reliable transports and cannot provide such security assurance
until an entire record is available, which may require the
buffering and/or assembly of several distinct messages prior to TLS
processing. This defers RDMA processing and introduces overheads
that RDMA is designed to avoid. TLS therefore is viewed as
potentially a less natural fit for protecting the RDMA protocols.
Resource issues leading to denial-of-service attacks, overwrites Resource issues leading to denial-of-service attacks, overwrites
and other concurrent operations, the ordering of completions as and other concurrent operations, the ordering of completions as
required by the RDMA protocol, and the granularity of transfer are required by the RDMA protocol, and the granularity of transfer are
all within the required scope of any security analysis of RDMA and all within the required scope of any security analysis of RDMA and
DDP. DDP.
The RDMA operations require checking of what is essentially user
information, explicitly including addressing information and
operation type (read or write), and implicitly including protection
and attributes. The semantics associated with each class of error
resulting from possible failure of such checks must be clearly
defined, and the expected action to be taken by the protocols in
each case must be specified.
In some cases, this will result in a catastrophic error on the RDMA
association, however in others, a local or remote error may be
signalled. Certain of these errors may require consideration of
abstract local semantics. The result of the error on the RDMA
association must be carefully specified so as to provide useful
behavior, while not constraining the implementation.
4. IANA Considerations 4. IANA Considerations
IANA considerations are not addressed in by this document. Any IANA considerations are not addressed in by this document. Any
IANA considerations resulting from the use of DDP or RDMA must be IANA considerations resulting from the use of DDP or RDMA must be
addressed in the relevant standards. addressed in the relevant standards.
5. Acknowledgements 5. Acknowledgements
The authors wish to acknowledge the valuable contributions of The authors wish to acknowledge the valuable contributions of
Caitlin Bestler, David Black, Jeff Mogul and Allyn Romanow. Caitlin Bestler, David Black, Jeff Mogul and Allyn Romanow.
skipping to change at page 18, line 40 skipping to change at page 19, line 39
Specification Volumes 1 and 2", Release 1.1, November 2002, Specification Volumes 1 and 2", Release 1.1, November 2002,
available from http://www.infinibandta.org/specs available from http://www.infinibandta.org/specs
[MYR] [MYR]
VMEbus International Trade Association, "Myrinet on VME VMEbus International Trade Association, "Myrinet on VME
Protocol Specification", ANSI/VITA 26-1998, August 1998, Protocol Specification", ANSI/VITA 26-1998, August 1998,
available from http://www.myri.com/open-specs available from http://www.myri.com/open-specs
[ROM] [ROM]
A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP
Problem Statement", draft-ietf-rddp-problem-statement-04, Work Problem Statement", draft-ietf-rddp-problem-statement-05, Work
in Progress, July 2004 in Progress, October 2004
[SCTP] [SCTP]
R. Stewart et al., "Stream Transmission Control Protocol", RFC R. Stewart et al., "Stream Transmission Control Protocol", RFC
2960, Standards Track 2960, Standards Track
[SDP] [SDP]
InfiniBand Trade Association, "Sockets Direct Protocol v1.0", InfiniBand Trade Association, "Sockets Direct Protocol v1.0",
Annex A of InfiniBand Architecture Specification Volume 1, Annex A of InfiniBand Architecture Specification Volume 1,
Release 1.1, November 2002, available from Release 1.1, November 2002, available from
http://www.infinibandta.org/specs http://www.infinibandta.org/specs
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/