draft-ietf-rddp-security-10.txt   rfc5042.txt 
Internet Draft James Pinkerton Network Working Group J. Pinkerton
draft-ietf-rddp-security-10.txt Microsoft Corporation Request for Comments: 5042 Microsoft Corporation
Category: Standards Track Ellen Deleganes Category: Standards Track E. Deleganes
Expires: December, 2006 Intel Corporation Self
DDP/RDMAP Security October 2007
Status of this Memo
By submitting this Internet-Draft, each author represents that
any applicable patent or other IPR claims of which he or she is
aware have been or will be disclosed, and any of which he or she
becomes aware will be disclosed, in accordance with Section 6 of
BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six Direct Data Placement Protocol (DDP) /
months and may be updated, replaced, or obsoleted by other Remote Direct Memory Access Protocol (RDMAP) Security
documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in
progress."
The list of current Internet-Drafts can be accessed at Status of This Memo
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at This document specifies an Internet standards track protocol for the
http://www.ietf.org/shadow.html. Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Abstract Abstract
This document analyzes security issues around implementation and
use of the Direct Data Placement Protocol(DDP) and Remote Direct
Memory Access Protocol (RDMAP). It first defines an architectural
model for an RDMA Network Interface Card (RNIC), which can
implement DDP or RDMAP and DDP. The document reviews various
attacks against the resources defined in the architectural model
and the countermeasures that can be used to protect the system.
Attacks are grouped into those that can be mitigated by using
secure communication channels across the network, attacks from
Remote Peers, and attacks from Local Peers. Attack categories
include spoofing, tampering, information disclosure, denial of
service, and elevation of privilege.
J. Pinkerton, et al. Expires December, 2006 1
Table of Contents
1 Introduction.................................................4 This document analyzes security issues around implementation and use
2 Architectural Model..........................................7 of the Direct Data Placement Protocol (DDP) and Remote Direct Memory
2.1 Components...................................................8 Access Protocol (RDMAP). It first defines an architectural model for
2.2 Resources...................................................10 an RDMA Network Interface Card (RNIC), which can implement DDP or
2.2.1 Stream Context Memory.....................................10 RDMAP and DDP. The document reviews various attacks against the
2.2.2 Data Buffers..............................................10 resources defined in the architectural model and the countermeasures
2.2.3 Page Translation Tables...................................11 that can be used to protect the system. Attacks are grouped into
2.2.4 Protection Domain (PD)....................................11 those that can be mitigated by using secure communication channels
2.2.5 STag Namespace and Scope..................................12 across the network, attacks from Remote Peers, and attacks from Local
2.2.6 Completion Queues.........................................13 Peers. Attack categories include spoofing, tampering, information
2.2.7 Asynchronous Event Queue..................................13 disclosure, denial of service, and elevation of privilege.
2.2.8 RDMA Read Request Queue...................................13
2.3 RNIC Interactions...........................................14
2.3.1 Privileged Control Interface Semantics....................14
2.3.2 Non-Privileged Data Interface Semantics...................14
2.3.3 Privileged Data Interface Semantics.......................15
2.3.4 Initialization of RNIC Data Structures for Data Transfer..15
2.3.5 RNIC Data Transfer Interactions...........................16
3 Trust and Resource Sharing..................................18
4 Attacker Capabilities.......................................19
5 Attacks That Can be Mitigated With End-to-End Security......20
5.1 Spoofing....................................................20
5.1.1 Impersonation.............................................20
5.1.2 Stream Hijacking..........................................21
5.1.3 Man-in-the-Middle Attack..................................21
5.2 Tampering - Network based modification of buffer content....22
5.3 Information Disclosure - Network Based Eavesdropping........22
5.4 Specific Requirements for Security Services.................22
5.4.1 Introduction to Security Options..........................23
5.4.2 TLS is Inappropriate for DDP/RDMAP Security...............23
5.4.3 DTLS and RDDP.............................................24
5.4.4 ULPs Which Provide Security...............................24
5.4.5 Requirements for IPsec Encapsulation of DDP...............25
6 Attacks from Remote Peers...................................26
6.1 Spoofing....................................................26
6.1.1 Using an STag on a Different Stream.......................26
6.2 Tampering...................................................27
6.2.1 Buffer Overrun - RDMA Write or Read Response..............28
6.2.2 Modifying a Buffer After Indication.......................28
6.2.3 Multiple STags to access the same buffer..................29
6.3 Information Disclosure......................................29
6.3.1 Probing memory outside of the buffer bounds...............29
6.3.2 Using RDMA Read to Access Stale Data......................29
6.3.3 Accessing a Buffer After the Transfer.....................30
6.3.4 Accessing Unintended Data With a Valid STag...............30
6.3.5 RDMA Read into an RDMA Write Buffer.......................30
6.3.6 Using Multiple STags Which Alias to the Same Buffer.......31
6.4 Denial of Service (DOS).....................................31
6.4.1 RNIC Resource Consumption.................................32
6.4.2 Resource Consumption by Idle ULPs.........................32
6.4.3 Resource Consumption By Active ULPs.......................33
6.4.3.1 Multiple Streams Sharing Receive Buffers...............33
6.4.3.2 Remote or Local Peer Attacking a Shared CQ.............35
6.4.3.3 Attacking the RDMA Read Request Queue..................37
6.4.4 Exercise of non-optimal code paths........................38
6.4.5 Remote Invalidate an STag Shared on Multiple Streams......38
6.4.6 Remote Peer attacking an Unshared CQ......................39
6.5 Elevation of Privilege......................................39
7 Attacks from Local Peers....................................40
7.1 Local ULP Attacking a Shared CQ.............................40
7.2 Local Peer Attacking the RDMA Read Request Queue............40
7.3 Local ULP Attacking the PTT & STag Mapping..................40
8 Security considerations.....................................42
9 IANA Considerations.........................................43
10 References..................................................44
10.1 Normative References......................................44
10.2 Informative References....................................44
11 Appendix A: ULP Issues for RDDP Client/Server Protocols.....46
12 Appendix B: Summary of RNIC and ULP Implementation
Requirements.....................................................50
13 Appendix C: Partial Trust Taxonomy..........................52
14 Author's Addresses..........................................54
15 Acknowledgments.............................................55
16 Full Copyright Statement....................................57
Table of Figures Table of Contents
Figure 1 - RDMA Security Model....................................8 1. Introduction ....................................................4
2. Architectural Model .............................................6
2.1. Components .................................................7
2.2. Resources ..................................................9
2.2.1. Stream Context Memory ...............................9
2.2.2. Data Buffers .......................................10
2.2.3. Page Translation Tables ............................10
2.2.4. Protection Domain (PD) .............................11
2.2.5. STag Namespace and Scope ...........................11
2.2.6. Completion Queues ..................................12
2.2.7. Asynchronous Event Queue ...........................12
2.2.8. RDMA Read Request Queue ............................13
2.3. RNIC Interactions .........................................13
2.3.1. Privileged Control Interface Semantics .............13
2.3.2. Non-Privileged Data Interface Semantics ............13
2.3.3. Privileged Data Interface Semantics ................14
2.3.4. Initialization of RNIC Data Structures for
Data Transfer ......................................14
2.3.5. RNIC Data Transfer Interactions ....................16
3. Trust and Resource Sharing .....................................17
4. Attacker Capabilities ..........................................18
5. Attacks That Can Be Mitigated with End-to-End Security .........18
5.1. Spoofing ..................................................19
5.1.1. Impersonation ......................................19
5.1.2. Stream Hijacking ...................................20
5.1.3. Man-in-the-Middle Attack ...........................20
5.2. Tampering - Network-Based Modification of Buffer Content ..21
5.3. Information Disclosure - Network-Based Eavesdropping ......21
5.4. Specific Requirements for Security Services ...............21
5.4.1. Introduction to Security Options ...................21
5.4.2. TLS Is Inappropriate for DDP/RDMAP Security ........22
5.4.3. DTLS and RDDP ......................................23
5.4.4. ULPs That Provide Security .........................23
5.4.5. Requirements for IPsec Encapsulation of DDP ........23
6. Attacks from Remote Peers ......................................24
6.1. Spoofing ..................................................25
6.1.1. Using an STag on a Different Stream ................25
6.2. Tampering .................................................26
6.2.1. Buffer Overrun - RDMA Write or Read Response .......26
6.2.2. Modifying a Buffer after Indication ................27
6.2.3. Multiple STags to Access the Same Buffer ...........27
6.3. Information Disclosure ....................................28
6.3.1. Probing Memory Outside of the Buffer Bounds ........28
6.3.2. Using RDMA Read to Access Stale Data ...............28
6.3.3. Accessing a Buffer after the Transfer ..............28
6.3.4. Accessing Unintended Data with a Valid STag ........29
6.3.5. RDMA Read into an RDMA Write Buffer ................29
6.3.6. Using Multiple STags That Alias to the Same
Buffer .............................................29
6.4. Denial of Service (DOS) ...................................30
6.4.1. RNIC Resource Consumption ..........................30
6.4.2. Resource Consumption by Idle ULPs ..................31
6.4.3. Resource Consumption by Active ULPs ................32
6.4.3.1. Multiple Streams Sharing Receive Buffers ..32
6.4.3.2. Remote or Local Peer Attacking a
Shared CQ .................................34
6.4.3.3. Attacking the RDMA Read Request Queue .....36
6.4.4. Exercise of Non-Optimal Code Paths .................37
6.4.5. Remote Invalidate an STag Shared on
Multiple Streams ...................................37
6.4.6. Remote Peer Attacking an Unshared CQ ...............38
6.5. Elevation of Privilege ....................................38
7. Attacks from Local Peers .......................................38
7.1. Local ULP Attacking a Shared CQ ...........................39
7.2. Local Peer Attacking the RDMA Read Request Queue ..........39
7.3. Local ULP Attacking the PTT and STag Mapping ..............39
8. Security considerations ........................................40
9. IANA Considerations ............................................40
10. References ....................................................40
10.1. Normative References .....................................40
10.2. Informative References ...................................41
Appendix A. ULP Issues for RDDP Client/Server Protocols ...........43
Appendix B. Summary of RNIC and ULP Implementation Requirements ...46
Appendix C. Partial Trust Taxonomy ................................47
Acknowledgments ...................................................49
1 Introduction 1. Introduction
RDMA enables new levels of flexibility when communicating between RDMA enables new levels of flexibility when communicating between two
two parties compared to current conventional networking practice parties compared to current conventional networking practice (e.g., a
(e.g. a stream-based model or datagram model). This flexibility stream-based model or datagram model). This flexibility brings new
brings new security issues that must be carefully understood when security issues that must be carefully understood when designing
designing Upper Layer Protocols (ULPs) utilizing RDMA and when Upper Layer Protocols (ULPs) utilizing RDMA and when implementing
implementing RDMA-aware NICs (RNICs). Note that for the purposes RDMA-aware NICs (RNICs). Note that for the purposes of this security
of this security analysis, an RNIC may implement RDMAP [RDMAP] analysis, an RNIC may implement RDMAP [RDMAP] and DDP [DDP], or just
and DDP [DDP], or just DDP. Also, a ULP may be an application or DDP. Also, a ULP may be an application or it may be a middleware
it may be a middleware library. library.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
"OPTIONAL" in this document are to be interpreted as described in document are to be interpreted as described in RFC 2119.
RFC 2119. Additionally the security terminology defined in Additionally, the security terminology defined in [RFC4949] is used
[RFC2828] is used in this specification. in this specification.
The document first develops an architectural model that is The document first develops an architectural model that is relevant
relevant for the security analysis - it details components, for the security analysis. Section 2 details components, resources,
resources, and system properties that may be attacked in Section and system properties that may be attacked. The document uses Local
2. The document uses Local Peer to represent the RDMA/DDP Peer to represent the RDMA/DDP protocol implementation on the local
protocol implementation on the local end of a Stream (implemented end of a Stream (implemented with a transport protocol, such as
with a transport protocol such as [RFC793] or [RFC2960]). The [RFC793] or [RFC4960]). The local Upper-Layer-Protocol (ULP) is used
local Upper-Layer-Protocol (ULP) is used to represent the to represent the application or middle-ware layer above the Local
application or middle-ware layer above the Local Peer. The Peer. The document does not attempt to differentiate between a
document does not attempt to differentiate between a Remote Peer Remote Peer and a Remote ULP (an RDMA/DDP protocol implementation on
and a Remote ULP (an RDMA/DDP protocol implementation on the the remote end of a Stream versus the application on the remote end)
remote end of a Stream versus the application on the remote end) for several reasons: often, the source of the attack is difficult to
for several reasons: often the source of the attack is difficult know for sure and, regardless of the source, the mitigations required
to know for sure; and regardless of the source, the mitigations of the Local Peer or local ULP are the same. Thus, the document
required of the Local Peer or local ULP are the same. Thus the generically refers to a Remote Peer rather than trying to further
document generically refers to a Remote Peer rather than trying delineate the attacker.
to further delineate the attacker.
The document then defines what resources a local ULP may share The document then defines what resources a local ULP may share across
across Streams and what resources the local ULP may share with Streams and what resources the local ULP may share with the Remote
the Remote Peer across Streams in Section 3. Peer across Streams in Section 3.
Intentional sharing of resources between multiple Streams may Intentional sharing of resources between multiple Streams may imply
imply some level of trust between the Streams. However, some some level of trust between the Streams. However, some types of
types of resource sharing have unmitigated security attacks which resource sharing have unmitigated security attacks, which would
would mandate not sharing a specific type of resource unless mandate not sharing a specific type of resource unless there is some
there is some level of trust between the Streams sharing level of trust between the Streams sharing resources.
resources.
This document defines a new term, "Partial Mutual Trust" to This document defines a new term, "Partial Mutual Trust", to address
address this concept: this concept:
Partial Mutual Trust - a collection of RDMAP/DDP Streams, Partial Mutual Trust - a collection of RDMAP/DDP Streams, which
which represent the local and remote end points of the represent the local and remote end points of the Stream that are
Stream, which are willing to assume that the Streams from willing to assume that the Streams from the collection will not
the collection will not perform malicious attacks against perform malicious attacks against any of the other Streams in the
any of the other Streams in the collection. collection.
ULPs have explicit control of which collection of endpoints is in ULPs have explicit control of which collection of endpoints is in a
a Partial Mutual Trust collection through tools discussed in Partial Mutual Trust collection through tools discussed in Appendix
Section 13 Appendix C: Partial Trust Taxonomy. C, Partial Trust Taxonomy.
An untrusted peer relationship is appropriate when a ULP wishes An untrusted peer relationship is appropriate when a ULP wishes to
to ensure that it will be robust and uncompromised even in the ensure that it will be robust and uncompromised even in the face of a
face of a deliberate attack by its peer. For example, a single deliberate attack by its peer. For example, a single ULP that
ULP that concurrently supports multiple unrelated Streams (e.g. a concurrently supports multiple unrelated Streams (e.g., a server)
server) would presumably treat each of its peers as an untrusted would presumably treat each of its peers as an untrusted peer. For a
peer. For a collection of Streams which share Partial Mutual collection of Streams that share Partial Mutual Trust, the assumption
Trust, the assumption is that any Stream not in the collection is is that any Stream not in the collection is untrusted. For the
untrusted. For the untrusted peer, a brief list of capabilities untrusted peer, a brief list of capabilities is enumerated in Section
is enumerated in Section 4. 4.
The rest of the document is focused on analyzing attacks and The rest of the document is focused on analyzing attacks and
recommending specific mitigations to the attacks. Attacks are recommending specific mitigations to the attacks. Attacks are
categorized into attacks mitigated by end-to-end security, categorized into attacks mitigated by end-to-end security, attacks
attacks initiated by Remote Peers, and attacks initiated by Local initiated by Remote Peers, and attacks initiated by Local Peers. For
Peers. For each attack, possible countermeasures are reviewed. each attack, possible countermeasures are reviewed.
ULPs within a host are divided into two categories - Privileged ULPs within a host are divided into two categories - Privileged and
and Non-Privileged. Both ULP types can send and receive data and Non-Privileged. Both ULP types can send and receive data and request
request resources. The key differences between the two are: resources. The key differences between the two are:
The Privileged ULP is trusted by the local system to not The Privileged ULP is trusted by the local system not to
maliciously attack the operating environment, but it is not maliciously attack the operating environment, but it is not
trusted to optimize resource allocation globally. For trusted to optimize resource allocation globally. For example,
example, the Privileged ULP could be a kernel ULP, thus the the Privileged ULP could be a kernel ULP; thus, the kernel
kernel presumably has in some way vetted the ULP before presumably has in some way vetted the ULP before allowing it to
allowing it to execute. execute.
A Non-Privileged ULP's capabilities are a logical sub-set of A Non-Privileged ULP's capabilities are a logical sub-set of the
the Privileged ULP's. It is assumed by the local system that Privileged ULP's. It is assumed by the local system that a Non-
a Non-Privileged ULP is untrusted. All Non-Privileged ULP Privileged ULP is untrusted. All Non-Privileged ULP interactions
interactions with the RNIC Engine that could affect other with the RNIC Engine that could affect other ULPs need to be done
ULPs need to be done through a trusted intermediary that can through a trusted intermediary that can verify the Non-Privileged
verify the Non-Privileged ULP requests. ULP requests.
The appendices provide focused summaries of this specification. The appendices provide focused summaries of this specification.
Section 11 Appendix A: ULP Issues for RDDP Client/Server Appendix A, ULP Issues for RDDP Client/Server Protocols, focuses on
Protocols focuses on implementers of traditional client/server implementers of traditional client/server protocols. Appendix B,
protocols. Section 12 Appendix B: Summary of RNIC and ULP Summary of RNIC and ULP Implementation Requirements, summarizes all
Implementation Requirements summarizes all normative requirements normative requirements in this specification. Appendix C, Partial
in this specification. Section 13 Appendix C: Partial Trust Trust Taxonomy, provides an abstract model for categorizing trust
Taxonomy provides an abstract model for categorizing trust
boundaries. boundaries.
If an RDMAP/DDP protocol implementation uses the mitigations If an RDMAP/DDP protocol implementation uses the mitigations
recommended in this document, that implementation should not recommended in this document, that implementation should not exhibit
exhibit additional security vulnerabilities above and beyond additional security vulnerabilities above and beyond those of an
those of an implementation of the transport protocol (i.e., TCP implementation of the transport protocol (i.e., TCP or SCTP) and
or SCTP) and protocols beneath it (e.g., IP) without RDMAP/DDP. protocols beneath it (e.g., IP) without RDMAP/DDP.
2 Architectural Model 2. Architectural Model
This section describes an RDMA architectural reference model that This section describes an RDMA architectural reference model that is
is used as security issues are examined. It introduces the used as security issues are examined. It introduces the components
components of the model, the resources that can be attacked, the of the model, the resources that can be attacked, the types of
types of interactions possible between components and resources, interactions possible between components and resources, and the
and the system properties which must be preserved. system properties that must be preserved.
Figure 1 shows the components comprising the architecture and the Figure 1 shows the components comprising the architecture and the
interfaces where potential security attacks could be launched. interfaces where potential security attacks could be launched.
External attacks can be injected into the system from a ULP that External attacks can be injected into the system from a ULP that sits
sits above the RNIC Interface or from the network. above the RNIC Interface or from the network.
The intent here is to describe high level components and The intent here is to describe high level components and capabilities
capabilities which affect threat analysis, and not focus on that affect threat analysis, and not focus on specific implementation
specific implementation options. Also note that the architectural options. Also note that the architectural model is an abstraction,
model is an abstraction, and an actual implementation may choose and an actual implementation may choose to subdivide its components
to subdivide its components along different boundary lines than along different boundary lines from those defined here. For example,
defined here. For example, the Privileged Resource Manager may be the Privileged Resource Manager may be partially or completely
partially or completely encapsulated in the Privileged ULP. encapsulated in the Privileged ULP. Regardless, it is expected that
Regardless, it is expected that the security analysis of the the security analysis of the potential threats and countermeasures
potential threats and countermeasures still apply. still apply.
Note that the model below is derived from several specific RDMA Note that the model below is derived from several specific RDMA
implementations. A few of note are [VERBS-RDMAC], [VERBS-RDMAC- implementations. A few of note are [VERBS-RDMAC], [VERBS-RDMAC-
Overview], and [INFINIBAND]. Overview], and [INFINIBAND].
+-------------+ +-------------+
| Privileged | | Privileged |
| Resource | | Resource |
Admin<-+>| Manager | ULP Control Interface Admin<-+>| Manager | ULP Control Interface
| | |<------+-------------------+ | | |<------+-------------------+
skipping to change at page 8, line 36 skipping to change at page 7, line 36
| RNIC Engine | | RNIC Engine |
| | | |
+--------------------------------------+ +--------------------------------------+
^ ^
| |
v v
Internet Internet
Figure 1 - RDMA Security Model Figure 1 - RDMA Security Model
2.1 Components 2.1. Components
The components shown in Figure 1 - RDMA Security Model are: The components shown in Figure 1 - RDMA Security Model are:
* RDMA Network Interface Controller Engine (RNIC) - the * RDMA Network Interface Controller Engine (RNIC) - The component
component that implements the RDMA protocol and/or DDP that implements the RDMA protocol and/or DDP protocol.
protocol.
* Privileged Resource Manager - the component responsible * Privileged Resource Manager - The component responsible for
for managing and allocating resources associated with the managing and allocating resources associated with the RNIC
RNIC Engine. The Resource Manager does not send or Engine. The Resource Manager does not send or receive data.
receive data. Note that whether the Resource Manager is Note that whether the Resource Manager is an independent
an independent component, part of the RNIC, or part of component, part of the RNIC, or part of the ULP is implementation
the ULP is implementation dependent. dependent.
* Privileged ULP - See Section 1 Introduction for a * Privileged ULP - See Section 1, Introduction, for a definition of
definition of Privileged ULP. The local host Privileged ULP. The local host infrastructure can enable the
infrastructure can enable the Privileged ULP to map a Privileged ULP to map a Data Buffer directly from the RNIC Engine
data buffer directly from the RNIC Engine to the host to the host through the RNIC Interface, but it does not allow the
through the RNIC Interface, but it does not allow the
Privileged ULP to directly consume RNIC Engine resources. Privileged ULP to directly consume RNIC Engine resources.
* Non-Privileged ULP - See Section 1 Introduction for a * Non-Privileged ULP - See Section 1, Introduction, for a
definition of Non-Privileged ULP. definition of Non-Privileged ULP.
A design goal of the DDP and RDMAP protocols is to allow, under A design goal of the DDP and RDMAP protocols is to allow, under
constrained conditions, Non-Privileged ULP to send and receive constrained conditions, Non-Privileged ULP to send and receive data
data directly to/from the RDMA Engine without Privileged Resource directly to/from the RDMA Engine without Privileged Resource Manager
Manager intervention - while ensuring that the host remains intervention, while ensuring that the host remains secure. Thus, one
secure. Thus, one of the primary goals of this document is to of the primary goals of this document is to analyze this usage model
analyze this usage model for the enforcement that is required in for the enforcement that is required in the RNIC Engine to ensure
the RNIC Engine to ensure the system remains secure. that the system remains secure.
DDP provides two mechanisms for transferring data: DDP provides two mechanisms for transferring data:
* Untagged Data Transfer - the incoming payload simply * Untagged Data Transfer - The incoming payload simply consumes the
consumes the first buffer in a queue of buffers that are first buffer in a queue of buffers that are in the order
in the order specified by the receiving Peer (commonly specified by the receiving Peer (commonly referred to as the
referred to as the Receive Queue), and Receive Queue), and
* Tagged Data Transfer - the Peer transmitting the payload * Tagged Data Transfer - The Peer transmitting the payload
explicitly states which destination buffer is targeted, explicitly states which destination buffer is targeted, through
through use of an STag. STag based transfers allow the use of an STag. STag-based transfers allow the receiving ULP to
receiving ULP to be indifferent to what order (or in what be indifferent to what order (or in what messages) the opposite
messages) the opposite Peer sent the data, or what order Peer sent the data, or in what order packets are received.
packets are received in.
Both data transfer mechanisms are also enabled through RDMAP, Both data transfer mechanisms are also enabled through RDMAP, with
with additional control semantics. Typically Tagged Data Transfer additional control semantics. Typically, Tagged Data Transfer can be
can be used for payload transfer, while Untagged Data Transfer is used for payload transfer, while Untagged Data Transfer is best used
best used for control messages. However, each upper layer for control messages. However, each Upper Layer Protocol can
protocol can determine the optimal use of tagged and untagged determine the optimal use of Tagged and Untagged messages for itself.
messages for itself. See [APPLICABILITY] for more information on See [APPLICABILITY] for more information on application applicability
application applicability for the two transfer mechanisms. for the two transfer mechanisms.
For DDP the two forms correspond to Untagged and Tagged DDP For DDP, the two forms correspond to Untagged and Tagged DDP
Messages, respectively. For RDMAP the two forms correspond to Messages, respectively. For RDMAP, the two forms correspond to Send
Send Type Messages and RDMA Messages (either RDMA Read or RDMA Type Messages and RDMA Messages (either RDMA Read or RDMA Write
Write Messages), respectively. Messages), respectively.
The host interfaces that could be exercised include: The host interfaces that could be exercised include:
* Privileged Control Interface - A Privileged Resource * Privileged Control Interface - A Privileged Resource Manager uses
Manager uses the RNIC Interface to allocate and manage the RNIC Interface to allocate and manage RNIC Engine resources,
RNIC Engine resources, control the state within the RNIC control the state within the RNIC Engine, and monitor various
Engine, and monitor various events from the RNIC Engine. events from the RNIC Engine. It also uses this interface to act
It also uses this interface to act as a proxy for some as a proxy for some operations that a Non-Privileged ULP may
operations that a Non-Privileged ULP may require (after require (after performing appropriate countermeasures).
performing appropriate countermeasures).
* ULP Control Interface - A ULP uses this interface to the * ULP Control Interface - A ULP uses this interface to the
Privileged Resource Manager to allocate RNIC Engine Privileged Resource Manager to allocate RNIC Engine resources.
resources. The Privileged Resource Manager implements The Privileged Resource Manager implements countermeasures to
countermeasures to ensure that if the Non-Privileged ULP ensure that, if the Non-Privileged ULP launches an attack, it can
launches an attack it can prevent the attack from prevent the attack from affecting other ULPs.
affecting other ULPs.
* Non-Privileged Data Transfer Interface - A Non-Privileged * Non-Privileged Data Transfer Interface - A Non-Privileged ULP
ULP uses this interface to initiate and to check the uses this interface to initiate and check the status of data
status of data transfer operations. transfer operations.
* Privileged Data Transfer Interface - A superset of the * Privileged Data Transfer Interface - A superset of the
functionality provided by the Non-Privileged Data functionality provided by the Non-Privileged Data Transfer
Transfer Interface. The ULP is allowed to directly Interface. The ULP is allowed to directly manipulate RNIC Engine
manipulate RNIC Engine mapping resources to map an STag mapping resources to map an STag to a ULP Data Buffer.
to a ULP data buffer.
If Internet control messages, such as ICMP, ARP, RIPv4, etc. are If Internet control messages, such as ICMP, ARP, RIPv4, etc. are
processed by the RNIC Engine, the threat analyses for those processed by the RNIC Engine, the threat analyses for those protocols
protocols is also applicable, but outside the scope of this is also applicable, but outside the scope of this document.
document.
2.2 Resources 2.2. Resources
This section describes the primary resources in the RNIC Engine This section describes the primary resources in the RNIC Engine that
that could be affected if under attack. For RDMAP, all of the could be affected if under attack. For RDMAP, all the defined
defined resources apply. For DDP, all of the resources except the resources apply. For DDP, all the resources except the RDMA Read
RDMA Read Queue apply. Queue apply.
2.2.1 Stream Context Memory 2.2.1. Stream Context Memory
The state information for each Stream is maintained in memory, The state information for each Stream is maintained in memory, which
which could be located in a number of places - on the NIC, inside could be located in a number of places - on the NIC, inside RAM
RAM attached to the NIC, in host memory, or in any combination of attached to the NIC, in host memory, or in any combination of the
the three, depending on the implementation. three, depending on the implementation.
Stream Context Memory includes state associated with Data Stream Context Memory includes state associated with Data Buffers.
Buffers. For Tagged Buffers, this includes how STag names, Data For Tagged Buffers, this includes how STag names, Data Buffers, and
Buffers, and Page Translation Tables (see Section 2.2.3) Page Translation Tables (see Section 2.2.3) interrelate. It also
interrelate. It also includes the list of Untagged Data Buffers includes the list of Untagged Data Buffers posted for reception of
posted for reception of Untagged Messages (commonly called the Untagged Messages (commonly called the Receive Queue), and a list of
Receive Queue), and a list of operations to perform to send data operations to perform to send data (commonly called the Send Queue).
(commonly called the Send Queue).
2.2.2 Data Buffers 2.2.2. Data Buffers
As mentioned previously, there are two different ways to expose a As mentioned previously, there are two different ways to expose a
local ULP's data buffers for data transfer; Untagged Data local ULP's Data Buffers for data transfer: Untagged Data Transfer,
Transfer - a buffer can be exposed for receiving RDMAP Send Type where a buffer can be exposed for receiving RDMAP Send Type Messages
Messages (a.k.a. DDP Untagged Messages) on DDP Queue zero - or (a.k.a. DDP Untagged Messages) on DDP Queue zero, or Tagged Data
Tagged Data Transfer - the buffer can be exposed for remote Transfer, where the buffer can be exposed for remote access through
access through STags (a.k.a. DDP Tagged Messages). This STags (a.k.a. DDP Tagged Messages). This distinction is important
distinction is important because the attacks and the because the attacks and the countermeasures used to protect against
countermeasures used to protect against the attack are different the attack are different depending on the method for exposing the
depending on the method for exposing the buffer to the network. buffer to the network.
For the purposes of the security discussion, for Tagged Data For the purposes of the security discussion, for Tagged Data
Transfer a single logical Data Buffer is exposed with a single Transfer, a single logical Data Buffer is exposed with a single STag
Stag on a given Stream. Actual implementations may support on a given Stream. Actual implementations may support scatter/gather
scatter/gather capabilities to enable multiple physical data capabilities to enable multiple physical data buffers to be accessed
buffers to be accessed with a single STag, but from a threat with a single STag, but from a threat analysis perspective, it is
analysis perspective it is assumed that a single STag enables assumed that a single STag enables access to a single logical Data
access to a single logical Data Buffer. Buffer.
In any event, it is the responsibility of the Privileged Resource In any event, it is the responsibility of the Privileged Resource
Manager to ensure that no STag can be created that exposes memory Manager to ensure that no STag can be created that exposes memory
that the consumer had no authority to expose. that the consumer had no authority to expose.
A data buffer has specific access rights. The local ULP can A Data Buffer has specific access rights. The local ULP can control
control whether a data buffer is exposed for local only, or local whether a Data Buffer is exposed for local only, or local and remote
and remote access, and assign specific access privileges (read, access, and assign specific access privileges (read, write, read and
write, read and write) on a per Stream basis. write) on a per Stream basis.
For DDP, when an STag is advertised, the Remote Peer is For DDP, when an STag is Advertised, the Remote Peer is presumably
presumably given write access rights to the data (otherwise there given write access rights to the data (otherwise, there would not be
was not much point to the advertisement). For RDMAP, when a ULP much point to the Advertisement). For RDMAP, when a ULP Advertises
advertises an STag, it can enable write-only, read-only, or both an STag, it can enable write-only, read-only, or both write and read
write and read access rights. access rights.
Similarly, some ULPs may wish to provide a single buffer with Similarly, some ULPs may wish to provide a single buffer with
different access rights on a per-Stream basis. For example, some different access rights on a per Stream basis. For example, some
Streams may have read-only access, some may have remote read and Streams may have read-only access, some may have remote read and
write access, while on other Streams only the local ULP/Local write access, while on other Streams, only the local ULP/Local Peer
Peer is allowed access. is allowed access.
2.2.3 Page Translation Tables 2.2.3. Page Translation Tables
Page Translation Tables are the structures used by the RNIC to be Page Translation Tables are the structures used by the RNIC to be
able to access ULP memory for data transfer operations. Even able to access ULP memory for data transfer operations. Even though
though these structures are called "Page" Translation Tables, these structures are called "Page" Translation Tables, they may not
they may not reference a page at all - conceptually they are used reference a page at all - conceptually, they are used to map a ULP
to map a ULP address space representation (e.g. a virtual address space representation (e.g., a virtual address) of a buffer to
address) of a buffer to the physical addresses that are used by the physical addresses that are used by the RNIC Engine to move data.
the RNIC Engine to move data. If on a specific system a mapping If, on a specific system, a mapping is not used, then a subset of the
is not used, then a subset of the attacks examined may be attacks examined may be appropriate. Note that the Page Translation
appropriate. Note that the Page Translation Table may or may not Table may or may not be a shared resource.
be a shared resource.
2.2.4 Protection Domain (PD) 2.2.4. Protection Domain (PD)
A Protection Domain (PD) is a local construct to the RDMA A Protection Domain (PD) is a local construct to the RDMA
implementation, and never visible over the wire. Protection implementation, and never visible over the wire. Protection Domains
Domains are assigned to three of the resources of concern - are assigned to three of the resources of concern - Stream Context
Stream Context Memory, STags associated with Page Translation Memory, STags associated with Page Translation Table entries, and
Table entries, and data buffers. A correct implementation of a Data Buffers. A correct implementation of a Protection Domain
Protection Domain requires that resources which belong to a given requires that resources that belong to a given Protection Domain
Protection Domain can not be used on a resource belonging to cannot be used on a resource belonging to another Protection Domain,
another Protection Domain, because Protection Domain membership because Protection Domain membership is checked by the RNIC prior to
is checked by the RNIC prior to taking any action involving such taking any action involving such a resource. Protection Domains are
a resource. Protection Domains are therefore used to ensure that therefore used to ensure that an STag can only be used to access an
an STag can only be used to access an associated data buffer on associated Data Buffer on one or more Streams that are associated
one or more Streams that are associated with the same Protection with the same Protection Domain as the specific STag.
Domain as the specific STag.
If an implementation chooses to not share resources between If an implementation chooses not to share resources between Streams,
Streams, it is recommended that each Stream be associated with it is recommended that each Stream be associated with its own, unique
its own, unique Protection Domain. If an implementation chooses Protection Domain. If an implementation chooses to allow resource
to allow resource sharing, it is recommended that Protection sharing, it is recommended that Protection Domain be limited to the
Domain be limited to the collection of Streams that have Partial collection of Streams that have Partial Mutual Trust with each other.
Mutual Trust with each other.
Note that a ULP (either Privileged or Non-Privileged) can Note that a ULP (either Privileged or Non-Privileged) can potentially
potentially have multiple Protection Domains. This could be used, have multiple Protection Domains. This could be used, for example,
for example, to ensure that multiple clients of a server do not to ensure that multiple clients of a server do not have the ability
have the ability to corrupt each other. The server would allocate to corrupt each other. The server would allocate a Protection Domain
a Protection Domain per client to ensure that resources covered per client to ensure that resources covered by the Protection Domain
by the Protection Domain could not be used by another (untrusted) could not be used by another (untrusted) client.
client.
2.2.5 STag Namespace and Scope 2.2.5. STag Namespace and Scope
The DDP specification defines a 32-bit namespace for the STag. The DDP specification defines a 32-bit namespace for the STag.
Implementations may vary in terms of the actual number of STags Implementations may vary in terms of the actual number of STags that
that are supported. In any case, this is a bounded resource that are supported. In any case, this is a bounded resource that can come
can come under attack. Depending upon STag namespace allocation under attack. Depending upon STag namespace allocation algorithms,
algorithms, the actual name space to attack may be significantly the actual name space to attack may be significantly less than 2^32.
less than 2^32.
The scope of an STag is the set of DDP/RDMAP Streams on which the The scope of an STag is the set of DDP/RDMAP Streams on which the
STag is valid. If an STag is valid on a particular DDP/RDMAP STag is valid. If an STag is valid on a particular DDP/RDMAP Stream,
Stream, then that stream can modify the buffer, subject to the then that stream can modify the buffer, subject to the access rights
access rights that the stream has for the STag (see Section 2.2.2 that the stream has for the STag (see Section 2.2.2, Data Buffers,
Data Buffers for additional information). for additional information).
The analysis presented in this document assumes two mechanisms The analysis presented in this document assumes two mechanisms for
for limiting the scope of Streams for which the STag is valid: limiting the scope of Streams for which the STag is valid:
* Protection Domain scope. The STag is valid if used on * Protection Domain scope. The STag is valid if used on any Stream
any Stream within a specific Protection Domain, and within a specific Protection Domain, and is invalid if used on
is invalid if used on any Stream that is not a member any Stream that is not a member of the Protection Domain.
of the Protection Domain.
* Single Stream scope. The STag is valid on a single * Single Stream scope. The STag is valid on a single Stream,
Stream, regardless of what the Stream association is regardless of what the Stream association is to a Protection
to a Protection Domain. If used on any other Stream, Domain. If used on any other Stream, it is invalid.
it is invalid.
2.2.6 Completion Queues 2.2.6. Completion Queues
Completion Queues (CQ) are used in this document to conceptually Completion Queues (CQ) are used in this document to conceptually
represent how the RNIC Engine notifies the ULP about the represent how the RNIC Engine notifies the ULP about the completion
completion of the transmission of data, or the completion of the of the transmission of data, or the completion of the reception of
reception of data through the Data Transfer Interface data through the Data Transfer Interface (specifically for Untagged
(specifically for Untagged Data Transfer - Tagged Data Transfer Data Transfer; Tagged Data Transfer cannot cause a completion to
can not cause a completion to occur). Because there could be many occur). Because there could be many transmissions or receptions in
transmissions or receptions in flight at any one time, flight at any one time, completions are modeled as a queue rather
completions are modeled as a queue rather than a single event. An than as a single event. An implementation may also use the
implementation may also use the Completion Queue to notify the Completion Queue to notify the ULP of other activities; for example,
ULP of other activities, for example, the completion of a mapping the completion of a mapping of an STag to a specific ULP buffer.
of an STag to a specific ULP buffer. Completion Queues may be Completion Queues may be shared by a group of Streams, or may be
shared by a group of Streams, or may be designated to handle a designated to handle a specific Stream's traffic. Limiting
specific Stream's traffic. Limiting Completion Queue association Completion Queue association to one, or a small number, of RDMAP/DDP
to one, or a small number of RDMAP/DDP Streams can prevent Streams can prevent several forms of attacks by sharply limiting the
several forms of attacks by sharply limiting the scope of the scope of the attack's effect.
attack's effect.
Some implementations may allow this queue to be manipulated Some implementations may allow this queue to be manipulated directly
directly by both Non-Privileged and Privileged ULPs. by both Non-Privileged and Privileged ULPs.
2.2.7 Asynchronous Event Queue 2.2.7. Asynchronous Event Queue
The Asynchronous Event Queue is a queue from the RNIC to the The Asynchronous Event Queue is a queue from the RNIC to the
Privileged Resource Manager of bounded size. It is used by the Privileged Resource Manager of bounded size. It is used by the RNIC
RNIC to notify the host of various events which might require to notify the host of various events that might require management
management action, including protocol violations, Stream state action, including protocol violations, Stream state changes, local
changes, local operation errors, low water marks on receive operation errors, low water marks on receive queues, and possibly
queues, and possibly other events. other events.
The Asynchronous Event Queue is a resource that can be attacked The Asynchronous Event Queue is a resource that can be attacked
because Remote or Local Peers and/or ULPs can cause events to because Remote or Local Peers and/or ULPs can cause events to occur
occur which have the potential of overflowing the queue. that have the potential of overflowing the queue.
Note that an implementation is at liberty to implement the Note that an implementation is at liberty to implement the functions
functions of the Asynchronous Event Queue in a variety of ways, of the Asynchronous Event Queue in a variety of ways, including
including multiple queues or even simple callbacks. All multiple queues or even simple callbacks. All vulnerabilities
vulnerabilities identified are intended to apply regardless of identified are intended to apply, regardless of the implementation of
the implementation of the Asynchronous Event Queue. For example, the Asynchronous Event Queue. For example, a callback function may
a callback function may be viewed as simply a very short queue. be viewed simply as a very short queue.
2.2.8 RDMA Read Request Queue 2.2.8. RDMA Read Request Queue
The RDMA Read Request Queue is the memory that holds state The RDMA Read Request Queue is the memory that holds state
information for one or more RDMA Read Request Messages that have information for one or more RDMA Read Request Messages that have
arrived, but for which the RDMA Read Response Messages have not arrived, but for which the RDMA Read Response Messages have not yet
yet been completely sent. Because potentially more than one RDMA been completely sent. Because potentially more than one RDMA Read
Read Request can be outstanding at one time, the memory is Request can be outstanding at one time, the memory is modeled as a
modeled as a queue of bounded size. Some implementations may queue of bounded size. Some implementations may enable sharing of a
enable sharing of a single RDMA Read Request Queue across single RDMA Read Request Queue across multiple Streams.
multiple Streams.
2.3 RNIC Interactions 2.3. RNIC Interactions
With RNIC resources and interfaces defined, it is now possible to With RNIC resources and interfaces defined, it is now possible to
examine the interactions supported by the generic RNIC functional examine the interactions supported by the generic RNIC functional
interfaces through each of the 3 interfaces - Privileged Control interfaces through each of the 3 interfaces: Privileged Control
Interface, Privileged Data Interface, and Non-Privileged Data Interface, Privileged Data Interface, and Non-Privileged Data
Interface. As mentioned previously in Section 2.1 Components, Interface. As mentioned previously in Section 2.1, Components, there
there are two data transfer mechanisms to be examined - Untagged are two data transfer mechanisms to be examined, Untagged Data
Data Transfer and Tagged Data Transfer. Transfer and Tagged Data Transfer.
2.3.1 Privileged Control Interface Semantics 2.3.1. Privileged Control Interface Semantics
Generically, the Privileged Control Interface controls the RNIC's Generically, the Privileged Control Interface controls the RNIC's
allocation, de-allocation, and initialization of RNIC global allocation, de-allocation, and initialization of RNIC global
resources. This includes allocation and de-allocation of Stream resources. This includes allocation and de-allocation of Stream
Context Memory, Page Translation Tables, STag names, Completion Context Memory, Page Translation Tables, STag names, Completion
Queues, RDMA Read Request Queues, and Asynchronous Event Queues. Queues, RDMA Read Request Queues, and Asynchronous Event Queues.
The Privileged Control Interface is also typically used for The Privileged Control Interface is also typically used for managing
managing Non-Privileged ULP resources for the Non-Privileged ULP Non-Privileged ULP resources for the Non-Privileged ULP (and possibly
(and possibly for the Privileged ULP as well). This includes for the Privileged ULP as well). This includes initialization and
initialization and removal of Page Translation Table resources, removal of Page Translation Table resources, and managing RNIC events
and managing RNIC events (possibly managing all events for the (possibly managing all events for the Asynchronous Event Queue).
Asynchronous Event Queue).
2.3.2 Non-Privileged Data Interface Semantics 2.3.2. Non-Privileged Data Interface Semantics
The Non-Privileged Data Interface enables data transfer (transmit The Non-Privileged Data Interface enables data transfer (transmit and
and receive) but does not allow initialization of the Page receive) but does not allow initialization of the Page Translation
Translation Table resources. However, once the Page Translation Table resources. However, once the Page Translation Table resources
Table resources have been initialized, the interface may enable a have been initialized, the interface may enable a specific STag
specific STag mapping to be enabled and disabled by directly mapping to be enabled and disabled by directly communicating with the
communicating with the RNIC, or create an STag mapping for a RNIC, or create an STag mapping for a buffer that has been previously
buffer that has been previously initialized in the RNIC. initialized in the RNIC.
For RDMAP, ULP data can be sent by one of the previously For RDMAP, ULP data can be sent by one of the previously described
described data transfer mechanisms - Untagged Data Transfer or data transfer mechanisms: Untagged Data Transfer or Tagged Data
Tagged Data Transfer. Two RDMAP data transfer mechanisms are Transfer. Two RDMAP data transfer mechanisms are defined, one using
defined, one using Untagged Data Transfer (Send Type Messages), Untagged Data Transfer (Send Type Messages), and one using Tagged
and one using Tagged Data Transfer (RDMA Read Responses and RDMA Data Transfer (RDMA Read Responses and RDMA Writes). ULP data
Writes). ULP data reception through RDMAP can be done by reception through RDMAP can be done by receiving Send Type Messages
receiving Send Type Messages into buffers that have been posted into buffers that have been posted on the Receive Queue or Shared
on the Receive Queue or Shared Receive Queue. Thus a Receive Receive Queue. Thus, a Receive Queue or Shared Receive Queue can
Queue or Shared Receive Queue can only be affected by Untagged only be affected by Untagged Data Transfer. Data reception can also
Data Transfer. Data reception can also be done by receiving RDMA be done by receiving RDMA Write and RDMA Read Response Messages into
Write and RDMA Read Response Messages into buffers that have buffers that have previously been exposed for external write access
previously been exposed for external write access through through Advertisement of an STag (i.e., Tagged Data Transfer).
advertisement of an STag (i.e. Tagged Data Transfer).
Additionally, to cause ULP data to be pulled (read) across the Additionally, to cause ULP data to be pulled (read) across the
network, RDMAP uses an RDMA Read Request Message (which only network, RDMAP uses an RDMA Read Request Message (which only contains
contains RDMAP control information necessary to access the ULP RDMAP control information necessary to access the ULP buffer to be
buffer to be read), to cause an RDMA Read Response Message to be read), to cause an RDMA Read Response Message to be generated that
generated that contains the ULP data. contains the ULP data.
For DDP, transmitting data means sending DDP Tagged or Untagged For DDP, transmitting data means sending DDP Tagged or Untagged
Messages. For data reception, DDP can receive Untagged Messages Messages. For data reception, DDP can receive Untagged Messages into
into buffers that have been posted on the Receive Queue or Shared buffers that have been posted on the Receive Queue or Shared Receive
Receive Queue. It can also receive Tagged DDP Messages into Queue. It can also receive Tagged DDP Messages into buffers that
buffers that have previously been exposed for external write have previously been exposed for external write access through
access through advertisement of an STag. Advertisement of an STag.
Completion of data transmission or reception generally entails Completion of data transmission or reception generally entails
informing the ULP of the completed work by placing completion informing the ULP of the completed work by placing completion
information on the Completion Queue. For data reception, only an information on the Completion Queue. For data reception, only an
Untagged Data Transfer can cause completion information to be put Untagged Data Transfer can cause completion information to be put in
in the Completion Queue. the Completion Queue.
2.3.3 Privileged Data Interface Semantics
The Privileged Data Interface semantics are a superset of the 2.3.3. Privileged Data Interface Semantics
Non-Privileged Data Transfer semantics. The interface can do
everything defined in the prior section, as well as
create/destroy buffer to STag mappings directly. This generally
entails initialization or clearing of Page Translation Table
state in the RNIC.
2.3.4 Initialization of RNIC Data Structures for Data Transfer The Privileged Data Interface semantics are a superset of the Non-
Privileged Data Transfer semantics. The interface can do everything
defined in the prior section, as well as create/destroy buffer to
STag mappings directly. This generally entails initialization or
clearing of Page Translation Table state in the RNIC.
Initialization of the mapping between an STag and a Data Buffer 2.3.4. Initialization of RNIC Data Structures for Data Transfer
can be viewed in the abstract as two separate operations:
a. Initialization of the allocated Page Translation Table Initialization of the mapping between an STag and a Data Buffer can
entries with the location of the Data Buffer, and be viewed in the abstract as two separate operations:
b. Initialization of a mapping from an allocated STag name a. Initialization of the allocated Page Translation Table entries
to a set of Page Translation Table entry(s) or partial- with the location of the Data Buffer, and
entries. b. Initialization of a mapping from an allocated STag name to a set
of Page Translation Table entry(s) or partial entries.
Note that an implementation may not have a Page Translation Table Note that an implementation may not have a Page Translation Table
(i.e. it may support a direct mapping between an STag and a Data (i.e., it may support a direct mapping between an STag and a Data
Buffer). If there is no Page Translation Table, then attacks Buffer). If there is no Page Translation Table, then attacks based
based on changing its contents or exhausting its resources are on changing its contents or exhausting its resources are not
not possible. possible.
Initialization of the contents of the Page Translation Table can Initialization of the contents of the Page Translation Table can be
be done by either the Privileged ULP or by the Privileged done by either the Privileged ULP or by the Privileged Resource
Resource Manager as a proxy for the Non-Privileged ULP. By Manager as a proxy for the Non-Privileged ULP. By definition, the
definition the Non-Privileged ULP is not trusted to directly Non-Privileged ULP is not trusted to directly manipulate the Page
manipulate the Page Translation Table. In general the concern is Translation Table. In general, the concern is that the Non-
that the Non-Privileged ULP may try to maliciously initialize the Privileged ULP may try to maliciously initialize the Page Translation
Page Translation Table to access a buffer for which it does not Table to access a buffer for which it does not have permission.
have permission.
The exact resource allocation algorithm for the Page Translation The exact resource allocation algorithm for the Page Translation
Table is outside the scope of this document. It may be allocated Table is outside the scope of this document. It may be allocated for
for a specific Data Buffer, or be allocated as a pooled resource a specific Data Buffer, or as a pooled resource to be consumed by
to be consumed by potentially multiple Data Buffers, or be potentially multiple Data Buffers, or be managed in some other way.
managed in some other way. This document attempts to abstract This document attempts to abstract implementation dependent issues,
implementation dependent issues, and group them into higher level and group them into higher level security issues, such as resource
security issues such as resource starvation and sharing of starvation and sharing of resources between Streams.
resources between Streams.
The next issue is how an STag name is associated with a Data The next issue is how an STag name is associated with a Data Buffer.
Buffer. For the case of an Untagged Data Buffer (i.e. Untagged For the case of an Untagged Data Buffer (i.e., Untagged Data
Data Transfer), there is no wire visible mapping between an STag Transfer), there is no wire visible mapping between an STag and the
and the Data Buffer. Note that there may, in fact, be an STag Data Buffer. Note that there may, in fact, be an STag that
which represents the buffer, if an implementation chooses to represents the buffer, if an implementation chooses to internally
internally represent Untagged Data Buffer using STags. However, represent Untagged Data Buffer using STags. However, because the
because the STag by definition is not visible on the wire, this STag, by definition, is not visible on the wire, this is a local
is a local host implementation specific issue which should be host, implementation-specific issue that should be analyzed in the
analyzed in the context of a local host implementation specific context of a local host implementation-specific security analysis,
security analysis, and thus is outside the scope of this and thus, is outside the scope of this document.
document.
For a Tagged Data Buffer (i.e. Tagged Data Transfer), either the For a Tagged Data Buffer (i.e., Tagged Data Transfer), either the
Privileged ULP or the Privileged Resource Manager acting on Privileged ULP or the Privileged Resource Manager acting on behalf of
behalf of the Non-Privileged ULP may initialize a mapping from an the Non-Privileged ULP may initialize a mapping from an STag to a
STag to a Page Translation Table, or may have the ability to Page Translation Table, or may have the ability to simply
simply enable/disable an existing STag to Page Translation Table enable/disable an existing STag to Page Translation Table mapping.
mapping. There may also be multiple STag names which map to a There may also be multiple STag names that map to a specific group of
specific group of Page Translation Table entries (or sub- Page Translation Table entries (or sub-entries). Specific security
entries). Specific security issues with this level of flexibility issues with this level of flexibility are examined in Section 6.2.3,
are examined in Section 6.2.3 Multiple STags to access the same Multiple STags to Access the Same Buffer.
buffer.
There are a variety of implementation options for initialization There are a variety of implementation options for initialization of
of Page Translation Table entries and mapping an STag to a group Page Translation Table entries and mapping an STag to a group of Page
of Page Translation Table entries which have security Translation Table entries that have security repercussions. This
repercussions. This includes support for separation of Mapping an includes support for separation of mapping an STag versus mapping a
STag versus mapping a set of Page Translation Table entries, and set of Page Translation Table entries, and support for ULPs directly
support for ULPs directly manipulating STag to Page Translation manipulating STag to Page Translation Table entry mappings (versus
Table entry mappings (versus requiring access through the requiring access through the Privileged Resource Manager).
Privileged Resource Manager).
2.3.5 RNIC Data Transfer Interactions 2.3.5. RNIC Data Transfer Interactions
RNIC Data Transfer operations can be subdivided into send RNIC Data Transfer operations can be subdivided into send and receive
operations and receive operations. operations.
For send operations, there is typically a queue that enables the For send operations, there is typically a queue that enables the ULP
ULP to post multiple operation requests to send data (referred to to post multiple operation requests to send data (referred to as the
as the Send Queue). Depending upon the implementation, Data Send Queue). Depending upon the implementation, Data Buffers used in
Buffers used in the operations may or may not have Page the operations may or may not have Page Translation Table entries
Translation Table entries associated with them, and may or may associated with them, and may or may not have STags associated with
not have STags associated with them. Because this is a local host them. Because this is a local host specific implementation issue
specific implementation issue rather than a protocol issue, the rather than a protocol issue, the security analysis of threats and
security analysis of threats and mitigations is left to the host mitigations is left to the host implementation.
implementation.
Receive operations are different for Tagged Data Buffers versus Receive operations are different for Tagged Data Buffers versus
Untagged Data Buffers (i.e. Tagged Data Transfer vs. Untagged Untagged Data Buffers (i.e., Tagged Data Transfer vs. Untagged Data
Data Transfer). For Untagged Data Transfer, if more than one Transfer). For Untagged Data Transfer, if more than one Untagged
Untagged Data Buffer can be posted by the ULP, the DDP Data Buffer can be posted by the ULP, the DDP specification requires
specification requires that they be consumed in sequential order that they be consumed in sequential order (the RDMAP specification
(the RDMAP specification also requires this). Thus the most also requires this). Thus, the most general implementation is that
general implementation is that there is a sequential queue of there is a sequential queue of receive Untagged Data Buffers (Receive
receive Untagged Data Buffers (Receive Queue). Some Queue). Some implementations may also support sharing of the
implementations may also support sharing of the sequential queue sequential queue between multiple Streams. In this case, defining
between multiple Streams. In this case defining "sequential" "sequential" becomes non-trivial - in general, the buffers for a
becomes non-trivial - in general the buffers for a single Stream single Stream are consumed from the queue in the order that they were
are consumed from the queue in the order that they were placed on placed on the queue, but there is no consumption order guarantee
the queue, but there is no consumption order guarantee between between Streams.
Streams.
For receive Tagged Data Transfer (i.e. Tagged Data Buffers, RDMA For receive Tagged Data Transfer (i.e., Tagged Data Buffers, RDMA
Write Buffers, or RDMA Read Buffers), at some time prior to data Write Buffers, or RDMA Read Buffers), at some time prior to data
transfer, the mapping of the STag to specific Page Translation transfer, the mapping of the STag to specific Page Translation Table
Table entries (if present) and the mapping from the Page entries (if present) and the mapping from the Page Translation Table
Translation Table entries to the Data Buffer must have been entries to the Data Buffer must have been initialized (see Section
initialized (see Section 2.3.4 for interaction details). 2.3.4 for interaction details).
3 Trust and Resource Sharing 3. Trust and Resource Sharing
It is assumed that in general the Local and Remote Peer are It is assumed that, in general, the Local and Remote Peer are
untrusted, and thus attacks by either should have mitigations in untrusted, and thus attacks by either should have mitigations in
place. place.
A separate, but related issue is resource sharing between A separate, but related issue is resource sharing between multiple
multiple Streams. If local resources are not shared, the Streams. If local resources are not shared, the resources are
resources are dedicated on a per Stream basis. Resources are dedicated on a per Stream basis. Resources are defined in Section
defined in Section 2.2 Resources. The advantage of not sharing 2.2, Resources. The advantage of not sharing resources between
resources between Streams is that it reduces the types of attacks Streams is that it reduces the types of attacks that are possible.
that are possible. The disadvantage of not sharing resources is The disadvantage of not sharing resources is that ULPs might run out
that ULPs might run out of resources. Thus there can be a strong of resources. Thus, there can be a strong incentive for sharing
incentive for sharing resources, if the security issues resources, if the security issues associated with the sharing of
associated with the sharing of resources can be mitigated. resources can be mitigated.
It is assumed in this document that the component that implements It is assumed in this document that the component that implements the
the mechanism to control sharing of the RNIC Engine resources is mechanism to control sharing of the RNIC Engine resources is the
the Privileged Resource Manager. The RNIC Engine exposes its Privileged Resource Manager. The RNIC Engine exposes its resources
resources through the RNIC Interface to the Privileged Resource through the RNIC Interface to the Privileged Resource Manager. All
Manager. All Privileged and Non-Privileged ULPs request resources Privileged and Non-Privileged ULPs request resources from the
from the Resource Manager (note that by definition both the Non- Resource Manager (note that by definition both the Non-Privileged and
Privileged and the Privileged application might try to greedily the Privileged application might try to greedily consume resources,
consume resources, thus creating a potential Denial of Service thus creating a potential Denial of Service (DOS) attack). The
(DOS) attack). The Resource Manager implements resource Resource Manager implements resource management policies to ensure
management policies to ensure fair access to resources. The fair access to resources. The Resource Manager should be designed to
Resource Manager should be designed to take into account security take into account security attacks detailed in this document. Note
attacks detailed in this document. Note that for some systems the that for some systems the Privileged Resource Manager may be
Privileged Resource Manager may be implemented within the implemented within the Privileged ULP.
Privileged ULP.
All Non-Privileged ULP interactions with the RNIC Engine that All Non-Privileged ULP interactions with the RNIC Engine that could
could affect other ULPs MUST be done using the Privileged affect other ULPs MUST be done using the Privileged Resource Manager
Resource Manager as a proxy. All ULP resource allocation requests as a proxy. All ULP resource allocation requests for scarce
for scarce resources MUST also be done using a Privileged resources MUST also be done using a Privileged Resource Manager.
Resource Manager.
The sharing of resources across Streams should be under the The sharing of resources across Streams should be under the control
control of the ULP, both in terms of the trust model the ULP of the ULP, both in terms of the trust model the ULP wishes to
wishes to operate under, as well as the level of resource sharing operate under, as well as the level of resource sharing the ULP
the ULP wishes to give local processes. For more discussion on wishes to give local processes. For more discussion on types of
types of trust models which combine partial trust and sharing of trust models that combine partial trust and sharing of resources, see
resources, see Appendix C: Partial Trust Taxonomy. Appendix C, Partial Trust Taxonomy.
The Privileged Resource Manager MUST NOT assume different Streams The Privileged Resource Manager MUST NOT assume that different
share Partial Mutual Trust unless there is a mechanism to ensure Streams share Partial Mutual Trust unless there is a mechanism to
that the Streams do indeed share Partial Mutual Trust. This can ensure that the Streams do indeed share Partial Mutual Trust. This
be done in several ways, including explicit notification from the can be done in several ways, including explicit notification from the
ULP that owns the Streams. ULP that owns the Streams.
4 Attacker Capabilities 4. Attacker Capabilities
An attacker's capabilities delimit the types of attacks that An attacker's capabilities delimit the types of attacks that the
attacker is able to launch. RDMAP and DDP require that the attacker is able to launch. RDMAP and DDP require that the initial
initial LLP Stream (and connection) be set up prior to LLP Stream (and connection) be set up prior to transferring RDMAP/DDP
transferring RDMAP/DDP Messages. This requires at least one Messages. This requires at least one round-trip handshake to occur.
round-trip handshake to occur.
If the attacker is not the Remote Peer that created the initial If the attacker is not the Remote Peer that created the initial
connection, then the attacker's capabilities can be segmented connection, then the attacker's capabilities can be segmented into
into send only capabilities or send and receive capabilities. send only capabilities or send and receive capabilities. Attacking
Attacking with send only capabilities requires the attacker to with send only capabilities requires the attacker to first guess the
first guess the current LLP Stream parameters before they can current LLP Stream parameters before they can attack RNIC resources
attack RNIC resources (e.g. TCP sequence number). If this class (e.g., TCP sequence number). If this class of attacker also has
of attacker also has receive capabilities and the ability to pose receive capabilities and the ability to pose as the receiver to the
as the receiver to the sender and the sender to the receiver, sender and the sender to the receiver, they are typically referred to
they are typically referred to as a "man-in-the-middle" attacker as a "man-in-the-middle" attacker [RFC3552]. A man-in-the-middle
[RFC3552]. A man-in-the-middle attacker has a much wider ability attacker has a much wider ability to attack RNIC resources. The
to attack RNIC resources. The breadth of attack is essentially breadth of attack is essentially the same as that of an attacking
the same as that of an attacking Remote Peer (i.e. the Remote Remote Peer (i.e., the Remote Peer that set up the initial LLP
Peer that setup the initial LLP Stream). Stream).
5 Attacks That Can be Mitigated With End-to-End Security 5. Attacks That Can Be Mitigated with End-to-End Security
This section describes the RDMAP/DDP attacks where the only This section describes the RDMAP/DDP attacks where the only solution
solution is to implement some form of end-to-end security. The is to implement some form of end-to-end security. The analysis
analysis includes a detailed description of each attack, what is includes a detailed description of each attack, what is being
being attacked, and a description of the countermeasures that can attacked, and a description of the countermeasures that can be taken
be taken to thwart the attack. to thwart the attack.
Some forms of attack involve modifying the RDMAP or DDP payload Some forms of attack involve modifying the RDMAP or DDP payload by a
by a network based attacker or involve monitoring the traffic to network-based attacker or involve monitoring the traffic to discover
discover private information. An effective tool to ensure private information. An effective tool to ensure confidentiality is
confidentiality is to encrypt the data stream through mechanisms to encrypt the data stream through mechanisms, such as IPsec
such as IPsec encryption. Additionally, authentication protocols encryption. Additionally, authentication protocols, such as IPsec
such as IPsec authentication are an effective tool to ensure the authentication, are an effective tool to ensure the remote entity is
remote entity is who they claim to be as well as ensuring that who they claim to be, as well as ensuring that the payload is
the payload is unmodified as it traverses the network. unmodified as it traverses the network.
Note that connection setup and teardown is presumed to be done in Note that connection setup and teardown is presumed to be done in
stream mode (i.e. no RDMA encapsulation of the payload), so there stream mode (i.e., no RDMA encapsulation of the payload), so there
are no new attacks related to connection setup/teardown beyond are no new attacks related to connection setup/tear down beyond what
what is already present in the LLP (e.g. TCP or SCTP). Note, is already present in the LLP (e.g., TCP or SCTP). Note, however,
however, that RDMAP/DDP parameters may be exchanged in stream that RDMAP/DDP parameters may be exchanged in stream mode, and if
mode, and if they are corrupted by an attacker unintended they are corrupted by an attacker unintended consequences will
consequences will result. Therefore, any existing mitigations for result. Therefore, any existing mitigations for LLP Spoofing,
LLP Spoofing, Tampering, Repudiation, Information Disclosure, Tampering, Repudiation, Information Disclosure, Denial of Service, or
Denial of Service, or Elevation of Privilege continue to apply Elevation of Privilege continue to apply (and are out of scope of
(and are out of scope of this document). Thus the analysis in this document). Thus, the analysis in this section focuses on
this section focuses on attacks that are present regardless of attacks that are present, regardless of the LLP Stream type.
the LLP Stream type.
Tampering is any modification of the legitimate traffic (machine Tampering is any modification of the legitimate traffic (machine
internal or network). Spoofing attack is a special case of internal or network). Spoofing attack is a special case of tampering
tampering where the attacker falsifies an identity of the Remote where the attacker falsifies an identity of the Remote Peer (identity
Peer (identity can be an IP address, machine name, ULP level can be an IP address, machine name, ULP level identity, etc.).
identity etc.).
5.1 Spoofing 5.1. Spoofing
Spoofing attacks can be launched by the Remote Peer, or by a Spoofing attacks can be launched by the Remote Peer, or by a
network based attacker. A network based spoofing attack applies network-based attacker. A network-based spoofing attack applies to
to all Remote Peers. This section analyzes the various types of all Remote Peers. This section analyzes the various types of
spoofing attacks applicable to RDMAP & DDP. spoofing attacks applicable to RDMAP and DDP.
5.1.1 Impersonation 5.1.1. Impersonation
A network based attacker can impersonate a legal RDMAP/DDP Peer A network-based attacker can impersonate a legal RDMAP/DDP Peer (by
(by spoofing a legal IP address). This can either be done as a spoofing a legal IP address). This can either be done as a blind
blind attack (see [RFC3552]) or by establishing an RDMAP/DDP attack (see [RFC3552]) or by establishing an RDMAP/DDP Stream with
Stream with the victim. Because an RDMAP/DDP Stream requires an the victim. Because an RDMAP/DDP Stream requires an LLP Stream to be
LLP Stream to be fully initialized (e.g. for [RFC793] it is in fully initialized (e.g., for [RFC793], it is in the ESTABLISHED
the ESTABLISHED state), existing transport layer protection state), existing transport layer protection mechanisms against blind
mechanisms against blind attacks remain in place. attacks remain in place.
For a blind attack to succeed, it requires the attacker to inject For a blind attack to succeed, it requires the attacker to inject a
a valid transport layer segment (e.g. for TCP it must match at valid transport layer segment (e.g., for TCP, it must match at least
least the 4-tuple as well as guess a sequence number within the the 4-tuple as well as guess a sequence number within the window)
window) while also guessing valid RDMAP or DDP parameters. There while also guessing valid RDMAP or DDP parameters. There are many
are many ways to attack the RDMAP/DDP protocol if the transport ways to attack the RDMAP/DDP protocol if the transport protocol is
protocol is assumed to be vulnerable. For example, for Tagged assumed to be vulnerable. For example, for Tagged Messages, this
Messages, this entails guessing the STag and TO values. If the entails guessing the STag and TO values. If the attacker wishes to
attacker wishes to simply terminate the connection, it can do so simply terminate the connection, it can do so by correctly guessing
by correctly guessing the transport & network layer values, and the transport and network layer values, and providing an invalid
providing an invalid STag. Per the DDP specification, if an STag. Per the DDP specification, if an invalid STag is received, the
invalid STag is received, the Stream is torn down and the Remote Stream is torn down and the Remote Peer is notified with an error.
Peer is notified with an error. If an attacker wishes to If an attacker wishes to overwrite an Advertised Buffer, it must
overwrite an Advertised Buffer, it must successfully guess the successfully guess the correct STag and TO. Given that the TO will
correct STag and TO. Given that the TO often will start at zero, often start at zero, this is straightforward. The value of the STag
this is straightforward. The value of the STag should be chosen should be chosen at random, as discussed in Section 6.1.1, Using an
at random, as discussed in Section 6.1.1 Using an STag on a STag on a Different Stream. For Untagged Messages, if the MSN is
Different Stream. For Untagged Messages, if the MSN is invalid invalid then the connection may be torn down. If it is valid, then
then the connection may be torn down. If it is valid, then the the receive buffers can be corrupted.
receive buffers can be corrupted.
End-to-end authentication (e.g. IPsec or ULP authentication) End-to-end authentication (e.g., IPsec or ULP authentication)
provides protection against either the blind attack or the provides protection against either the blind attack or the connected
connected attack. attack.
5.1.2 Stream Hijacking 5.1.2. Stream Hijacking
Stream hijacking happens when a network based attacker eavesdrops Stream hijacking happens when a network-based attacker eavesdrops on
the LLP connection through the Stream establishment phase, and the LLP connection through the Stream establishment phase, and waits
waits until the authentication phase (if such a phase exists) is until the authentication phase (if such a phase exists) is completed
completed successfully. The attacker then spoofs the IP address successfully. The attacker then spoofs the IP address and re-directs
and re-directs the Stream from the victim to its own machine. For the Stream from the victim to its own machine. For example, an
example, an attacker can wait until an iSCSI authentication is attacker can wait until an iSCSI authentication is completed
completed successfully, and then hijack the iSCSI Stream. successfully, and then hijack the iSCSI Stream.
The best protection against this form of attack is end-to-end The best protection against this form of attack is end-to-end
integrity protection and authentication, such as IPsec, to integrity protection and authentication, such as IPsec, to prevent
prevent spoofing. Another option is to provide a physically spoofing. Another option is to provide a physically segregated
segregated network for security. Discussion of physical security network for security. Discussion of physical security is out of
is out of scope for this document. scope for this document.
Because the connection and/or Stream itself is established by the Because the connection and/or Stream itself is established by the
LLP, some LLPs are more difficult to hijack than others. Please LLP, some LLPs are more difficult to hijack than others. Please see
see the relevant LLP documentation on security issues around the relevant LLP documentation on security issues around connection
connection and/or Stream hijacking. and/or Stream hijacking.
5.1.3 Man-in-the-Middle Attack 5.1.3. Man-in-the-Middle Attack
If a network based attacker has the ability to delete or modify If a network-based attacker has the ability to delete or modify
packets which will still be accepted by the LLP (e.g., TCP packets that will still be accepted by the LLP (e.g., TCP sequence
sequence number is correct) then the Stream can be exposed to a number is correct), then the Stream can be exposed to a man-in-the-
man-in-the-middle attack. One style of attack is for the man-in- middle attack. One style of attack is for the man-in-the-middle to
the-middle to send Tagged Messages (either RDMAP or DDP). If it send Tagged Messages (either RDMAP or DDP). If it can discover a
can discover a buffer that has been exposed for STag enabled buffer that has been exposed for STag enabled access, then the man-
access, then the man-in-the-middle can use an RDMA Read operation in-the-middle can use an RDMA Read operation to read the contents of
to read the contents of the associated data buffer, perform an the associated Data Buffer, perform an RDMA Write Operation to modify
RDMA Write Operation to modify the contents of the associated the contents of the associated Data Buffer, or invalidate the STag to
data buffer, or invalidate the STag to disable further access to disable further access to the buffer.
the buffer.
The best protection against this form of attack is end-to-end The best protection against this form of attack is end-to-end
integrity protection and authentication, such as IPsec, to integrity protection and authentication, such as IPsec, to prevent
prevent spoofing or tampering. If authentication and integrity spoofing or tampering. If authentication and integrity protections
protections are not used, then physical protection must be are not used, then physical protection must be employed to prevent
employed to prevent man-in-the-middle attacks. man-in-the-middle attacks.
Because the connection/Stream itself is established by the LLP, Because the connection/Stream itself is established by the LLP, some
some LLPs are more exposed to man-in-the-middle attack than LLPs are more exposed to man-in-the-middle attack than others.
others. Please see the relevant LLP documentation on security Please see the relevant LLP documentation on security issues around
issues around connection and/or Stream hijacking. connection and/or Stream hijacking.
Another approach is to restrict access to only the local Another approach is to restrict access to only the local subnet/link,
subnet/link, and provide some mechanism to limit access, such as and provide some mechanism to limit access, such as physical security
physical security or 802.1.x. This model is an extremely limited or 802.1.x. This model is an extremely limited deployment scenario,
deployment scenario, and will not be further examined here. and will not be further examined here.
5.2 Tampering - Network based modification of buffer content 5.2. Tampering - Network-Based Modification of Buffer Content
This is actually a man in the middle attack - but only on the This is actually a man-in-the-middle attack, but only on the content
content of the buffer, as opposed to the man in the middle attack of the buffer, as opposed to the man-in-the-middle attack presented
presented above, where both the signaling and content can be above, where both the signaling and content can be modified. See
modified. See Section 5.1.3 Man-in-the-Middle Attack. Section 5.1.3, Man-in-the-Middle Attack.
5.3 Information Disclosure - Network Based Eavesdropping 5.3. Information Disclosure - Network-Based Eavesdropping
An attacker that is able to eavesdrop on the network can read the An attacker that is able to eavesdrop on the network can read the
content of all read and write accesses to a Peer's buffers. To content of all read and write accesses to a Peer's buffers. To
prevent information disclosure, the read/written data must be prevent information disclosure, the read/written data must be
encrypted. See also Section 5.1.3 Man-in-the-Middle Attack. The encrypted. See also Section 5.1.3, Man-in-the-Middle Attack. The
encryption can be done either by the ULP, or by a protocol that encryption can be done either by the ULP, or by a protocol that can
can provide security services to RDMAP & DDP (e.g. IPsec). provide security services to RDMAP and DDP (e.g., IPsec).
5.4 Specific Requirements for Security Services 5.4. Specific Requirements for Security Services
Generally speaking, Stream confidentiality protects against Generally speaking, Stream confidentiality protects against
eavesdropping. Stream and/or session authentication and integrity eavesdropping. Stream and/or session authentication and integrity
protection is a counter measurement against various spoofing and protection is a counter measurement against various spoofing and
tampering attacks. The effectiveness of authentication and tampering attacks. The effectiveness of authentication and integrity
integrity against a specific attack depend on whether the against a specific attack depends on whether the authentication is
authentication is machine level authentication (such as IPsec), machine level authentication (such as IPsec), or ULP authentication.
or ULP authentication.
5.4.1 Introduction to Security Options 5.4.1. Introduction to Security Options
The following security services can be applied to an RDMAP/DDP The following security services can be applied to an RDMAP/DDP
Stream: Stream:
1. Session confidentiality - protects against eavesdropping 1. Session confidentiality - Protects against eavesdropping (Section
(Section 5.3). 5.3).
2. Per-packet data source authentication - protects against the 2. Per-packet data source authentication - Protects against the
following spoofing attacks: network based impersonation following spoofing attacks: network-based impersonation (Section
(Section 5.1.1), Stream hijacking (Section 5.1.2), and man in 5.1.1) and Stream hijacking (Section 5.1.2).
the middle (Section 5.1.3).
3. Per-packet integrity - protects against tampering done by 3. Per-packet integrity - Protects against tampering done by
network based modification of buffer content (Section 5.2) network-based modification of buffer content (Section 5.2) and
when combined with authentication, also protects against man-in-
the-middle attacks (Section 5.1.3).
4. Packet sequencing - protects against replay attacks, which is 4. Packet sequencing - protects against replay attacks, which is a
a special case of the above tampering attack. special case of the above tampering attack.
If an RDMAP/DDP Stream may be subject to impersonation attacks, If an RDMAP/DDP Stream may be subject to impersonation attacks, or
or Stream hijacking attacks, it is recommended that the Stream be Stream hijacking attacks, it is recommended that the Stream be
authenticated, integrity protected, and protected from replay authenticated, integrity protected, and protected from replay
attacks; it may use confidentiality protection to protect from attacks; it may use confidentiality protection to protect from
eavesdropping (in case the RDMAP/DDP Stream traverses a public eavesdropping (in case the RDMAP/DDP Stream traverses a public
network). network).
IPsec is a protocol suite which is used to secure communication IPsec is a protocol suite that is used to secure communication at the
at the network layer between two peers. The IPsec protocol suite network layer between two peers. The IPsec protocol suite is
is specified within the IP Security Architecture [RFC2401], IKE specified within the IP Security Architecture [RFC2401], IKE
[RFC2409], IPsec Authentication Header (AH) [RFC2402] and IPsec [RFC2409], IPsec Authentication Header (AH) [RFC2402], and IPsec
Encapsulating Security Payload (ESP) [RFC2406] documents. IKE is Encapsulating Security Payload (ESP) [RFC2406] documents. IKE is the
the key management protocol while AH and ESP are used to protect key management protocol, while AH and ESP are used to protect IP
IP traffic. Please see those RFCs for a complete description of traffic. Please see those RFCs for a complete description of the
the respective protocols. respective protocols.
IPsec is capable of providing the above security services for IP IPsec is capable of providing the above security services for IP and
and TCP traffic respectively. ULP protocols are able to provide TCP traffic, respectively. ULP protocols are able to provide only
only part of the above security services. part of the above security services.
5.4.2 TLS is Inappropriate for DDP/RDMAP Security 5.4.2. TLS Is Inappropriate for DDP/RDMAP Security
TLS [RFC 2246] provides Stream authentication, integrity and TLS [RFC4346] provides Stream authentication, integrity and
confidentiality for TCP based ULPs. TLS supports one-way (server confidentiality for TCP based ULPs. TLS supports one-way (server
only) or mutual certificates based authentication. only) or mutual certificates based authentication.
If TLS is layered underneath RDMAP, there are at least two If TLS is layered underneath RDMAP, TLS's connection orientation
limitations that make TLS inappropriate for DDP/RDMA security: makes TLS inappropriate for DDP/RDMA security. If a stream cipher or
block cipher in CBC mode is used for bulk encryption, then a packet
1. The maximum length supported by the TLS record layer protocol can be decrypted only after all the packets preceding it have already
is 2^14 bytes - longer packets must be fragmented (as a arrived. If TLS is used to protect DDP/RDMAP traffic, then TCP must
comparison, the maximum length of an Untagged DDP Message is gather all out-of-order packets before TLS can decrypt them. Only
roughly 2^32). after this is done can RDMAP/DDP place them into the ULP buffer.
Thus, one of the primary features of DDP/RDMAP - enabling
2. TLS is a connection oriented protocol. If a stream cipher or implementations to have a flow-through architecture with little to no
block cipher in CBC mode is used for bulk encryption, then a buffering - cannot be achieved if TLS is used to protect the data
packet can be decrypted only after all the packets preceding stream.
it have already arrived. If TLS is used to protect DDP/RDMAP
traffic, then TCP must gather all out-of-order packets before
TLS can decrypt them. Only after this is done can RDMAP/DDP
place them into the ULP buffer. Thus one of the primary
features of DDP/RDMAP - enabling implementations to have a
flow-through architecture with little to no buffering, can
not be achieved if TLS is used to protect the data stream.
If TLS is layered on top of RDMAP or DDP, TLS does not protect If TLS is layered on top of RDMAP or DDP, TLS does not protect the
the RDMAP and/or DDP headers. Thus a man-in-the-middle attack can RDMAP and/or DDP headers. Thus, a man-in-the-middle attack can still
still occur by modifying the RDMAP/DDP header to incorrectly occur by modifying the RDMAP/DDP header to place the data into the
place the data into the wrong buffer, thus effectively corrupting wrong buffer, thus effectively corrupting the data stream.
the data stream.
For these reasons, it is not RECOMMENDED that TLS be layered on For these reasons, it is not RECOMMENDED that TLS be layered on top
top of RDMAP or DDP. of RDMAP or DDP.
5.4.3 DTLS and RDDP 5.4.3. DTLS and RDDP
DTLS [DTLS] provides security services for datagram protocols, DTLS [DTLS] provides security services for datagram protocols,
including unreliable datagram protocols. These services include including unreliable datagram protocols. These services include
anti-replay based on a mechanism adapted from IPsec that is anti-replay based on a mechanism adapted from IPsec that is intended
intended to operate on packets as they are received from the to operate on packets as they are received from the network. For
network. For these and other reasons, DTLS is best applied to these and other reasons, DTLS is best applied to RDDP by employing
RDDP by employing DTLS beneath TCP, yielding a layering of RDDP DTLS beneath TCP, yielding a layering of RDDP over TCP over DTLS over
over TCP over DTLS over UDP/IP. Such a layering inserts DTLS at UDP/IP. Such a layering inserts DTLS at roughly the same level in
roughly the same level in the protocol stack as IPsec, making the protocol stack as IPsec, making DTLS's security services an
DTLS's security services an alternative to IPsec's services from alternative to IPsec's services from an RDDP standpoint.
an RDDP standpoint.
For RDDP, IPsec is the better choice for a security framework, For RDDP, IPsec is the better choice for a security framework, and
and hence is mandatory-to-implement (as specified elsewhere in hence is mandatory-to-implement (as specified elsewhere in this
this document). An important contributing factor to the document). An important contributing factor to the specification of
specification of IPsec rather than DTLS is that the non-RDDP IPsec rather than DTLS is that the non-RDDP versions of two initial
versions of two initial adopters of RDDP (iSCSI [iSCSI][iSER] and adopters of RDDP (iSCSI [iSCSI][iSER] and NFSv4 [NFSv4][NFSv4.1]) are
NFSv4 [NFSv4][NFSv4.1]) are compatible with IPsec but neither of compatible with IPsec but neither of these protocols currently uses
these protocols currently uses either TLS or DTLS. For the either TLS or DTLS. For the specific case of iSCSI, IPsec is the
specific case of iSCSI, IPsec is the basis for mandatory-to- basis for mandatory-to-implement security services [RFC3723].
implement security services [RFC3723]. Therefore this document Therefore, this document and the RDDP protocol specifications contain
and the RDDP protocol specifications contain mandatory mandatory implementation requirements for IPsec rather than for DTLS.
implementation requirements for IPsec rather than for DTLS.
5.4.4 ULPs Which Provide Security 5.4.4. ULPs That Provide Security
ULPs which provide integrated security but wish to leverage ULPs that provide integrated security but wish to leverage lower-
lower-layer protocol security should be aware of security layer protocol security, should be aware of security concerns around
concerns around correlating a specific channel's security correlating a specific channel's security mechanisms to the
mechanisms to the authentication performed by the ULP. See authentication performed by the ULP. See [NFSv4CHANNEL] for
[NFSv4CHANNEL] for additional information on a promising approach additional information on a promising approach called "channel
called "channel binding". From [NFSv4CHANNEL]: binding". From [NFSv4CHANNEL]:
"The concept of channel bindings allows applications to "The concept of channel bindings allows applications to prove that
prove that the end-points of two secure channels at the end-points of two secure channels at different network layers
different network layers are the same by binding are the same by binding authentication at one channel to the
authentication at one channel to the session protection at session protection at the other channel. The use of channel
the other channel. The use of channel bindings allows bindings allows applications to delegate session protection to
applications to delegate session protection to lower layers, lower layers, which may significantly improve performance for some
which may significantly improve performance for some
applications." applications."
5.4.5 Requirements for IPsec Encapsulation of DDP 5.4.5. Requirements for IPsec Encapsulation of DDP
The IP Storage working group has spent significant time and The IP Storage working group has spent significant time and effort to
effort to define the normative IPsec requirements for IP Storage define the normative IPsec requirements for IP Storage [RFC3723].
[RFC3723]. Portions of that specification are applicable to a Portions of that specification are applicable to a wide variety of
wide variety of protocols, including the RDDP protocol suite. In protocols, including the RDDP protocol suite. In order not to
order to not replicate this effort, an RNIC implementation MUST replicate this effort, an RNIC implementation MUST follow the
follow the requirements defined in RFC3723 Section 2.3 and requirements defined in RFC 3723, Section 2.3 and Section 5,
Section 5, including the associated normative references for including the associated normative references for those sections.
those sections. Note that this means that support for IPSEC ESP Note that this means that support for IPSEC ESP mode is normative.
mode is normative.
Additionally, since IPsec acceleration hardware may only be able Additionally, since IPsec acceleration hardware may only be able to
to handle a limited number of active IKE Phase 2 SAs, Phase 2 handle a limited number of active IKE Phase 2 SAs, Phase 2 delete
delete messages may be sent for idle SAs, as a means of keeping messages may be sent for idle SAs as a means of keeping the number of
the number of active Phase 2 SAs to a minimum. The receipt of an active Phase 2 SAs to a minimum. The receipt of an IKE Phase 2
IKE Phase 2 delete message MUST NOT be interpreted as a reason delete message MUST NOT be interpreted as a reason for tearing down a
for tearing down a DDP/RDMA Stream. Rather, it is preferable to DDP/RDMA Stream. Rather, it is preferable to leave the Stream up,
leave the Stream up, and if additional traffic is sent on it, to and if additional traffic is sent on it, to bring up another IKE
bring up another IKE Phase 2 SA to protect it. This avoids the Phase 2 SA to protect it. This avoids the potential for continually
potential for continually bringing Streams up and down. bringing Streams up and down.
Note that there are serious security issues if IPsec is not Note that there are serious security issues if IPsec is not
implemented end-to-end. For example, if IPsec is implemented as a implemented end-to-end. For example, if IPsec is implemented as a
tunnel in the middle of the network, any hosts between the Peer tunnel in the middle of the network, any hosts between the Peer and
and the IPsec tunneling device can freely attack the unprotected the IPsec tunneling device can freely attack the unprotected Stream.
Stream.
6 Attacks from Remote Peers The IPsec requirements for RDDP are based on the version of IPsec
specified in RFC 2401 [RFC2401] and related RFCs, as profiled by RFC
3723 [RFC3723], despite the existence of a newer version of IPsec
specified in RFC 4301 [RFC4301] and related RFCs. One of the
important early applications of the RDDP protocols is their use with
iSCSI [iSER]; RDDP's IPsec requirements follow those of IPsec in
order to facilitate that usage by allowing a common profile of IPsec
to be used with iSCSI and the RDDP protocols. In the future, RFC
3723 may be updated to the newer version of IPsec; the IPsec security
requirements of any such update should apply uniformly to iSCSI and
the RDDP protocols.
This section describes remote attacks that are possible against 6. Attacks from Remote Peers
the RDMA system defined in Figure 1 - RDMA Security Model and the
RNIC Engine resources defined in Section 2.2. The analysis
includes a detailed description of each attack, what is being
attacked, and a description of the countermeasures that can be
taken to thwart the attack.
The attacks are classified into five categories: Spoofing, This section describes remote attacks that are possible against the
Tampering, Information Disclosure, Denial of Service (DoS) RDMA system defined in Figure 1 - RDMA Security Model and the RNIC
attacks, and Elevation of Privileges. As mentioned previously, Engine resources defined in Section 2.2. The analysis includes a
tampering is any modification of the legitimate traffic (machine detailed description of each attack, what is being attacked, and a
internal or network). A spoofing attack is a special case of description of the countermeasures that can be taken to thwart the
tampering where the attacker falsifies an identity of the Remote attack.
Peer (identity can be an IP address, machine name, ULP level
identity etc.).
6.1 Spoofing The attacks are classified into five categories: Spoofing, Tampering,
Information Disclosure, Denial of Service (DoS) attacks, and
Elevation of Privileges. As mentioned previously, tampering is any
modification of the legitimate traffic (machine internal or network).
A spoofing attack is a special case of tampering where the attacker
falsifies an identity of the Remote Peer (identity can be an IP
address, machine name, ULP level identity, etc.).
6.1. Spoofing
This section analyzes the various types of spoofing attacks This section analyzes the various types of spoofing attacks
applicable to RDMAP & DDP. Spoofing attacks can be launched by applicable to RDMAP and DDP. Spoofing attacks can be launched by the
the Remote Peer, or by a network based attacker. For Remote Peer or by a network-based attacker. For countermeasures
countermeasures against a network based attacker, see Section 5 against a network-based attacker, see Section 5, Attacks That Can Be
Attacks That Can be Mitigated With End-to-End Security. Mitigated with End-to-End Security.
6.1.1 Using an STag on a Different Stream 6.1.1. Using an STag on a Different Stream
One style of attack from the Remote Peer is for it to attempt to One style of attack from the Remote Peer is for it to attempt to use
use STag values that it is not authorized to use. Note that if STag values that it is not authorized to use. Note that if the
the Remote Peer sends an invalid STag to the Local Peer, per the Remote Peer sends an invalid STag to the Local Peer, per the DDP and
DDP and RDMAP specifications, the Stream must be torn down. Thus RDMAP specifications, the Stream must be torn down. Thus, the threat
the threat exists if an STag has been enabled for Remote Access exists if an STag has been enabled for Remote Access on one Stream
on one Stream and a Remote Peer is able to use it on an unrelated and a Remote Peer is able to use it on an unrelated Stream. If the
Stream. If the attack is successful, the attacker could attack is successful, the attacker could potentially be able to
potentially be able to perform either RDMA Read Operations to either perform RDMA Read operations to read the contents of the
read the contents of the associated data buffer, perform RDMA associated Data Buffer, perform RDMA Write operations to modify the
Write Operations to modify the contents of the associated data contents of the associated data buffer, or invalidate the STag to
buffer, or to invalidate the STag to disable further access to disable further access to the buffer.
the buffer.
An attempt by a Remote Peer to access a buffer with an STag on a An attempt by a Remote Peer to access a buffer with an STag on a
different Stream in the same Protection Domain may or may not be different Stream in the same Protection Domain may or may not be an
an attack depending on whether resource sharing is intended (i.e. attack, depending on whether resource sharing is intended (i.e.,
whether the Streams shared Partial Mutual Trust or not). For some whether the Streams shared Partial Mutual Trust). For some ULPs,
ULPs, using an STag on multiple Streams within the same using an STag on multiple Streams within the same Protection Domain
Protection Domain could be desired behavior. For other ULPs, could be desired behavior. For other ULPs, attempting to use an STag
attempting to use an STag on a different Stream could be on a different Stream could be considered an attack. Since this
considered to be an attack. Since this varies by ULP, a ULP varies by ULP, a ULP typically would need to be able to control the
typically would need to be able to control the scope of the STag. scope of the STag.
In the case where an implementation does not share resources In the case where an implementation does not share resources between
between Streams (including STags), this attack can be defeated by Streams (including STags), this attack can be defeated by assigning
assigning each Stream to a different Protection Domain. Before each Stream to a different Protection Domain. Before allowing remote
allowing remote access to the buffer, the Protection Domain of access to the buffer, the Protection Domain of the Stream where the
the Stream where the access attempt was made is matched against access attempt was made is matched against the Protection Domain of
the Protection Domain of the STag. If the Protection Domains do the STag. If the Protection Domains do not match, access to the
not match, access to the buffer is denied, an error is generated, buffer is denied, an error is generated, and the RDMAP Stream
and the RDMAP Stream associated with the attacking Stream is associated with the attacking Stream is terminated.
terminated.
For implementations that share resources between multiple For implementations that share resources between multiple Streams, it
Streams, it may not be practical to separate each Stream into its may not be practical to separate each Stream into its own Protection
own Protection Domain. In this case, the ULP can still limit the Domain. In this case, the ULP can still limit the scope of any of
scope of any of the STags to a single Stream (if it is enabling the STags to a single Stream (if it is enabling it for remote
it for remote access). If the STag scope has been limited to a access). If the STag scope has been limited to a single Stream, any
single Stream, any attempt to use that STag on a different Stream attempt to use that STag on a different Stream will result in an
will result in an error, and the RDMAP Stream is terminated. error, and the RDMAP Stream is terminated.
Thus for implementations that do not share STags between Streams, Thus, for implementations that do not share STags between Streams,
each Stream MUST either be in a separate Protection Domain or the each Stream MUST either be in a separate Protection Domain or the
scope of an STag MUST be limited to a single Stream. scope of an STag MUST be limited to a single Stream.
An RNIC MUST ensure that a specific Stream in a specific An RNIC MUST ensure that a specific Stream in a specific Protection
Protection Domain can not access an STag in a different Domain cannot access an STag in a different Protection Domain.
Protection Domain.
An RNIC MUST ensure that if an STag is limited in scope to a An RNIC MUST ensure that, if an STag is limited in scope to a single
single Stream, no other Stream can use the STag. Stream, no other Stream can use the STag.
An additional issue may be unintended sharing of STags (i.e. a An additional issue may be unintended sharing of STags (i.e., a bug
bug in the ULP) or a bug in the Remote Peer which causes an off- in the ULP) or a bug in the Remote Peer that causes an off-by-one
by-one STag to be used. For additional protection, an STag to be used. For additional protection, an implementation should
implementation should allocate STags in such a fashion that it is allocate STags in such a fashion that it is difficult to predict the
difficult to predict the next allocated STag number, and also next allocated STag number, and also ensure that STags are reused at
ensure that STags are reused at as slow a rate as possible. Any as slow a rate as possible. Any allocation method that would lead to
allocation method which would lead to intentional or intentional or unintentional reuse of an STag by the peer should be
unintentional reuse of an STag by the peer should be avoided avoided (e.g., a method that always starts with a given STag and
(e.g. a method which always starts with a given STag and monotonically increases it for each new allocation, or a method that
monotonically increases it for each new allocation, or a method always uses the same STag for each operation).
which always uses the same STag for each operation).
6.2 Tampering 6.2. Tampering
A Remote Peer or a network based attacker can attempt to tamper A Remote Peer or a network-based attacker can attempt to tamper with
with the contents of data buffers on a Local Peer that have been the contents of Data Buffers on a Local Peer that have been enabled
enabled for remote write access. The types of tampering attacks for remote write access. The types of tampering attacks from a
from a Remote Peer are outlined in the sections that follow. For Remote Peer are outlined in the sections that follow. For
countermeasures against a network based attacker, see Section 5 countermeasures against a network-based attacker, see Section 5,
Attacks That Can be Mitigated With End-to-End Security. Attacks That Can Be Mitigated with End-to-End Security.
6.2.1 Buffer Overrun - RDMA Write or Read Response 6.2.1. Buffer Overrun - RDMA Write or Read Response
This attack is an attempt by the Remote Peer to perform an RDMA This attack is an attempt by the Remote Peer to perform an RDMA Write
Write or RDMA Read Response to memory outside of the valid length or RDMA Read Response to memory outside of the valid length range of
range of the data buffer enabled for remote write access. This the Data Buffer enabled for remote write access. This attack can
attack can occur even when no resources are shared across occur even when no resources are shared across Streams. This issue
Streams. This issue can also arise if the ULP has a bug. can also arise if the ULP has a bug.
The countermeasure for this type of attack must be in the RNIC The countermeasure for this type of attack must be in the RNIC
implementation, leveraging the STag. When the local ULP specifies implementation, leveraging the STag. When the local ULP specifies to
to the RNIC the base address and the number of bytes in the the RNIC the base address and the umber of bytes in the buffer that
buffer that it wishes to make accessible, the RNIC must ensure it wishes to make accessible, the RNIC must ensure that the base and
that the base and bounds check are applied to any access to the bounds check are applied to any access to the buffer referenced by
buffer referenced by the STag before the STag is enabled for the STag before the STag is enabled for access. When an RDMA data
access. When an RDMA data transfer operation (which includes an transfer operation (which includes an STag) arrives on a Stream, a
STag) arrives on a Stream, a base and bounds byte granularity base and bounds byte granularity access check must be performed to
access check must be performed to ensure the operation accesses ensure that the operation accesses only memory locations within the
only memory locations within the buffer described by that STag. buffer described by that STag.
Thus an RNIC implementation MUST ensure that a Remote Peer is not Thus an RNIC implementation MUST ensure that a Remote Peer is not
able to access memory outside of the buffer specified when the able to access memory outside of the buffer specified when the STag
STag was enabled for remote access. was enabled for remote access.
6.2.2 Modifying a Buffer After Indication 6.2.2. Modifying a Buffer after Indication
This attack can occur if a Remote Peer attempts to modify the This attack can occur if a Remote Peer attempts to modify the
contents of an STag referenced buffer by performing an RDMA Write contents of an STag referenced buffer by performing an RDMA Write or
or an RDMA Read Response after the Remote Peer has indicated to an RDMA Read Response after the Remote Peer has indicated to the
the Local Peer or local ULP (by a variety of means) that the STag Local Peer or local ULP (by a variety of means) that the STag Data
data buffer contents are ready for use. This attack can occur Buffer contents are ready for use. This attack can occur even when
even when no resources are shared across Streams. Note that a bug no resources are shared across Streams. Note that a bug in a Remote
in a Remote Peer, or network based tampering, could also result Peer, or network-based tampering, could also result in this problem.
in this problem.
For example, assume the STag referenced buffer contains ULP For example, assume that the STag referenced buffer contains ULP
control information as well as ULP payload, and the ULP sequence control information as well as ULP payload, and the ULP sequence of
of operation is to first validate the control information and operation is to first validate the control information and then
then perform operations on the control information. If the Remote perform operations on the control information. If the Remote Peer
Peer can perform an additional RDMA Write or RDMA Read Response can perform an additional RDMA Write or RDMA Read Response (thus,
(thus changing the buffer) after the validity checks have been changing the buffer) after the validity checks have been completed
completed but before the control data is operated on, the Remote but before the control data is operated on, the Remote Peer could
Peer could force the ULP down operational paths that were never force the ULP down operational paths that were never intended.
intended.
The local ULP can protect itself from this type of attack by The local ULP can protect itself from this type of attack by revoking
revoking remote access when the original data transfer has remote access when the original data transfer has completed and
completed and before it validates the contents of the buffer. The before it validates the contents of the buffer. The local ULP can do
local ULP can either do this by explicitly revoking remote access this either by explicitly revoking remote access rights for the STag
rights for the STag when the Remote Peer indicates the operation when the Remote Peer indicates the operation has completed, or by
has completed, or by checking to make sure the Remote Peer checking to make sure the Remote Peer invalidated the STag through
invalidated the STag through the RDMAP Remote Invalidate the RDMAP Remote Invalidate capability. If the Remote Peer did not
capability (see Section 6.4.5 Remote Invalidate an STag Shared on invalidate the STag, the local ULP then explicitly revokes the STag
Multiple Streams for a definition of Remote Invalidate), and if remote access rights. (See Section 6.4.5, Remote Invalidate an STag
it did not, the local ULP then explicitly revokes the STag remote Shared on Multiple Streams for a definition of Remote Invalidate.)
access rights.
The local ULP SHOULD follow the above procedure to protect the The local ULP SHOULD follow the above procedure to protect the buffer
buffer before it validates the contents of the buffer (or uses before it validates the contents of the buffer (or uses the buffer in
the buffer in any way). any way).
An RNIC MUST ensure that network packets using the STag for a An RNIC MUST ensure that network packets using the STag for a
previously advertised buffer can no longer modify the buffer previously Advertised Buffer can no longer modify the buffer after
after the ULP revokes remote access rights for the specific STag. the ULP revokes remote access rights for the specific STag.
6.2.3 Multiple STags to access the same buffer 6.2.3. Multiple STags to Access the Same Buffer
See Section 6.3.6 Using Multiple STags Which Alias to the Same See Section 6.3.6 Using Multiple STags That Alias to the Same Buffer,
Buffer for this analysis. for this analysis.
6.3 Information Disclosure 6.3. Information Disclosure
The main potential source for information disclosure is through a The main potential source for information disclosure is through a
local buffer that has been enabled for remote access. If the local buffer that has been enabled for remote access. If the buffer
buffer can be probed by a Remote Peer on another Stream, then can be probed by a Remote Peer on another Stream, then there is
there is potential for information disclosure. potential for information disclosure.
The potential attacks that could result in unintended information The potential attacks that could result in unintended information
disclosure and countermeasures are detailed in the following disclosure and countermeasures are detailed in the following
sections. sections.
6.3.1 Probing memory outside of the buffer bounds 6.3.1. Probing Memory Outside of the Buffer Bounds
This is essentially the same attack as described in Section 6.2.1 This is essentially the same attack as described in Section 6.2.1,
Buffer Overrun - RDMA Write or Read Response, except an RDMA Read Buffer Overrun - RDMA Write or Read Response, except that an RDMA
Request is used to mount the attack. The same countermeasure Read Request is used to mount the attack. The same countermeasure
applies. applies.
6.3.2 Using RDMA Read to Access Stale Data 6.3.2. Using RDMA Read to Access Stale Data
If a buffer is being used for some combination of reads and If a buffer is being used for some combination of reads and writes
writes (either remote or local), and is exposed to a Remote Peer (either remote or local), and is exposed to a Remote Peer with at
with at least remote read access rights before it is initialized least remote read access rights before it is initialized with the
with the correct data, there is a potential race condition where correct data, there is a potential race condition where the Remote
the Remote Peer can view the prior contents of the buffer. This Peer can view the prior contents of the buffer. This becomes a
becomes a security issue if the prior contents of the buffer were security issue if the prior contents of the buffer were not intended
not intended to be shared with the Remote Peer. to be shared with the Remote Peer.
To eliminate this race condition, the local ULP SHOULD ensure To eliminate this race condition, the local ULP SHOULD ensure that no
that no stale data is contained in the buffer before remote read stale data is contained in the buffer before remote read access
access rights are granted (this can be done by zeroing the rights are granted (this can be done by zeroing the contents of the
contents of the memory, for example). This ensures that the memory, for example). This ensures that the Remote Peer cannot
Remote Peer can not access the buffer until the stale data has access the buffer until the stale data has been removed.
been removed.
6.3.3 Accessing a Buffer After the Transfer 6.3.3. Accessing a Buffer after the Transfer
If the Remote Peer has remote read access to a buffer, and by If the Remote Peer has remote read access to a buffer and, by some
some mechanism tells the local ULP that the transfer has been mechanism, tells the local ULP that the transfer has been completed,
completed, but the local ULP does not disable remote access to but the local ULP does not disable remote access to the buffer before
the buffer before modifying the data, it is possible for the modifying the data, it is possible for the Remote Peer to retrieve
Remote Peer to retrieve the new data. the new data.
This is similar to the attack defined in Section 6.2.2 Modifying This is similar to the attack defined in Section 6.2.2, Modifying a
a Buffer After Indication. The same countermeasures apply. In Buffer after Indication. The same countermeasures apply. In
addition, the local ULP SHOULD grant remote read access rights addition, the local ULP SHOULD grant remote read access rights only
only for the amount of time needed to retrieve the data. for the amount of time needed to retrieve the data.
6.3.4 Accessing Unintended Data With a Valid STag 6.3.4. Accessing Unintended Data with a Valid STag
If the ULP enables remote access to a buffer using an STag that If the ULP enables remote access to a buffer using an STag that
references the entire buffer, but intends only a portion of the references the entire buffer, but intends only a portion of the
buffer to be accessed, it is possible for the Remote Peer to buffer to be accessed, it is possible for the Remote Peer to access
access the other parts of the buffer anyway. the other parts of the buffer anyway.
To prevent this attack, the ULP SHOULD set the base and bounds of To prevent this attack, the ULP SHOULD set the base and bounds of the
the buffer when the STag is initialized to expose only the data buffer when the STag is initialized to expose only the data to be
to be retrieved. retrieved.
6.3.5 RDMA Read into an RDMA Write Buffer 6.3.5. RDMA Read into an RDMA Write Buffer
One form of disclosure can occur if the access rights on the One form of disclosure can occur if the access rights on the buffer
buffer enabled remote read, when only remote write access was enabled remote read, when only remote write access was intended. If
intended. If the buffer contained ULP data, or data from a the buffer contained ULP data, or data from a transfer on an
transfer on an unrelated Stream, the Remote Peer could retrieve unrelated Stream, the Remote Peer could retrieve the data through an
the data through an RDMA Read operation. Note that an RNIC RDMA Read operation. Note that an RNIC implementation is not
implementation is not required to support STags that have both required to support STags that have both read and write access.
read and write access.
The most obvious countermeasure for this attack is to not grant The most obvious countermeasure for this attack is to not grant
remote read access if the buffer is intended to be write-only. remote read access if the buffer is intended to be write-only. Then
Then the Remote Peer would not be able to retrieve data the Remote Peer would not be able to retrieve data associated with
associated with the buffer. An attempt to do so would result in the buffer. An attempt to do so would result in an error and the
an error and the RDMAP Stream associated with the Stream would be RDMAP Stream associated with the Stream would be terminated.
terminated.
Thus if a ULP only intends a buffer to be exposed for remote Thus, if a ULP only intends a buffer to be exposed for remote write
write access, it MUST set the access rights to the buffer to only access, it MUST set the access rights to the buffer to only enable
enable remote write access. Note that this requirement is not remote write access. Note that this requirement is not meant to
meant to restrict the use of zero-length RDMA Reads. Zero-length restrict the use of zero-length RDMA Reads. Zero-length RDMA Reads
RDMA Reads do not expose ULP data. Because they are intended to do not expose ULP data. Because they are intended to be used as a
be used as a mechanism to ensure that all RDMA Writes have been mechanism to ensure that all RDMA Writes have been received, and do
received, and do not even require a valid STag, their use is not even require a valid STag, their use is permitted even if a
permitted even if a buffer has only been enabled for write buffer has only been enabled for write access.
access.
6.3.6 Using Multiple STags Which Alias to the Same Buffer 6.3.6. Using Multiple STags That Alias to the Same Buffer
Multiple STags which alias to the same buffer at the same time Multiple STags that alias to the same buffer at the same time can
can result in unintentional information disclosure if the STags result in unintentional information disclosure if the STags are used
are used by different, mutually untrusted, Remote Peers. This by different, mutually untrusted Remote Peers. This model applies
model applies specifically to client/server communication, where specifically to client/server communication, where the server is
the server is communicating with multiple clients, each of which communicating with multiple clients, each of which do not mutually
do not mutually trust each other. trust each other.
If only read access is enabled, then the local ULP has complete If only read access is enabled, then the local ULP has complete
control over information disclosure. Thus a server which intended control over information disclosure. Thus, a server that intended to
to expose the same data (i.e. buffer) to multiple clients by expose the same data (i.e., buffer) to multiple clients by using
using multiple STags to the same buffer creates no new security multiple STags to the same buffer creates no new security issues
issues beyond what has already been described in this document. beyond what has already been described in this document. Note that
Note that if the server did not intend to expose the same data to if the server did not intend to expose the same data to the clients,
the clients, it should use separate buffers for each client (and it should use separate buffers for each client (and separate STags).
separate STags).
When one STag has remote read access enabled and a different STag When one STag has remote read access enabled and a different STag has
has remote write access enabled to the same buffer, it is remote write access enabled to the same buffer, it is possible for
possible for one Remote Peer to view the contents that have been one Remote Peer to view the contents that have been written by
written by another Remote Peer. another Remote Peer.
If both STags have remote write access enabled and the two Remote If both STags have remote write access enabled and the two Remote
Peers do not mutually trust each other, it is possible for one Peers do not mutually trust each other, it is possible for one Remote
Remote Peer to overwrite the contents that have been written by Peer to overwrite the contents that have been written by the other
the other Remote Peer. Remote Peer.
Thus a ULP with multiple Remote Peers which do not share Partial Thus, a ULP with multiple Remote Peers that do not share Partial
Mutual Trust MUST NOT grant write access to the same buffer Mutual Trust MUST NOT grant write access to the same buffer through
through different STags. A buffer should be exposed to only one different STags. A buffer should be exposed to only one untrusted
untrusted Remote Peer at a time to ensure that no information Remote Peer at a time to ensure that no information disclosure or
disclosure or information tampering occurs between peers. information tampering occurs between peers.
6.4 Denial of Service (DOS) 6.4. Denial of Service (DOS)
A DOS attack is one of the primary security risks of RDMAP. This A DOS attack is one of the primary security risks of RDMAP. This is
is because RNIC resources are valuable and scarce, and many ULP because RNIC resources are valuable and scarce, and many ULP
environments require communication with untrusted Remote Peers. environments require communication with untrusted Remote Peers. If
If the Remote Peer can be authenticated or the ULP payload the Remote Peer can be authenticated or the ULP payload encrypted,
encrypted, clearly, the DOS profile can be reduced. For the clearly, the DOS profile can be reduced. For the purposes of this
purposes of this analysis, it is assumed that the RNIC must be analysis, it is assumed that the RNIC must be able to operate in
able to operate in untrusted environments, which are open to DOS untrusted environments, which are open to DOS-style attacks.
style attacks.
Denial of service attacks against RNIC resources are not the Denial of service attacks against RNIC resources are not the typical
typical unknown party spraying packets at a random host (such as unknown party spraying packets at a random host (such as a TCP SYN
a TCP SYN attack). Because the connection/Stream must be fully attack). Because the connection/Stream must be fully established
established (e.g. a 3 message transport layer handshake has (e.g., a 3-message transport layer handshake has occurred), the
occurred), the attacker must be able to both send and receive attacker must be able to both send and receive messages over that
messages over that connection/Stream, or be able to guess a valid connection/Stream, or be able to guess a valid packet on an existing
packet on an existing RDMAP Stream. RDMAP Stream.
This section outlines the potential attacks and the This section outlines the potential attacks and the countermeasures
countermeasures available for dealing with each attack. available for dealing with each attack.
6.4.1 RNIC Resource Consumption 6.4.1. RNIC Resource Consumption
This section covers attacks that fall into the general category This section covers attacks that fall into the general category of a
of a local ULP attempting to unfairly allocate scarce (i.e. local ULP attempting to unfairly allocate scarce (i.e., bounded) RNIC
bounded) RNIC resources. The local ULP may be attempting to resources. The local ULP may be attempting to allocate resources on
allocate resources on its own behalf, or on behalf of a Remote its own behalf, or on behalf of a Remote Peer. Resources that fall
Peer. Resources that fall into this category include: Protection into this category include Protection Domains, Stream Context Memory,
Domains, Stream Context Memory, Translation and Protection Translation and Protection Tables, and STag namespace. These can be
Tables, and STag namespace. These can be due to attacks by due to attacks by currently active local ULPs or ones that allocated
currently active local ULPs or ones that allocated resources resources earlier but are now idle.
earlier, but are now idle.
This type of attack can occur regardless of whether or not This type of attack can occur regardless of whether resources are
resources are shared across Streams. shared across Streams.
The allocation of all scarce resources MUST be placed under the The allocation of all scarce resources MUST be placed under the
control of a Privileged Resource Manager. This allows the control of a Privileged Resource Manager. This allows the Privileged
Privileged Resource Manager to: Resource Manager to:
* prevent a local ULP from allocating more than its fair * prevent a local ULP from allocating more than its fair share of
share of resources. resources.
* detect if a Remote Peer is attempting to launch a DOS * detect if a Remote Peer is attempting to launch a DOS attack by
attack by attempting to create an excessive number of attempting to create an excessive number of Streams (with
Streams (with associated resources) and take corrective associated resources) and take corrective action (such as
action (such as refusing the request or applying network refusing the request or applying network layer filters against
layer filters against the Remote Peer). the Remote Peer).
This analysis assumes that the Resource Manager is responsible This analysis assumes that the Resource Manager is responsible for
for handing out Protection Domains, and RNIC implementations will handing out Protection Domains, and that RNIC implementations will
provide enough Protection Domains to allow the Resource Manager provide enough Protection Domains to allow the Resource Manager to be
to be able to assign a unique Protection Domain for each able to assign a unique Protection Domain for each unrelated,
unrelated, untrusted local ULP (for a bounded, reasonable number untrusted local ULP (for a bounded, reasonable number of local ULPs).
of local ULPs). This analysis further assumes that the Resource This analysis further assumes that the Resource Manager implements
Manager implements policies to ensure that untrusted local ULPs policies to ensure that untrusted local ULPs are not able to consume
are not able to consume all of the Protection Domains through a all the Protection Domains through a DOS attack. Note that
DOS attack. Note that Protection Domain consumption cannot result Protection Domain consumption cannot result from a DOS attack
from a DOS attack launched by a Remote Peer, unless a local ULP launched by a Remote Peer, unless a local ULP is acting on the Remote
is acting on the Remote Peer's behalf. Peer's behalf.
6.4.2 Resource Consumption by Idle ULPs 6.4.2. Resource Consumption by Idle ULPs
The simplest form of a DOS attack given a fixed amount of The simplest form of a DOS attack, given a fixed amount of resources,
resources is for the Remote Peer to create a RDMAP Stream to a is for the Remote Peer to create an RDMAP Stream to a Local Peer,
Local Peer, and request dedicated resources then do no actual request dedicated resources, and then do no actual work. This allows
work. This allows the Remote Peer to be very light weight (i.e. the Remote Peer to be very light weight (i.e., only negotiate
only negotiate resources, but do no data transfer) and consumes a resources, but do no data transfer) and consumes a disproportionate
disproportionate amount of resources at the Local Peer. amount of resources at the Local Peer.
A general countermeasure for this style of attack is to monitor A general countermeasure for this style of attack is to monitor
active RDMAP Streams and if resources are getting low, reap the active RDMAP Streams and, if resources are getting low, to reap the
resources from RDMAP Streams that are not transferring data and resources from RDMAP Streams that are not transferring data and
possibly terminate the Stream. This would presumably be under possibly terminate the Stream. This would presumably be under
administrative control. administrative control.
Refer to Section 6.4.1 for the analysis and countermeasures for Refer to Section 6.4.1 for the analysis and countermeasures for this
this style of attack on the following RNIC resources: Stream style of attack on the following RNIC resources: Stream Context
Context Memory, Page Translation Tables and STag namespace. Memory, Page Translation Tables, and STag namespace.
Note that some RNIC resources are not at risk of this type of Note that some RNIC resources are not at risk of this type of attack
attack from a Remote Peer because an attack requires the Remote from a Remote Peer because an attack requires the Remote Peer to send
Peer to send messages in order to consume the resource. Receive messages in order to consume the resource. Receive Data Buffers,
Data Buffers, Completion Queue, and RDMA Read Request Queue Completion Queue, and RDMA Read Request Queue resources are examples.
resources are examples. These resources are, however, at risk These resources are, however, at risk from a local ULP that attempts
from a local ULP that attempts to allocate resources, then goes to allocate resources, then goes idle. This could also be created if
idle. This could also be created if the ULP negotiates the the ULP negotiates the resource levels with the Remote Peer, which
resource levels with the Remote Peer, which causes the Local Peer causes the Local Peer to consume resources; however, the Remote Peer
to consume resources, however the Remote Peer never sends data to never sends data to consume them. The general countermeasure
consume them. The general countermeasure described in this described in this section can be used to free resources allocated by
section can be used to free resources allocated by an idle Local an idle Local Peer.
Peer.
6.4.3 Resource Consumption By Active ULPs 6.4.3. Resource Consumption by Active ULPs
This section describes DOS attacks from Local and Remote Peers This section describes DOS attacks from Local and Remote Peers that
that are actively exchanging messages. Attacks on each RDMA NIC are actively exchanging messages. Attacks on each RDMA NIC resource
resource are examined and specific countermeasures are are examined and specific countermeasures are identified. Note that
identified. Note that attacks on Stream Context Memory, Page attacks on Stream Context Memory, Page Translation Tables, and STag
Translation Tables, and STag namespace are covered in Section namespace are covered in Section 6.4.1, RNIC Resource Consumption, so
6.4.1 RNIC Resource Consumption, so are not included here. they are not included here.
6.4.3.1 Multiple Streams Sharing Receive Buffers 6.4.3.1. Multiple Streams Sharing Receive Buffers
The Remote Peer can attempt to consume more than its fair share The Remote Peer can attempt to consume more than its fair share of
of receive data buffers (i.e. Untagged buffers for DDP are or receive Data Buffers (i.e., Untagged Buffers for DDP or Send Type
Send Type Messages for RDMAP) if receive buffers are shared Messages for RDMAP) if receive buffers are shared across multiple
across multiple Streams. Streams.
If resources are not shared across multiple Streams, then this If resources are not shared across multiple Streams, then this attack
attack is not possible because the Remote Peer will not be able is not possible because the Remote Peer will not be able to consume
to consume more buffers than were allocated to the Stream. The more buffers than were allocated to the Stream. The worst case
worst case scenario is that the Remote Peer can consume more scenario is that the Remote Peer can consume more receive buffers
receive buffers than the local ULP allowed, resulting in no than the local ULP allowed, resulting in no buffers being available,
buffers being available, which could cause the Remote Peer's which could cause the Remote Peer's Stream to the Local Peer to be
Stream to the Local Peer to be torn down, and all allocated torn down, and all allocated resources to be released.
resources to be released.
If local receive data buffers are shared among multiple Streams, If local receive Data Buffers are shared among multiple Streams, then
then the Remote Peer can attempt to consume more than its fair the Remote Peer can attempt to consume more than its fair share of
share of the receive buffers, causing a different Stream to be the receive buffers, causing a different Stream to be short of
short of receive buffers, thus possibly causing the other Stream receive buffers, and thus, possibly causing the other Stream to be
to be torn down. For example, if the Remote Peer sent enough one torn down. For example, if the Remote Peer sent enough one-byte
byte Untagged Messages, they might be able to consume all local Untagged Messages, they might be able to consume all locally shared,
shared receive queue resources with little effort on their part. receive queue resources with little effort on their part.
One method the Local Peer could use is to recognize that a Remote One method the Local Peer could use is to recognize that a Remote
Peer is attempting to use more than its fair share of resources Peer is attempting to use more than its fair share of resources and
and terminate the Stream (causing the allocated resources to be terminate the Stream (causing the allocated resources to be
released). However, if the Local Peer is sufficiently slow, it released). However, if the Local Peer is sufficiently slow, it may
may be possible for the Remote Peer to still mount a denial of be possible for the Remote Peer to still mount a denial of service
service attack. One countermeasure that can protect against this attack. One countermeasure that can protect against this attack is
attack is implementing a low-water notification. The low-water implementing a low-water notification. The low-water notification
notification alerts the ULP if the number of buffers in the alerts the ULP if the number of buffers in the receive queue is less
receive queue is less than a threshold. than a threshold.
If all of the following conditions are true, then the Local Peer If all the following conditions are true, then the Local Peer or
or local ULP can size the amount of local receive buffers posted local ULP can size the amount of local receive buffers posted on the
on the receive queue to ensure a DOS attack can be stopped. receive queue to ensure a DOS attack can be stopped.
* a low-water notification is enabled, and * A low-water notification is enabled, and
* the Local Peer is able to bound the amount of time that * The Local Peer is able to bound the amount of time that it takes
it takes to replenish receive buffers, and to replenish receive buffers, and
* the Local Peer maintains statistics to determine which * The Local Peer maintains statistics to determine which Remote
Remote Peer is consuming buffers. Peer is consuming buffers.
The above conditions enable the low-water notification to arrive The above conditions enable the low-water notification to arrive
before resources are depleted and thus the Local Peer or local before resources are depleted, and thus, the Local Peer or local ULP
ULP can take corrective action (e.g., terminate the Stream of the can take corrective action (e.g., terminate the Stream of the
attacking Remote Peer). attacking Remote Peer).
A different, but similar attack is if the Remote Peer sends a A different, but similar, attack is if the Remote Peer sends a
significant number of out-of-order packets and the RNIC has the significant number of out-of-order packets and the RNIC has the
ability to use the ULP buffer (i.e. the Untagged Buffer for DDP ability to use the ULP buffer (i.e., the Untagged Buffer for DDP or
or the buffer consumed by a Send Type Message for RDMAP) as a the buffer consumed by a Send Type Message for RDMAP) as a reassembly
reassembly buffer. In this case the Remote Peer can consume a buffer. In this case, the Remote Peer can consume a significant
significant number of ULP buffers, but never send enough data to number of ULP buffers, but never send enough data to enable the ULP
enable the ULP buffer to be completed to the ULP. buffer to be completed to the ULP.
An effective countermeasure is to create a high-water An effective countermeasure is to create a high-water notification
notification which alerts the ULP if there is more than a that alerts the ULP if there is more than a specified number of
specified number of receive buffers "in process" (partially receive buffers "in process" (partially consumed, but not completed).
consumed, but not completed). The notification is generated when The notification is generated when more than the specified number of
more than the specified number of buffers are in process buffers are in process simultaneously on a specific Stream (i.e.,
simultaneously on a specific Stream (i.e., packets have started packets have started to arrive for the buffer, but the buffer has not
to arrive for the buffer, but the buffer has not yet been yet been delivered to the ULP).
delivered to the ULP).
A different countermeasure is for the RNIC Engine to provide the A different countermeasure is for the RNIC Engine to provide the
capability to limit the Remote Peer's ability to consume receive capability to limit the Remote Peer's ability to consume receive
buffers on a per Stream basis. Unfortunately this requires a buffers on a per Stream basis. Unfortunately, this requires a large
large amount of state to be tracked in each RNIC on a per Stream amount of state to be tracked in each RNIC on a per Stream basis.
basis.
Thus, if an RNIC Engine provides the ability to share receive Thus, if an RNIC Engine provides the ability to share receive buffers
buffers across multiple Streams, the combination of the RNIC across multiple Streams, the combination of the RNIC Engine and the
Engine and the Privileged Resource Manager MUST be able to detect Privileged Resource Manager MUST be able to detect if the Remote Peer
if the Remote Peer is attempting to consume more than its fair is attempting to consume more than its fair share of resources so
share of resources so that the Local Peer or local ULP can apply that the Local Peer or local ULP can apply countermeasures to detect
countermeasures to detect and prevent the attack. and prevent the attack.
6.4.3.2 Remote or Local Peer Attacking a Shared CQ 6.4.3.2. Remote or Local Peer Attacking a Shared CQ
For an overview of the shared CQ attack model, see Section 7.1. For an overview of the shared CQ attack model, see Section 7.1.
The Remote Peer can attack a shared CQ by consuming more than its The Remote Peer can attack a shared CQ by consuming more than its
fair share of CQ entries by using one of the following methods: fair share of CQ entries by using one of the following methods:
* The ULP protocol allows the Remote Peer to cause the * The ULP protocol allows the Remote Peer to cause the local ULP to
local ULP to reserve a specified number of CQ entries, reserve a specified number of CQ entries, possibly leaving
possibly leaving insufficient entries for other Streams insufficient entries for other Streams that are sharing the CQ.
that are sharing the CQ.
* If the Remote Peer, Local Peer, or local ULP (or any * If the Remote Peer, Local Peer, or local ULP (or any combination)
combination) can attack the CQ by overwhelming the CQ can attack the CQ by overwhelming the CQ with completions, then
with completions, then completion processing on other completion processing on other Streams sharing that Completion
Streams sharing that Completion Queue can be affected Queue can be affected (e.g., the Completion Queue overflows and
(e.g. the Completion Queue overflows and stops stops functioning).
functioning).
The first method of attack can be avoided if the ULP does not The first method of attack can be avoided if the ULP does not allow a
allow a Remote Peer to reserve CQ entries or there is a trusted Remote Peer to reserve CQ entries, or if there is a trusted
intermediary such as a Privileged Resource Manager. Unfortunately intermediary, such as a Privileged Resource Manager. Unfortunately,
it is often unrealistic to not allow a Remote Peer to reserve CQ it is often unrealistic not to allow a Remote Peer to reserve CQ
entries - particularly if the number of completion entries is entries, particularly if the number of completion entries is
dependent on other ULP negotiated parameters, such as the amount dependent on other ULP negotiated parameters, such as the amount of
of buffering required by the ULP. Thus an implementation MUST buffering required by the ULP. Thus, an implementation MUST
implement a Privileged Resource Manager to control the allocation implement a Privileged Resource Manager to control the allocation of
of CQ entries. See Section 2.1 Components for a definition of CQ entries. See Section 2.1, Components, for a definition of a
Privileged Resource Manager. Privileged Resource Manager.
One way that a Local or Remote Peer can attempt to overwhelm a CQ One way that a Local or Remote Peer can attempt to overwhelm a CQ
with completions is by sending minimum length RDMAP/DDP Messages with completions is by sending minimum length RDMAP/DDP Messages to
to cause as many completions (receive completions for the Remote cause as many completions (receive completions for the Remote Peer,
Peer, send completions for the Local Peer) per second as send completions for the Local Peer) per second as possible. If it
possible. If it is the Remote Peer attacking, and we assume that is the Remote Peer attacking, and we assume that the Local Peer's
the Local Peer's receive queue(s) do not run out of receive receive queue(s) do not run out of receive buffers (if they do, then
buffers (if they do, then this is a different attack, documented this is a different attack, documented in Section 6.4.3.1 Multiple
in Section 6.4.3.1 Multiple Streams Sharing Receive Buffers), Streams Sharing Receive Buffers), then it might be possible for the
then it might be possible for the Remote Peer to consume more Remote Peer to consume more than its fair share of Completion Queue
than its fair share of Completion Queue entries. Depending upon entries. Depending upon the CQ implementation, this could either
the CQ implementation, this could either cause the CQ to overflow cause the CQ to overflow (if it is not large enough to handle all the
(if it is not large enough to handle all of the completions completions generated) or for another Stream not to be able to
generated) or for another Stream to not be able to generate CQ generate CQ entries (if the RNIC had flow control on generation of CQ
entries (if the RNIC had flow control on generation of CQ entries entries into the CQ). In either case, the CQ will stop functioning
into the CQ). In either case, the CQ will stop functioning correctly, and any Streams expecting completions on the CQ will stop
correctly and any Streams expecting completions on the CQ will functioning.
stop functioning.
This attack can occur regardless of whether all of the Streams This attack can occur regardless of whether all the Streams
associated with the CQ are in the same Protection Domain or are associated with the CQ are in the same or different Protection
in different Protection Domains - the key issue is that the Domains - the key issue is that the number of Completion Queue
number of Completion Queue entries is less than the number of all entries is less than the number of all outstanding operations that
outstanding operations that can cause a completion. can cause a completion.
The Local Peer can protect itself from this type of attack using The Local Peer can protect itself from this type of attack using
either of the following methods: either of the following methods:
* Size the CQ to the appropriate level, as specified below * Size the CQ to the appropriate level, as specified below (note
(note that if the CQ currently exists, and it needs to be that if the CQ currently exists and needs to be resized, resizing
resized, resizing the CQ is not required to succeed in the CQ is not required to succeed in all cases, so the CQ resize
all cases, so the CQ resize should be done before sizing should be done before sizing the Send Queue and Receive Queue on
the Send Queue and Receive Queue on the Stream), OR the Stream), OR
* Grant fewer resources than the Remote Peer requested (not * Grant fewer resources than the Remote Peer requested (not
supplying the number of Receive Data Buffers requested). supplying the number of Receive Data Buffers requested).
The proper sizing of the CQ is dependent on whether the local The proper sizing of the CQ is dependent on whether the local ULP(s)
ULP(s) will post as many resources to the various queues as the will post as many resources to the various queues as the size of the
size of the queue enables or not. If the local ULP(s) can be queue enables. If the local ULP(s) can be trusted to post a number
trusted to post a number of resources that is smaller than the of resources that is smaller than the size of the specific resource's
size of the specific resource's queue, then a correctly sized CQ queue, then a correctly sized CQ means that the CQ is large enough to
means that the CQ is large enough to hold completion status for hold completion status for all the outstanding Data Buffers (both
all of the outstanding Data Buffers (both send and receive send and receive buffers), or:
buffers), or:
CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ) CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ)
+ SUM(MaxPostedOnEachSRQ) + SUM(MaxPostedOnEachSRQ)
+ SUM(MaxPostedOnEachSQ) + SUM(MaxPostedOnEachSQ)
Where: Where:
MaxPostedOnEachRQ = the maximum number of requests which MaxPostedOnEachRQ = the maximum number of requests that
can cause a completion that will be posted on a can cause a completion that will be posted on a
specific Receive Queue. specific Receive Queue.
MaxPostedOnEachSRQ = the maximum number of requests which MaxPostedOnEachSRQ = the maximum number of requests that
can cause a completion that will be posted on a can cause a completion that will be posted on a
specific Shared Receive Queue. specific Shared Receive Queue.
MaxPostedOnEachSQ = the maximum number of requests which MaxPostedOnEachSQ = the maximum number of requests that
can cause a completion that will be posted on a can cause a completion that will be posted on a
specific Send Queue. specific Send Queue.
If the local ULP must be able to completely fill the queues, or If the local ULP must be able to completely fill the queues, or
can not be trusted to observe a limit smaller than the queues, cannot be trusted to observe a limit smaller than the queues, then
then the CQ must be sized to accommodate the maximum number of the CQ must be sized to accommodate the maximum number of operations
operations that it is possible to post at any one time. Thus the that it is possible to post at any one time. Thus, the equation
equation becomes: becomes:
CQ_MIN_SIZE = SUM(SizeOfEachRQ) CQ_MIN_SIZE = SUM(SizeOfEachRQ)
+ SUM(SizeOfEachSRQ) + SUM(SizeOfEachSRQ)
+ SUM(SizeOfEachSQ) + SUM(SizeOfEachSQ)
Where: Where:
SizeOfEachRQ = the maximum number of requests which SizeOfEachRQ = the maximum number of requests that
can cause a completion that can ever be posted can cause a completion that can ever be posted
on a specific Receive Queue. on a specific Receive Queue.
SizeOfEachSRQ = the maximum number of requests which SizeOfEachSRQ = the maximum number of requests that
can cause a completion that can ever be posted can cause a completion that can ever be posted
on a specific Shared Receive Queue. on a specific Shared Receive Queue.
SizeOfEachSQ = the maximum number of requests which SizeOfEachSQ = the maximum number of requests that
can cause a completion that can ever be posted can cause a completion that can ever be posted
on a specific Send Queue. on a specific Send Queue.
Where MaxPosted*OnEach*Q and SizeOfEach*Q varies on a per Stream MaxPosted*OnEach*Q and SizeOfEach*Q vary on a per Stream or per
or per Shared Receive Queue basis. Shared Receive Queue basis.
If the ULP is sharing a CQ across multiple Streams which do not If the ULP is sharing a CQ across multiple Streams that do not share
share Partial Mutual Trust, then the ULP MUST implement a Partial Mutual Trust, then the ULP MUST implement a mechanism to
mechanism to ensure that the Completion Queue can not overflow. ensure that the Completion Queue does not overflow. Note that it is
Note that it is possible to share CQs even if the Remote Peers possible to share CQs even if the Remote Peers accessing the CQs are
accessing the CQs are untrusted if either of the above two untrusted if either of the above two formulas are implemented. If
formulas are implemented. If the ULP can be trusted to not post the ULP can be trusted not to post more than MaxPostedOnEachRQ,
more than MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and MaxPostedOnEachSRQ, and MaxPostedOnEachSQ, then the first formula
MaxPostedOnEachSQ, then the first formula applies. If the ULP can applies. If the ULP cannot be trusted to obey the limit, then the
not be trusted to obey the limit, then the second formula second formula applies.
applies.
6.4.3.3 Attacking the RDMA Read Request Queue 6.4.3.3. Attacking the RDMA Read Request Queue
The RDMA Read Request Queue can be attacked if the Remote Peer The RDMA Read Request Queue can be attacked if the Remote Peer sends
sends more RDMA Read Requests than the depth of the RDMA Read more RDMA Read Requests than the depth of the RDMA Read Request Queue
Request Queue at the Local Peer. If the RDMA Read Request Queue at the Local Peer. If the RDMA Read Request Queue is a shared
is a shared resource, this could corrupt the queue. If the queue resource, this could corrupt the queue. If the queue is not shared,
is not shared, then the worst case is that the current Stream is then the worst case is that the current Stream is no longer
no longer functional (e.g. torn down). One approach to solving functional (e.g., torn down). One approach to solving the shared
the shared RDMA Read Request Queue would be to create thresholds, RDMA Read Request Queue would be to create thresholds, similar to
similar to those described in Section 6.4.3.1 Multiple Streams those described in Section 6.4.3.1, Multiple Streams Sharing Receive
Sharing Receive Buffers. A simpler approach is to not share RDMA Buffers. A simpler approach is to not share RDMA Read Request Queue
Read Request Queue resources among Streams or enforce hard limits resources among Streams or to enforce hard limits of consumption per
of consumption per Stream. Thus RDMA Read Request Queue resource Stream. Thus, RDMA Read Request Queue resource consumption MUST be
consumption MUST be controlled by the Privileged Resource Manager controlled by the Privileged Resource Manager such that RDMAP/DDP
such that RDMAP/DDP Streams which do not share Partial Mutual Streams that do not share Partial Mutual Trust do not share RDMA Read
Trust do not share RDMA Read Request Queue resources. Request Queue resources.
If the issue is a bug in the Remote Peer's implementation, but If the issue is a bug in the Remote Peer's implementation, but not a
not a malicious attack, the issue can be solved by requiring the malicious attack, the issue can be solved by requiring the Remote
Remote Peer's RNIC to throttle RDMA Read Requests. By properly Peer's RNIC to throttle RDMA Read Requests. By properly configuring
configuring the Stream at the Remote Peer through a trusted the Stream at the Remote Peer through a trusted agent, the RNIC can
agent, the RNIC can be made to not transmit RDMA Read Requests be made not to transmit RDMA Read Requests that exceed the depth of
that exceed the depth of the RDMA Read Request Queue at the Local the RDMA Read Request Queue at the Local Peer. If the Stream is
Peer. If the Stream is correctly configured, and if the Remote correctly configured, and if the Remote Peer submits more requests
Peer submits more requests than the Local Peer's RDMA Read than the Local Peer's RDMA Read Request Queue can handle, the
Request Queue can handle, the requests would be queued at the requests would be queued at the Remote Peer's RNIC until previous
Remote Peer's RNIC until previous requests complete. If the requests complete. If the Remote Peer's Stream is not configured
Remote Peer's Stream is not configured correctly, the RDMAP correctly, the RDMAP Stream is terminated when more RDMA Read
Stream is terminated when more RDMA Read Requests arrive at the Requests arrive at the Local Peer than the Local Peer can handle
Local Peer than the Local Peer can handle (assuming the prior (assuming that the prior paragraph's recommendation is implemented).
paragraph's recommendation is implemented). Thus an RNIC Thus, an RNIC implementation SHOULD provide a mechanism to cap the
implementation SHOULD provide a mechanism to cap the number of number of outstanding RDMA Read Requests. The configuration of this
outstanding RDMA Read Requests. The configuration of this limit limit is outside the scope of this document.
is outside the scope of this document.
6.4.4 Exercise of non-optimal code paths 6.4.4. Exercise of Non-Optimal Code Paths
Another form of DOS attack is to attempt to exercise data paths Another form of a DOS attack is to attempt to exercise data paths
that can consume a disproportionate amount of resources. An that can consume a disproportionate amount of resources. An example
example might be if error cases are handled on a "slow path" might be if error cases are handled on a "slow path" (consuming
(consuming either host or RNIC computational resources), and an either host or RNIC computational resources), and an attacker
attacker generates excessive numbers of errors in an attempt to generates excessive numbers of errors in an attempt to consume these
consume these resources. Note that for most RDMAP or DDP errors, resources. Note that for most RDMAP or DDP errors, the attacking
the attacking Stream will simply be torn down. Thus for this form Stream will simply be torn down. Thus, for this form of attack to be
of attack to be effective, the Remote Peer needs to exercise data effective, the Remote Peer needs to exercise data paths that do not
paths which do not cause the Stream to be torn down. cause the Stream to be torn down.
If an RNIC implementation contains "slow paths" which do not If an RNIC implementation contains "slow paths" that do not result in
result in the tear down of the Stream, it is recommended that an the tear down of the Stream, it is recommended that an implementation
implementation provide the ability to detect the above condition provide the ability to detect the above condition and allow an
and allow an administrator to act, including potentially administrator to act, including potentially administratively tearing
administratively tearing down the RDMAP Stream associated with down the RDMAP Stream associated with the Stream that is exercising
the Stream exercising data paths consuming a disproportionate data paths, which consume a disproportionate amount of resources.
amount of resources.
6.4.5 Remote Invalidate an STag Shared on Multiple Streams 6.4.5. Remote Invalidate an STag Shared on Multiple Streams
If a Local Peer has enabled an STag for remote access, the Remote If a Local Peer has enabled an STag for remote access, the Remote
Peer could attempt to remote invalidate the STag by using the Peer could attempt to remotely invalidate the STag by using the RDMAP
RDMAP Send with Invalidate or Send with SE and Invalidate Send with Invalidate or Send with SE and Invalidate Message. If the
Message. If the STag is only valid on the current Stream, then STag is only valid on the current Stream, then the only side effect
the only side effect is that the Remote Peer can no longer use is that the Remote Peer can no longer use the STag; thus, there are
the STag; thus there are no security issues. no security issues.
If the STag is valid across multiple Streams, then the Remote If the STag is valid across multiple Streams, then the Remote Peer
Peer can prevent other Streams from using that STag by using the can prevent other Streams from using that STag by using the Remote
remote invalidate functionality. Invalidate functionality.
Thus if RDDP Streams do not share Partial Mutual Trust (i.e. the Thus, if RDDP Streams do not share Partial Mutual Trust (i.e., the
Remote Peer may attempt to remote invalidate the STag Remote Peer may attempt to remotely invalidate the STag prematurely),
prematurely), the ULP MUST NOT enable an STag which would be the ULP MUST NOT enable an STag that would be valid across multiple
valid across multiple Streams. Streams.
6.4.6 Remote Peer attacking an Unshared CQ 6.4.6. Remote Peer Attacking an Unshared CQ
The Remote Peer can attack an unshared CQ if the Local Peer does The Remote Peer can attack an unshared CQ if the Local Peer does not
not size the CQ correctly. For example, if the Local Peer enables size the CQ correctly. For example, if the Local Peer enables the CQ
the CQ to handle completions of received buffers, and the receive to handle completions of received buffers, and the receive buffer
buffer queue is longer than the Completion Queue, then an queue is longer than the Completion Queue, then an overflow can
overflow can potentially occur. The effect on the attacker's potentially occur. The effect on the attacker's Stream is
Stream is catastrophic. However if an RNIC does not have the catastrophic. However, if an RNIC does not have the proper
proper protections in place, then an attack to overflow the CQ protections in place, then an attack to overflow the CQ can also
can also cause corruption and/or termination of an unrelated cause corruption and/or termination of an unrelated Stream. Thus, an
Stream. Thus an RNIC MUST ensure that if a CQ overflows, any RNIC MUST ensure that if a CQ overflows, any Streams that do not use
Streams which do not use the CQ MUST remain unaffected. the CQ MUST remain unaffected.
6.5 Elevation of Privilege 6.5. Elevation of Privilege
The RDMAP/DDP Security Architecture explicitly differentiates The RDMAP/DDP Security Architecture explicitly differentiates between
between three levels of privilege - Non-Privileged, Privileged, three levels of privilege: Non-Privileged, Privileged, and the
and the Privileged Resource Manager. If a Non-Privileged ULP is Privileged Resource Manager. If a Non-Privileged ULP is able to
able to elevate its privilege level to a Privileged ULP, then elevate its privilege level to a Privileged ULP, then mapping a
mapping a physical address list to an STag can provide local and physical address list to an STag can provide local and remote access
remote access to any physical address location on the node. If a to any physical address location on the node. If a Privileged Mode
Privileged Mode ULP is able to promote itself to be a Resource ULP is able to promote itself to be a Resource Manager, then it is
Manager, then it is possible for it to perform denial of service possible for it to perform denial of service type attacks where
type attacks where substantial amounts of local resources could substantial amounts of local resources could be consumed.
be consumed.
In general, elevation of privilege is a local implementation In general, elevation of privilege is a local implementation specific
specific issue and thus outside the scope of this document. issue and is thus outside the scope of this document.
7 Attacks from Local Peers 7. Attacks from Local Peers
This section describes local attacks that are possible against This section describes local attacks that are possible against the
the RDMA system defined in Figure 1 - RDMA Security Model and the RDMA system defined in Figure 1 - RDMA Security Model and the RNIC
RNIC Engine resources defined in Section 2.2. Engine resources defined in Section 2.2.
7.1 Local ULP Attacking a Shared CQ 7.1. Local ULP Attacking a Shared CQ
DOS attacks against a Shared Completion Queue (CQ - see Section DOS attacks against a Shared Completion Queue (CQ - see Section
2.2.6 Completion Queues) can be caused by either the local ULP or 2.2.6, Completion Queues) can be caused by either the local ULP or
the Remote Peer if either attempts to cause more completions than the Remote Peer if either attempts to cause more completions than its
its fair share of the number of entries, thus potentially fair share of the number of entries; thus, potentially starving
starving another unrelated ULP such that no Completion Queue another unrelated ULP such that no Completion Queue entries are
entries are available. available.
A Completion Queue entry can potentially be maliciously consumed A Completion Queue entry can potentially be maliciously consumed by a
by a completion from the Send Queue or a completion from the completion from the Send Queue or a completion from the Receive
Receive Queue. In the former, the attacker is the local ULP. In Queue. In the former, the attacker is the local ULP. In the latter,
the latter, the attacker is the Remote Peer. the attacker is the Remote Peer.
A form of attack can occur where the local ULPs can consume A form of attack can occur where the local ULPs can consume resources
resources on the CQ. A local ULP that is slow to free resources on the CQ. A local ULP that is slow to free resources on the CQ by
on the CQ by not reaping the completion status quickly enough not reaping the completion status quickly enough could stall all
could stall all other local ULPs attempting to use that CQ. other local ULPs attempting to use that CQ.
For these reasons, an RNIC MUST NOT enable sharing a CQ across For these reasons, an RNIC MUST NOT enable sharing a CQ across ULPs
ULPs that do not share Partial Mutual Trust. that do not share Partial Mutual Trust.
7.2 Local Peer Attacking the RDMA Read Request Queue 7.2. Local Peer Attacking the RDMA Read Request Queue
If RDMA Read Request Queue resources are pooled across multiple If RDMA Read Request Queue resources are pooled across multiple
Streams, one attack is if the local ULP attempts to unfairly Streams, one attack is if the local ULP attempts to unfairly allocate
allocate RDMA Read Request Queue resources for its Streams. For RDMA Read Request Queue resources for its Streams. For example, a
example, a local ULP attempts to allocate all available resources local ULP attempts to allocate all available resources on a specific
on a specific RDMA Read Request Queue for its Streams, thereby RDMA Read Request Queue for its Streams, thereby denying the resource
denying the resource to ULPs sharing the RDMA Read Request Queue. to ULPs sharing the RDMA Read Request Queue. The same type of
The same type of argument applies even if the RDMA Read Request argument applies even if the RDMA Read Request is not shared, but a
is not shared - but a local ULP attempts to allocate all of the local ULP attempts to allocate all the RNIC's resources when the
RNIC's resources when the queue is created. queue is created.
Thus access to interfaces that allocate RDMA Read Request Queue Thus, access to interfaces that allocate RDMA Read Request Queue
entries MUST be restricted to a trusted Local Peer, such as a entries MUST be restricted to a trusted Local Peer, such as a
Privileged Resource Manager. The Privileged Resource Manager Privileged Resource Manager. The Privileged Resource Manager SHOULD
SHOULD prevent a local ULP from allocating more than its fair prevent a local ULP from allocating more than its fair share of
share of resources. resources.
7.3 Local ULP Attacking the PTT & STag Mapping 7.3. Local ULP Attacking the PTT and STag Mapping
If a Non-Privileged ULP is able to directly manipulate the RNIC If a Non-Privileged ULP is able to directly manipulate the RNIC Page
Page Translation Tables (which translate from an STag to a host Translation Tables (which translate from an STag to a host address),
address), it is possible that the Non-Privileged ULP could point it is possible that the Non-Privileged ULP could point the Page
the Page Translation Table at an unrelated Stream's or ULP's Translation Table at an unrelated Stream's or ULP's buffers and,
buffers and thereby be able to gain access to information of the thereby, be able to gain access to information of the unrelated
unrelated Stream/ULP. Stream/ULP.
As discussed in Section 2 Architectural Model, introduction of a As discussed in Section 2, Architectural Model, introduction of a
Privileged Resource Manager to arbitrate the mapping requests is Privileged Resource Manager to arbitrate the mapping requests is an
an effective countermeasure. This enables the Privileged Resource effective countermeasure. This enables the Privileged Resource
Manager to ensure a local ULP can only initialize the Page Manager to ensure that a local ULP can only initialize the Page
Translation Table (PTT)to point to its own buffers. Translation Table (PTT)to point to its own buffers.
Thus if Non-Privileged ULPs are supported, the Privileged Thus, if Non-Privileged ULPs are supported, the Privileged Resource
Resource Manager MUST verify that the Non-Privileged ULP has the Manager MUST verify that the Non-Privileged ULP has the right to
right to access a specific Data Buffer before allowing an STag access a specific Data Buffer before allowing an STag for which the
for which the ULP has access rights to be associated with a ULP has access rights to be associated with a specific Data Buffer.
specific Data Buffer. This can be done when the Page Translation This can be done when the Page Translation Table is initialized to
Table is initialized to access the Data Buffer or when the STag access the Data Buffer or when the STag is initialized to point to a
is initialized to point to a group of Page Translation Table group of Page Translation Table entries, or both.
entries, or both.
8 Security considerations 8. Security considerations
Please see Sections 5 Attacks That Can be Mitigated With End-to- Please see Sections 5, Attacks That Can be Mitigated with End-to-End
End Security, Section 6 Attacks from Remote Peers, and Section 7 Security; Section 6, Attacks from Remote Peers; and Section 7,
Attacks from Local Peers, for a detailed analysis of attacks and Attacks from Local Peers, for a detailed analysis of attacks and
normative countermeasures to mitigate the attacks. normative countermeasures to mitigate the attacks.
Additionally, the appendices provide a summary of the security Additionally, the appendices provide a summary of the security
requirements for specific audiences. Section 11 Appendix A: ULP requirements for specific audiences. Appendix A, ULP Issues for RDDP
Issues for RDDP Client/Server Protocols provides a summary of Client/Server Protocols, provides a summary of implementation issues
implementation issues and requirements for applications which and requirements for applications that implement a traditional
implement a traditional client/server style of interaction. It client/server style of interaction. It provides additional insight
provides additional insight and applicability of the normative and applicability of the normative text in Sections 5, 6, and 7.
text in Sections 5, 6, and 7. Section 12, Appendix B: Summary of Appendix B, Summary of RNIC and ULP Implementation Requirements,
RNIC and ULP Implementation Requirements provides a convenient provides a convenient summary of normative requirements for
summary of normative requirements for implementers. implementers.
9 IANA Considerations 9. IANA Considerations
IANA considerations are not addressed by this document. Any IANA IANA considerations are not addressed by this document. Any IANA
considerations resulting from the use of DDP or RDMA must be considerations resulting from the use of DDP or RDMA must be
addressed in the relevant standards. addressed in the relevant standards.
10 References 10. References
10.1 Normative References 10.1. Normative References
[DDP] Shah, H., J. Pinkerton, R. Recio, and P. Culley, "Direct [DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley,
Data Placement over Reliable Transports", Internet-Draft Work "Direct Data Placement over Reliable Transports", RFC
in Progress draft-ietf-rddp-ddp-05.txt, July 2005. 5041, October 2007.
[RDMAP] Recio, R., P. Culley, D. Garcia, J. Hilland, "An RDMA [RDMAP] Recio, R., Culley, P., Garcia, D., and J. Hilland, "A
Protocol Specification", Internet-Draft Work in Progress Remote Direct Memory Access Protocol Specification",
draft-ietf-rddp-rdmap-05.txt, July 2005. RFC 5040, October 2007.
[RFC2406] Kent, S., Atkinson, R. "IP Encapsulating Security [RFC2401] Kent, S. and R. Atkinson, "Security Architecture for
the Internet Protocol", RFC 2401, November 1998.
[RFC2402] Kent, S. and R. Atkinson, "IP Authentication Header",
RFC 2402, November 1998.
[RFC2406] Kent, S. and R. Atkinson, "IP Encapsulating Security
Payload (ESP)", RFC 2406, November 1998. Payload (ESP)", RFC 2406, November 1998.
[RFC2409] Harkins, D., Carrel, D., "The Internet Key Exchange [RFC2409] Harkins, D. and D. Carrel, "The Internet Key Exchange
(IKE)", RFC 2409, November 1998. (IKE)", RFC 2409, November 1998.
[RFC2401] Kent, S., Atkinson, R. "Security Architecture for the [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
Internet Protocol", RFC 2401, November 1998. Travostino, "Securing Block Storage Protocols over IP",
RFC 3723, April 2004.
[RFC2402] Kent, S., Atkinson, R. "IP Authentication Header", RFC [RFC4960] Stewart, R., Ed., "Stream Control Transmission
2402, November 1998. Protocol", RFC 4960, September 2007.
[RFC3723] Aboba, B., et al, "Securing Block Storage Protocols [RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC
over IP", RFC3723, April 2004. 793, September 1981.
[RFC2960] Stewart, R. et al., "Stream Control Transmission 10.2. Informative References
Protocol", RFC 2960, October 2000.
[RFC793] Postel, J., "Transmission Control Protocol - DARPA [RFC4301] Kent, S. and K. Seo, "Security Architecture for the
Internet Program Protocol Specification", RFC 793, September Internet Protocol", RFC 4301, December 2005.
1981.
10.2 Informative References [RFC4346] Dierks, T. and E. Rescorla, "The Transport Layer
Security (TLS) Protocol Version 1.1", RFC 4346, April
2006.
[RFC2828] Shirley, R., "Internet Security Glossary", FYI 36, RFC [RFC4949] Shirey, R., "Internet Security Glossary, Version 2",
2828, May 2000. RFC 4949, August 2007.
[APPLICABILITY] Bestler, C. , Coene, L. "Applicability of Remote [APPLICABILITY]
Bestler, C. and L. Coene, "Applicability of Remote
Direct Memory Access Protocol (RDMA) and Direct Data Direct Memory Access Protocol (RDMA) and Direct Data
Placement (DDP)", Internet-Draft Work in Progress draft-ietf- Placement (DDP)", RFC 5045, October 2007.
rddp-applicability-06.txt, April 2006.
[IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor
Discovery Trust Models and threats", Informational RFC,
RFC3756, May 2004.
[NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to [NFSv4CHANNEL]
Secure Channels", Internet-Draft draft-ietf-nfsv4-channel- Williams, N., "On the Use of Channel Bindings to Secure
bindings-02.txt, July 2004. Channels", Work in Progress, July 2004.
[VERBS-RDMAC] "RDMA Protocol Verbs Specification", RDMA [VERBS-RDMAC] "RDMA Protocol Verbs Specification", RDMA Consortium
Consortium standard, April 2003. standard, April 2003, <http://www.rdmaconsortium.org/
http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs- home/draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>.
v1.0-RDMAC.pdf
[VERBS-RDMAC-Overview] "RDMA enabled NIC (RNIC) Verbs Overview", [VERBS-RDMAC-Overview]
slide presentation by Renato Recio, April 2003. "RDMA enabled NIC (RNIC) Verbs Overview", slide
http://www.rdmaconsortium.org/home/RNIC_Verbs_Overview2.pdf presentation by Renato Recio, April 2003,
<http://www.rdmaconsortium.org/home/
RNIC_Verbs_Overview2.pdf>.
[RFC3552] "Guidelines for Writing RFC Text on Security [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC
Considerations", Best Current Practice RFC, RFC 3552, July Text on Security Considerations", BCP 72, RFC 3552,
2003. July 2003.
[INFINIBAND] "InfiniBand Architecture Specification Volume 1", [INFINIBAND] "InfiniBand Architecture Specification Volume 1",
release 1.2, InfiniBand Trade Association standard. release 1.2, InfiniBand Trade Association standard,
http://www.infinibandta.org/specs. Verbs are documented in <http://www.infinibandta.org/specs>. Verbs are
chapter 11. documented in chapter 11.
[DTLS] E. Rescorla and N. Modadugu, "Datagram Transport Layer [DTLS] Rescorla, E. and N. Modadugu, "Datagram Transport Layer
Security", RFC 4347, April 2006. Security", RFC 4347, April 2006.
[iSCSI] J. Satran, et al, "Internet Small Computer Systems [iSCSI] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M.,
and E. Zeidner, "Internet Small Computer Systems
Interface (iSCSI)", RFC 3720, April 2004. Interface (iSCSI)", RFC 3720, April 2004.
[ISER] M. Ko, et al, "iSCSI Extensions for RDMA Specification", [iSER] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah,
Internet-Draft Work in Progress draft-ietf-ips-iser-05.txt, H., and P. Thaler, "Internet Small Computer System
October 2005. Interface (iSCSI) Extensions for Remote Direct Memory
Access (RDMA)", RFC 5046, October 2007.
[NFSv4] S. Shepler, et al, "Network File System (NFS) version 4 [NFSv4] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Protocol", RFC 3530, April 2003. Beame, C., Eisler, M., and D. Noveck, "Network File
System (NFS) version 4 Protocol", RFC 3530, April 2003.
[NFSv4.1] S. Shepler, ed., "NFSv4 Minor Version 1", Internet- [NFSv4.1] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
Draft draft-ietf-nfsv4-minorversion1-03.txt, Work in "NFSv4 Minor Version 1", Work in Progress, September
Progress, June 2006. 2007.
11 Appendix A: ULP Issues for RDDP Client/Server Protocols Appendix A: ULP Issues for RDDP Client/Server Protocols
This section is a normative appendix to the document that is This section is a normative appendix to the document that is focused
focused on client/server ULP implementation requirements to on client/server ULP implementation requirements to ensure a secure
ensure a secure server implementation. server implementation.
The prior sections outlined specific attacks and their The prior sections outlined specific attacks and their
countermeasures. This section summarizes the attacks and countermeasures. This section summarizes the attacks and
countermeasures that have been defined in the prior section which countermeasures that have been defined in the prior section, which
are applicable to creation of a secure ULP (e.g. application) are applicable to creation of a secure ULP (e.g., application)
server. A ULP server is defined as a ULP which must be able to server. A ULP server is defined as a ULP that must be able to
communicate with many clients which do not necessarily have a communicate with many clients that do not necessarily have a trust
trust relationship with each other, and ensure that each client relationship with each other, and to ensure that each client cannot
can not attack another client through server interactions. attack another client through server interactions. Further, the
Further, the server may wish to use multiple Streams to server may wish to use multiple Streams to communicate with a
communicate with a specific client, and those Streams may share specific client, and those Streams may share mutual trust. Note that
mutual trust. Note that this section assumes a compliant RNIC and this section assumes a compliant RNIC and Privileged Resource Manager
Privileged Resource Manager implementation - thus it focuses implementation - thus, it focuses specifically on ULP server (e.g.,
specifically on ULP server (e.g. application) implementation application) implementation issues.
issues.
All of the prior section's details on attacks and countermeasures All of the prior section's details on attacks and countermeasures
apply to the server, thus requirements which are repeated in this apply to the server; thus, requirements that are repeated in this
section use non-normative "must", "should", "may". In some cases section use non-normative "must", "should", and "may". In some
normative SHOULD statements for the ULP from the main body of cases, normative SHOULD statements for the ULP from the main body of
this document are made MUST statements for the ULP server because this document are made MUST statements for the ULP server because the
the operating conditions can be refined to make the motives for a operating conditions can be refined to make the motives for a SHOULD
SHOULD inapplicable. If a prior SHOULD is changed to a MUST in inapplicable. If a prior SHOULD is changed to a MUST in this
this section, it is explicitly noted and it uses upper-case section, it is explicitly noted and it uses uppercase normative
normative statements. statements.
The following list summarizes the relevant attacks that clients The following list summarizes the relevant attacks that clients can
can mount on the shared server, by re-stating the previous mount on the shared server by re-stating the previous normative
normative statements to be client/server specific. Note that each statements to be client/server specific. Note that each
client/server ULP may employ explicit RDMA operations (RDMA Read, client/server ULP may employ explicit RDMA Operations (RDMA Read,
RDMA Write) in differing fashions. Therefore where appropriate, RDMA Write) in differing fashions. Therefore, where appropriate,
"Local ULP", "Local Peer" and "Remote Peer" are used in place of "Local ULP", "Local Peer", and "Remote Peer" are used in place of
"server" or "client", in order to retain full generality of each "server" or "client", in order to retain full generality of each
requirement. requirement.
* Spoofing * Spoofing
* Sections 5.1.1 to 5.1.3. For protection against many * Sections 5.1.1 to 5.1.3. For protection against many forms of
forms of spoofing attacks, enable IPsec. spoofing attacks, enable IPsec.
* Section 6.1.1 Using an STag on a Different Stream. To * Section 6.1.1, Using an STag on a Different Stream. To ensure
ensure that one client can not access another that one client cannot access another client's data via use of
client's data via use of the other client's STag, the the other client's STag, the server ULP must either scope an
server ULP must either scope an STag to a single STag to a single Stream or use a unique Protection Domain per
Stream or use a unique Protection Domain per client. client. If a single client has multiple Streams that share
If a single client has multiple Streams that share Partial Mutual Trust, then the STag can be shared between the
Partial Mutual Trust, then the STag can be shared associated Streams by using a single Protection Domain among
between the associated Streams by using a single the associated Streams (see Section 5.4.4, ULPs That Provide
Protection Domain among the associated Streams (see Security, for additional issues). To prevent unintended
Section 5.4.4 ULPs Which Provide Security for sharing of STags within the associated Streams, a server ULP
additional issues). To prevent unintended sharing of should use STags in such a fashion that it is difficult to
STags within the associated Streams, a server ULP predict the next allocated STag number.
should use STags in such a fashion that it is
difficult to predict the next allocated STag number.
* Tampering * Tampering
* 6.2.2 Modifying a Buffer After Indication. Before the * 6.2.2 Modifying a Buffer after Indication. Before the local
local ULP operates on a buffer that was written by ULP operates on a buffer that was written by the Remote Peer
the Remote Peer using an RDMA Write or RDMA Read, the using an RDMA Write or RDMA Read, the local ULP MUST ensure the
local ULP MUST ensure the buffer can no longer be buffer can no longer be modified by invalidating the STag for
modified, by invalidating the STag for remote access remote access (note that this is stronger than the SHOULD in
(note that this is stronger than the SHOULD in Section 6.2.2). This can be done either by explicitly revoking
Section 6.2.2). This can either be done explicitly by remote access rights for the STag when the Remote Peer
revoking remote access rights for the STag when the indicates the operation has completed, or by checking to make
Remote Peer indicates the operation has completed, or sure the Remote Peer Invalidated the STag through the RDMAP
by checking to make sure the Remote Peer Invalidated Invalidate capability. If the Remote Peer did not invalidate
the STag through the RDMAP Invalidate capability, and the STag, the local ULP then explicitly revokes the STag remote
if it did not, the local ULP then explicitly revoking access rights.
the STag remote access rights.
* Information Disclosure * Information Disclosure
* 6.3.2 Using RDMA Read to Access Stale Data. In a * 6.3.2, Using RDMA Read to Access Stale Data. In a general
general purpose server environment there is no purpose server environment, there is no compelling rationale
compelling rationale to not require a buffer to be not to require a buffer to be initialized before remote read is
initialized before remote read is enabled (and an enabled (and an enormous downside of unintentionally sharing
enormous down side of unintentionally sharing data). data). Thus, a local ULP MUST (this is stronger than the SHOULD
Thus a local ULP MUST (this is stronger than the in Section 6.3.2) ensure that no stale data is contained in a
SHOULD in Section 6.3.2) ensure that no stale data is buffer before remote read access rights are granted to a Remote
contained in a buffer before remote read access Peer (this can be done by zeroing the contents of the memory,
rights are granted to a Remote Peer (this can be done for example).
by zeroing the contents of the memory, for example).
* 6.3.3 Accessing a Buffer After the Transfer. This * 6.3.3, Accessing a Buffer after the Transfer. This mitigation
mitigation is already covered by Section 6.2.2 is already covered by Section 6.2.2 (above).
(above).
* 6.3.4 Accessing Unintended Data With a Valid STag. * 6.3.4, Accessing Unintended Data with a Valid STag. The ULP
The ULP must set the base and bounds of the buffer must set the base and bounds of the buffer when the STag is
when the STag is initialized to expose only the data initialized to expose only the data to be retrieved.
to be retrieved.
* 6.3.5 RDMA Read into an RDMA Write Buffer. If a peer * 6.3.5, RDMA Read into an RDMA Write Buffer. If a peer only
only intends a buffer to be exposed for remote write intends a buffer to be exposed for remote write access, it must
access, it must set the access rights to the buffer set the access rights to the buffer to only enable remote write
to only enable remote write access. access.
* 6.3.6 Using Multiple STags Which Alias to the Same * 6.3.6, Using Multiple STags That Alias to the Same Buffer. The
Buffer. The requirement in Section 6.1.1 (above) requirement in Section 6.1.1 (above) mitigates this attack. A
mitigates this attack. A server buffer is exposed to server buffer is exposed to only one client at a time to ensure
only one client at a time to ensure that no that no information disclosure or information tampering occurs
information disclosure or information tampering between peers.
occurs between peers.
* 5.3 - Network Based Eavesdropping. Confidentiality * 5.3, Network-Based Eavesdropping. Confidentiality services
services should be enabled by the ULP if this threat should be enabled by the ULP if this threat is a concern.
is a concern.
* Denial of Service * Denial of Service
* 6.4.3.1 Multiple Streams Sharing Receive Buffers. ULP * 6.4.3.1, Multiple Streams Sharing Receive Buffers. ULP memory
memory footprint size can be important for some footprint size can be important for some server ULPs. If a
server ULPs. If a server ULP is expecting significant server ULP is expecting significant network traffic from
network traffic from multiple clients, using a multiple clients, using a receive buffer queue per Stream where
receive buffer queue per Stream where there is a there is a large number of Streams can consume substantial
large number of Streams can consume substantial amounts of memory. Thus, a receive queue that can be shared by
amounts of memory. Thus a receive queue that can be multiple Streams is attractive.
shared by multiple Streams is attractive.
However, because of the attacks outlined in this However, because of the attacks outlined in this section,
section, sharing a single receive queue between sharing a single receive queue between multiple clients must
multiple clients must only be done if a mechanism is only be done if a mechanism is in place to ensure that one
in place to ensure one client cannot consume receive client cannot consume receive buffers in excess of its limits,
buffers in excess of its limits, as defined by each as defined by each ULP. For multiple Streams within a single
ULP. For multiple Streams within a single client ULP client ULP (which presumably shared Partial Mutual Trust), this
(which presumably shared Partial Mutual Trust) this
added overhead may be avoided. added overhead may be avoided.
* 7.1 Local ULP Attacking a Shared CQ. The normative * 7.1 Local ULP Attacking a Shared CQ. The normative RNIC
RNIC mitigations require the RNIC to not enable mitigations require that the RNIC not enable sharing of a CQ if
sharing of a CQ if the local ULPs do not share the local ULPs do not share Partial Mutual Trust. Thus, while
Partial Mutual Trust. Thus while the ULP is not the ULP is not allowed to enable this feature in an unsafe
allowed to enable this feature in an unsafe mode, if mode, if the two local ULPs share Partial Mutual Trust, they
the two local ULPs share Partial Mutual Trust, they
must behave in the following manner: must behave in the following manner:
1) The sizing of the completion queue is based on the 1) The sizing of the completion queue is based on the size of
size of the receive queue and send queues as the receive queue and send queues, as documented in 6.4.3.2,
documented in 6.4.3.2 Remote or Local Peer Attacking Remote or Local Peer Attacking a Shared CQ.
a Shared CQ.
2) The local ULP ensures that CQ entries are reaped 2) The local ULP ensures that CQ entries are reaped frequently
frequently enough to adhere to Section 6.4.3.2's enough to adhere to Section 6.4.3.2's rules.
rules.
* 6.4.3.2 Remote or Local Peer Attacking a Shared CQ. * 6.4.3.2, Remote or Local Peer Attacking a Shared CQ. There are
There are two mitigations specified in this section - two mitigations specified in this section - one requires a
one requires a worst-case size of the CQ, and can be worst-case size of the CQ, and can be implemented entirely
implemented entirely within the Privileged Resource within the Privileged Resource Manager. The second approach
Manager. The second approach requires cooperation requires cooperation with the local ULP server (not to post too
with the local ULP server (to not post too many many buffers), and enables a smaller CQ to be used.
buffers), and enables a smaller CQ to be used.
In some server environments, partial trust of the In some server environments, partial trust of the server ULP
server ULP (but not the clients) is acceptable, thus (but not the clients) is acceptable; thus, the smaller CQ fully
the smaller CQ fully mitigates the remote attacker. mitigates the remote attacker. In other environments, the
In other environments, the local server ULP could local server ULP could also contain untrusted elements that can
also contain untrusted elements which can attack the attack the local machine (or have bugs). In those
local machine (or have bugs). In those environments, environments, the worst-case size of the CQ must be used.
the worst-case size of the CQ must be used.
* 6.4.3.3 The section requires a server's Privileged * 6.4.3.3, Attacking the RDMA Read Request Queue. The section
Resource Manager to not allow sharing of RDMA Read requires a server's Privileged Resource Manager not to allow
Request Queues across multiple Streams that do not sharing of RDMA Read Request Queues across multiple Streams
share Partial Mutual Trust, for a ULP which performs that do not share Partial Mutual Trust for a ULP that performs
RDMA Read operations to server buffers. However, RDMA Read operations to server buffers. However, because the
because the server ULP knows best which of its server ULP knows which of its Streams best share Partial Mutual
Streams share Partial Mutual Trust, this requirement Trust, this requirement can be reflected back to the ULP. The
can be reflected back to the ULP. The ULP (i.e. ULP (i.e., server) requirement, in this case, is that it MUST
server) requirement in this case is that it MUST NOT NOT allow RDMA Read Request Queues to be shared between ULPs
allow RDMA Read Request Queues to be shared between that do not have Partial Mutual Trust.
ULPs which do not have Partial Mutual Trust.
* 6.4.5 Remote Invalidate an STag Shared on Multiple * 6.4.5, Remote Invalidate an STag Shared on Multiple Streams.
Streams. This mitigation is already covered by This mitigation is already covered by Section 6.2.2 (above).
Section 6.2.2 (above).
12 Appendix B: Summary of RNIC and ULP Implementation Requirements Appendix B: Summary of RNIC and ULP Implementation Requirements
This appendix is informative. This appendix is informative.
Below is a summary of implementation requirements for the RNIC: Below is a summary of implementation requirements for the RNIC:
* 3 Trust and Resource Sharing * 3 Trust and Resource Sharing
* 5.4.5 Requirements for IPsec Encapsulation of DDP * 5.4.5 Requirements for IPsec Encapsulation of DDP
* 6.1.1 Using an STag on a Different Stream * 6.1.1 Using an STag on a Different Stream
* 6.2.1 Buffer Overrun - RDMA Write or Read Response * 6.2.1 Buffer Overrun - RDMA Write or Read Response
* 6.2.2 Modifying a Buffer After Indication * 6.2.2 Modifying a Buffer after Indication
* 6.4.1 RNIC Resource Consumption * 6.4.1 RNIC Resource Consumption
* 6.4.3.1 Multiple Streams Sharing Receive Buffers * 6.4.3.1 Multiple Streams Sharing Receive Buffers
* 6.4.3.2 Remote or Local Peer Attacking a Shared CQ * 6.4.3.2 Remote or Local Peer Attacking a Shared CQ
* 6.4.3.3 Attacking the RDMA Read Request Queue * 6.4.3.3 Attacking the RDMA Read Request Queue
* 6.4.6 Remote Peer attacking an Unshared CQ. * 6.4.6 Remote Peer Attacking an Unshared CQ
* 6.5 Elevation of Privilege 39 * 6.5 Elevation of Privilege 39
* 7.1 Local ULP Attacking a Shared CQ * 7.1 Local ULP Attacking a Shared CQ
* 7.3 Local ULP Attacking the PTT & STag Mapping * 7.3 Local ULP Attacking the PTT and STag Mapping
Below is a summary of implementation requirements for the ULP Below is a summary of implementation requirements for the ULP above
above the RNIC: the RNIC:
* 5.3 Information Disclosure - Network Based Eavesdropping * 5.3 Information Disclosure - Network-Based Eavesdropping
* 6.1.1 Using an STag on a Different Stream * 6.1.1 Using an STag on a Different Stream
* 6.2.2 Modifying a Buffer After Indication * 6.2.2 Modifying a Buffer after Indication
* 6.3.2 Using RDMA Read to Access Stale Data * 6.3.2 Using RDMA Read to Access Stale Data
* 6.3.3 Accessing a Buffer After the Transfer * 6.3.3 Accessing a Buffer after the Transfer
* 6.3.4 Accessing Unintended Data With a Valid STag * 6.3.4 Accessing Unintended Data with a Valid STag
* 6.3.5 RDMA Read into an RDMA Write Buffer * 6.3.5 RDMA Read into an RDMA Write Buffer
* 6.3.6 Using Multiple STags Which Alias to the Same Buffer * 6.3.6 Using Multiple STags That Alias to the Same Buffer
* 6.4.5 Remote Invalidate an STag Shared on Multiple
Streams
13 Appendix C: Partial Trust Taxonomy * 6.4.5 Remote Invalidate an STag Shared on Multiple Streams
Appendix C: Partial Trust Taxonomy
This appendix is informative. This appendix is informative.
Partial Trust is defined as when one party is willing to assume Partial Trust is defined as when one party is willing to assume that
that another party will refrain from a specific attack or set of another party will refrain from a specific attack or set of attacks,
attacks, the parties are said to be in a state of Partial Trust. the parties are said to be in a state of Partial Trust. Note that
Note that the partially trusted peer may attempt a different set the partially trusted peer may attempt a different set of attacks.
of attacks. This may be appropriate for many ULPs where any This may be appropriate for many ULPs where any adverse effects of
adverse effects of the betrayal is easily confined and does not the betrayal is easily confined and does not place other clients or
place other clients or ULPs at risk. ULPs at risk.
The Trust Models described in this section have three primary The Trust Models described in this section have three primary
distinguishing characteristics. The Trust Model refers to a local distinguishing characteristics. The Trust Model refers to a local
ULP and Remote Peer, which are intended to be the local and ULP and Remote Peer, which are intended to be the local and remote
remote ULP instances communicating via RDMA/DDP. ULP instances communicating via RDMA/DDP.
* Local Resource Sharing (yes/no) - When local resources * Local Resource Sharing (yes/no) - When local resources are
are shared, they are shared across a grouping of shared, they are shared across a grouping of RDMAP/DDP Streams.
RDMAP/DDP Streams. If local resources are not shared, the If local resources are not shared, the resources are dedicated on
resources are dedicated on a per Stream basis. Resources a per Stream basis. Resources are defined in Section 2.2,
are defined in Section 2.2 - Resources. The advantage of Resources. The advantage of not sharing resources between
not sharing resources between Streams is that it reduces Streams is that it reduces the types of attacks that are
the types of attacks that are possible. The disadvantage possible. The disadvantage is that ULPs might run out of
is that ULPs might run out of resources. resources.
* Local Partial Trust (yes/no) - Local Partial Trust is * Local Partial Trust (yes/no) - Local Partial Trust is determined
determined based on whether the local grouping of based on whether the local grouping of RDMAP/DDP Streams (which
RDMAP/DDP Streams (which typically equates to one ULP or typically equates to one ULP or group of ULPs) mutually trust
group of ULPs) mutually trust each other to not perform a each other not to perform a specific set of attacks.
specific set of attacks.
* Remote Partial Trust (yes/no) - The Remote Partial Trust * Remote Partial Trust (yes/no) - The Remote Partial Trust level is
level is determined based on whether the local ULP of a determined based on whether the local ULP of a specific RDMAP/DDP
specific RDMAP/DDP Stream partially trusts the Remote Stream partially trusts the Remote Peer of the Stream (see the
Peer of the Stream (see the definition of Partial Trust definition of Partial Trust in Section 1, Introduction).
in Section 1 Introduction).
Not all of the combinations of the trust characteristics are Not all the combinations of the trust characteristics are expected to
expected to be used by ULPs. This document specifically analyzes be used by ULPs. This document specifically analyzes five ULP Trust
five ULP Trust Models that are expected to be in common use. The Models that are expected to be in common use. The Trust Models are
Trust Models are as follows: as follows:
* NS-NT - Non-Shared Local Resources, no Local Trust, no * NS-NT - Non-Shared Local Resources, no Local Trust, no Remote
Remote Trust - typically a server ULP that wants to run Trust; typically, a server ULP that wants to run in the safest
in the safest mode possible. All attack mitigations are mode possible. All attack mitigations are in place to ensure
in place to ensure robust operation. robust operation.
* NS-RT - Non-Shared Local Resources, no Local Trust, * NS-RT - Non-Shared Local Resources, no Local Trust, Remote
Remote Partial Trust - typically a peer-to-peer ULP, Partial Trust; typically, a peer-to-peer ULP that has, by some
which has, by some method outside of the scope of this method outside of the scope of this document, authenticated the
document, authenticated the Remote Peer. Note that unless Remote Peer. Note that unless some form of key based
some form of key based authentication is used on a per authentication is used on a per RDMA/DDP Stream basis, it may not
RDMA/DDP Stream basis, it may not be possible be possible be possible for man-in-the-middle attacks to occur.
for man-in-the-middle attacks to occur.
* S-NT - Shared Local Resources, no Local Trust, no Remote * S-NT - Shared Local Resources, no Local Trust, no Remote Trust;
Trust - typically a server ULP that runs in an untrusted typically, a server ULP that runs in an untrusted environment
environment where the amount of resources required is where the amount of resources required is either too large or too
either too large or too dynamic to dedicate for each dynamic to dedicate for each RDMAP/DDP Stream.
RDMAP/DDP Stream.
* S-LT - Shared Local Resources, Local Partial Trust, no * S-LT - Shared Local Resources, Local Partial Trust, no Remote
Remote Trust - typically a ULP, which provides a session Trust; typically, a ULP that provides a session layer and uses
layer and uses multiple Streams, to provide additional multiple Streams, to provides additional throughput or fail-over
throughput or fail-over capabilities. All of the Streams capabilities. All the Streams within the local ULP partially
within the local ULP partially trust each other, but do trust each other, but do not trust the Remote Peer. This Trust
not trust the Remote Peer. This trust model may be Model may be appropriate for embedded environments.
appropriate for embedded environments.
* S-T - Shared Local Resources, Local Partial Trust, Remote * S-T - Shared Local Resources, Local Partial Trust, Remote Partial
Partial Trust - typically a distributed application, such Trust; typically, a distributed application, such as a
as a distributed database application or a High distributed database application or High Performance Computer
Performance Computer (HPC) application, which is intended (HPC) application, which is intended to run on a cluster. Due to
to run on a cluster. Due to extreme resource and extreme resource and performance requirements, the application
performance requirements, the application typically typically authenticates with all of its peers and then runs in a
authenticates with all of its peers and then runs in a highly trusted environment. The application peers are all in a
highly trusted environment. The application peers are all single application fault domain and depend on one another to be
in a single application fault domain and depend on one well-behaved when accessing data structures. If a trusted Remote
another to be well-behaved when accessing data Peer has an implementation defect that results in poor behavior,
structures. If a trusted Remote Peer has an the entire application could be corrupted.
implementation defect that results in poor behavior, the
entire application could be corrupted.
Models NS-NT and S-NT above are typical for Internet networking - Models NS-NT and S-NT, above, are typical for Internet networking -
neither local ULPs nor the Remote Peer is trusted. Sometimes neither the local ULP nor the Remote Peer is trusted. Sometimes,
optimizations can be done that enable sharing of Page Translation optimizations can be done that enable sharing of Page Translation
Tables across multiple local ULPs, thus Model S-LT can be Tables across multiple local ULPs; thus, Model S-LT can be
advantageous. Model S-T is typically used when resource scaling advantageous. Model S-T is typically used when resource scaling
across a large parallel ULP makes it infeasible to use any other across a large parallel ULP makes it infeasible to use any other
model. Resource scaling issues can either be due to performance model. Resource scaling issues can either be due to performance
around scaling or because there simply are not enough resources. around scaling or because there simply are not enough resources.
Model NS-RT is probably the least likely model to be used, but is Model NS-RT is probably the least likely model to be used, but is
presented for completeness. presented for completeness.
14 Author's Addresses Acknowledgments
James Pinkerton
Microsoft Corporation
One Microsoft Way
Redmond, WA. 98052 USA
Phone: +1 (425) 705-5442
Email: jpink@windows.microsoft.com
Ellen Deleganes
Intel Corporation
MS JF5-355
2111 NE 25th Ave.
Hillsboro, OR 97124 USA
Phone: +1 (503) 712-4173
Email: ellen.m.deleganes@intel.com
15 Acknowledgments
Sara Bitan Sara Bitan
Microsoft Corporation Microsoft Corporation
Email: sarab@microsoft.com EMail: sarab@microsoft.com
Allyn Romanow Allyn Romanow
Cisco Systems Cisco Systems
170 W Tasman Drive 170 W Tasman Drive
San Jose, CA 95134 USA San Jose, CA 95134 USA
Phone: +1 408 525 8836 Phone: +1 (408) 525-8836
Email: allyn@cisco.com EMail: allyn@cisco.com
Catherine Meadows Catherine Meadows
Naval Research Laboratory Naval Research Laboratory
Code 5543 Code 5543
Washington, DC 20375 Washington, DC 20375 USA
Email: meadows@itd.nrl.navy.mil EMail: meadows@itd.nrl.navy.mil
Patricia Thaler Patricia Thaler
Agilent Technologies, Inc. Agilent Technologies, Inc.
1101 Creekside Ridge Drive, #100 1101 Creekside Ridge Drive, #100
M/S-RG10 M/S-RG10
Roseville, CA 95678 Roseville, CA 95678 USA
Phone: +1-916-788-5662 Phone: +1 (916) 788-5662
email: pat_thaler@agilent.com EMail: pat_thaler@agilent.com
James Livingston James Livingston
NEC Solutions (America), Inc. NEC Solutions (America), Inc.
7525 166th Ave. N.E., Suite D210 7525 166th Ave. N.E., Suite D210
Redmond, WA 98052-7811 Redmond, WA 98052-7811 USA
Phone: +1 (425) 897-2033 Phone: +1 (425) 897-2033
Email: james.livingston@necsam.com EMail: james.livingston@necsam.com
John Carrier John Carrier
Adaptec, Inc. Cray Inc.
691 S. Milpitas Blvd. 411 First Avenue S, Suite 600
Milpitas, CA 95035 USA Seattle, WA 98104-2860
Phone: +1 (360) 378-8526 Phone: 206-701-2090
Email: john_carrier@adaptec.com EMail: carrier@cray.com
Caitlin Bestler Caitlin Bestler
Broadcom Broadcom
49 Discovery 49 Discovery
Irvine, CA 92618 Irvine, CA 92618
Email: cait@asomi.com EMail: cait@asomi.com
Bernard Aboba Bernard Aboba
Microsoft Corporation Microsoft Corporation
One Microsoft Way One Microsoft Way USA
Redmond, WA. 98052 USA Redmond, WA 98052
Phone: +1 (425) 706-6606 Phone: +1 (425) 706-6606
Email: bernarda@windows.microsoft.com EMail: bernarda@windows.microsoft.com
16 Full Copyright Statement Authors' Addresses
Copyright (C) The Internet Society (2006). James Pinkerton
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052 USA
Phone: +1 (425) 705-5442
EMail: jpink@windows.microsoft.com
Ellen Deleganes
Self
P.O. Box 9245
Brooks, OR 97305
Phone: (503) 642-3950
EMail: deleganes@yahoo.com
Full Copyright Statement
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided This document and the information contained herein are provided on an
on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
FOR A PARTICULAR PURPOSE.
Intellectual Property Intellectual Property
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be Intellectual Property Rights or other rights that might be claimed to
claimed to pertain to the implementation or use of the technology pertain to the implementation or use of the technology described in
described in this document or the extent to which any license this document or the extent to which any license under such rights
under such rights might or might not be available; nor does it might or might not be available; nor does it represent that it has
represent that it has made any independent effort to identify any made any independent effort to identify any such rights. Information
such rights. Information on the procedures with respect to on the procedures with respect to rights in RFC documents can be
rights in RFC documents can be found in BCP 78 and BCP 79. found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the attempt made to obtain a general license or permission for the use of
use of such proprietary rights by implementers or users of this such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR specification can be obtained from the IETF on-line IPR repository at
repository at http://www.ietf.org/ipr. http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention
any copyrights, patents or patent applications, or other
proprietary rights that may cover technology that may be required
to implement this standard. Please address the information to
the IETF at ietf-ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is currently provided by the The IETF invites any interested party to bring to its attention any
Internet Society. copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
 End of changes. 389 change blocks. 
1784 lines changed or deleted 1669 lines changed or added

This html diff was produced by rfcdiff 1.34. The latest version is available from http://tools.ietf.org/tools/rfcdiff/