draft-ietf-rddp-security-07.txt   draft-ietf-rddp-security-08.txt 
Internet Draft James Pinkerton Internet Draft James Pinkerton
draft-ietf-rddp-security-07.txt Microsoft Corporation draft-ietf-rddp-security-08.txt Microsoft Corporation
Category: Standards Track Ellen Deleganes Category: Standards Track Ellen Deleganes
Expires: October, 2005 Intel Corporation Expires: September, 2006 Intel Corporation
Sara Bitan Sara Bitan
Microsoft Corporation Microsoft Corporation
April 2005 March 2006
DDP/RDMAP Security DDP/RDMAP Security
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that By submitting this Internet-Draft, each author represents that
any applicable patent or other IPR claims of which he or she is any applicable patent or other IPR claims of which he or she is
aware have been or will be disclosed, and any of which he or she aware have been or will be disclosed, and any of which he or she
becomes aware will be disclosed, in accordance with Section 6 of becomes aware will be disclosed, in accordance with Section 6 of
BCP 79. BCP 79.
skipping to change at page 2, line ? skipping to change at page 2, line ?
Memory Access Protocol (RDMAP). It first defines an architectural Memory Access Protocol (RDMAP). It first defines an architectural
model for an RDMA Network Interface Card (RNIC), which can model for an RDMA Network Interface Card (RNIC), which can
implement DDP or RDMAP and DDP. The document reviews various implement DDP or RDMAP and DDP. The document reviews various
attacks against the resources defined in the architectural model attacks against the resources defined in the architectural model
and the countermeasures that can be used to protect the system. and the countermeasures that can be used to protect the system.
Attacks are grouped into spoofing, tampering, information Attacks are grouped into spoofing, tampering, information
disclosure, denial of service, and elevation of privilege. disclosure, denial of service, and elevation of privilege.
Finally, the document concludes with a summary of security Finally, the document concludes with a summary of security
services for DDP and RDMAP, such as IPsec. services for DDP and RDMAP, such as IPsec.
J. Pinkerton, et al. Expires October, 2005 1 J. Pinkerton, et al. Expires September, 2006 1
Table of Contents Table of Contents
1 Introduction.................................................4 1 Introduction.................................................4
2 Architectural Model..........................................6 2 Architectural Model..........................................6
2.1 Components...................................................7 2.1 Components...................................................7
2.2 Resources....................................................8 2.2 Resources....................................................9
2.2.1 Stream Context Memory......................................8 2.2.1 Stream Context Memory......................................9
2.2.2 Data Buffers...............................................8 2.2.2 Data Buffers..............................................10
2.2.3 Page Translation Tables....................................9 2.2.3 Page Translation Tables...................................10
2.2.4 STag Namespace.............................................9 2.2.4 Protection Domain (PD)....................................11
2.2.5 Completion Queues..........................................9 2.2.5 STag Namespace and Scope..................................11
2.2.6 Asynchronous Event Queue..................................10 2.2.6 Completion Queues.........................................12
2.2.7 RDMA Read Request Queue...................................10 2.2.7 Asynchronous Event Queue..................................12
2.2.8 RNIC Interactions.........................................10 2.2.8 RDMA Read Request Queue...................................13
2.2.8.1 Privileged Control Interface Semantics.................10 2.3 RNIC Interactions...........................................13
2.2.8.2 Non-Privileged Data Interface Semantics................11 2.3.1 Privileged Control Interface Semantics....................13
2.2.8.3 Privileged Data Interface Semantics....................11 2.3.2 Non-Privileged Data Interface Semantics...................13
2.2.9 Initialization of RNIC Data Structures for Data Transfer..11 2.3.3 Privileged Data Interface Semantics.......................14
2.2.10 RNIC Data Transfer Interactions..........................13 2.3.4 Initialization of RNIC Data Structures for Data Transfer..14
3 Trust and Resource Sharing..................................14 2.3.5 RNIC Data Transfer Interactions...........................16
4 Attacker Capabilities.......................................15 3 Trust and Resource Sharing..................................17
5 Attacks and Countermeasures.................................16 4 Attacker Capabilities.......................................18
5.1 Tools for Countermeasures...................................16 5 Attacks That Can be Mitigated With End-to-End Security......19
5.1.1 Protection Domain (PD)....................................16 5.1 Spoofing....................................................19
5.1.2 Limiting STag Scope.......................................17 5.1.1 Impersonation.............................................19
5.1.3 Access Rights.............................................18 5.1.2 Stream Hijacking..........................................20
5.1.4 Limiting the Scope of the Completion Queue................18 5.1.3 Man-in-the-Middle Attack..................................20
5.1.5 Limiting the Scope of an Error............................18 5.2 Tampering - Network based modification of buffer content....21
5.2 Spoofing....................................................19 5.3 Information Disclosure - Network Based Eavesdropping........21
5.2.1 Impersonation.............................................19 5.4 Specific Requirements for Security Services.................21
5.2.2 Stream Hijacking..........................................19 5.4.1 Introduction to Security Options..........................22
5.2.3 Man in the Middle Attack..................................20 5.4.2 TLS is Inappropriate for DDP/RDMAP Security...............22
5.2.4 Using an STag on a Different Stream.......................20 5.4.3 ULPs Which Provide Security...............................23
5.3 Tampering...................................................21 5.4.4 Requirements for IPsec Encapsulation of DDP...............23
5.3.1 Buffer Overrun - RDMA Write or Read Response..............22 6 Attacks from Remote Peers...................................25
5.3.2 Modifying a Buffer After Indication.......................22 6.1 Spoofing....................................................25
5.3.3 Multiple STags to access the same buffer..................23 6.1.1 Using an STag on a Different Stream.......................25
5.3.4 Network based modification of buffer content..............23 6.2 Tampering...................................................26
5.4 Information Disclosure......................................23 6.2.1 Buffer Overrun - RDMA Write or Read Response..............27
5.4.1 Probing memory outside of the buffer bounds...............23 6.2.2 Modifying a Buffer After Indication.......................27
5.4.2 Using RDMA Read to Access Stale Data......................23 6.2.3 Multiple STags to access the same buffer..................28
5.4.3 Accessing a Buffer After the Transfer.....................24 6.3 Information Disclosure......................................28
5.4.4 Accessing Unintended Data With a Valid STag...............24 6.3.1 Probing memory outside of the buffer bounds...............28
5.4.5 RDMA Read into an RDMA Write Buffer.......................24 6.3.2 Using RDMA Read to Access Stale Data......................28
5.4.6 Using Multiple STags Which Alias to the Same Buffer.......25 6.3.3 Accessing a Buffer After the Transfer.....................29
5.4.7 Remote Node Loading Firmware onto the RNIC................25 6.3.4 Accessing Unintended Data With a Valid STag...............29
5.4.8 Controlling Access to PTT & STag Mapping..................25 6.3.5 RDMA Read into an RDMA Write Buffer.......................29
5.4.9 Network based eavesdropping...............................26 6.3.6 Using Multiple STags Which Alias to the Same Buffer.......30
5.5 Denial of Service (DOS).....................................26 6.3.7 Controlling Access to PTT & STag Mapping..................30
5.5.1 RNIC Resource Consumption.................................26 6.4 Denial of Service (DOS).....................................31
5.5.2 Resource Consumption By Active ULPs.......................27 6.4.1 RNIC Resource Consumption.................................31
5.5.2.1 Multiple Streams Sharing Receive Buffers...............27 6.4.2 Resource Consumption by Idle ULPs.........................32
5.5.2.2 Local ULP Attacking a Shared CQ........................29 6.4.3 Resource Consumption By Active ULPs.......................32
5.5.2.3 Local or Remote Peer Attacking a Shared CQ.............29 6.4.3.1 Multiple Streams Sharing Receive Buffers...............33
5.5.2.4 Attacking the RDMA Read Request Queue..................32 6.4.3.2 Remote or Local Peer Attacking a Shared CQ.............34
5.5.3 Resource Consumption by Idle ULPs.........................33 6.4.3.3 Attacking the RDMA Read Request Queue..................37
5.5.4 Exercise of non-optimal code paths........................34 6.4.4 Exercise of non-optimal code paths........................37
5.5.5 Remote Invalidate an STag Shared on Multiple Streams......34 6.4.5 Remote Invalidate an STag Shared on Multiple Streams......38
5.5.6 Remote Peer attacking an Unshared CQ......................34 6.4.6 Remote Peer attacking an Unshared CQ......................38
5.6 Elevation of Privilege......................................35 6.5 Elevation of Privilege......................................38
6 Security Services for RDMAP and DDP.........................36 7 Attacks from Local Peers....................................40
6.1 Introduction to Security Options............................36 7.1 Local ULP Attacking a Shared CQ.............................40
6.1.1 Introduction to IPsec.....................................36 7.2 Local Peer Attacking the RDMA Read Request Queue............40
6.1.2 Introduction to SSL Limitations on RDMAP..................38 8 Security considerations.....................................41
6.1.3 ULPs Which Provide Security...............................38 9 IANA Considerations.........................................42
6.2 Requirements for IPsec Encapsulation of DDP.................39 10 References..................................................43
7 Security considerations.....................................40 10.1 Normative References......................................43
8 IANA Considerations.........................................41 10.2 Informative References....................................43
9 References..................................................42 11 Appendix A: ULP Issues for RDDP Client/Server Protocols.....45
9.1 Normative References........................................42 12 Appendix B: Summary of RNIC and ULP Implementation
9.2 Informative References......................................42 Requirements.....................................................49
10 Appendix A: ULP Issues for RDDP Client/Server Protocols.....43 13 Appendix C: Partial Trust Taxonomy..........................51
11 Appendix B: Summary of RNIC and ULP Implementation 14 Author's Addresses..........................................53
Requirements.....................................................47 15 Acknowledgments.............................................54
12 Appendix C: Partial Trust Taxonomy..........................49 16 Full Copyright Statement....................................55
13 Author's Addresses..........................................51
14 Acknowledgments.............................................52
15 Full Copyright Statement....................................53
Table of Figures Table of Figures
Figure 1 - RDMA Security Model....................................7 Figure 1 - RDMA Security Model....................................7
1 Introduction 1 Introduction
RDMA enables new levels of flexibility when communicating between RDMA enables new levels of flexibility when communicating between
two parties compared to current conventional networking practice two parties compared to current conventional networking practice
(e.g. a stream-based model or datagram model). This flexibility (e.g. a stream-based model or datagram model). This flexibility
skipping to change at page 4, line 55 skipping to change at page 4, line 55
address this concept: address this concept:
Partial Mutual Trust - a collection of RDMAP/DDP Streams, Partial Mutual Trust - a collection of RDMAP/DDP Streams,
which represent the local and remote end points of the which represent the local and remote end points of the
Stream, which are willing to assume that the Streams from Stream, which are willing to assume that the Streams from
the collection will not perform malicious attacks against the collection will not perform malicious attacks against
any of the other Streams in the collection. any of the other Streams in the collection.
ULPs have explicit control of which collection of endpoints is in ULPs have explicit control of which collection of endpoints is in
a Partial Mutual Trust collection through tools discussed in a Partial Mutual Trust collection through tools discussed in
Section 5.1 Tools for Countermeasures on page 16. Section 13 Appendix C: Partial Trust Taxonomy.
An untrusted peer relationship is appropriate when a ULP wishes An untrusted peer relationship is appropriate when a ULP wishes
to ensure that it will be robust and uncompromised even in the to ensure that it will be robust and uncompromised even in the
face of a deliberate attack by its peer. For example, a single face of a deliberate attack by its peer. For example, a single
ULP that concurrently supports multiple unrelated Streams (e.g. a ULP that concurrently supports multiple unrelated Streams (e.g. a
server) would presumably treat each of its peers as an untrusted server) would presumably treat each of its peers as an untrusted
peer. For a collection of Streams which share Partial Mutual peer. For a collection of Streams which share Partial Mutual
Trust, the assumption is that any Stream not in the collection is Trust, the assumption is that any Stream not in the collection is
untrusted. For the untrusted peer, a brief list of capabilities untrusted. For the untrusted peer, a brief list of capabilities
is enumerated in Section 4. is enumerated in Section 4.
The rest of the document is focused on analyzing attacks and The rest of the document is focused on analyzing attacks and
recommending specific mitigations to the attacks. First, the recommending specific mitigations to the attacks. Attacks are
tools for mitigating attacks are listed (Section 5.1), and then a categorized into attacks mitigated by LLP mechanisms, attacks
series of attacks on components, resources, or system properties initiated by Remote Peers, and attacks initiated by Local Peers.
is listed in the rest of Section 5. For each attack, possible For each attack, possible countermeasures are reviewed.
countermeasures are reviewed.
ULPs within a host are divided into two categories - Privileged ULPs within a host are divided into two categories - Privileged
and Non-Privileged. Both ULP types can send and receive data and and Non-Privileged. Both ULP types can send and receive data and
request resources. The key differences between the two are: request resources. The key differences between the two are:
The Privileged ULP is trusted by the local system to not The Privileged ULP is trusted by the local system to not
maliciously attack the operating environment, but it is not maliciously attack the operating environment, but it is not
trusted to optimize resource allocation globally. For trusted to optimize resource allocation globally. For
example, the Privileged ULP could be a kernel ULP, thus the example, the Privileged ULP could be a kernel ULP, thus the
kernel presumably has in some way vetted the ULP before kernel presumably has in some way vetted the ULP before
skipping to change at page 6, line 28 skipping to change at page 6, line 28
The intent here is to describe high level components and The intent here is to describe high level components and
capabilities which affect threat analysis, and not focus on capabilities which affect threat analysis, and not focus on
specific implementation options. Also note that the architectural specific implementation options. Also note that the architectural
model is an abstraction, and an actual implementation may choose model is an abstraction, and an actual implementation may choose
to subdivide its components along different boundary lines than to subdivide its components along different boundary lines than
defined here. For example, the Privileged Resource Manager may be defined here. For example, the Privileged Resource Manager may be
partially or completely encapsulated in the Privileged ULP. partially or completely encapsulated in the Privileged ULP.
Regardless, it is expected that the security analysis of the Regardless, it is expected that the security analysis of the
potential threats and countermeasures still apply. potential threats and countermeasures still apply.
Note that the model below is derived from several specific RDMA
implementations. A few of note is [VERBS-RDMAC], [VERBS-RDMAC-
Overview], and [INFINIBAND].
+-------------+ +-------------+
| Privileged | | Privileged |
| Resource | | Resource |
Admin<-+>| Manager | ULP Control Interface Admin<-+>| Manager | ULP Control Interface
| | |<------+-------------------+ | | |<------+-------------------+
| +-------------+ | | | +-------------+ | |
| ^ v v | ^ v v
| | +-------------+ +-----------------+ | | +-------------+ +-----------------+
|---------------->| Privileged | | Non-Privileged | |---------------->| Privileged | | Non-Privileged |
| | ULP | | ULP | | | ULP | | ULP |
skipping to change at page 6, line 49 skipping to change at page 7, line 26
| ^ ^ | ^ ^
|Privileged |Privileged |Non-Privileged |Privileged |Privileged |Non-Privileged
|Control |Data |Data |Control |Data |Data
|Interface |Interface |Interface |Interface |Interface |Interface
RNIC | | | RNIC | | |
Interface v v v Interface v v v
================================================================= =================================================================
+--------------------------------------+ +--------------------------------------+
| | | |
| RNIC Engine | <-- Firmware | RNIC Engine |
| | | |
+--------------------------------------+ +--------------------------------------+
^ ^
| |
v v
Internet Internet
Figure 1 - RDMA Security Model Figure 1 - RDMA Security Model
2.1 Components 2.1 Components
The components shown in Figure 1 - RDMA Security Model are: The components shown in Figure 1 - RDMA Security Model are:
* RDMA Network Interface Controller Engine (RNIC) - the * RDMA Network Interface Controller Engine (RNIC) - the
component that implements the RDMA protocol and/or DDP component that implements the RDMA protocol and/or DDP
protocol. protocol.
skipping to change at page 7, line 39 skipping to change at page 8, line 18
definition of Non-Privileged ULP. definition of Non-Privileged ULP.
A design goal of the DDP and RDMAP protocols is to allow, under A design goal of the DDP and RDMAP protocols is to allow, under
constrained conditions, Non-Privileged ULP to send and receive constrained conditions, Non-Privileged ULP to send and receive
data directly to/from the RDMA Engine without Privileged Resource data directly to/from the RDMA Engine without Privileged Resource
Manager intervention - while ensuring that the host remains Manager intervention - while ensuring that the host remains
secure. Thus, one of the primary goals of this document is to secure. Thus, one of the primary goals of this document is to
analyze this usage model for the enforcement that is required in analyze this usage model for the enforcement that is required in
the RNIC Engine to ensure the system remains secure. the RNIC Engine to ensure the system remains secure.
DDP supports both untagged and tagged buffers. Tagged buffers
allow the Data Sink ULP to be indifferent to what order (or in
what messages) the Data Source sent the data, or what order
packets are received in. Typically tagged data can be used for
payload transfer, while untagged is best used for control
messages. However each upper layer protocol can determine the
optimal use of tagged and untagged messages for itself. This
document will discuss when Data Source flexibility is of benefit
to applications
Note that in DDP there are two mechanisms for transferring data.
These two data transfer mechanisms are also enabled through
RDMAP, with additional control semantics:
* Untagged data transfer - the Data Source payload simply
consumes the first buffer in a queue of buffers that are
in Data Sink specified order (commonly referred to as the
Receive Queue), and
* Tagged data transfer - the Data Source explicitly states
which destination buffer is targeted, through use of an
STag. STag based transfers allow the Data Sink ULP to be
indifferent to what order (or in what messages) the Data
Source sent the data, or what order packets are received
in.
For DDP the two forms correspond to Untagged and Tagged DDP
Messages, respectively. For RDMAP the two forms correspond to
Send Type Messages and RDMA Messages (either RDMA Read or RDMA
Write Messages), respectively. Typically tagged data can be used
for payload transfer, while untagged is best used for control
messages. However each upper layer protocol can determine the
optimal use of tagged and untagged messages for itself. See
[APPLICABILITY] for more information on application applicability
for the two transfer mechanisms.
The host interfaces that could be exercised include: The host interfaces that could be exercised include:
* Privileged Control Interface - A Privileged Resource * Privileged Control Interface - A Privileged Resource
Manager uses the RNIC Interface to allocate and manage Manager uses the RNIC Interface to allocate and manage
RNIC Engine resources, control the state within the RNIC RNIC Engine resources, control the state within the RNIC
Engine, and monitor various events from the RNIC Engine. Engine, and monitor various events from the RNIC Engine.
It also uses this interface to act as a proxy for some It also uses this interface to act as a proxy for some
operations that a Non-Privileged ULP may require (after operations that a Non-Privileged ULP may require (after
performing appropriate countermeasures). performing appropriate countermeasures).
skipping to change at page 8, line 15 skipping to change at page 9, line 30
* Non-Privileged Data Transfer Interface - A Non-Privileged * Non-Privileged Data Transfer Interface - A Non-Privileged
ULP uses this interface to initiate and to check the ULP uses this interface to initiate and to check the
status of data transfer operations. status of data transfer operations.
* Privileged Data Transfer Interface - A superset of the * Privileged Data Transfer Interface - A superset of the
functionality provided by the Non-Privileged Data functionality provided by the Non-Privileged Data
Transfer Interface. The ULP is allowed to directly Transfer Interface. The ULP is allowed to directly
manipulate RNIC Engine mapping resources to map an STag manipulate RNIC Engine mapping resources to map an STag
to a ULP data buffer. to a ULP data buffer.
* Figure 1 also shows the ability to load new firmware in
the RNIC Engine. Not all RNICs will support this, but it
is shown for completeness and is also reviewed under
potential attacks.
If Internet control messages, such as ICMP, ARP, RIPv4, etc. are If Internet control messages, such as ICMP, ARP, RIPv4, etc. are
processed by the RNIC Engine, the threat analyses for those processed by the RNIC Engine, the threat analyses for those
protocols is also applicable, but outside the scope of this protocols is also applicable, but outside the scope of this
document. document.
2.2 Resources 2.2 Resources
This section describes the primary resources in the RNIC Engine This section describes the primary resources in the RNIC Engine
that could be affected if under attack. For RDMAP, all of the that could be affected if under attack. For RDMAP, all of the
defined resources apply. For DDP, all of the resources except the defined resources apply. For DDP, all of the resources except the
skipping to change at page 8, line 41 skipping to change at page 9, line 51
2.2.1 Stream Context Memory 2.2.1 Stream Context Memory
The state information for each Stream is maintained in memory, The state information for each Stream is maintained in memory,
which could be located in a number of places - on the NIC, inside which could be located in a number of places - on the NIC, inside
RAM attached to the NIC, in host memory, or in any combination of RAM attached to the NIC, in host memory, or in any combination of
the three, depending on the implementation. the three, depending on the implementation.
Stream Context Memory includes state associated with Data Stream Context Memory includes state associated with Data
Buffers. For Tagged Buffers, this includes how STag names, Data Buffers. For Tagged Buffers, this includes how STag names, Data
Buffers, and Page Translation Tables (see section 2.2.3 on page Buffers, and Page Translation Tables (see section 2.2.3)
9) interrelate. It also includes the list of Untagged Data interrelate. It also includes the list of Untagged Data Buffers
Buffers posted for reception of Untagged Messages (commonly posted for reception of Untagged Messages (commonly called the
called the Receive Queue), and a list of operations to perform to Receive Queue), and a list of operations to perform to send data
send data (commonly called the Send Queue). (commonly called the Send Queue).
2.2.2 Data Buffers 2.2.2 Data Buffers
There are two different ways to expose a local ULP's data buffer; As mentioned previously, there are two different ways to expose a
a buffer can be exposed for receiving RDMAP Send Type Messages local ULP's data buffers for data transfer; Untagged Data
(a.k.a. DDP Untagged Messages) on DDP Queue zero or the buffer Transfer - a buffer can be exposed for receiving RDMAP Send Type
can be exposed for remote access through STags (a.k.a. DDP Tagged Messages (a.k.a. DDP Untagged Messages) on DDP Queue zero - or
Messages). This distinction is important because the attacks and Tagged Data Transfer - the buffer can be exposed for remote
the countermeasures used to protect against the attack are access through STags (a.k.a. DDP Tagged Messages). This
different depending on the method for exposing the buffer to the distinction is important because the attacks and the
network. countermeasures used to protect against the attack are different
depending on the method for exposing the buffer to the network.
For the purposes of the security discussion, a single logical For the purposes of the security discussion, for Tagged Data
Data Buffer is exposed with a single Stag on a given Stream. Transfer a single logical Data Buffer is exposed with a single
Actual implementations may support scatter/gather capabilities to Stag on a given Stream. Actual implementations may support
enable multiple physical data buffers to be accessed with a scatter/gather capabilities to enable multiple physical data
single STag, but from a threat analysis perspective it is assumed buffers to be accessed with a single STag, but from a threat
that a single STag enables access to a single logical Data analysis perspective it is assumed that a single STag enables
Buffer. access to a single logical Data Buffer.
In any event, it is the responsibility of the Privileged Resource In any event, it is the responsibility of the Privileged Resource
Manager to ensure that no STag can be created that exposes memory Manager to ensure that no STag can be created that exposes memory
that the consumer had no authority to expose. that the consumer had no authority to expose.
A data buffer has specific access rights. The local ULP can
control whether a data buffer is exposed for local only, or local
and remote access, and assign specific access privileges (read,
write, read and write) on a per Stream basis.
For DDP, when an STag is advertised, the Remote Peer is
presumably given write access rights to the data (otherwise there
was not much point to the advertisement). For RDMAP, when a ULP
advertises an STag, it can enable write-only, read-only, or both
write and read access rights.
Similarly, some ULPs may wish to provide a single buffer with
different access rights on a per-Stream basis. For example, some
Streams may have read-only access, some may have remote read and
write access, while on other Streams only the local ULP/Local
Peer is allowed access.
2.2.3 Page Translation Tables 2.2.3 Page Translation Tables
Page Translation Tables are the structures used by the RNIC to be Page Translation Tables are the structures used by the RNIC to be
able to access ULP memory for data transfer operations. Even able to access ULP memory for data transfer operations. Even
though these structures are called "Page" Translation Tables, though these structures are called "Page" Translation Tables,
they may not reference a page at all - conceptually they are used they may not reference a page at all - conceptually they are used
to map a ULP address space representation (e.g. a virtual to map a ULP address space representation (e.g. a virtual
address) of a buffer to the physical addresses that are used by address) of a buffer to the physical addresses that are used by
the RNIC Engine to move data. If on a specific system a mapping the RNIC Engine to move data. If on a specific system a mapping
is not used, then a subset of the attacks examined may be is not used, then a subset of the attacks examined may be
appropriate. Note that the Page Translation Table may or may not appropriate. Note that the Page Translation Table may or may not
be a shared resource. be a shared resource.
2.2.4 STag Namespace 2.2.4 Protection Domain (PD)
A Protection Domain (PD) is a local construct to the RDMA
implementation, and never visible over the wire. Protection
Domains are assigned to two of the resources of concern, Stream
Context Memory and STags associated with Page Translation Table
entries and data buffers. A correct implementation of a
Protection Domain requires that resources which belong to a given
Protection Domain can not be used on a resource belonging to
another Protection Domain, because Protection Domain membership
is checked by the RNIC prior to taking any action involving such
a resource. Protection Domains are therefore used to ensure that
an STag can only be used to access an associated data buffer on
one or more Streams that are associated with the same Protection
Domain as the specific STag.
If an implementation chooses to not share resources between
Streams, it is recommended that each Stream be associated with
its own, unique Protection Domain. If an implementation chooses
to allow resource sharing, it is recommended that Protection
Domain be limited to the collection of Streams that have Partial
Mutual Trust with each other.
Note that a ULP (either Privileged or Non-Privileged) can
potentially have multiple Protection Domains. This could be used,
for example, to ensure that multiple clients of a server do not
have the ability to corrupt each other. The server would allocate
a Protection Domain per client to ensure that resources covered
by the Protection Domain could not be used by another (untrusted)
client.
2.2.5 STag Namespace and Scope
The DDP specification defines a 32-bit namespace for the STag. The DDP specification defines a 32-bit namespace for the STag.
Implementations may vary in terms of the actual number of STags Implementations may vary in terms of the actual number of STags
that are supported. In any case, this is a bounded resource that that are supported. In any case, this is a bounded resource that
can come under attack. Depending upon STag namespace allocation can come under attack. Depending upon STag namespace allocation
algorithms, the actual name space to attack may be significantly algorithms, the actual name space to attack may be significantly
less than 2^32. less than 2^32.
2.2.5 Completion Queues The scope of an STag is the set of DDP/RDMAP Streams on which the
STag is valid. If an STag is valid on a particular DDP/RDMAP
Stream, then that stream can modify the buffer, subject to the
access rights that the stream has for the STag (see Section 2.2.2
Data Buffers for additional information).
Completion Queues are used in this document to conceptually The analysis presented in this document assumes two mechanisms
for limiting the scope of Streams for which the STag is valid:
* Protection Domain scope. The STag is valid if used on
any Stream within a specific Protection Domain, and
is invalid if used on any Stream that is not a member
of the Protection Domain.
* Single Stream scope. The STag is valid on a single
Stream, regardless of what the Stream association is
to a Protection Domain. If used on any other Stream,
it is invalid.
2.2.6 Completion Queues
Completion Queues (CQ) are used in this document to conceptually
represent how the RNIC Engine notifies the ULP about the represent how the RNIC Engine notifies the ULP about the
completion of the transmission of data, or the completion of the completion of the transmission of data, or the completion of the
reception of data through the Data Transfer Interface. Because reception of data through the Data Transfer Interface
there could be many transmissions or receptions in flight at any (specifically for Untagged Data Transfer - Tagged Data Transfer
one time, completions are modeled as a queue rather than a single can not cause a completion to occur). Because there could be many
event. An implementation may also use the Completion Queue to transmissions or receptions in flight at any one time,
notify the ULP of other activities, for example, the completion completions are modeled as a queue rather than a single event. An
of a mapping of an STag to a specific ULP buffer. Completion implementation may also use the Completion Queue to notify the
Queues may be shared by a group of Streams, or may be designated ULP of other activities, for example, the completion of a mapping
to handle a specific Stream's traffic. of an STag to a specific ULP buffer. Completion Queues may be
shared by a group of Streams, or may be designated to handle a
specific Stream's traffic. Limiting Completion Queue association
to one, or a small number of RDMAP/DDP Streams can prevent
several forms of attacks by sharply limiting the scope of the
attack's effect.
Some implementations may allow this queue to be manipulated Some implementations may allow this queue to be manipulated
directly by both Non-Privileged and Privileged ULPs. directly by both Non-Privileged and Privileged ULPs.
2.2.6 Asynchronous Event Queue 2.2.7 Asynchronous Event Queue
The Asynchronous Event Queue is a queue from the RNIC to the The Asynchronous Event Queue is a queue from the RNIC to the
Privileged Resource Manager of bounded size. It is used by the Privileged Resource Manager of bounded size. It is used by the
RNIC to notify the host of various events which might require RNIC to notify the host of various events which might require
management action, including protocol violations, Stream state management action, including protocol violations, Stream state
changes, local operation errors, low water marks on receive changes, local operation errors, low water marks on receive
queues, and possibly other events. queues, and possibly other events.
The Asynchronous Event Queue is a resource that can be attacked The Asynchronous Event Queue is a resource that can be attacked
because Remote or Local Peers and/or ULPs can cause events to because Remote or Local Peers and/or ULPs can cause events to
occur which have the potential of overflowing the queue. occur which have the potential of overflowing the queue.
Note that an implementation is at liberty to implement the Note that an implementation is at liberty to implement the
functions of the Asynchronous Event Queue in a variety of ways, functions of the Asynchronous Event Queue in a variety of ways,
including multiple queues or even simple callbacks. All including multiple queues or even simple callbacks. All
vulnerabilities identified are intended to apply regardless of vulnerabilities identified are intended to apply regardless of
the implementation of the Asynchronous Event Queue. For example, the implementation of the Asynchronous Event Queue. For example,
a callback function may be viewed as simply a very short queue. a callback function may be viewed as simply a very short queue.
2.2.7 RDMA Read Request Queue 2.2.8 RDMA Read Request Queue
The RDMA Read Request Queue is the memory that holds state The RDMA Read Request Queue is the memory that holds state
information for one or more RDMA Read Request Messages that have information for one or more RDMA Read Request Messages that have
arrived, but for which the RDMA Read Response Messages have not arrived, but for which the RDMA Read Response Messages have not
yet been completely sent. Because potentially more than one RDMA yet been completely sent. Because potentially more than one RDMA
Read Request can be outstanding at one time, the memory is Read Request can be outstanding at one time, the memory is
modeled as a queue of bounded size. Some implementations may modeled as a queue of bounded size. Some implementations may
enable sharing of a single RDMA Read Request Queue across enable sharing of a single RDMA Read Request Queue across
multiple Streams. multiple Streams.
2.2.8 RNIC Interactions 2.3 RNIC Interactions
With RNIC resources and interfaces defined, it is now possible to With RNIC resources and interfaces defined, it is now possible to
examine the interactions supported by the generic RNIC functional examine the interactions supported by the generic RNIC functional
interfaces through each of the 3 interfaces - Privileged Control interfaces through each of the 3 interfaces - Privileged Control
Interface, Privileged Data Interface, and Non-Privileged Data Interface, Privileged Data Interface, and Non-Privileged Data
Interface. Interface. As mentioned previously in Section 2.1 Components,
there are two data transfer mechanisms to be examined - Untagged
data transfer and Tagged data transfer.
2.2.8.1 Privileged Control Interface Semantics 2.3.1 Privileged Control Interface Semantics
Generically, the Privileged Control Interface controls the RNIC's Generically, the Privileged Control Interface controls the RNIC's
allocation, deallocation, and initialization of RNIC global allocation, deallocation, and initialization of RNIC global
resources. This includes allocation and deallocation of Stream resources. This includes allocation and deallocation of Stream
Context Memory, Page Translation Tables, STag names, Completion Context Memory, Page Translation Tables, STag names, Completion
Queues, RDMA Read Request Queues, and Asynchronous Event Queues. Queues, RDMA Read Request Queues, and Asynchronous Event Queues.
The Privileged Control Interface is also typically used for The Privileged Control Interface is also typically used for
managing Non-Privileged ULP resources for the Non-Privileged ULP managing Non-Privileged ULP resources for the Non-Privileged ULP
(and possibly for the Privileged ULP as well). This includes (and possibly for the Privileged ULP as well). This includes
initialization and removal of Page Translation Table resources, initialization and removal of Page Translation Table resources,
and managing RNIC events (possibly managing all events for the and managing RNIC events (possibly managing all events for the
Asynchronous Event Queue). Asynchronous Event Queue).
2.2.8.2 Non-Privileged Data Interface Semantics 2.3.2 Non-Privileged Data Interface Semantics
The Non-Privileged Data Interface enables data transfer (transmit The Non-Privileged Data Interface enables data transfer (transmit
and receive) but does not allow initialization of the Page and receive) but does not allow initialization of the Page
Translation Table resources. However, once the Page Translation Translation Table resources. However, once the Page Translation
Table resources have been initialized, the interface may enable a Table resources have been initialized, the interface may enable a
specific STag mapping to be enabled and disabled by directly specific STag mapping to be enabled and disabled by directly
communicating with the RNIC, or create an STag mapping for a communicating with the RNIC, or create an STag mapping for a
buffer that has been previously initialized in the RNIC. buffer that has been previously initialized in the RNIC.
For RDMAP, ULP data can be sent by using RDMAP Send Type For RDMAP, ULP data can be sent by one of the previously
Messages, RDMA Read Responses, and RDMA Writes. ULP data described data transfer mechanisms - Untagged data transfer or
reception through RDMAP can be done by receiving Send Type Tagged Data Transfer. Three RDMAP data transfer mechanisms are
Messages into buffers that have been posted on the Receive Queue defined, one using Untagged data transfer (Send Type Messages),
or Shared Receive Queue. It can also be done by receiving RDMA and one using Tagged data transfer (RDMA Read Responses and RDMA
Writes). ULP data reception through RDMAP can be done by
receiving Send Type Messages into buffers that have been posted
on the Receive Queue or Shared Receive Queue. Thus a Receive
Queue or Shared Receive Queue can only be affected by Untagged
data transfer. Data reception can also be done by receiving RDMA
Write and RDMA Read Response Messages into buffers that have Write and RDMA Read Response Messages into buffers that have
previously been exposed for external write access through previously been exposed for external write access through
advertisement of an STag. Additionally, to cause ULP data to be advertisement of an STag (i.e. Tagged data transfer).
pulled (read) across the network, RDMAP uses an RDMA Read Request Additionally, to cause ULP data to be pulled (read) across the
Message (which only contains RDMAP control information necessary network, RDMAP uses an RDMA Read Request Message (which only
to access the ULP buffer to be read), to cause an RDMA Read contains RDMAP control information necessary to access the ULP
Response Message to be generated that contains the ULP data. buffer to be read), to cause an RDMA Read Response Message to be
generated that contains the ULP data.
For DDP, transmitting data means sending DDP Tagged or Untagged For DDP, transmitting data means sending DDP Tagged or Untagged
Messages. For data reception, for DDP it can receive Untagged Messages. For data reception, for DDP it can receive Untagged
Messages into buffers that have been posted on the Receive Queue Messages into buffers that have been posted on the Receive Queue
or Shared Receive Queue. It can also receive Tagged DDP Messages or Shared Receive Queue. It can also receive Tagged DDP Messages
into buffers that have previously been exposed for external write into buffers that have previously been exposed for external write
access through advertisement of an STag. access through advertisement of an STag.
Completion of data transmission or reception generally entails Completion of data transmission or reception generally entails
informing the ULP of the completed work by placing completion informing the ULP of the completed work by placing completion
information on the Completion Queue. information on the Completion Queue. For data reception, only an
Untagged data transfer can cause completion information to be put
in the Completion Queue.
2.2.8.3 Privileged Data Interface Semantics 2.3.3 Privileged Data Interface Semantics
The Privileged Data Interface semantics are a superset of the The Privileged Data Interface semantics are a superset of the
Non-Privileged Data Transfer semantics. The interface can do Non-Privileged Data Transfer semantics. The interface can do
everything defined in the prior section, as well as everything defined in the prior section, as well as
create/destroy buffer to STag mappings directly. This generally create/destroy buffer to STag mappings directly. This generally
entails initialization or clearing of Page Translation Table entails initialization or clearing of Page Translation Table
state in the RNIC. state in the RNIC.
2.2.9 Initialization of RNIC Data Structures for Data Transfer 2.3.4 Initialization of RNIC Data Structures for Data Transfer
Initialization of the mapping between an STag and a Data Buffer Initialization of the mapping between an STag and a Data Buffer
can be viewed in the abstract as two separate operations: can be viewed in the abstract as two separate operations:
a. Initialization of the allocated Page Translation Table a. Initialization of the allocated Page Translation Table
entries with the location of the Data Buffer, and entries with the location of the Data Buffer, and
b. Initialization of a mapping from an allocated STag name b. Initialization of a mapping from an allocated STag name
to a set of Page Translation Table entry(s) or partial- to a set of Page Translation Table entry(s) or partial-
entries. entries.
skipping to change at page 12, line 46 skipping to change at page 15, line 34
Buffer. For the case of an Untagged Data Buffer, there is no wire Buffer. For the case of an Untagged Data Buffer, there is no wire
visible mapping between an STag and the Data Buffer. Note that visible mapping between an STag and the Data Buffer. Note that
there may, in fact, be an STag which represents the buffer, if an there may, in fact, be an STag which represents the buffer, if an
implementation chooses to internally represent Untagged Data implementation chooses to internally represent Untagged Data
Buffer using STags. However, because the STag by definition is Buffer using STags. However, because the STag by definition is
not visible on the wire, this is a local host implementation not visible on the wire, this is a local host implementation
specific issue which should be analyzed in the context of a local specific issue which should be analyzed in the context of a local
host implementation specific security analysis, and thus is host implementation specific security analysis, and thus is
outside the scope of this document. outside the scope of this document.
For a Tagged Data Buffer, either the Privileged ULP, the Non- For a Tagged Data Buffer, either the Privileged ULP or the
Privileged ULP, or the Privileged Resource Manager acting on Privileged Resource Manager acting on behalf of the Non-
behalf of the Non-Privileged ULP may initialize a mapping from an Privileged ULP may initialize a mapping from an STag to a Page
STag to a Page Translation Table, or may have the ability to Translation Table, or may have the ability to simply
simply enable/disable an existing STag to Page Translation Table enable/disable an existing STag to Page Translation Table
mapping. There may also be multiple STag names which map to a mapping. There may also be multiple STag names which map to a
specific group of Page Translation Table entries (or sub- specific group of Page Translation Table entries (or sub-
entries). Specific security issues with this level of flexibility entries). Specific security issues with this level of flexibility
are examined in Section 5.3.3 Multiple STags to access the same are examined in Section 6.2.3 Multiple STags to access the same
buffer on page 23. buffer.
There are a variety of implementation options for initialization There are a variety of implementation options for initialization
of Page Translation Table entries and mapping an STag to a group of Page Translation Table entries and mapping an STag to a group
of Page Translation Table entries which have security of Page Translation Table entries which have security
repercussions. This includes support for separation of Mapping an repercussions. This includes support for separation of Mapping an
STag versus mapping a set of Page Translation Table entries, and STag versus mapping a set of Page Translation Table entries, and
support for ULPs directly manipulating STag to Page Translation support for ULPs directly manipulating STag to Page Translation
Table entry mappings (versus requiring access through the Table entry mappings (versus requiring access through the
Privileged Resource Manager). Privileged Resource Manager).
2.2.10 RNIC Data Transfer Interactions 2.3.5 RNIC Data Transfer Interactions
RNIC Data Transfer operations can be subdivided into send RNIC Data Transfer operations can be subdivided into send
operations and receive operations. operations and receive operations.
For send operations, there is typically a queue that enables the For send operations, there is typically a queue that enables the
ULP to post multiple operation requests to send data (referred to ULP to post multiple operation requests to send data (referred to
as the Send Queue). Depending upon the implementation, Data as the Send Queue). Depending upon the implementation, Data
Buffers used in the operations may or may not have Page Buffers used in the operations may or may not have Page
Translation Table entries associated with them, and may or may Translation Table entries associated with them, and may or may
not have STags associated with them. Because this is a local host not have STags associated with them. Because this is a local host
specific implementation issue rather than a protocol issue, the specific implementation issue rather than a protocol issue, the
security analysis of threats and mitigations is left to the host security analysis of threats and mitigations is left to the host
implementation. implementation.
Receive operations are different for Tagged Data Buffers versus Receive operations are different for Tagged Data Buffers versus
Untagged Data Buffers. If more than one Untagged Data Buffer can Untagged Data Buffers (i.e. Tagged data transfer vs. Untagged
be posted by the ULP, the DDP specification requires that they be data transfer). For Untagged data transfer, if more than one
consumed in sequential order. Thus the most general Untagged Data Buffer can be posted by the ULP, the DDP
implementation is that there is a sequential queue of receive specification requires that they be consumed in sequential order
Untagged Data Buffers (Receive Queue). Some implementations may (the RDMAP specification also requires this). Thus the most
also support sharing of the sequential queue between multiple general implementation is that there is a sequential queue of
Streams. In this case defining "sequential" becomes non-trivial - receive Untagged Data Buffers (Receive Queue). Some
in general the buffers for a single Stream are consumed from the implementations may also support sharing of the sequential queue
queue in the order that they were placed on the queue, but there between multiple Streams. In this case defining "sequential"
is no consumption order guarantee between Streams. becomes non-trivial - in general the buffers for a single Stream
are consumed from the queue in the order that they were placed on
the queue, but there is no consumption order guarantee between
Streams.
For receive Tagged Data Buffers, at some time prior to data For receive Tagged data transfer (i.e. Tagged Data Buffers, RDMA
Write Buffers, or RDMA Read Buffers), at some time prior to data
transfer, the mapping of the STag to specific Page Translation transfer, the mapping of the STag to specific Page Translation
Table entries (if present) and the mapping from the Page Table entries (if present) and the mapping from the Page
Translation Table entries to the Data Buffer must have been Translation Table entries to the Data Buffer must have been
initialized (see section 2.2.9 for interaction details). initialized (see section 2.3.4 for interaction details).
3 Trust and Resource Sharing 3 Trust and Resource Sharing
It is assumed that in general the Local and Remote Peer are It is assumed that in general the Local and Remote Peer are
untrusted, and thus attacks by either should have mitigations in untrusted, and thus attacks by either should have mitigations in
place. place.
A separate, but related issue is resource sharing between A separate, but related issue is resource sharing between
multiple Streams. If local resources are not shared, the multiple Streams. If local resources are not shared, the
resources are dedicated on a per Stream basis. Resources are resources are dedicated on a per Stream basis. Resources are
defined in Section 2.2 Resources on page 8. The advantage of not defined in Section 2.2 Resources. The advantage of not sharing
sharing resources between Streams is that it reduces the types of resources between Streams is that it reduces the types of attacks
attacks that are possible. The disadvantage of not sharing that are possible. The disadvantage of not sharing resources is
resources is that ULPs might run out of resources. Thus there can that ULPs might run out of resources. Thus there can be a strong
be a strong incentive for sharing resources, if the security incentive for sharing resources, if the security issues
issues associated with the sharing of resources can be mitigated. associated with the sharing of resources can be mitigated.
It is assumed in this document that the component that implements It is assumed in this document that the component that implements
the mechanism to control sharing of the RNIC Engine resources is the mechanism to control sharing of the RNIC Engine resources is
the Privileged Resource Manager. The RNIC Engine exposes its the Privileged Resource Manager. The RNIC Engine exposes its
resources through the RNIC Interface to the Privileged Resource resources through the RNIC Interface to the Privileged Resource
Manager. All Privileged and Non-Privileged ULPs request resources Manager. All Privileged and Non-Privileged ULPs request resources
from the Resource Manager (note that by definition both the Non- from the Resource Manager (note that by definition both the Non-
Privileged and the Privileged application might try to greedily Privileged and the Privileged application might try to greedily
consume resources, thus creating a potential Denial of Service consume resources, thus creating a potential Denial of Service
(DOS) attack). The Resource Manager implements resource (DOS) attack). The Resource Manager implements resource
skipping to change at page 14, line 47 skipping to change at page 17, line 47
could affect other ULPs MUST be done using the Privileged could affect other ULPs MUST be done using the Privileged
Resource Manager as a proxy. All ULP resource allocation requests Resource Manager as a proxy. All ULP resource allocation requests
for scarce resources MUST also be done using a Privileged for scarce resources MUST also be done using a Privileged
Resource Manager. Resource Manager.
The sharing of resources across Streams should be under the The sharing of resources across Streams should be under the
control of the ULP, both in terms of the trust model the ULP control of the ULP, both in terms of the trust model the ULP
wishes to operate under, as well as the level of resource sharing wishes to operate under, as well as the level of resource sharing
the ULP wishes to give local processes. For more discussion on the ULP wishes to give local processes. For more discussion on
types of trust models which combine partial trust and sharing of types of trust models which combine partial trust and sharing of
resources, see Appendix C: Partial Trust Taxonomy on page 49. resources, see Appendix C: Partial Trust Taxonomy.
The Privileged Resource Manager MUST NOT assume different Streams The Privileged Resource Manager MUST NOT assume different Streams
share Partial Mutual Trust unless there is a mechanism to ensure share Partial Mutual Trust unless there is a mechanism to ensure
that the Streams do indeed share Partial Mutual Trust. This can that the Streams do indeed share Partial Mutual Trust. This can
be done in several ways, including explicit notification from the be done in several ways, including explicit notification from the
ULP that owns the Streams. ULP that owns the Streams.
4 Attacker Capabilities 4 Attacker Capabilities
An attacker's capabilities delimit the types of attacks that An attacker's capabilities delimit the types of attacks that
skipping to change at page 15, line 19 skipping to change at page 18, line 19
initial LLP Stream (and connection) be set up prior to initial LLP Stream (and connection) be set up prior to
transferring RDMAP/DDP Messages. This requires at least one transferring RDMAP/DDP Messages. This requires at least one
round-trip handshake to occur. round-trip handshake to occur.
If the attacker is not the Remote Peer that created the initial If the attacker is not the Remote Peer that created the initial
connection, then the attacker's capabilities can be segmented connection, then the attacker's capabilities can be segmented
into send only capabilities or send and receive capabilities. into send only capabilities or send and receive capabilities.
Attacking with send only capabilities requires the attacker to Attacking with send only capabilities requires the attacker to
first guess the current LLP Stream parameters before they can first guess the current LLP Stream parameters before they can
attack RNIC resources (e.g. TCP sequence number). If this class attack RNIC resources (e.g. TCP sequence number). If this class
of attacker also has receive capabilities, they are typically of attacker also has receive capabilities and the ability to pose
referred to as a "man-in-the-middle" attacker, and they have a as the receiver to the sender and the sender to the receiver,
much wider ability to attack RNIC resources. The breadth of they are typically referred to as a "man-in-the-middle" attacker
attack is essentially the same as that of an attacking Remote [RFC3552]. A man-in-the-middle attacker has a much wider ability
Peer (i.e. the Remote Peer that setup the initial LLP Stream). to attack RNIC resources. The breadth of attack is essentially
the same as that of an attacking Remote Peer (i.e. the Remote
Peer that setup the initial LLP Stream).
5 Attacks and Countermeasures 5 Attacks That Can be Mitigated With End-to-End Security
This section describes the attacks that are possible against the This section describes the RDMAP/DDP attacks where the only
RDMA system defined in Figure 1 - RDMA Security Model and the solution is to implement some form of end-to-end security. The
RNIC Engine resources defined in Section 2.2. The analysis analysis includes a detailed description of each attack, what is
includes a detailed description of each attack, what is being being attacked, and a description of the countermeasures that can
attacked, and a description of the countermeasures that can be be taken to thwart the attack.
taken to thwart the attack.
Some forms of attack involve modifying the RDMAP or DDP payload
by a network based attacker or involve monitoring the traffic to
discover private information. An effective tool to ensure
confidentiality is to encrypt the data stream through mechanisms
such as IPsec encryption. An effective tool to ensure the remote
entity is who they claim to be as well as ensuring that the
payload is unmodified as it traverses the network is the use of
authentication protocols such as IPSec Authentication.
Note that connection setup and teardown is presumed to be done in Note that connection setup and teardown is presumed to be done in
stream mode (i.e. no RDMA encapsulation of the payload), so there stream mode (i.e. no RDMA encapsulation of the payload), so there
are no new attacks related to connection setup/teardown beyond are no new attacks related to connection setup/teardown beyond
what is already present in the LLP (e.g. TCP or SCTP). Note, what is already present in the LLP (e.g. TCP or SCTP). Note,
however, that RDMAP/DDP parameters may be exchanged in stream however, that RDMAP/DDP parameters may be exchanged in stream
mode, and if they are corrupted by an attacker unintended mode, and if they are corrupted by an attacker unintended
consequences will result. Therefore, any existing mitigations for consequences will result. Therefore, any existing mitigations for
LLP Spoofing, Tampering, Repudiation, Information Disclosure, LLP Spoofing, Tampering, Repudiation, Information Disclosure,
Denial of Service, or Elevation of Privilege continue to apply Denial of Service, or Elevation of Privilege continue to apply
(and are out of scope of this document). Thus the analysis in (and are out of scope of this document). Thus the analysis in
this section focuses on attacks that are present regardless of this section focuses on attacks that are present regardless of
the LLP Stream type. the LLP Stream type.
The attacks are classified into five categories: Spoofing, Tampering is any modification of the legitimate traffic (machine
Tampering, Information Disclosure, Denial of Service (DoS) internal or network). Spoofing attack is a special case of
attacks, and Elevation of Privileges. Tampering is any tampering where the attacker falsifies an identity of the Remote
modification of the legitimate traffic (machine internal or Peer (identity can be an IP address, machine name, ULP level
network). Spoofing attack is a special case of tampering where identity etc.).
the attacker falsifies an identity of the Remote Peer (identity
can be an IP address, machine name, ULP level identity etc.).
5.1 Tools for Countermeasures 5.1 Spoofing
The tools described in this section are the primary mechanisms Spoofing attacks can be launched by the Remote Peer, or by a
that can be used to provide countermeasures to potential attacks. network based attacker. A network based spoofing attack applies
to all Remote Peers. This section analyzes the various types of
spoofing attacks applicable to RDMAP & DDP.
5.1.1 Protection Domain (PD) 5.1.1 Impersonation
A Protection Domain (PD) is a local construct to the RDMA A network based attacker can impersonate a legal RDMAP/DDP Peer
implementation, and never visible over the wire. Protection (by spoofing a legal IP address). This can either be done as a
Domains are assigned to two of the resources of concern, Stream blind attack (see [RFC3552]) or by establishing an RDMAP/DDP
Context Memory and STags associated with Page Translation Table Stream with the victim. Because an RDMAP/DDP Stream requires an
entries and data buffers. A correct implementation of a LLP Stream to be fully initialized (e.g. for [TCP] it is in the
Protection Domain requires that resources which belong to a given ESTABLISHED state), existing transport layer protection
Protection Domain can not be used on a resource belonging to mechanisms against blind attacks remain in place.
another Protection Domain, because Protection Domain membership
is checked by the RNIC prior to taking any action involving such
a resource. Protection Domains are therefore used to ensure that
an STag can only be used to access an associated data buffer on
one or more Streams that are associated with the same Protection
Domain as the specific STag.
If an implementation chooses to not share resources between For a blind attack to succeed, it requires the attacker to inject
Streams, it is recommended that each Stream be associated with a valid transport layer segment (e.g. for TCP it must match at
its own, unique Protection Domain. If an implementation chooses least the 4-tuple as well as guess a sequence number within the
to allow resource sharing, it is recommended that Protection window) while also guessing valid RDMAP or DDP parameters. There
Domain be limited to the collection of Streams that have Partial are many ways to attack the RDMAP/DDP protocol if the transport
Mutual Trust with each other. protocol is assumed to be vulnerable. For example, for Tagged
Messages, this entails guessing the STag and TO values. If the
attacker wishes to simply terminate the connection, it can do so
by correctly guessing the transport & network layer values, and
providing an invalid STag. Per the DDP specification, if an
invalid STag is received, the Stream is torn down and the Remote
Peer is notified with an error. If an attacker wishes to
overwrite an Advertised Buffer, it must successfully guess the
correct STag and TO. Given that the TO often will start at zero,
this is straightforward. The value of the STag should be chosen
at random, as discussed in Section 6.1.1 Using an STag on a
Different Stream. For Untagged Messages, if the MSN is invalid
then the connection may be torn down. If it is valid, then the
receive buffers can be corrupted.
Note that a ULP (either Privileged or Non-Privileged) can End-to-end authentication (e.g. IPsec or ULP authentication)
potentially have multiple Protection Domains. This could be used, provides protection against either the blind attack or the
for example, to ensure that multiple clients of a server do not connected attack.
have the ability to corrupt each other. The server would allocate
a Protection Domain per client to ensure that resources covered
by the Protection Domain could not be used by another (untrusted)
client.
5.1.2 Limiting STag Scope 5.1.2 Stream Hijacking
The key to protecting a local data buffer is to limit the scope Stream hijacking happens when a network based attacker eavesdrops
of its STag to the level appropriate for the Streams which share the LLP connection through the Stream establishment phase, and
Partial Mutual Trust. The scope of the STag can be measured in waits until the authentication phase (if such a phase exists) is
multiple ways. completed successfully. The attacker then spoofs the IP address
and re-directs the Stream from the victim to its own machine. For
example, an attacker can wait until an iSCSI authentication is
completed successfully, and then hijack the iSCSI Stream.
* Number of Connections and/or Streams on which the STag is The best protection against this form of attack is end-to-end
valid. One way to limit the scope of the STag is to limit integrity protection and authentication, such as IPsec, to
the connections and/or Streams that are allowed to use prevent spoofing. Another option is to provide a physically
the STag. As noted in the previous section, use of segregated network for security. Discussion of physical security
Protection Domains appropriately can limit the scope of is out of scope for this document.
the STag. The analysis presented in this document assumes
two mechanisms for limiting the scope of Streams for
which the STag is valid:
* Protection Domain scope. The STag is valid if used on Because the connection and/or Stream itself is established by the
any Stream within a specific Protection Domain, and LLP, some LLPs are more difficult to hijack than others. Please
is invalid if used on any Stream that is not a member see the relevant LLP documentation on security issues around
of the Protection Domain. connection and/or Stream hijacking.
* Single Stream scope. The STag is valid on a single 5.1.3 Man-in-the-Middle Attack
Stream, regardless of what the Stream association is
to a Protection Domain. If used on any other Stream,
it is invalid.
* Limit the time an STag is valid. By Invalidating an If a network based attacker has the ability to delete or modify
advertised STag (e.g., revoking remote access to the packets which will still be accepted by the LLP (e.g., TCP
buffers described by an STag when done with the sequence number is correct) then the Stream can be exposed to a
transfer), an entire class of attacks can be eliminated. man-in-the-middle attack. One style of attack is for the man-in-
the-middle to send Tagged Messages (either RDMAP or DDP). If it
can discover a buffer that has been exposed for STag enabled
access, then the man-in-the-middle can use an RDMA Read operation
to read the contents of the associated data buffer, perform an
RDMA Write Operation to modify the contents of the associated
data buffer, or invalidate the STag to disable further access to
the buffer.
* Limit the buffer the STag can reference. Limiting the The best protection against this form of attack is end-to-end
scope of an STag access to just the intended portion of integrity protection and authentication, such as IPsec, to
the ULP buffers to be exposed is critical to prevent prevent spoofing or tampering. If authentication and integrity
certain forms of attacks. protections are not used, then physical protection must be
employed to prevent man-in-the-middle attacks.
* Allocating and/or advertising STag numbers in an Because the connection/Stream itself is established by the LLP,
unpredictable way. If STags are allocated/advertised some LLPs are more exposed to man-in-the-middle attack than
using an algorithm which makes it hard for the attacker others. Please see the relevant LLP documentation on security
to guess which STag(s) are currently in use, it makes it issues around connection and/or Stream hijacking.
more difficult for an attacker to guess the correct
value. As stated in the RDMAP specification [RDMAP], an
invalid STag will cause the RDMAP Stream to be
terminated. For the case of [DDP], at a minimum it must
signal an error to the ULP. This permits the ULP to
detect such attempts, and take countermeasures. Commonly,
the ULP will cause the DDP Stream to be immediately
terminated.
5.1.3 Access Rights Another approach is to restrict access to only the local
subnet/link, and provide some mechanism to limit access, such as
physical security or 802.1.x. This model is an extremely limited
deployment scenario, and will not be further examined here.
Access Rights associated with a specific advertised STag or 5.2 Tampering - Network based modification of buffer content
RDMAP/DDP Stream provide another mechanism for ULPs to limit the
attack capabilities of the Remote Peer. The local ULP can control
whether a data buffer is exposed for local only, or local and
remote access, and assign specific access privileges (read,
write, read and write) on a per Stream basis.
For DDP, when an STag is advertised, the Remote Peer is This is actually a man in the middle attack - but only on the
presumably given write access rights to the data (otherwise there content of the buffer, as opposed to the man in the middle attack
was not much point to the advertisement). For RDMAP, when a ULP presented above, where both the signaling and content can be
advertises an STag, it can enable write-only, read-only, or both modified. See Section 5.1.3 Man-in-the-Middle Attack.
write and read access rights.
Similarly, some ULPs may wish to provide a single buffer with 5.3 Information Disclosure - Network Based Eavesdropping
different access rights on a per-Stream basis. For example, some
Streams may have read-only access, some may have remote read and
write access, while on other Streams only the local ULP/Local
Peer is allowed access.
5.1.4 Limiting the Scope of the Completion Queue An attacker that is able to eavesdrop on the network can read the
content of all read and write accesses to a Peer's buffers. To
prevent information disclosure, the read/written data must be
encrypted. See also Section 5.1.3 Man-in-the-Middle Attack. The
encryption can be done either by the ULP, or by a protocol that
can provide security services to RDMAP & DDP (e.g. IPsec).
Completions associated with sending and receiving data, or 5.4 Specific Requirements for Security Services
setting up buffers for sending and receiving data, could be
accumulated in a shared Completion Queue for a group of RDMAP/DDP
Streams, or a specific RDMAP/DDP Stream could have a dedicated
Completion Queue. Limiting Completion Queue association to one,
or a small number of RDMAP/DDP Streams can prevent several forms
of Denial of Service attacks, by sharply limiting the scope of
the attack's effect.
5.1.5 Limiting the Scope of an Error Generally speaking, Stream confidentiality protects against
eavesdropping. Stream and/or session authentication and integrity
protection is a counter measurement against various spoofing and
tampering attacks. The effectiveness of authentication and
integrity against a specific attack depend on whether the
authentication is machine level authentication (such as IPsec),
or ULP authentication.
To prevent a variety of attacks, it is important that an 5.4.1 Introduction to Security Options
RDMAP/DDP implementation be robust in the face of errors. If an
error on a specific Stream can cause other unrelated Streams to
fail, then a broad class of attacks are enabled against the
implementation.
For example, an error on a specific RDMAP Stream should not cause The following security services can be applied to an RDMAP/DDP
the RNIC to stop processing incoming packets, or corrupt a Stream:
receive queue for an unrelated Stream.
5.2 Spoofing 1. Session confidentiality - protects against eavesdropping
(section 5.3).
Spoofing attacks can be launched by the Remote Peer, or by a 2. Per-packet data source authentication - protects against the
network based attacker. A network based spoofing attack applies following spoofing attacks: network based impersonation
to all Remote Peers. (section 5.1.1), Stream hijacking (section 5.1.2), and man in
the middle (section 5.1.3).
Because the RDMAP Stream requires an LLP Stream to be fully 3. Per-packet integrity - protects against tampering done by
initialized (e.g. for [TCP] it is in the ESTABLISHED state), network based modification of buffer content (section 5.2)
certain types of traditional forms of wire attacks do not apply -
- an end-to-end handshake must have occurred to establish the
RDMAP Stream. So, the only form of spoofing that applies is one
when an attacker can both send and receive packets. Yet even with
this limitation the Stream is still exposed to the following
spoofing attacks.
5.2.1 Impersonation 4. Packet sequencing - protects against replay attacks, which is
a special case of the above tampering attack.
A network based attacker can impersonate a legal RDMAP/DDP Peer If an RDMAP/DDP Stream may be subject to impersonation attacks,
(by spoofing a legal IP address), and establish an RDMAP/DDP or Stream hijacking attacks, it is recommended that the Stream be
Stream with the victim. End-to-end authentication (i.e. IPsec, authenticated, integrity protected, and protected from replay
SSL or ULP authentication) provides protection against this attacks; it may use confidentiality protection to protect from
attack. For additional information see Section 6, Security eavesdropping (in case the RDMAP/DDP Stream traverses a public
Services for RDMAP and DDP, on page 36. network).
5.2.2 Stream Hijacking IPsec is a protocol suite which is used to secure communication
at the network layer between two peers. The IPsec protocol suite
is specified within the IP Security Architecture [RFC2401], IKE
[RFC2409], IPsec Authentication Header (AH) [RFC2402] and IPsec
Encapsulating Security Payload (ESP) [RFC2406] documents. IKE is
the key management protocol while AH and ESP are used to protect
IP traffic. Please see those RFCs for a complete description of
the respective protocols.
Stream hijacking happens when a network based attacker eavesdrops IPsec is capable of providing the above security services for IP
the LLP connection through the Stream establishment phase, and and TCP traffic respectively. ULP protocols are able to provide
waits until the authentication phase (if such a phase exists) is only part of the above security services.
completed successfully. The attacker then spoofs the IP address
and re-direct the Stream from the victim to its own machine. For
example, an attacker can wait until an iSCSI authentication is
completed successfully, and then hijack the iSCSI Stream.
The best protection against this form of attack is end-to-end 5.4.2 TLS is Inappropriate for DDP/RDMAP Security
integrity protection and authentication, such as IPsec (see
Section 6, Security Services for RDMAP and DDP, on page 36), to
prevent spoofing. Another option is to provide physical security.
Discussion of physical security is out of scope for this
document.
Because the connection and/or Stream itself is established by the TLS [RFC 2246] provide Stream authentication, integrity and
LLP, some LLPs are more difficult to hijack than others. Please confidentiality for TCP based ULPs. TLS supports one-way (server
see the relevant LLP documentation on security issues around only) or mutual certificates based authentication.
connection and/or Stream hijacking.
5.2.3 Man in the Middle Attack There are at least two limitations that make TLS underneath RDMAP
inappropriate for DDP/RDMA security:
If a network based attacker has the ability to delete, inject 1. The maximum length supported by the TLS record layer protocol
replay, or modify packets which will still be accepted by the LLP is 2^14 bytes - longer packets must be fragmented (as a
(e.g., TCP sequence number is correct) then the Stream can be comparison, the maximum length of an Untagged DDP Message is
exposed to a man in the middle attack. One style of attack is for roughly 2^32).
the man-in-the-middle to send Tagged Messages (either RDMAP or
DDP). If it can discover a buffer that has been exposed for STag
enabled access, then the man-in-the-middle can use an RDMA Read
operation to read the contents of the associated data buffer,
perform an RDMA Write Operation to modify the contents of the
associated data buffer, or invalidate the STag to disable further
access to the buffer.
The best protection against this form of attack is end-to-end 2. TLS is a connection oriented protocol. If a stream cipher or
integrity protection and authentication, such as IPsec (see block cipher in CBC mode is used for bulk encryption, then a
Section 6 Security Services for RDMAP and DDP on page 36), to packet can be decrypted only after all the packets preceding
prevent spoofing or tampering. If Stream or session level it have already arrived. If TLS is used to protect DDP/RDMAP
authentication and integrity protection are not used, then traffic, then TLS must gather all out-of-order packets before
physical protection must be employed, lest a man-in-the-middle RDMAP/DDP can place them into the ULP buffer. Thus one of the
attack occur, enabling spoofing and tampering. primary features of DDP/RDMAP - enabling implementations to
have a flow-through architecture with little to no buffering,
can not be achieved if TLS is used to protect the data
stream.
Because the connection/Stream itself is established by the LLP, If TLS is layered on top of RDMAP or DDP, TLS does not protect
some LLPs are more exposed to man-in-the-middle attack than the RDMAP and/or DDP headers. Thus a man-in-the-middle attack can
others. Please see the relevant LLP documentation on security still occur by modifying the RDMAP/DDP header to incorrectly
issues around connection and/or Stream hijacking. place the data into the wrong buffer, thus effectively corrupting
the data stream.
Another approach is to restrict access to only the local For these reasons, it is NOT RECOMMENDED that TLS be layered on
subnet/link, and provide some mechanism to limit access, such as top of RDMAP or DDP.
physical security or 802.1.x. This model is an extremely limited
deployment scenario, and will not be further examined here.
5.2.4 Using an STag on a Different Stream 5.4.3 ULPs Which Provide Security
ULPs which provide integrated security but wish to leverage
lower-layer protocol security should be aware of security
concerns around correlating a specific channel's security
mechanisms to the authentication performed by the ULP. See
[NFSv4CHANNEL] for additional information on a promising approach
called "channel binding". From [NFSv4CHANNEL]:
"The concept of channel bindings allows applications to
prove that the end-points of two secure channels at
different network layers are the same by binding
authentication at one channel to the session protection at
the other channel. The use of channel bindings allows
applications to delegate session protection to lower layers,
which may significantly improve performance for some
applications."
5.4.4 Requirements for IPsec Encapsulation of DDP
The IP Storage working group has spent significant time and
effort to define the normative IPsec requirements for IP Storage
[RFC3723]. Portions of that specification are applicable to a
wide variety of protocols, including the RDDP protocol suite. In
order to not replicate this effort, an RNIC implementation MUST
follow the requirements defined in RFC3723 Section 2.3 and
Section 5, including the associated normative references for
those sections. Note that this means that support for IPSEC ESP
mode is normative.
Additionally, since IPsec acceleration hardware may only be able
to handle a limited number of active IKE Phase 2 SAs, Phase 2
delete messages may be sent for idle SAs, as a means of keeping
the number of active Phase 2 SAs to a minimum. The receipt of an
IKE Phase 2 delete message MUST NOT be interpreted as a reason
for tearing down an DDP/RDMA Stream. Rather, it is preferable to
leave the Stream up, and if additional traffic is sent on it, to
bring up another IKE Phase 2 SA to protect it. This avoids the
potential for continually bringing Streams up and down.
Note that there are serious security issues if IPsec is not
implemented end-to-end. For example, if IPsec is implemented as a
tunnel in the middle of the network, any hosts between the Peer
and the IPsec tunneling device can freely attack the unprotected
Stream.
6 Attacks from Remote Peers
This section describes remote attacks that are possible against
the RDMA system defined in Figure 1 - RDMA Security Model and the
RNIC Engine resources defined in Section 2.2. The analysis
includes a detailed description of each attack, what is being
attacked, and a description of the countermeasures that can be
taken to thwart the attack.
The attacks are classified into five categories: Spoofing,
Tampering, Information Disclosure, Denial of Service (DoS)
attacks, and Elevation of Privileges. As mentioned previously,
tampering is any modification of the legitimate traffic (machine
internal or network). A spoofing attack is a special case of
tampering where the attacker falsifies an identity of the Remote
Peer (identity can be an IP address, machine name, ULP level
identity etc.).
6.1 Spoofing
This section analyzes the various types of spoofing attacks
applicable to RDMAP & DDP. Spoofing attacks can be launched by
the Remote Peer, or by a network based attacker. For
countermeasures against a network based attacker, see Section 5
Attacks That Can be Mitigated With End-to-End Security.
6.1.1 Using an STag on a Different Stream
One style of attack from the Remote Peer is for it to attempt to One style of attack from the Remote Peer is for it to attempt to
use STag values that it is not authorized to use. Note that if use STag values that it is not authorized to use. Note that if
the Remote Peer sends an invalid STag to the Local Peer, per the the Remote Peer sends an invalid STag to the Local Peer, per the
DDP and RDMAP specifications, the Stream must be torn down. Thus DDP and RDMAP specifications, the Stream must be torn down. Thus
the threat exists if an STag has been enabled for Remote Access the threat exists if an STag has been enabled for Remote Access
on one Stream and a Remote Peer is able to use it on an unrelated on one Stream and a Remote Peer is able to use it on an unrelated
Stream. If the attack is successful, the attacker could Stream. If the attack is successful, the attacker could
potentially be able to perform either RDMA Read Operations to potentially be able to perform either RDMA Read Operations to
read the contents of the associated data buffer, perform RDMA read the contents of the associated data buffer, perform RDMA
skipping to change at page 21, line 50 skipping to change at page 26, line 46
by-one STag to be used. For additional protection, an by-one STag to be used. For additional protection, an
implementation should allocate STags in such a fashion that it is implementation should allocate STags in such a fashion that it is
difficult to predict the next allocated STag number, and also difficult to predict the next allocated STag number, and also
ensure that STags are reused at as slow a rate as possible. Any ensure that STags are reused at as slow a rate as possible. Any
allocation method which would lead to intentional or allocation method which would lead to intentional or
unintentional reuse of an STag by the peer should be avoided unintentional reuse of an STag by the peer should be avoided
(e.g. a method which always starts with a given STag and (e.g. a method which always starts with a given STag and
monotonically increases it for each new allocation, or a method monotonically increases it for each new allocation, or a method
which always uses the same STag for each operation). which always uses the same STag for each operation).
5.3 Tampering 6.2 Tampering
A Remote Peer or a network based attacker can attempt to tamper A Remote Peer or a network based attacker can attempt to tamper
with the contents of data buffers on a Local Peer that have been with the contents of data buffers on a Local Peer that have been
enabled for remote write access. The types of tampering attacks enabled for remote write access. The types of tampering attacks
that are possible are outlined in the sections that follow. from a Remote Peer are outlined in the sections that follow. For
countermeasures against a network based attacker, see Section 5
Attacks That Can be Mitigated With End-to-End Security.
5.3.1 Buffer Overrun - RDMA Write or Read Response 6.2.1 Buffer Overrun - RDMA Write or Read Response
This attack is an attempt by the Remote Peer to perform an RDMA This attack is an attempt by the Remote Peer to perform an RDMA
Write or RDMA Read Response to memory outside of the valid length Write or RDMA Read Response to memory outside of the valid length
range of the data buffer enabled for remote write access. This range of the data buffer enabled for remote write access. This
attack can occur even when no resources are shared across attack can occur even when no resources are shared across
Streams. This issue can also arise if the ULP has a bug. Streams. This issue can also arise if the ULP has a bug.
The countermeasure for this type of attack must be in the RNIC The countermeasure for this type of attack must be in the RNIC
implementation, leveraging the STag. When the local ULP specifies implementation, leveraging the STag. When the local ULP specifies
to the RNIC the base address and the number of bytes in the to the RNIC the base address and the number of bytes in the
skipping to change at page 22, line 28 skipping to change at page 27, line 28
buffer referenced by the STag before the STag is enabled for buffer referenced by the STag before the STag is enabled for
access. When an RDMA data transfer operation (which includes an access. When an RDMA data transfer operation (which includes an
STag) arrives on a Stream, a base and bounds byte granularity STag) arrives on a Stream, a base and bounds byte granularity
access check must be performed to ensure the operation accesses access check must be performed to ensure the operation accesses
only memory locations within the buffer described by that STag. only memory locations within the buffer described by that STag.
Thus an RNIC implementation MUST ensure that a Remote Peer is not Thus an RNIC implementation MUST ensure that a Remote Peer is not
able to access memory outside of the buffer specified when the able to access memory outside of the buffer specified when the
STag was enabled for remote access. STag was enabled for remote access.
5.3.2 Modifying a Buffer After Indication 6.2.2 Modifying a Buffer After Indication
This attack can occur if a Remote Peer attempts to modify the This attack can occur if a Remote Peer attempts to modify the
contents of an STag referenced buffer by performing an RDMA Write contents of an STag referenced buffer by performing an RDMA Write
or an RDMA Read Response after the Remote Peer has indicated to or an RDMA Read Response after the Remote Peer has indicated to
the Local Peer or local ULP (by a variety of means) that the STag the Local Peer or local ULP (by a variety of means) that the STag
data buffer contents are ready for use. This attack can occur data buffer contents are ready for use. This attack can occur
even when no resources are shared across Streams. Note that a bug even when no resources are shared across Streams. Note that a bug
in a Remote Peer, or network based tampering, could also result in a Remote Peer, or network based tampering, could also result
in this problem. in this problem.
skipping to change at page 22, line 56 skipping to change at page 27, line 56
Peer could force the ULP down operational paths that were never Peer could force the ULP down operational paths that were never
intended. intended.
The local ULP can protect itself from this type of attack by The local ULP can protect itself from this type of attack by
revoking remote access when the original data transfer has revoking remote access when the original data transfer has
completed and before it validates the contents of the buffer. The completed and before it validates the contents of the buffer. The
local ULP can either do this by explicitly revoking remote access local ULP can either do this by explicitly revoking remote access
rights for the STag when the Remote Peer indicates the operation rights for the STag when the Remote Peer indicates the operation
has completed, or by checking to make sure the Remote Peer has completed, or by checking to make sure the Remote Peer
invalidated the STag through the RDMAP Remote Invalidate invalidated the STag through the RDMAP Remote Invalidate
capability (see section 5.5.5 Remote Invalidate an STag Shared on capability (see section 6.4.5 Remote Invalidate an STag Shared on
Multiple Streams on page 34 for a definition of Remote Multiple Streams for a definition of Remote Invalidate), and if
Invalidate), and if it did not, the local ULP then explicitly it did not, the local ULP then explicitly revokes the STag remote
revokes the STag remote access rights. access rights.
The local ULP SHOULD follow the above procedure to protect the The local ULP SHOULD follow the above procedure to protect the
buffer before it validates the contents of the buffer (or uses buffer before it validates the contents of the buffer (or uses
the buffer in any way). the buffer in any way).
An RNIC MUST ensure that network packets using the STag for a An RNIC MUST ensure that network packets using the STag for a
previously advertised buffer can no longer modify the buffer previously advertised buffer can no longer modify the buffer
after the ULP revokes remote access rights for the specific STag. after the ULP revokes remote access rights for the specific STag.
5.3.3 Multiple STags to access the same buffer 6.2.3 Multiple STags to access the same buffer
See section 5.4.6 Using Multiple STags Which Alias to the Same
Buffer on page 25 for this analysis.
5.3.4 Network based modification of buffer content
This is actually a man in the middle attack - but only on the See section 6.3.6 Using Multiple STags Which Alias to the Same
content of the buffer, as opposed to the man in the middle attack Buffer for this analysis.
presented above, where both the signaling and content can be
modified. See Section 5.2.3 Man in the Middle Attack on page 20.
5.4 Information Disclosure 6.3 Information Disclosure
The main potential source for information disclosure is through a The main potential source for information disclosure is through a
local buffer that has been enabled for remote access. If the local buffer that has been enabled for remote access. If the
buffer can be probed by a Remote Peer on another Stream, then buffer can be probed by a Remote Peer on another Stream, then
there is potential for information disclosure. there is potential for information disclosure.
The potential attacks that could result in unintended information The potential attacks that could result in unintended information
disclosure and countermeasures are detailed in the following disclosure and countermeasures are detailed in the following
sections. sections.
5.4.1 Probing memory outside of the buffer bounds 6.3.1 Probing memory outside of the buffer bounds
This is essentially the same attack as described in Section This is essentially the same attack as described in Section
5.3.1, except an RDMA Read Request is used to mount the attack. 6.2.1, except an RDMA Read Request is used to mount the attack.
The same countermeasure applies. The same countermeasure applies.
5.4.2 Using RDMA Read to Access Stale Data 6.3.2 Using RDMA Read to Access Stale Data
If a buffer is being used for a combination of reads and writes If a buffer is being used for a combination of reads and writes
(either remote or local), and is exposed to the Remote Peer with (either remote or local), and is exposed to the Remote Peer with
at least remote read access rights, the Remote Peer may be able at least remote read access rights, the Remote Peer may be able
to examine the contents of the buffer before they are initialized to examine the contents of the buffer before they are initialized
with the correct data. In this situation, whatever contents were with the correct data. In this situation, whatever contents were
present in the buffer before the buffer is initialized can be present in the buffer before the buffer is initialized can be
viewed by the Remote Peer, if the Remote Peer performs an RDMA viewed by the Remote Peer, if the Remote Peer performs an RDMA
Read. Read.
Because of this, the local ULP SHOULD ensure that no stale data Because of this, the local ULP SHOULD ensure that no stale data
is contained in the buffer before remote read access rights are is contained in the buffer before remote read access rights are
granted (this can be done by zeroing the contents of the memory, granted (this can be done by zeroing the contents of the memory,
for example). for example).
5.4.3 Accessing a Buffer After the Transfer 6.3.3 Accessing a Buffer After the Transfer
If the Remote Peer has remote read access to a buffer, and by If the Remote Peer has remote read access to a buffer, and by
some mechanism tells the local ULP that the transfer has been some mechanism tells the local ULP that the transfer has been
completed, but the local ULP does not disable remote access to completed, but the local ULP does not disable remote access to
the buffer before modifying the data, it is possible for the the buffer before modifying the data, it is possible for the
Remote Peer to retrieve the new data. Remote Peer to retrieve the new data.
This is similar to the attack defined in Section 5.3.2 Modifying This is similar to the attack defined in Section 6.2.2 Modifying
a Buffer After Indication on page 22. The same countermeasures a Buffer After Indication. The same countermeasures apply. In
apply. In addition, the local ULP SHOULD grant remote read access addition, the local ULP SHOULD grant remote read access rights
rights only for the amount of time needed to retrieve the data. only for the amount of time needed to retrieve the data.
5.4.4 Accessing Unintended Data With a Valid STag 6.3.4 Accessing Unintended Data With a Valid STag
If the ULP enables remote access to a buffer using an STag that If the ULP enables remote access to a buffer using an STag that
references the entire buffer, but intends only a portion of the references the entire buffer, but intends only a portion of the
buffer to be accessed, it is possible for the Remote Peer to buffer to be accessed, it is possible for the Remote Peer to
access the other parts of the buffer anyway. access the other parts of the buffer anyway.
To prevent this attack, the ULP SHOULD set the base and bounds of To prevent this attack, the ULP SHOULD set the base and bounds of
the buffer when the STag is initialized to expose only the data the buffer when the STag is initialized to expose only the data
to be retrieved. to be retrieved.
5.4.5 RDMA Read into an RDMA Write Buffer 6.3.5 RDMA Read into an RDMA Write Buffer
One form of disclosure can occur if the access rights on the One form of disclosure can occur if the access rights on the
buffer enabled remote read, when only remote write access was buffer enabled remote read, when only remote write access was
intended. If the buffer contained ULP data, or data from a intended. If the buffer contained ULP data, or data from a
transfer on an unrelated Stream, the Remote Peer could retrieve transfer on an unrelated Stream, the Remote Peer could retrieve
the data through an RDMA Read operation. Note that an RNIC the data through an RDMA Read operation. Note that an RNIC
implementation is not required to support STags that have both implementation is not required to support STags that have both
read and write access. read and write access.
The most obvious countermeasure for this attack is to not grant The most obvious countermeasure for this attack is to not grant
skipping to change at page 25, line 8 skipping to change at page 30, line 5
Thus if a ULP only intends a buffer to be exposed for remote Thus if a ULP only intends a buffer to be exposed for remote
write access, it MUST set the access rights to the buffer to only write access, it MUST set the access rights to the buffer to only
enable remote write access. Note that this requirement is not enable remote write access. Note that this requirement is not
meant to restrict the use of zero-length RDMA Reads. Zero-length meant to restrict the use of zero-length RDMA Reads. Zero-length
RDMA Reads do not expose ULP data. Because they are intended to RDMA Reads do not expose ULP data. Because they are intended to
be used as a mechanism to ensure that all RDMA Writes have been be used as a mechanism to ensure that all RDMA Writes have been
received, and do not even require a valid STag, their use is received, and do not even require a valid STag, their use is
permitted even if a buffer has only been enabled for write permitted even if a buffer has only been enabled for write
access. access.
5.4.6 Using Multiple STags Which Alias to the Same Buffer 6.3.6 Using Multiple STags Which Alias to the Same Buffer
Multiple STags which alias to the same buffer at the same time Multiple STags which alias to the same buffer at the same time
can result in unintentional information disclosure if the STags can result in unintentional information disclosure if the STags
are used by different, mutually untrusted, Remote Peers. This are used by different, mutually untrusted, Remote Peers. This
model applies specifically to client/server communication, where model applies specifically to client/server communication, where
the server is communicating with multiple clients, each of which the server is communicating with multiple clients, each of which
do not mutually trust each other. do not mutually trust each other.
If only read access is enabled, then the local ULP has complete If only read access is enabled, then the local ULP has complete
control over information disclosure. Thus a server which intended control over information disclosure. Thus a server which intended
skipping to change at page 25, line 42 skipping to change at page 30, line 39
Peers do not mutually trust each other, it is possible for one Peers do not mutually trust each other, it is possible for one
Remote Peer to overwrite the contents that have been written by Remote Peer to overwrite the contents that have been written by
the other Remote Peer. the other Remote Peer.
Thus a ULP with multiple Remote Peers which do not share Partial Thus a ULP with multiple Remote Peers which do not share Partial
Mutual Trust MUST NOT grant write access to the same buffer Mutual Trust MUST NOT grant write access to the same buffer
through different STags. A buffer should be exposed to only one through different STags. A buffer should be exposed to only one
untrusted Remote Peer at a time to ensure that no information untrusted Remote Peer at a time to ensure that no information
disclosure or information tampering occurs between peers. disclosure or information tampering occurs between peers.
5.4.7 Remote Node Loading Firmware onto the RNIC 6.3.7 Controlling Access to PTT & STag Mapping
If the Remote Peer can cause firmware to be loaded onto the RNIC,
there is an opportunity for information disclosure. See Elevation
of Privilege in Section 5.5.6 for this analysis.
5.4.8 Controlling Access to PTT & STag Mapping
If a Non-Privileged ULP is able to directly manipulate the RNIC If a Non-Privileged ULP is able to directly manipulate the RNIC
Page Translation Tables (which translate from an STag to a host Page Translation Tables (which translate from an STag to a host
address), it is possible that the Non-Privileged ULP could point address), it is possible that the Non-Privileged ULP could point
the Page Translation Table at an unrelated Stream's or ULP's the Page Translation Table at an unrelated Stream's or ULP's
buffers and thereby be able to gain access to information of the buffers and thereby be able to gain access to information of the
unrelated Stream/ULP. unrelated Stream/ULP.
As discussed in Section 2 Architectural Model on page 6, As discussed in Section 2 Architectural Model, introduction of a
introduction of a Privileged Resource Manager to arbitrate the Privileged Resource Manager to arbitrate the mapping requests is
mapping requests is an effective countermeasure. This enables the an effective countermeasure. This enables the Privileged Resource
Privileged Resource Manager to ensure a local ULP can only Manager to ensure a local ULP can only initialize the Page
initialize the Page Translation Table (PTT)to point to its own Translation Table (PTT)to point to its own buffers.
buffers.
Thus if Non-Privileged ULPs are supported, the Privileged Thus if Non-Privileged ULPs are supported, the Privileged
Resource Manager MUST verify that the Non-Privileged ULP has the Resource Manager MUST verify that the Non-Privileged ULP has the
right to access a specific Data Buffer before allowing an STag right to access a specific Data Buffer before allowing an STag
for which the ULP has access rights to be associated with a for which the ULP has access rights to be associated with a
specific Data Buffer. This can be done when the Page Translation specific Data Buffer. This can be done when the Page Translation
Table is initialized to access the Data Buffer or when the STag Table is initialized to access the Data Buffer or when the STag
is initialized to point to a group of Page Translation Table is initialized to point to a group of Page Translation Table
entries, or both. entries, or both.
5.4.9 Network based eavesdropping 6.4 Denial of Service (DOS)
An attacker that is able to eavesdrop on the network can read the
content of all read and write accesses to a Peer's buffers. To
prevent information disclosure, the read/written data must be
encrypted. See also Section 5.2.3 Man in the Middle Attack on
page 20. The encryption can be done either by the ULP, or by a
protocol that provides security services to the LLP (e.g. IPsec
or SSL). Refer to section 6 for discussion of security services
for DDP/RDMA.
5.5 Denial of Service (DOS)
A DOS attack is one of the primary security risks of RDMAP. This A DOS attack is one of the primary security risks of RDMAP. This
is because RNIC resources are valuable and scarce, and many ULP is because RNIC resources are valuable and scarce, and many ULP
environments require communication with untrusted Remote Peers. environments require communication with untrusted Remote Peers.
If the remote ULP can be authenticated or encrypted, clearly, the If the Remote Peer can be authenticated or the ULP payload
DOS profile can be reduced. For the purposes of this analysis, it encrypted, clearly, the DOS profile can be reduced. For the
is assumed that the RNIC must be able to operate in untrusted purposes of this analysis, it is assumed that the RNIC must be
environments, which are open to DOS style attacks. able to operate in untrusted environments, which are open to DOS
style attacks.
Denial of service attacks against RNIC resources are not the Denial of service attacks against RNIC resources are not the
typical unknown party spraying packets at a random host (such as typical unknown party spraying packets at a random host (such as
a TCP SYN attack). Because the connection/Stream must be fully a TCP SYN attack). Because the connection/Stream must be fully
established, the attacker must be able to both send and receive established (e.g. a 3 message transport layer handshake has
occurred), the attacker must be able to both send and receive
messages over that connection/Stream, or be able to guess a valid messages over that connection/Stream, or be able to guess a valid
packet on an existing RDMAP Stream. packet on an existing RDMAP Stream.
This section outlines the potential attacks and the This section outlines the potential attacks and the
countermeasures available for dealing with each attack. countermeasures available for dealing with each attack.
5.5.1 RNIC Resource Consumption 6.4.1 RNIC Resource Consumption
This section covers attacks that fall into the general category This section covers attacks that fall into the general category
of a local ULP attempting to unfairly allocate scarce (i.e. of a local ULP attempting to unfairly allocate scarce (i.e.
bounded) RNIC resources. The local ULP may be attempting to bounded) RNIC resources. The local ULP may be attempting to
allocate resources on its own behalf, or on behalf of a Remote allocate resources on its own behalf, or on behalf of a Remote
Peer. Resources that fall into this category include: Protection Peer. Resources that fall into this category include: Protection
Domains, Stream Context Memory, Translation and Protection Domains, Stream Context Memory, Translation and Protection
Tables, and STag namespace. These can be due to attacks by Tables, and STag namespace. These can be due to attacks by
currently active local ULPs or ones that allocated resources currently active local ULPs or ones that allocated resources
earlier, but are now idle. earlier, but are now idle.
skipping to change at page 27, line 39 skipping to change at page 32, line 20
provide enough Protection Domains to allow the Resource Manager provide enough Protection Domains to allow the Resource Manager
to be able to assign a unique Protection Domain for each to be able to assign a unique Protection Domain for each
unrelated, untrusted local ULP (for a bounded, reasonable number unrelated, untrusted local ULP (for a bounded, reasonable number
of local ULPs). This analysis further assumes that the Resource of local ULPs). This analysis further assumes that the Resource
Manager implements policies to ensure that untrusted local ULPs Manager implements policies to ensure that untrusted local ULPs
are not able to consume all of the Protection Domains through a are not able to consume all of the Protection Domains through a
DOS attack. Note that Protection Domain consumption cannot result DOS attack. Note that Protection Domain consumption cannot result
from a DOS attack launched by a Remote Peer, unless a local ULP from a DOS attack launched by a Remote Peer, unless a local ULP
is acting on the Remote Peer's behalf. is acting on the Remote Peer's behalf.
5.5.2 Resource Consumption By Active ULPs 6.4.2 Resource Consumption by Idle ULPs
The simplest form of a DOS attack given a fixed amount of
resources is for the Remote Peer to create a RDMAP Stream to a
Local Peer, and request dedicated resources then do no actual
work. This allows the Remote Peer to be very light weight (i.e.
only negotiate resources, but do no data transfer) and consumes a
disproportionate amount of resources at the Local Peer.
A general countermeasure for this style of attack is to monitor
active RDMAP Streams and if resources are getting low, reap the
resources from RDMAP Streams that are not transferring data and
possibly terminate the Stream. This would presumably be under
administrative control.
Refer to Section 6.4.1 for the analysis and countermeasures for
this style of attack on the following RNIC resources: Stream
Context Memory, Page Translation Tables and STag namespace.
Note that some RNIC resources are not at risk of this type of
attack from a Remote Peer because an attack requires the Remote
Peer to send messages in order to consume the resource. Receive
Data Buffers, Completion Queue, and RDMA Read Request Queue
resources are examples. These resources are, however, at risk
from a local ULP that attempts to allocate resources, then goes
idle. This could also be created if the ULP negotiates the
resource levels with the Remote Peer, which causes the Local Peer
to consume resources, however the Remote Peer never sends data to
consume them. The general countermeasure described in this
section can be used to free resources allocated by an idle Local
Peer.
6.4.3 Resource Consumption By Active ULPs
This section describes DOS attacks from Local and Remote Peers This section describes DOS attacks from Local and Remote Peers
that are actively exchanging messages. Attacks on each RDMA NIC that are actively exchanging messages. Attacks on each RDMA NIC
resource are examined and specific countermeasures are resource are examined and specific countermeasures are
identified. Note that attacks on Stream Context Memory, Page identified. Note that attacks on Stream Context Memory, Page
Translation Tables, and STag namespace are covered in Section Translation Tables, and STag namespace are covered in Section
5.5.1 RNIC Resource Consumption, so are not included here. 6.4.1 RNIC Resource Consumption, so are not included here.
5.5.2.1 Multiple Streams Sharing Receive Buffers 6.4.3.1 Multiple Streams Sharing Receive Buffers
The Remote Peer can attempt to consume more than its fair share The Remote Peer can attempt to consume more than its fair share
of receive data buffers (i.e. Untagged buffers for DDP are or of receive data buffers (i.e. Untagged buffers for DDP are or
Send Type Messages for RDMAP) if receive buffers are shared Send Type Messages for RDMAP) if receive buffers are shared
across multiple Streams. across multiple Streams.
If resources are not shared across multiple Streams, then this If resources are not shared across multiple Streams, then this
attack is not possible because the Remote Peer will not be able attack is not possible because the Remote Peer will not be able
to consume more buffers than were allocated to the Stream. The to consume more buffers than were allocated to the Stream. The
worst case scenario is that the Remote Peer can consume more worst case scenario is that the Remote Peer can consume more
skipping to change at page 29, line 27 skipping to change at page 34, line 40
large amount of state to be tracked in each RNIC on a per Stream large amount of state to be tracked in each RNIC on a per Stream
basis. basis.
Thus, if an RNIC Engine provides the ability to share receive Thus, if an RNIC Engine provides the ability to share receive
buffers across multiple Streams, the combination of the RNIC buffers across multiple Streams, the combination of the RNIC
Engine and the Privileged Resource Manager MUST be able to detect Engine and the Privileged Resource Manager MUST be able to detect
if the Remote Peer is attempting to consume more than its fair if the Remote Peer is attempting to consume more than its fair
share of resources so that the Local Peer or local ULP can apply share of resources so that the Local Peer or local ULP can apply
countermeasures to detect and prevent the attack. countermeasures to detect and prevent the attack.
5.5.2.2 Local ULP Attacking a Shared CQ 6.4.3.2 Remote or Local Peer Attacking a Shared CQ
DOS attacks against a Shared Completion Queue (CQ) can be caused
by either the local ULP or the Remote Peer if either attempts to
cause more completions than its fair share of the number of
entries, thus potentially starving another unrelated ULP such
that no Completion Queue entries are available.
A Completion Queue entry can potentially be maliciously consumed
by a completion from the Send Queue or a completion from the
Receive Queue. In the former, the attacker is the local ULP. In
the latter, the attacker is the Remote Peer.
A form of attack can occur where the local ULPs can consume
resources on the CQ. A local ULP that is slow to free resources
on the CQ by not reaping the completion status quickly enough
could stall all other local ULPs attempting to use that CQ.
For these reasons, an RNIC MUST NOT enable sharing a CQ across
ULPs that do not share Partial Mutual Trust.
5.5.2.3 Local or Remote Peer Attacking a Shared CQ
For an overview of the shared CQ attack model, see Section For an overview of the shared CQ attack model, see Section 7.1.
5.5.2.2.
The Remote Peer can attack a shared CQ by consuming more than its The Remote Peer can attack a shared CQ by consuming more than its
fair share of CQ entries by using one of the following methods: fair share of CQ entries by using one of the following methods:
* The ULP protocol allows the Remote Peer to cause the * The ULP protocol allows the Remote Peer to cause the
local ULP to reserve a specified number of CQ entries, local ULP to reserve a specified number of CQ entries,
possibly leaving insufficient entries for other Streams possibly leaving insufficient entries for other Streams
that are sharing the CQ. that are sharing the CQ.
* If the Remote Peer, Local Peer, or local ULP (or any * If the Remote Peer, Local Peer, or local ULP (or any
skipping to change at page 30, line 25 skipping to change at page 35, line 15
functioning). functioning).
The first method of attack can be avoided if the ULP does not The first method of attack can be avoided if the ULP does not
allow a Remote Peer to reserve CQ entries or there is a trusted allow a Remote Peer to reserve CQ entries or there is a trusted
intermediary such as a Privileged Resource Manager. Unfortunately intermediary such as a Privileged Resource Manager. Unfortunately
it is often unrealistic to not allow a Remote Peer to reserve CQ it is often unrealistic to not allow a Remote Peer to reserve CQ
entries - particularly if the number of completion entries is entries - particularly if the number of completion entries is
dependent on other ULP negotiated parameters, such as the amount dependent on other ULP negotiated parameters, such as the amount
of buffering required by the ULP. Thus an implementation MUST of buffering required by the ULP. Thus an implementation MUST
implement a Privileged Resource Manager to control the allocation implement a Privileged Resource Manager to control the allocation
of CQ entries. See Section 2.1 Components on page 7 for a of CQ entries. See Section 2.1 Components for a definition of
definition of Privileged Resource Manager. Privileged Resource Manager.
One way that a Local or Remote Peer can attempt to overwhelm a CQ One way that a Local or Remote Peer can attempt to overwhelm a CQ
with completions is by sending minimum length RDMAP/DDP Messages with completions is by sending minimum length RDMAP/DDP Messages
to cause as many completions (receive completions for the Remote to cause as many completions (receive completions for the Remote
Peer, send completions for the Local Peer) per second as Peer, send completions for the Local Peer) per second as
possible. If it is the Remote Peer attacking, and we assume that possible. If it is the Remote Peer attacking, and we assume that
the Local Peer's receive queue(s) do not run out of receive the Local Peer's receive queue(s) do not run out of receive
buffers (if they do, then this is a different attack, documented buffers (if they do, then this is a different attack, documented
in Section 5.5.2.1 Multiple Streams Sharing Receive Buffers on in Section 6.4.3.1 Multiple Streams Sharing Receive Buffers),
page 27), then it might be possible for the Remote Peer to then it might be possible for the Remote Peer to consume more
consume more than its fair share of Completion Queue entries. than its fair share of Completion Queue entries. Depending upon
Depending upon the CQ implementation, this could either cause the the CQ implementation, this could either cause the CQ to overflow
CQ to overflow (if it is not large enough to handle all of the (if it is not large enough to handle all of the completions
completions generated) or for another Stream to not be able to generated) or for another Stream to not be able to generate CQ
generate CQ entries (if the RNIC had flow control on generation entries (if the RNIC had flow control on generation of CQ entries
of CQ entries into the CQ). In either case, the CQ will stop into the CQ). In either case, the CQ will stop functioning
functioning correctly and any Streams expecting completions on correctly and any Streams expecting completions on the CQ will
the CQ will stop functioning. stop functioning.
This attack can occur regardless of whether all of the Streams This attack can occur regardless of whether all of the Streams
associated with the CQ are in the same Protection Domain or are associated with the CQ are in the same Protection Domain or are
in different Protection Domains - the key issue is that the in different Protection Domains - the key issue is that the
number of Completion Queue entries is less than the number of all number of Completion Queue entries is less than the number of all
outstanding operations that can cause a completion. outstanding operations that can cause a completion.
The Local Peer can protect itself from this type of attack using The Local Peer can protect itself from this type of attack using
either of the following methods: either of the following methods:
skipping to change at page 32, line 27 skipping to change at page 37, line 16
share Partial Mutual Trust, then the ULP MUST implement a share Partial Mutual Trust, then the ULP MUST implement a
mechanism to ensure that the Completion Queue can not overflow. mechanism to ensure that the Completion Queue can not overflow.
Note that it is possible to share CQs even if the Remote Peers Note that it is possible to share CQs even if the Remote Peers
accessing the CQs are untrusted if either of the above two accessing the CQs are untrusted if either of the above two
formulas are implemented. If the ULP can be trusted to not post formulas are implemented. If the ULP can be trusted to not post
more than MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and more than MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and
MaxPostedOnEachSQ, then the first formula applies. If the ULP can MaxPostedOnEachSQ, then the first formula applies. If the ULP can
not be trusted to obey the limit, then the second formula not be trusted to obey the limit, then the second formula
applies. applies.
5.5.2.4 Attacking the RDMA Read Request Queue 6.4.3.3 Attacking the RDMA Read Request Queue
If RDMA Read Request Queue resources are pooled across multiple
Streams, one attack is if the local ULP attempts to unfairly
allocate RDMA Read Request Queue resources for its Streams. For
example, a local ULP attempts to allocate all available resources
on a specific RDMA Read Request Queue for its Streams, thereby
denying the resource to ULPs sharing the RDMA Read Request Queue.
The same type of argument applies even if the RDMA Read Request
is not shared - but a local ULP attempts to allocate all of the
RNIC's resources when the queue is created.
Thus access to interfaces that allocate RDMA Read Request Queue
entries MUST be restricted to a trusted Local Peer, such as a
Privileged Resource Manager. The Privileged Resource Manager
SHOULD prevent a local ULP from allocating more than its fair
share of resources.
Another form of attack is if the Remote Peer sends more RDMA Read The RDMA Read Request Queue can be attacked if the Remote Peer
Requests than the depth of the RDMA Read Request Queue at the sends more RDMA Read Requests than the depth of the RDMA Read
Local Peer. If the RDMA Read Request Queue is a shared resource, Request Queue at the Local Peer. If the RDMA Read Request Queue
this could corrupt the queue. If the queue is not shared, then is a shared resource, this could corrupt the queue. If the queue
the worst case is that the current Stream is no longer functional is not shared, then the worst case is that the current Stream is
(e.g. torn down). One approach to solving the shared RDMA Read no longer functional (e.g. torn down). One approach to solving
Request Queue would be to create thresholds, similar to those the shared RDMA Read Request Queue would be to create thresholds,
described in Section 5.5.2.1 Multiple Streams Sharing Receive similar to those described in Section 6.4.3.1 Multiple Streams
Buffers on page 27. A simpler approach is to not share RDMA Read Sharing Receive Buffers. A simpler approach is to not share RDMA
Request Queue resources among Streams or enforce hard limits of Read Request Queue resources among Streams or enforce hard limits
consumption per Stream. Thus RDMA Read Request Queue resource of consumption per Stream. Thus RDMA Read Request Queue resource
consumption MUST be controlled by the Privileged Resource Manager consumption MUST be controlled by the Privileged Resource Manager
such that RDMAP/DDP Streams which do not share Partial Mutual such that RDMAP/DDP Streams which do not share Partial Mutual
Trust do not share RDMA Read Request Queue resources. Trust do not share RDMA Read Request Queue resources.
If the issue is a bug in the Remote Peer's implementation, but If the issue is a bug in the Remote Peer's implementation, but
not a malicious attack, the issue can be solved by requiring the not a malicious attack, the issue can be solved by requiring the
Remote Peer's RNIC to throttle RDMA Read Requests. By properly Remote Peer's RNIC to throttle RDMA Read Requests. By properly
configuring the Stream at the Remote Peer through a trusted configuring the Stream at the Remote Peer through a trusted
agent, the RNIC can be made to not transmit RDMA Read Requests agent, the RNIC can be made to not transmit RDMA Read Requests
that exceed the depth of the RDMA Read Request Queue at the Local that exceed the depth of the RDMA Read Request Queue at the Local
skipping to change at page 33, line 25 skipping to change at page 37, line 51
Request Queue can handle, the requests would be queued at the Request Queue can handle, the requests would be queued at the
Remote Peer's RNIC until previous requests complete. If the Remote Peer's RNIC until previous requests complete. If the
Remote Peer's Stream is not configured correctly, the RDMAP Remote Peer's Stream is not configured correctly, the RDMAP
Stream is terminated when more RDMA Read Requests arrive at the Stream is terminated when more RDMA Read Requests arrive at the
Local Peer than the Local Peer can handle (assuming the prior Local Peer than the Local Peer can handle (assuming the prior
paragraph's recommendation is implemented). Thus an RNIC paragraph's recommendation is implemented). Thus an RNIC
implementation SHOULD provide a mechanism to cap the number of implementation SHOULD provide a mechanism to cap the number of
outstanding RDMA Read Requests. The configuration of this limit outstanding RDMA Read Requests. The configuration of this limit
is outside the scope of this document. is outside the scope of this document.
5.5.3 Resource Consumption by Idle ULPs 6.4.4 Exercise of non-optimal code paths
The simplest form of a DOS attack given a fixed amount of
resources is for the Remote Peer to create a RDMAP Stream to a
Local Peer, and request dedicated resources then do no actual
work. This allows the Remote Peer to be very light weight (i.e.
only negotiate resources, but do no data transfer) and consumes a
disproportionate amount of resources at the Local Peer.
A general countermeasure for this style of attack is to monitor
active RDMAP Streams and if resources are getting low, reap the
resources from RDMAP Streams that are not transferring data and
possibly terminate the Stream. This would presumably be under
administrative control.
Refer to Section 5.5.1 for the analysis and countermeasures for
this style of attack on the following RNIC resources: Stream
Context Memory, Page Translation Tables and STag namespace.
Note that some RNIC resources are not at risk of this type of
attack from a Remote Peer because an attack requires the Remote
Peer to send messages in order to consume the resource. Receive
Data Buffers, Completion Queue, and RDMA Read Request Queue
resources are examples. These resources are, however, at risk
from a local ULP that attempts to allocate resources, then goes
idle. This could also be created if the ULP negotiates the
resource levels with the Remote Peer, which causes the Local Peer
to consume resources, however the Remote Peer never sends data to
consume them. The general countermeasure described in this
section can be used to free resources allocated by an idle Local
Peer.
5.5.4 Exercise of non-optimal code paths
Another form of DOS attack is to attempt to exercise data paths Another form of DOS attack is to attempt to exercise data paths
that can consume a disproportionate amount of resources. An that can consume a disproportionate amount of resources. An
example might be if error cases are handled on a "slow path" example might be if error cases are handled on a "slow path"
(consuming either host or RNIC computational resources), and an (consuming either host or RNIC computational resources), and an
attacker generates excessive numbers of errors in an attempt to attacker generates excessive numbers of errors in an attempt to
consume these resources. Note that for most RDMAP or DDP errors, consume these resources. Note that for most RDMAP or DDP errors,
the attacking Stream will simply be torn down. Thus for this form the attacking Stream will simply be torn down. Thus for this form
of attack to be effective, the Remote Peer needs to exercise data of attack to be effective, the Remote Peer needs to exercise data
paths which do not cause the Stream to be torn down. paths which do not cause the Stream to be torn down.
If an RNIC implementation contains "slow paths" which do not If an RNIC implementation contains "slow paths" which do not
result in the tear down of the Stream, it is recommended that an result in the tear down of the Stream, it is recommended that an
implementation provide the ability to detect the above condition implementation provide the ability to detect the above condition
and allow an administrator to act, including potentially and allow an administrator to act, including potentially
administratively tearing down the RDMAP Stream associated with administratively tearing down the RDMAP Stream associated with
the Stream exercising data paths consuming a disproportionate the Stream exercising data paths consuming a disproportionate
amount of resources. amount of resources.
5.5.5 Remote Invalidate an STag Shared on Multiple Streams 6.4.5 Remote Invalidate an STag Shared on Multiple Streams
If a Local Peer has enabled an STag for remote access, the Remote If a Local Peer has enabled an STag for remote access, the Remote
Peer could attempt to remote invalidate the STag by using the Peer could attempt to remote invalidate the STag by using the
RDMAP Send with Invalidate or Send with SE and Invalidate RDMAP Send with Invalidate or Send with SE and Invalidate
Message. If the STag is only valid on the current Stream, then Message. If the STag is only valid on the current Stream, then
the only side effect is that the Remote Peer can no longer use the only side effect is that the Remote Peer can no longer use
the STag; thus there are no security issues. the STag; thus there are no security issues.
If the STag is valid across multiple Streams, then the Remote If the STag is valid across multiple Streams, then the Remote
Peer can prevent other Streams from using that STag by using the Peer can prevent other Streams from using that STag by using the
remote invalidate functionality. remote invalidate functionality.
Thus if RDDP Streams do not share Partial Mutual Trust (i.e. the Thus if RDDP Streams do not share Partial Mutual Trust (i.e. the
Remote Peer may attempt to remote invalidate the STag Remote Peer may attempt to remote invalidate the STag
prematurely), the ULP MUST NOT enable an STag which would be prematurely), the ULP MUST NOT enable an STag which would be
valid across multiple Streams. valid across multiple Streams.
5.5.6 Remote Peer attacking an Unshared CQ 6.4.6 Remote Peer attacking an Unshared CQ
The Remote Peer can attack an unshared CQ if the Local Peer does The Remote Peer can attack an unshared CQ if the Local Peer does
not size the CQ correctly. For example, if the Local Peer enables not size the CQ correctly. For example, if the Local Peer enables
the CQ to handle completions of received buffers, and the receive the CQ to handle completions of received buffers, and the receive
buffer queue is longer than the Completion Queue, then an buffer queue is longer than the Completion Queue, then an
overflow can potentially occur. The effect on the attacker's overflow can potentially occur. The effect on the attacker's
Stream is catastrophic. However if an RNIC does not have the Stream is catastrophic. However if an RNIC does not have the
proper protections in place, then an attack to overflow the CQ proper protections in place, then an attack to overflow the CQ
can also cause corruption and/or termination of an unrelated can also cause corruption and/or termination of an unrelated
Stream. Thus an RNIC MUST ensure that if a CQ overflows, any Stream. Thus an RNIC MUST ensure that if a CQ overflows, any
Streams which do not use the CQ MUST remain unaffected. Streams which do not use the CQ MUST remain unaffected.
5.6 Elevation of Privilege 6.5 Elevation of Privilege
The RDMAP/DDP Security Architecture explicitly differentiates The RDMAP/DDP Security Architecture explicitly differentiates
between three levels of privilege - Non-Privileged, Privileged, between three levels of privilege - Non-Privileged, Privileged,
and the Privileged Resource Manager. If a Non-Privileged ULP is and the Privileged Resource Manager. If a Non-Privileged ULP is
able to elevate its privilege level to a Privileged ULP, then able to elevate its privilege level to a Privileged ULP, then
mapping a physical address list to an STag can provide local and mapping a physical address list to an STag can provide local and
remote access to any physical address location on the node. If a remote access to any physical address location on the node. If a
Privileged Mode ULP is able to promote itself to be a Resource Privileged Mode ULP is able to promote itself to be a Resource
Manager, then it is possible for it to perform denial of service Manager, then it is possible for it to perform denial of service
type attacks where substantial amounts of local resources could type attacks where substantial amounts of local resources could
be consumed. be consumed.
In general, elevation of privilege is a local implementation In general, elevation of privilege is a local implementation
specific issue and thus outside the scope of this document. specific issue and thus outside the scope of this document.
There is one issue worth noting, however. If the RNIC 7 Attacks from Local Peers
implementation, by some insecure mechanism (or implementation
defect), can enable a Remote Peer or un-trusted local ULP to load
firmware into the RNIC Engine, it is possible to use the RNIC to
attack the host. Thus, an RNIC implementation MUST NOT enable
firmware to be loaded on the RNIC Engine directly from an
untrusted local ULP or Remote Peer, unless they are properly
authenticated (by a mechanism outside the scope of this document.
The mechanism presumably entails authenticating that the remote
ULP has the right to perform the update), and the update is done
via a secure protocol, such as IPsec (See Section 6 Security
Services for RDMAP and DDP on page 36).
6 Security Services for RDMAP and DDP
RDMAP and DDP are used to control, read and write data buffers
over IP networks. Therefore, the control and the data packets of
these protocols are vulnerable to the spoofing, tampering and
information disclosure attacks listed in Section 7.
Generally speaking, Stream confidentiality protects against
eavesdropping. Stream and/or session authentication and integrity
protection is a counter measurement against various spoofing and
tampering attacks. The effectiveness of authentication and
integrity against a specific attack, depend on whether the
authentication is machine level authentication (as the one
provided by IPsec and SSL), or ULP authentication.
6.1 Introduction to Security Options
The following security services can be applied to an RDMAP/DDP
Stream:
1. Session confidentiality - protects against eavesdropping
(section 5.4.9).
2. Per-packet data source authentication - protects against the
following spoofing attacks: network based impersonation
(section 5.2.1), Stream hijacking (section 5.2.2), and man in
the middle (section 5.2.3).
3. Per-packet integrity - protects against tampering done by
network based modification of buffer content (section 5.3.4)
4. Packet sequencing - protects against replay attacks, which is
a special case of the above tampering attack.
If an RDMAP/DDP Stream may be subject to impersonation attacks,
or Stream hijacking attacks, it is recommended that the Stream be
authenticated, integrity protected, and protected from replay
attacks; it may use confidentiality protection to protect from
eavesdropping (in case the RDMAP/DDP Stream traverses a public
network).
Both IPsec and SSL are capable of providing the above security
services for IP and TCP traffic respectively. ULP protocols are
able to provide only part of the above security services. The
next sections describe the different security options.
6.1.1 Introduction to IPsec
IPsec is a protocol suite which is used to secure communication
at the network layer between two peers. The IPsec protocol suite
is specified within the IP Security Architecture [RFC2401], IKE
[RFC2409], IPsec Authentication Header (AH) [RFC2402] and IPsec
Encapsulating Security Payload (ESP) [RFC2406] documents. IKE is
the key management protocol while AH and ESP are used to protect
IP traffic.
An IPsec SA is a one-way security association, uniquely
identified by the 3-tuple: Security Parameter Index (SPI),
protocol (ESP/AH) and destination IP address. The parameters for
an IPsec security association are typically established by a key
management protocol. These include the encapsulation mode,
encapsulation type, session keys and SPI values.
IKE is a two phase negotiation protocol based on the modular
exchange of messages defined by ISAKMP [RFC2408],and the IP
Security Domain of Interpretation (DOI) [RFC2407]. IKE has two
phases, and accomplishes the following functions:
1. Protected cipher suite and options negotiation - using keyed
MACs and encryption and anti-replay mechanisms.
2. Master key generation - via Diffie-Hellman calculations.
3. Authentication of end-points (usually machine level
authentication).
4. IPsec SA management (selector negotiation, options
negotiation, create, delete, and rekeying).
Items 1 through 3 are accomplished in IKE Phase 1, while item 4
is handled in IKE Phase 2.
IKE phase 1 defines four authentication methods; three of them
require both sides to have certified signature or encryption
public keys; the fourth requires the side to exchange out-of-band
a secret random string - called pre-shared-secret (PSS).
An IKE Phase 2 negotiation is performed to establish both an
inbound and an outbound IPsec SA. The traffic to be protected by
an IPsec SA is determined by a selector which has been proposed
by the IKE initiator and accepted by the IKE Responder. The IPsec
SA selector can be a "filter" or traffic classifier, defined as
the 5-tuple: <Source IP address, Destination IP address,
transport protocol (e.g. UDP/SCTP/TCP), Source port, Destination
port>. The successful establishment of a IKE Phase-2 SA results
in the creation of two uni-directional IPsec SAs fully qualified
by the tuple <Protocol (ESP/AH), destination address, SPI>.
The session keys for each IPsec SA are derived from a master key,
typically via a MODP Diffie-Hellman computation. Rekeying of an
existing IPsec SA pair is accomplished by creating two new IPsec
SAs, making them active, and then optionally deleting the older
IPsec SA pair. Typically the new outbound SA is used immediately,
and the old inbound SA is left active to receive packets for some
locally defined time, perhaps 30 seconds or 1 minute. Optionally,
rekeying can use Diffie-Hellman for keying material generation.
6.1.2 Introduction to SSL Limitations on RDMAP
SSL and TLS [RFC 2246] provide Stream authentication, integrity
and confidentiality for TCP based ULPs. SSL supports one-way
(server only) or mutual certificates based authentication.
There are at least two limitations that make SSL underneath RDMAP
less appropriate than IPsec for DDP/RDMA security:
1. The maximum length supported by the TLS record layer protocol
is 2^14 bytes - longer packets must be fragmented (as a
comparison, the maximal length of an IPsec packet is
determined by the maximum length of an IP packet).
2. SSL is a connection oriented protocol. If a stream cipher or This section describes local attacks that are possible against
block cipher in CBC mode is used for bulk encryption, then a the RDMA system defined in Figure 1 - RDMA Security Model and the
packet can be decrypted only after all the packets preceding RNIC Engine resources defined in Section 2.2.
it have already arrived. If SSL is used to protect DDP/RDMA
traffic, then SSL must gather all out-of-order packets before
RDMAP/DDP can place them into the ULP buffer, which might
cause a significant decrease in its efficiency.
If SSL is layered on top of RDMAP or DDP, SSL does not protect 7.1 Local ULP Attacking a Shared CQ
the RDMAP and/or DDP headers. Thus a man-in-the-middle attack can
still occur by modifying the RDMAP/DDP header to incorrectly
place the data into the wrong buffer, thus effectively corrupting
the data stream.
6.1.3 ULPs Which Provide Security DOS attacks against a Shared Completion Queue (CQ - see Section
2.2.6 Completion Queues) can be caused by either the local ULP or
the Remote Peer if either attempts to cause more completions than
its fair share of the number of entries, thus potentially
starving another unrelated ULP such that no Completion Queue
entries are available.
ULPs which provide integrated security but wish to leverage A Completion Queue entry can potentially be maliciously consumed
lower-layer protocol security should be aware of security by a completion from the Send Queue or a completion from the
concerns around correlating a specific channel's security Receive Queue. In the former, the attacker is the local ULP. In
mechanisms to the authentication performed by the ULP. See the latter, the attacker is the Remote Peer.
[NFSv4CHANNEL] for additional information on a promising approach
called "channel binding". From [NFSv4CHANNEL]:
"The concept of channel bindings allows applications to A form of attack can occur where the local ULPs can consume
prove that the end-points of two secure channels at resources on the CQ. A local ULP that is slow to free resources
different network layers are the same by binding on the CQ by not reaping the completion status quickly enough
authentication at one channel to the session protection at could stall all other local ULPs attempting to use that CQ.
the other channel. The use of channel bindings allows
applications to delegate session protection to lower layers,
which may significantly improve performance for some
applications."
6.2 Requirements for IPsec Encapsulation of DDP For these reasons, an RNIC MUST NOT enable sharing a CQ across
ULPs that do not share Partial Mutual Trust.
The IP Storage working group has spent significant time and 7.2 Local Peer Attacking the RDMA Read Request Queue
effort to define the normative IPsec requirements for IP Storage
[RFC3723]. Portions of that specification are applicable to a
wide variety of protocols, including the RDDP protocol suite. In
order to not replicate this effort, an RNIC implementation MUST
follow the requirements defined in RFC3723 Section 2.3 and
Section 5, including the associated normative references for
those sections.
Additionally, since IPsec acceleration hardware may only be able If RDMA Read Request Queue resources are pooled across multiple
to handle a limited number of active IKE Phase 2 SAs, Phase 2 Streams, one attack is if the local ULP attempts to unfairly
delete messages may be sent for idle SAs, as a means of keeping allocate RDMA Read Request Queue resources for its Streams. For
the number of active Phase 2 SAs to a minimum. The receipt of an example, a local ULP attempts to allocate all available resources
IKE Phase 2 delete message MUST NOT be interpreted as a reason on a specific RDMA Read Request Queue for its Streams, thereby
for tearing down an DDP/RDMA Stream. Rather, it is preferable to denying the resource to ULPs sharing the RDMA Read Request Queue.
leave the Stream up, and if additional traffic is sent on it, to The same type of argument applies even if the RDMA Read Request
bring up another IKE Phase 2 SA to protect it. This avoids the is not shared - but a local ULP attempts to allocate all of the
potential for continually bringing Streams up and down. RNIC's resources when the queue is created.
Note that there are serious security issues if IPsec is not Thus access to interfaces that allocate RDMA Read Request Queue
implemented end-to-end. For example, if IPsec is implemented as a entries MUST be restricted to a trusted Local Peer, such as a
tunnel in the middle of the network, any hosts between the Peer Privileged Resource Manager. The Privileged Resource Manager
and the IPsec tunneling device can freely attack the unprotected SHOULD prevent a local ULP from allocating more than its fair
Stream. share of resources.
7 Security considerations 8 Security considerations
This entire document is focused on security considerations. This entire document is focused on security considerations.
8 IANA Considerations 9 IANA Considerations
IANA considerations are not addressed by this document. Any IANA IANA considerations are not addressed by this document. Any IANA
considerations resulting from the use of DDP or RDMA must be considerations resulting from the use of DDP or RDMA must be
addressed in the relevant standards. addressed in the relevant standards.
9 References 10 References
9.1 Normative References 10.1 Normative References
[RFC2828] Shirley, R., "Internet Security Glossary", FYI 36, RFC [RFC2828] Shirley, R., "Internet Security Glossary", FYI 36, RFC
2828, May 2000. 2828, May 2000.
[DDP] Shah, H., J. Pinkerton, R.Recio, and P. Culley, "Direct [DDP] Shah, H., J. Pinkerton, R.Recio, and P. Culley, "Direct
Data Placement over Reliable Transports", Internet-Draft Work Data Placement over Reliable Transports", Internet-Draft Work
in Progress draft-ietf-rddp-ddp-04.txt, December 2004. in Progress draft-ietf-rddp-ddp-04.txt, December 2004.
[RDMAP] Recio, R., P. Culley, D. Garcia, J. Hilland, "An RDMA [RDMAP] Recio, R., P. Culley, D. Garcia, J. Hilland, "An RDMA
Protocol Specification", Internet-Draft Work in Progress Protocol Specification", Internet-Draft Work in Progress
draft-ietf-rddp-rdmap-03.txt, December 2004. draft-ietf-rddp-rdmap-03.txt, December 2004.
[RFC3723] Aboba B., et al, "Securing Block Storage Protocols over [RFC3723] Aboba, B., et al, "Securing Block Storage Protocols
IP", Internet draft (work in progress), RFC3723, April 2004. over IP", Internet draft (work in progress), RFC3723, April
2004.
[SCTP] R. Stewart et al., "Stream Control Transmission Protocol", [SCTP] Stewart, R. et al., "Stream Control Transmission
RFC 2960, October 2000. Protocol", RFC 2960, October 2000.
[TCP] Postel, J., "Transmission Control Protocol - DARPA Internet [TCP] Postel, J., "Transmission Control Protocol - DARPA Internet
Program Protocol Specification", RFC 793, September 1981. Program Protocol Specification", RFC 793, September 1981.
9.2 Informative References 10.2 Informative References
[APPLICABILITY] Bestler, C. , Coene, L. "Applicability of Remote
Direct Memory Access Protocol (RDMA) and Direct Data
Placement (DDP)", Internet-Draft Work in Progress draft-ietf-
rddp-applicability-05.txt, December 2005.
[IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor [IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor
Discovery Trust Models and threats", Informational RFC, Discovery Trust Models and threats", Informational RFC,
RFC3756, May 2004. RFC3756, May 2004.
[NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to [NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to
Secure Channels", Internet-Draft draft-ietf-nfsv4-channel- Secure Channels", Internet-Draft draft-ietf-nfsv4-channel-
bindings-02.txt, July 2004. bindings-02.txt, July 2004.
10 Appendix A: ULP Issues for RDDP Client/Server Protocols [VERBS-RDMAC] "RDMA Protocol Verbs Specification", RDMA
Consortium standard, April 2003.
http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-
v1.0-RDMAC.pdf
[VERBS-RDMAC-Overview] "RDMA enabled NIC (RNIC) Verbs Overview",
slide presentation by Renato Recio, April 2003.
http://www.rdmaconsortium.org/home/RNIC_Verbs_Overview2.pdf
[RFC3552] "Guidelines for Writing RFC Text on Security
Considerations", Best Current Practice RFC, RFC3552, July
2003.
[INFINIBAND] "InfiniBand Architecture Specification Volume 1",
release 1.2, InfiniBand Trade Association standard.
http://www.infinibandta.org/specs. Verbs are documented in
chapter 11.
11 Appendix A: ULP Issues for RDDP Client/Server Protocols
This section is a normative appendix to the document that is This section is a normative appendix to the document that is
focused on client/server ULP implementation requirements to focused on client/server ULP implementation requirements to
ensure a secure server implementation. ensure a secure server implementation.
The prior sections outlined specific attacks and their The prior sections outlined specific attacks and their
countermeasures. This section summarizes the attacks and countermeasures. This section summarizes the attacks and
countermeasures that have been defined in the prior section which countermeasures that have been defined in the prior section which
are applicable to creation of a secure ULP (e.g. application) are applicable to creation of a secure ULP (e.g. application)
server. A ULP server is defined as a ULP which must be able to server. A ULP server is defined as a ULP which must be able to
skipping to change at page 43, line 47 skipping to change at page 45, line 47
can mount on the shared server, by re-stating the previous can mount on the shared server, by re-stating the previous
normative statements to be client/server specific. Note that each normative statements to be client/server specific. Note that each
client/server ULP may employ explicit RDMA operations (RDMA Read, client/server ULP may employ explicit RDMA operations (RDMA Read,
RDMA Write) in differing fashions. Therefore where appropriate, RDMA Write) in differing fashions. Therefore where appropriate,
"Local ULP", "Local Peer" and "Remote Peer" are used in place of "Local ULP", "Local Peer" and "Remote Peer" are used in place of
"server" or "client", in order to retain full generality of each "server" or "client", in order to retain full generality of each
requirement. requirement.
* Spoofing * Spoofing
* Sections 5.2.1 to 5.2.3. For protection against many * Sections 5.1.1 to 5.1.3. For protection against many
forms of spoofing attacks, enable IPsec. forms of spoofing attacks, enable IPsec.
* Section 5.2.4 Using an STag on a Different Stream on * Section 6.1.1 Using an STag on a Different Stream. To
page 20. To ensure that one client can not access ensure that one client can not access another
another client's data via use of the other client's client's data via use of the other client's STag, the
STag, the server ULP must either scope an STag to a server ULP must either scope an STag to a single
single Stream or use a unique Protection Domain per Stream or use a unique Protection Domain per client.
client. If a single client has multiple Streams that If a single client has multiple Streams that share
share Partial Mutual Trust, then the STag can be Partial Mutual Trust, then the STag can be shared
shared between the associated Streams by using a between the associated Streams by using a single
single Protection Domain among the associated Streams Protection Domain among the associated Streams (see
(see section 6.1.3 ULPs Which Provide Security on section 5.4.3 ULPs Which Provide Security for
page 38 for additional issues). To prevent unintended additional issues). To prevent unintended sharing of
sharing of STags within the associated Streams, a STags within the associated Streams, a server ULP
server ULP should use STags in such a fashion that it should use STags in such a fashion that it is
is difficult to predict the next allocated STag difficult to predict the next allocated STag number.
number.
* Tampering * Tampering
* 5.3.2 Modifying a Buffer After Indication on page 22. * 6.2.2 Modifying a Buffer After Indication. Before the
Before the local ULP operates on a buffer that was local ULP operates on a buffer that was written by
written by the Remote Peer using an RDMA Write or the Remote Peer using an RDMA Write or RDMA Read, the
RDMA Read, the local ULP MUST ensure the buffer can local ULP MUST ensure the buffer can no longer be
no longer be modified, by invalidating the STag for modified, by invalidating the STag for remote access
remote access (note that this is stronger than the (note that this is stronger than the SHOULD in
SHOULD in section 5.3.2). This can either be done section 6.2.2). This can either be done explicitly by
explicitly by revoking remote access rights for the revoking remote access rights for the STag when the
STag when the Remote Peer indicates the operation has Remote Peer indicates the operation has completed, or
completed, or by checking to make sure the Remote by checking to make sure the Remote Peer Invalidated
Peer Invalidated the STag through the RDMAP the STag through the RDMAP Invalidate capability, and
Invalidate capability, and if it did not, the local if it did not, the local ULP then explicitly revoking
ULP then explicitly revoking the STag remote access the STag remote access rights.
rights.
* Information Disclosure * Information Disclosure
* 5.4.2 Using RDMA Read to Access Stale Data on page * 6.3.2 Using RDMA Read to Access Stale Data. In a
23. In a general purpose server environment there is general purpose server environment there is no
no compelling rationale to not require a buffer to be compelling rationale to not require a buffer to be
initialized before remote read is enabled (and an initialized before remote read is enabled (and an
enormous down side of unintentionally sharing data). enormous down side of unintentionally sharing data).
Thus a local ULP MUST (this is stronger than the Thus a local ULP MUST (this is stronger than the
SHOULD in section 5.4.2) ensure that no stale data is SHOULD in section 6.3.2) ensure that no stale data is
contained in a buffer before remote read access contained in a buffer before remote read access
rights are granted to a Remote Peer (this can be done rights are granted to a Remote Peer (this can be done
by zeroing the contents of the memory, for example). by zeroing the contents of the memory, for example).
* 5.4.3 Accessing a Buffer After the Transfer on page * 6.3.3 Accessing a Buffer After the Transfer. This
24. This mitigation is already covered by section mitigation is already covered by section 6.2.2
5.3.2 (above). (above).
* 5.4.4 Accessing Unintended Data With a Valid STag on * 6.3.4 Accessing Unintended Data With a Valid STag.
page 24. The ULP must set the base and bounds of the The ULP must set the base and bounds of the buffer
buffer when the STag is initialized to expose only when the STag is initialized to expose only the data
the data to be retrieved. to be retrieved.
* 5.4.5 RDMA Read into an RDMA Write Buffer on page 24. * 6.3.5 RDMA Read into an RDMA Write Buffer. If a peer
If a peer only intends a buffer to be exposed for only intends a buffer to be exposed for remote write
remote write access, it must set the access rights to access, it must set the access rights to the buffer
the buffer to only enable remote write access. to only enable remote write access.
* 5.4.6 Using Multiple STags Which Alias to the Same * 6.3.6 Using Multiple STags Which Alias to the Same
Buffer on page 25. The requirement in section 5.2.4 Buffer. The requirement in section 6.1.1 (above)
(above) mitigates this attack. A server buffer is mitigates this attack. A server buffer is exposed to
exposed to only one client at a time to ensure that only one client at a time to ensure that no
no information disclosure or information tampering information disclosure or information tampering
occurs between peers. occurs between peers.
* 5.4.9 Network based eavesdropping on page 26. * 5.3 - Network Based Eavesdropping. Confidentiality
Confidentiality services should be enabled by the ULP services should be enabled by the ULP if this threat
if this threat is a concern. is a concern.
* Denial of Service * Denial of Service
* 5.5.2.1 Multiple Streams Sharing Receive Buffers on * 6.4.3.1 Multiple Streams Sharing Receive Buffers. ULP
page 27. ULP memory footprint size can be important memory footprint size can be important for some
for some server ULPs. If a server ULP is expecting server ULPs. If a server ULP is expecting significant
significant network traffic from multiple clients, network traffic from multiple clients, using a
using a receive buffer queue per Stream where there receive buffer queue per Stream where there is a
is a large number of Streams can consume substantial large number of Streams can consume substantial
amounts of memory. Thus a receive queue that can be amounts of memory. Thus a receive queue that can be
shared by multiple Streams is attractive. shared by multiple Streams is attractive.
However, because of the attacks outlined in this However, because of the attacks outlined in this
section, sharing a single receive queue between section, sharing a single receive queue between
multiple clients must only be done if a mechanism is multiple clients must only be done if a mechanism is
in place to ensure one client cannot consume receive in place to ensure one client cannot consume receive
buffers in excess of its limits, as defined by each buffers in excess of its limits, as defined by each
ULP. For multiple Streams within a single client ULP ULP. For multiple Streams within a single client ULP
(which presumably shared Partial Mutual Trust) this (which presumably shared Partial Mutual Trust) this
added overhead may be avoided. added overhead may be avoided.
* 5.5.2.2 Local ULP Attacking a Shared CQ on page 29. * 7.1 Local ULP Attacking a Shared CQ. The normative
The normative RNIC mitigations require the RNIC to RNIC mitigations require the RNIC to not enable
not enable sharing of a CQ if the local ULPs do not sharing of a CQ if the local ULPs do not share
share Partial Mutual Trust. Thus while the ULP is not Partial Mutual Trust. Thus while the ULP is not
allowed to enable this feature in an unsafe mode, if allowed to enable this feature in an unsafe mode, if
the two local ULPs share Partial Tutual Trust, they the two local ULPs share Partial Mutual Trust, they
must behave in the following manner: must behave in the following manner:
1) The sizing of the completion queue is based on the 1) The sizing of the completion queue is based on the
size of the receive queue and send queues as size of the receive queue and send queues as
documented in 5.5.2.3 Local or Remote Peer Attacking documented in 6.4.3.2 Remote or Local Peer Attacking
a Shared CQ on page 29. a Shared CQ.
2) The local ULP ensures that CQ entries are reaped 2) The local ULP ensures that CQ entries are reaped
frequently enough to adhere to section 5.5.2.3's frequently enough to adhere to section 6.4.3.2's
rules. rules.
* 5.5.2.3 Local or Remote Peer Attacking a Shared CQ on * 6.4.3.2 Remote or Local Peer Attacking a Shared CQ.
page 29. There are two mitigations specified in this There are two mitigations specified in this section -
section - one requires a worst-case size of the CQ, one requires a worst-case size of the CQ, and can be
and can be implemented entirely within the Privileged implemented entirely within the Privileged Resource
Resource Manager. The second approach requires Manager. The second approach requires cooperation
cooperation with the local ULP server (to not post with the local ULP server (to not post too many
too many buffers), and enables a smaller CQ to be buffers), and enables a smaller CQ to be used.
used.
In some server environments, partial trust of the In some server environments, partial trust of the
server ULP (but not the clients) is acceptable, thus server ULP (but not the clients) is acceptable, thus
the smaller CQ fully mitigates the remote attacker. the smaller CQ fully mitigates the remote attacker.
In other environments, the local server ULP could In other environments, the local server ULP could
also contain untrusted elements which can attack the also contain untrusted elements which can attack the
local machine (or have bugs). In those environments, local machine (or have bugs). In those environments,
the worst-case size of the CQ must be used. the worst-case size of the CQ must be used.
* 5.5.2.4 The section requires a server's Privileged * 6.4.3.3 The section requires a server's Privileged
Resource Manager to not allow sharing of RDMA Read Resource Manager to not allow sharing of RDMA Read
Request Queues across multiple Streams that do not Request Queues across multiple Streams that do not
share Partial Mutual Trust, for a ULP which performs share Partial Mutual Trust, for a ULP which performs
RDMA Read operations to server buffers. However, RDMA Read operations to server buffers. However,
because the server ULP knows best which of its because the server ULP knows best which of its
Streams share Partial Mutual Trust, this requirement Streams share Partial Mutual Trust, this requirement
can be reflected back to the ULP. The ULP (i.e. can be reflected back to the ULP. The ULP (i.e.
server) requirement in this case is that it MUST NOT server) requirement in this case is that it MUST NOT
allow RDMA Read Request Queues to be shared between allow RDMA Read Request Queues to be shared between
ULPs which do not have Partial Mutual Trust. ULPs which do not have Partial Mutual Trust.
* 5.5.5 Remote Invalidate an STag Shared on Multiple * 6.4.5 Remote Invalidate an STag Shared on Multiple
Streams on page 34. This mitigation is already Streams. This mitigation is already covered by
covered by section 5.3.2 (above). section 6.2.2 (above).
11 Appendix B: Summary of RNIC and ULP Implementation Requirements 12 Appendix B: Summary of RNIC and ULP Implementation Requirements
This appendix is informative. This appendix is informative.
Below is a summary of implementation requirements for the RNIC: Below is a summary of implementation requirements for the RNIC:
* 3 Trust and Resource Sharing * 3 Trust and Resource Sharing
* 5.2.4 Using an STag on a Different Stream * 6.1.1 Using an STag on a Different Stream
* 5.3.1 Buffer Overrun - RDMA Write or Read Response * 6.2.1 Buffer Overrun - RDMA Write or Read Response
* 5.3.2 Modifying a Buffer After Indication * 6.2.2 Modifying a Buffer After Indication
* 5.4.8 Controlling Access to PTT & STag Mapping * 6.3.7 Controlling Access to PTT & STag Mapping
* 5.5.1 RNIC Resource Consumption * 6.4.1 RNIC Resource Consumption
* 5.5.2.1 Multiple Streams Sharing Receive Buffers * 6.4.3.1 Multiple Streams Sharing Receive Buffers
* 5.5.2.2 Local ULP Attacking a Shared CQ * 7.1 Local ULP Attacking a Shared CQ
* 5.5.2.3 Local or Remote Peer Attacking a Shared CQ * 6.4.3.2 Remote or Local Peer Attacking a Shared CQ
* 5.5.2.4 Attacking the RDMA Read Request Queue * 6.4.3.3 Attacking the RDMA Read Request Queue
* 5.5.6 Remote Peer attacking an Unshared CQ on page 34. * 6.4.6 Remote Peer attacking an Unshared CQ.
* 5.6 Elevation of Privilege 35 * 6.5 Elevation of Privilege 38
* 6.2 Requirements for IPsec Encapsulation of DDP * 5.4.4 Requirements for IPsec Encapsulation of DDP
Below is a summary of implementation requirements for the ULP Below is a summary of implementation requirements for the ULP
above the RNIC: above the RNIC:
* 5.2.4 Using an STag on a Different Stream * 6.1.1 Using an STag on a Different Stream
* 5.3.2 Modifying a Buffer After Indication * 6.2.2 Modifying a Buffer After Indication
* 5.4.2 Using RDMA Read to Access Stale Data * 6.3.2 Using RDMA Read to Access Stale Data
* 5.4.3 Accessing a Buffer After the Transfer * 6.3.3 Accessing a Buffer After the Transfer
* 5.4.4 Accessing Unintended Data With a Valid STag * 6.3.4 Accessing Unintended Data With a Valid STag
* 5.4.5 RDMA Read into an RDMA Write Buffer * 6.3.5 RDMA Read into an RDMA Write Buffer
* 5.4.6 Using Multiple STags Which Alias to the Same Buffer * 6.3.6 Using Multiple STags Which Alias to the Same Buffer
* 5.4.9 Network based eavesdropping * 5.3 - Network Based Eavesdropping
* 5.5.2.2 Local ULP Attacking a Shared CQ * 7.1 Local ULP Attacking a Shared CQ
* 5.5.5 Remote Invalidate an STag Shared on Multiple * 6.4.5 Remote Invalidate an STag Shared on Multiple
Streams Streams
12 Appendix C: Partial Trust Taxonomy 13 Appendix C: Partial Trust Taxonomy
This appendix is informative. This appendix is informative.
Partial Trust is defined as when one party is willing to assume Partial Trust is defined as when one party is willing to assume
that another party will refrain from a specific attack or set of that another party will refrain from a specific attack or set of
attacks, the parties are said to be in a state of Partial Trust. attacks, the parties are said to be in a state of Partial Trust.
Note that the partially trusted peer may attempt a different set Note that the partially trusted peer may attempt a different set
of attacks. This may be appropriate for many ULPs where any of attacks. This may be appropriate for many ULPs where any
adverse effects of the betrayal is easily confined and does not adverse effects of the betrayal is easily confined and does not
place other clients or ULPs at risk. place other clients or ULPs at risk.
The Trust Models described in this section have three primary The Trust Models described in this section have three primary
distinguishing characteristics. The Trust Model refers to a local distinguishing characteristics. The Trust Model refers to a local
ULP and Remote Peer, which are intended to be the local and ULP and Remote Peer, which are intended to be the local and
remote ULP instances communicating via RDMA/DDP. remote ULP instances communicating via RDMA/DDP.
* Local Resource Sharing (yes/no) - When local resources * Local Resource Sharing (yes/no) - When local resources
are shared, they are shared across a grouping of are shared, they are shared across a grouping of
RDMAP/DDP Streams. If local resources are not shared, the RDMAP/DDP Streams. If local resources are not shared, the
resources are dedicated on a per Stream basis. Resources resources are dedicated on a per Stream basis. Resources
are defined in Section 2.2 - Resources on page 8. The are defined in Section 2.2 - Resources. The advantage of
advantage of not sharing resources between Streams is not sharing resources between Streams is that it reduces
that it reduces the types of attacks that are possible. the types of attacks that are possible. The disadvantage
The disadvantage is that ULPs might run out of resources. is that ULPs might run out of resources.
* Local Partial Trust (yes/no) - Local Partial Trust is * Local Partial Trust (yes/no) - Local Partial Trust is
determined based on whether the local grouping of determined based on whether the local grouping of
RDMAP/DDP Streams (which typically equates to one ULP or RDMAP/DDP Streams (which typically equates to one ULP or
group of ULPs) mutually trust each other to not perform a group of ULPs) mutually trust each other to not perform a
specific set of attacks. specific set of attacks.
* Remote Partial Trust (yes/no) - The Remote Partial Trust * Remote Partial Trust (yes/no) - The Remote Partial Trust
level is determined based on whether the local ULP of a level is determined based on whether the local ULP of a
specific RDMAP/DDP Stream partially trusts the Remote specific RDMAP/DDP Stream partially trusts the Remote
skipping to change at page 50, line 6 skipping to change at page 52, line 6
Remote Trust - typically a server ULP that wants to run Remote Trust - typically a server ULP that wants to run
in the safest mode possible. All attack mitigations are in the safest mode possible. All attack mitigations are
in place to ensure robust operation. in place to ensure robust operation.
* NS-RT - Non-Shared Local Resources, no Local Trust, * NS-RT - Non-Shared Local Resources, no Local Trust,
Remote Partial Trust - typically a peer-to-peer ULP, Remote Partial Trust - typically a peer-to-peer ULP,
which has, by some method outside of the scope of this which has, by some method outside of the scope of this
document, authenticated the Remote Peer. Note that unless document, authenticated the Remote Peer. Note that unless
some form of key based authentication is used on a per some form of key based authentication is used on a per
RDMA/DDP Stream basis, it may not be possible be possible RDMA/DDP Stream basis, it may not be possible be possible
for man-in-the-middle attacks to occur. See section 6, for man-in-the-middle attacks to occur.
Security Services for RDMAP and DDP on page 36.
* S-NT - Shared Local Resources, no Local Trust, no Remote * S-NT - Shared Local Resources, no Local Trust, no Remote
Trust - typically a server ULP that runs in an untrusted Trust - typically a server ULP that runs in an untrusted
environment where the amount of resources required is environment where the amount of resources required is
either too large or too dynamic to dedicate for each either too large or too dynamic to dedicate for each
RDMAP/DDP Stream. RDMAP/DDP Stream.
* S-LT - Shared Local Resources, Local Partial Trust, no * S-LT - Shared Local Resources, Local Partial Trust, no
Remote Trust - typically a ULP, which provides a session Remote Trust - typically a ULP, which provides a session
layer and uses multiple Streams, to provide additional layer and uses multiple Streams, to provide additional
skipping to change at page 51, line 5 skipping to change at page 53, line 5
neither local ULPs nor the Remote Peer is trusted. Sometimes neither local ULPs nor the Remote Peer is trusted. Sometimes
optimizations can be done that enable sharing of Page Translation optimizations can be done that enable sharing of Page Translation
Tables across multiple local ULPs, thus Model S-LT can be Tables across multiple local ULPs, thus Model S-LT can be
advantageous. Model S-T is typically used when resource scaling advantageous. Model S-T is typically used when resource scaling
across a large parallel ULP makes it infeasible to use any other across a large parallel ULP makes it infeasible to use any other
model. Resource scaling issues can either be due to performance model. Resource scaling issues can either be due to performance
around scaling or because there simply are not enough resources. around scaling or because there simply are not enough resources.
Model NS-RT is probably the least likely model to be used, but is Model NS-RT is probably the least likely model to be used, but is
presented for completeness. presented for completeness.
13 Author's Addresses 14 Author's Addresses
James Pinkerton James Pinkerton
Microsoft Corporation Microsoft Corporation
One Microsoft Way One Microsoft Way
Redmond, WA. 98052 USA Redmond, WA. 98052 USA
Phone: +1 (425) 705-5442 Phone: +1 (425) 705-5442
Email: jpink@windows.microsoft.com Email: jpink@windows.microsoft.com
Ellen Deleganes Ellen Deleganes
Intel Corporation Intel Corporation
MS JF5-355 MS JF5-355
2111 NE 25th Ave. 2111 NE 25th Ave.
Hillsboro, OR 97124 USA Hillsboro, OR 97124 USA
Phone: +1 (503) 712-4173 Phone: +1 (503) 712-4173
Email: ellen.m.deleganes@intel.com Email: ellen.m.deleganes@intel.com
Sara Bitan Sara Bitan
Microsoft Corporation Microsoft Corporation
Email: sarab@microsoft.com Email: sarab@microsoft.com
14 Acknowledgments 15 Acknowledgments
Allyn Romanow Allyn Romanow
Cisco Systems Cisco Systems
170 W Tasman Drive 170 W Tasman Drive
San Jose, CA 95134 USA San Jose, CA 95134 USA
Phone: +1 408 525 8836 Phone: +1 408 525 8836
Email: allyn@cisco.com Email: allyn@cisco.com
Catherine Meadows Catherine Meadows
Naval Research Laboratory Naval Research Laboratory
skipping to change at page 52, line 43 skipping to change at page 54, line 43
Email: james.livingston@necsam.com Email: james.livingston@necsam.com
John Carrier John Carrier
Adaptec, Inc. Adaptec, Inc.
691 S. Milpitas Blvd. 691 S. Milpitas Blvd.
Milpitas, CA 95035 USA Milpitas, CA 95035 USA
Phone: +1 (360) 378-8526 Phone: +1 (360) 378-8526
Email: john_carrier@adaptec.com Email: john_carrier@adaptec.com
Caitlin Bestler Caitlin Bestler
Broadcom
49 Discovery
Irvine, CA 92618
Email: cait@asomi.com Email: cait@asomi.com
Bernard Aboba Bernard Aboba
Microsoft Corporation Microsoft Corporation
One Microsoft Way One Microsoft Way
Redmond, WA. 98052 USA Redmond, WA. 98052 USA
Phone: +1 (425) 706-6606 Phone: +1 (425) 706-6606
Email: bernarda@windows.microsoft.com Email: bernarda@windows.microsoft.com
15 Full Copyright Statement 16 Full Copyright Statement
Copyright (C) The Internet Society (2005). Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided This document and the information contained herein are provided
on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
 End of changes. 182 change blocks. 
826 lines changed or deleted 826 lines changed or added

This html diff was produced by rfcdiff 1.29, available from http://www.levkowetz.com/ietf/tools/rfcdiff/