draft-ietf-rddp-security-05.txt   draft-ietf-rddp-security-06.txt 
Internet Draft James Pinkerton Internet Draft James Pinkerton
draft-ietf-rddp-security-05.txt Microsoft Corporation draft-ietf-rddp-security-06.txt Microsoft Corporation
Category: Standards Track Ellen Deleganes Category: Standards Track Ellen Deleganes
Expires: February, 2005 Intel Corporation Expires: June, 2005 Intel Corporation
Sara Bitan Sara Bitan
Microsoft Corporation Microsoft Corporation
August 2004 December 2004
DDP/RDMAP Security DDP/RDMAP Security
1 Status of this Memo 1 Status of this Memo
This document is an Internet-Draft and is in full conformance By submitting this Internet-Draft, I certify that any applicable
with all provisions of Section 10 of RFC2026. patent or other IPR claims of which I am aware have been
disclosed, or will be disclosed, and any of which I become aware
will be disclosed, in accordance with RFC 3668.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in as reference material or to cite them other than as "work in
skipping to change at page 2, line ? skipping to change at page 2, line ?
Memory Access Protocol (RDMAP). It first defines an architectural Memory Access Protocol (RDMAP). It first defines an architectural
model for an RDMA Network Interface Card (RNIC), which can model for an RDMA Network Interface Card (RNIC), which can
implement DDP or RDMAP and DDP. The document reviews various implement DDP or RDMAP and DDP. The document reviews various
attacks against the resources defined in the architectural model attacks against the resources defined in the architectural model
and the countermeasures that can be used to protect the system. and the countermeasures that can be used to protect the system.
Attacks are grouped into spoofing, tampering, information Attacks are grouped into spoofing, tampering, information
disclosure, denial of service, and elevation of privilege. disclosure, denial of service, and elevation of privilege.
Finally, the document concludes with a summary of security Finally, the document concludes with a summary of security
services for DDP and RDMAP, such as IPsec. services for DDP and RDMAP, such as IPsec.
J. Pinkerton, et al. Expires February 2005 1 J. Pinkerton, et al. Expires June, 2005 1
Table of Contents Table of Contents
1 Status of this Memo.........................................1 1 Status of this Memo..........................................1
2 Abstract....................................................1 2 Abstract.....................................................1
2.1 Revision History............................................3 2.1 Revision History.............................................3
2.1.1 Changes from -04 to -05 version............................3 2.1.1 Changes from -05 to -06 version............................3
2.1.2 Changes from -03 to -04 version............................4 2.1.2 Changes from -04 to -05 version............................4
2.1.3 Changes from -02 to -03 version............................4 2.1.3 Changes from -03 to -04 version............................5
2.1.4 Changes from the -01 to the -02 version....................5 2.1.4 Changes from -02 to -03 version............................5
2.1.5 Changes from the -00 to -01 version........................5 2.1.5 Changes from the -01 to the -02 version....................6
3 Introduction................................................7 2.1.6 Changes from the -00 to -01 version........................6
4 Architectural Model.........................................9 3 Introduction.................................................8
4.1 Components.................................................10 4 Architectural Model.........................................10
4.2 Resources..................................................11 4.1 Components..................................................11
4.2.1 Stream Context Memory.....................................11 4.2 Resources...................................................12
4.2.2 Data Buffers..............................................11 4.2.1 Stream Context Memory.....................................12
4.2.3 Page Translation Tables...................................12 4.2.2 Data Buffers..............................................12
4.2.4 STag Namespace............................................12 4.2.3 Page Translation Tables...................................13
4.2.5 Completion Queues.........................................12 4.2.4 STag Namespace............................................13
4.2.6 Asynchronous Event Queue..................................13 4.2.5 Completion Queues.........................................13
4.2.7 RDMA Read Request Queue...................................13 4.2.6 Asynchronous Event Queue..................................14
4.2.8 RNIC Interactions.........................................13 4.2.7 RDMA Read Request Queue...................................14
4.2.8.1 Privileged Control Interface Semantics................13 4.2.8 RNIC Interactions.........................................14
4.2.8.2 Non-Privileged Data Interface Semantics...............14 4.2.8.1 Privileged Control Interface Semantics.................14
4.2.8.3 Privileged Data Interface Semantics...................14 4.2.8.2 Non-Privileged Data Interface Semantics................15
4.2.9 Initialization of RNIC Data Structures for Data Transfer..14 4.2.8.3 Privileged Data Interface Semantics....................15
4.2.10 RNIC Data Transfer Interactions.........................16 4.2.9 Initialization of RNIC Data Structures for Data Transfer..15
5 Trust and Resource Sharing.................................17 4.2.10 RNIC Data Transfer Interactions..........................17
6 Attacker Capabilities......................................18 5 Trust and Resource Sharing..................................18
7 Attacks and Countermeasures................................19 6 Attacker Capabilities.......................................19
7.1 Tools for Countermeasures..................................19 7 Attacks and Countermeasures.................................20
7.1.1 Protection Domain (PD)....................................19 7.1 Tools for Countermeasures...................................20
7.1.2 Limiting STag Scope.......................................20 7.1.1 Protection Domain (PD)....................................20
7.1.3 Access Rights.............................................21 7.1.2 Limiting STag Scope.......................................21
7.1.4 Limiting the Scope of the Completion Queue................21 7.1.3 Access Rights.............................................22
7.1.5 Limiting the Scope of an Error............................21 7.1.4 Limiting the Scope of the Completion Queue................22
7.2 Spoofing...................................................21 7.1.5 Limiting the Scope of an Error............................22
7.2.1 Impersonation.............................................22 7.2 Spoofing....................................................23
7.2.2 Stream Hijacking..........................................22 7.2.1 Impersonation.............................................23
7.2.3 Man in the Middle Attack..................................22 7.2.2 Stream Hijacking..........................................23
7.2.4 Using an STag on a Different Stream.......................23 7.2.3 Man in the Middle Attack..................................24
7.3 Tampering..................................................24 7.2.4 Using an STag on a Different Stream.......................24
7.3.1 Buffer Overrun - RDMA Write or Read Response..............24 7.3 Tampering...................................................25
7.3.2 Modifying a Buffer After Indication.......................25 7.3.1 Buffer Overrun - RDMA Write or Read Response..............26
7.3.3 Multiple STags to access the same buffer..................25 7.3.2 Modifying a Buffer After Indication.......................26
7.3.4 Network based modification of buffer content..............25 7.3.3 Multiple STags to access the same buffer..................27
7.4 Information Disclosure.....................................26 7.3.4 Network based modification of buffer content..............27
7.4.1 Probing memory outside of the buffer bounds...............26 7.4 Information Disclosure......................................27
7.4.2 Using RDMA Read to Access Stale Data......................26 7.4.1 Probing memory outside of the buffer bounds...............27
7.4.3 Accessing a Buffer After the Transfer.....................26 7.4.2 Using RDMA Read to Access Stale Data......................27
7.4.4 Accessing Unintended Data With a Valid STag...............26 7.4.3 Accessing a Buffer After the Transfer.....................28
7.4.5 RDMA Read into an RDMA Write Buffer.......................27 7.4.4 Accessing Unintended Data With a Valid STag...............28
7.4.6 Using Multiple STags Which Alias to the Same Buffer.......27 7.4.5 RDMA Read into an RDMA Write Buffer.......................28
7.4.7 Remote Node Loading Firmware onto the RNIC................28 7.4.6 Using Multiple STags Which Alias to the Same Buffer.......29
7.4.8 Controlling Access to PTT & STag Mapping..................28 7.4.7 Remote Node Loading Firmware onto the RNIC................29
7.4.9 Network based eavesdropping...............................28 7.4.8 Controlling Access to PTT & STag Mapping..................29
7.5 Denial of Service (DOS)....................................29 7.4.9 Network based eavesdropping...............................30
7.5.1 RNIC Resource Consumption.................................29 7.5 Denial of Service (DOS).....................................30
7.5.2 Resource Consumption By Active ULPs.......................30 7.5.1 RNIC Resource Consumption.................................30
7.5.2.1 Multiple Streams Sharing Receive Buffers..............30 7.5.2 Resource Consumption By Active ULPs.......................31
7.5.2.2 Local ULP Attacking a Shared CQ.......................31 7.5.2.1 Multiple Streams Sharing Receive Buffers...............31
7.5.2.3 Remote Peer Attacking a Shared CQ.....................32 7.5.2.2 Local ULP Attacking a Shared CQ........................33
7.5.2.4 Attacking the RDMA Read Request Queue.................35 7.5.2.3 Local or Remote Peer Attacking a Shared CQ.............33
7.5.3 Resource Consumption by Idle ULPs.........................36 7.5.2.4 Attacking the RDMA Read Request Queue..................36
7.5.4 Exercise of non-optimal code paths........................36 7.5.3 Resource Consumption by Idle ULPs.........................37
7.5.5 Remote Invalidate an STag Shared on Multiple Streams......37 7.5.4 Exercise of non-optimal code paths........................38
7.6 Elevation of Privilege.....................................37 7.5.5 Remote Invalidate an STag Shared on Multiple Streams......38
8 Security Services for RDMA and DDP.........................38 7.5.6 Remote Peer attacking an Unshared CQ......................38
8.1 Introduction to Security Options...........................38 7.6 Elevation of Privilege......................................39
8.1.1 Introduction to IPsec.....................................38 8 Security Services for RDMAP and DDP.........................40
8.1.2 Introduction to SSL Limitations on RDMAP..................40 8.1 Introduction to Security Options............................40
8.1.3 ULPs Which Provide Security...............................40 8.1.1 Introduction to IPsec.....................................40
8.2 Requirements for IPsec Encapsulation of DDP................41 8.1.2 Introduction to SSL Limitations on RDMAP..................42
9 Security considerations....................................42 8.1.3 ULPs Which Provide Security...............................42
10 References.................................................43 8.2 Requirements for IPsec Encapsulation of DDP.................43
10.1 Normative References......................................43 9 Security considerations.....................................44
10.2 Informative References....................................43 10 References..................................................45
11 Appendix A: ULP Issues for RDDP Client/Server Protocols....44 10.1 Normative References......................................45
10.2 Informative References....................................45
11 Appendix A: ULP Issues for RDDP Client/Server Protocols.....46
12 Appendix B: Summary of RNIC and ULP Implementation 12 Appendix B: Summary of RNIC and ULP Implementation
Requirements.....................................................48 Requirements.....................................................50
13 Appendix C: Partial Trust Taxonomy.........................50 13 Appendix C: Partial Trust Taxonomy..........................52
14 Author’s Addresses.........................................52 14 AuthorÆs Addresses..........................................54
15 Acknowledgments............................................53 15 Acknowledgments.............................................55
16 Full Copyright Statement...................................54 16 Full Copyright Statement....................................56
Table of Figures Table of Figures
Figure 1 - RDMA Security Model...................................10 Figure 1 - RDMA Security Model...................................11
2.1 Revision History 2.1 Revision History
2.1.1 Changes from -04 to -05 version 2.1.1 Changes from -05 to -06 version
* Appendix A: ULP Issues for RDDP Client/Server Protocols,
Section 7.5.2.2. Changed usage of MUST, MAY, etc to be
lower case if just repeating prior requirements, upper
case if a new requirement. Completed writeup on section
7.5.2.2 for client/server protocols.
* Added new attack - section 7.5.6 Remote Peer attacking an
Unshared CQ on page 38.
* Minor clarification in section 7.6 - clarified that the
threat is for local and remote peer (unauthorized loading
of firmware). Also removed redundant sentence at end of
section.
* Added new normative statements per the last IETF meeting
(7.2.4, 7.3.2)
* Provided better insight on what is at the ULP level
verses what is at the protocol level. Provided
definitions for "Local Peer", "Remote Peer", and added
new concept of "local ULP". Then swept the document for
Local Peer and either left it unchanged, changed it local
ULP, or add both. Remote Peer left unchanged because it's
difficult to separate the ULP from the protocol on the
remote end.
* Included detailed review changes from Tom Talpey and
Mallikarjun Chadalapaka. Includes:
* More formal definition of Remote Peer and Local Peer,
and subdividing Local Peer better between local ULP
and Local Peer. Recommend careful review of where
"local ULP" is used to make sure I got it right.
* Changed some instances where "ULP" was used talk
about shared resources to "Stream".
* Clarified the Attacker Capabilities a bit.
* Fixed misc minor issues, including capitalization
issues.
* Clarification on zero-length RDMA Read messages.
2.1.2 Changes from -04 to -05 version
* Small modifications to normative statements per phone * Small modifications to normative statements per phone
call review. call review.
* 4.1 - Moved MUST statement from Privileged Resource * 4.1 - Moved MUST statement from Privileged Resource
Manager to section 5. Also added additional normative Manager to section 5. Also added additional normative
statements around resource sharing and assumptions of statements around resource sharing and assumptions of
who trusts whom. who trusts whom.
* 7.2.4 - changed last paragraph SHOULD to should. * 7.2.4 - changed last paragraph SHOULD to should.
* 7.4.4 - changed last paragraph MUST to SHOULD. * 7.4.4 - changed last paragraph MUST to SHOULD.
* 7.5.2.2 - clarified it is the ULP at issue, and * 7.5.2.2 - clarified it is the ULP at issue, and
removed reference to Protection Domain - key issue is removed reference to Protection Domain - key issue is
whether they share partial mutual trust. whether they share partial mutual trust.
* 7.5.2.4 - remove MUST statement at the end of the 3rd * 7.5.2.4 - remove MUST statement at the end of the 3rd
paragraph - it was replaced with a more general MUST paragraph - it was replaced with a more general MUST.
in section <TBD>. Also changed the cap on the number Also changed the cap on the number of outstanding
of outstanding RDMA Read Requests at the sender to a RDMA Read Requests at the sender to a SHOULD (from
SHOULD (from MUST). MUST).
* 8.1 - first paragraph after enumerated list. Change * 8.1 - first paragraph after enumerated list. Change
MAY to may. It is a ULP issue. MAY to may. It is a ULP issue.
* Removed "application" from the document and replaced it * Removed "application" from the document and replaced it
with "ULP". In some cases also changed "Local Peer" to with "ULP". In some cases also changed "Local Peer" to
ULP to clarify what the text meant. ULP to clarify what the text meant.
2.1.2 Changes from -03 to -04 version 2.1.3 Changes from -03 to -04 version
* Removed "issues" section because all issues have been * Removed "issues" section because all issues have been
resolved. resolved.
* Completed section "ULPs Which Provide Security" by * Completed section "ULPs Which Provide Security" by
providing a cross reference to channel bindings. providing a cross reference to channel bindings.
* Substantial rewrite of Section 11 Appendix A: ULP Issues * Substantial rewrite of Section 11 Appendix A: ULP Issues
for RDDP Client/Server Protocols. Retargeted it to focus for RDDP Client/Server Protocols. Retargeted it to focus
on server application requirements, rather than RNIC on server application requirements, rather than RNIC
skipping to change at page 4, line 49 skipping to change at page 5, line 45
* Changed "IPSec" to "IPsec" everywhere to match the RFC. * Changed "IPSec" to "IPsec" everywhere to match the RFC.
* Added new ULP requirement in section 7.5.2.4 Attacking * Added new ULP requirement in section 7.5.2.4 Attacking
the RDMA Read Request Queue. the RDMA Read Request Queue.
* Reviesed Sectio 12 Appendix B: Summary of RNIC and ULP * Reviesed Sectio 12 Appendix B: Summary of RNIC and ULP
Implementation Requirements slightly to add one ULP Implementation Requirements slightly to add one ULP
requirement and one RNIC requirement which is stated in requirement and one RNIC requirement which is stated in
the document but was missed in this summary. the document but was missed in this summary.
2.1.3 Changes from -02 to -03 version 2.1.4 Changes from -02 to -03 version
* ID changed from Informational to Standards Track. This * ID changed from Informational to Standards Track. This
caused previous RECOMMENDATIONS to be categorized into caused previous RECOMMENDATIONS to be categorized into
the categories of MUST, SHOULD, MAY, RECOMMENDED, and in the categories of MUST, SHOULD, MAY, RECOMMENDED, and in
one case, "recommended". one case, "recommended".
* Completed Appendix B: Summary of Attacks to provide a * Completed Appendix B: Summary of Attacks to provide a
summary of implementation requirements for applications summary of implementation requirements for applications
using RDDP and for RNICs in Appendix B: Summary of using RDDP and for RNICs in Appendix B: Summary of
Attacks. Attacks.
skipping to change at page 5, line 38 skipping to change at page 6, line 35
* Changed section 8.2 to normative xref to IPS Security, * Changed section 8.2 to normative xref to IPS Security,
plus comment on the value of end-to-end IPsec. plus comment on the value of end-to-end IPsec.
* Added clarifying example on STag invalidation (e.g. One- * Added clarifying example on STag invalidation (e.g. One-
Shot STag discussion). Shot STag discussion).
* Added clarifying text on why SSL is a bad idea. * Added clarifying text on why SSL is a bad idea.
* Normative statement on mitigation for Shared RQ. * Normative statement on mitigation for Shared RQ.
2.1.4 Changes from the -01 to the -02 version 2.1.5 Changes from the -01 to the -02 version
Minimal - some typos, deleted some text previously marked for Minimal - some typos, deleted some text previously marked for
deletion. deletion.
2.1.5 Changes from the -00 to -01 version 2.1.6 Changes from the -00 to -01 version
* Added two pages to the architectural model to describe * Added two pages to the architectural model to describe
the Asynchronous Event Queue, and the types of the Asynchronous Event Queue, and the types of
interactions that can occur between the RNIC and the interactions that can occur between the RNIC and the
modules above it. modules above it.
* Addressed Mike Krauses comments submitted on 12/8/2003 * Addressed Mike Krauses comments submitted on 12/8/2003
* Moved "Trust Models" from the body of the document to an * Moved "Trust Models" from the body of the document to an
appendix. Removed references to it throughout the appendix. Removed references to it throughout the
skipping to change at page 6, line 22 skipping to change at page 7, line 20
section names for attacks to fit in table. section names for attacks to fit in table.
* Added a new concept of "Partial Mutual Trust" between a * Added a new concept of "Partial Mutual Trust" between a
collection of Streams to better characterize a set of collection of Streams to better characterize a set of
attacks in a client/server environment. attacks in a client/server environment.
* Filled in Security Services for RDMA and DDP section * Filled in Security Services for RDMA and DDP section
(almost all is new, except IPsec overview). (almost all is new, except IPsec overview).
* Globally tried to change "connection" to "Stream". In * Globally tried to change "connection" to "Stream". In
some cases it can be either a connection or stream. some cases it can be either a connection or Stream.
3 Introduction 3 Introduction
RDMA enables new levels of flexibility when communicating between RDMA enables new levels of flexibility when communicating between
two parties compared to current conventional networking practice two parties compared to current conventional networking practice
(e.g. a stream-based model or datagram model). This flexibility (e.g. a stream-based model or datagram model). This flexibility
brings new security issues that must be carefully understood when brings new security issues that must be carefully understood when
designing Upper Layer Protocols (ULPs) utilizing RDMA and when designing Upper Layer Protocols (ULPs) utilizing RDMA and when
implementing RDMA-aware NICs (RNICs). Note that for the purposes implementing RDMA-aware NICs (RNICs). Note that for the purposes
of this security analysis, an RNIC may implement RDMAP and DDP, of this security analysis, an RNIC may implement RDMAP and DDP,
or just DDP. Also, a ULP may be an application or it may be a or just DDP. Also, a ULP may be an application or it may be a
middleware library. middleware library.
The specification first develops an architectural model that is The document first develops an architectural model that is
relevant for the security analysis - it details components, relevant for the security analysis - it details components,
resources, and system properties that may be attacked in Section resources, and system properties that may be attacked in Section
4. 4. The document uses Local Peer to represent the RDMA/DDP
protocol implementation on the local end of a Stream. The local
Upper-Layer-Protocol (ULP) is used to represent the application
or middle-ware layer above the Local Peer. The document does not
attempt to differentiate between a Remote Peer and a Remote ULP
(an RDMA/DDP protocol implementation on the remote end of a
Stream versus the application on the remote end) for several
reasons: often the source of the attack is difficult to know for
sure; and regardless of the source, the mitigations required of
the Local Peer or local ULP are the same. Thus the document
generically refers to a Remote Peer rather than trying to further
delineate the attacker.
It then defines what resources a ULP may share locally across The document then defines what resources a local ULP may share
Streams and what resources the ULP may share with the Remote Peer across Streams and what resources the local ULP may share with
across Streams in Section 5. Intentional sharing of resources the Remote Peer across Streams in Section 5.
between multiple Streams may imply some level of trust between
the Streams. However, some types of resource sharing have Intentional sharing of resources between multiple Streams may
unmitigated security attacks which would mandate not sharing a imply some level of trust between the Streams. However, some
specific type of resource unless there is some level of trust types of resource sharing have unmitigated security attacks which
between the Streams sharing resources. Partial Mutual Trust is would mandate not sharing a specific type of resource unless
defined to address this concept: there is some level of trust between the Streams sharing
resources.
This document defines a new term, "Partial Mutual Trust" to
address this concept:
Partial Mutual Trust - a collection of RDMAP/DDP Streams, Partial Mutual Trust - a collection of RDMAP/DDP Streams,
which represent the local and remote end points of the which represent the local and remote end points of the
Stream, are willing to assume that the Streams from the Stream, which are willing to assume that the Streams from
collection will not perform malicious attacks against any of the collection will not perform malicious attacks against
the Streams in the collection. ULPs have explicit control of any of the other Streams in the collection.
which collection of endpoints is in the collection through
tools discussed in Section 7.1 Tools for Countermeasures on ULPs have explicit control of which collection of endpoints is in
page 19. a Partial Mutual Trust collection through tools discussed in
Section 7.1 Tools for Countermeasures on page 20.
An untrusted peer relationship is appropriate when a ULP wishes An untrusted peer relationship is appropriate when a ULP wishes
to ensure that it will be robust and uncompromised even in the to ensure that it will be robust and uncompromised even in the
face of a deliberate attack by its peer. For example, a single face of a deliberate attack by its peer. For example, a single
ULP that concurrently supports multiple unrelated Streams (e.g. a ULP that concurrently supports multiple unrelated Streams (e.g. a
server) would presumably treat each of its peers as an untrusted server) would presumably treat each of its peers as an untrusted
peer. For a collection of Streams which share Partial Mutual peer. For a collection of Streams which share Partial Mutual
Trust, the assumption is that any Stream not in the collection is Trust, the assumption is that any Stream not in the collection is
untrusted. For the untrusted peer, a brief list of capabilities untrusted. For the untrusted peer, a brief list of capabilities
is enumerated in Section 6. is enumerated in Section 6.
The rest of the specification is focused on analyzing attacks. The rest of the document is focused on analyzing attacks and
First, the tools for mitigating attacks are listed (Section 7.1), recommending specific mitigations to the attacks. First, the
and then a series of attacks on components, resources, or system tools for mitigating attacks are listed (Section 7.1), and then a
properties is enumerated in the rest of Section 7. For each series of attacks on components, resources, or system properties
attack, possible countermeasures are reviewed. If all recommended is listed in the rest of Section 7. For each attack, possible
mitigations are in place the implemented usage models, the countermeasures are reviewed.
RDMAP/DDP protocol can be shown to not expose any new security
vulnerabilities.
ULPs within a host are divided into two categories - Privileged ULPs within a host are divided into two categories - Privileged
and Non-Privileged. Both ULP types can send and receive data and and Non-Privileged. Both ULP types can send and receive data and
request resources. The key differences between the two are: request resources. The key differences between the two are:
The Privileged ULP is trusted by the system to not The Privileged ULP is trusted by the local system to not
maliciously attack the operating environment, but it is not maliciously attack the operating environment, but it is not
trusted to optimize resource allocation globally. For trusted to optimize resource allocation globally. For
example, the Privileged ULP could be a kernel ULP, thus the example, the Privileged ULP could be a kernel ULP, thus the
kernel presumably has in some way vetted the ULP before kernel presumably has in some way vetted the ULP before
allowing it to execute. allowing it to execute.
A Non-Privileged ULP's capabilities are a logical sub-set of A Non-Privileged ULP's capabilities are a logical sub-set of
the Privileged ULP's. It is assumed by the local system that the Privileged ULP's. It is assumed by the local system that
a Non-Privileged ULP is untrusted. All Non-Privileged ULP a Non-Privileged ULP is untrusted. All Non-Privileged ULP
interactions with the RNIC Engine that could affect other interactions with the RNIC Engine that could affect other
ULPs need to be done through a trusted intermediary that can ULPs need to be done through a trusted intermediary that can
verify the Non-Privileged ULP requests. verify the Non-Privileged ULP requests.
If all recommended mitigations are in place the implemented usage
models, the RDMAP/DDP protocol can be shown to not expose any new
security vulnerabilities.
4 Architectural Model 4 Architectural Model
This section describes an RDMA architectural reference model that This section describes an RDMA architectural reference model that
is used as security issues are examined. It introduces the is used as security issues are examined. It introduces the
components of the model, the resources that can be attacked, the components of the model, the resources that can be attacked, the
types of interactions possible between components and resources, types of interactions possible between components and resources,
and the system properties which must be preserved. and the system properties which must be preserved.
Figure 1 shows the components comprising the architecture and the Figure 1 shows the components comprising the architecture and the
interfaces where potential security attacks could be launched. interfaces where potential security attacks could be launched.
skipping to change at page 10, line 10 skipping to change at page 11, line 10
^ ^
| |
v v
Internet Internet
Figure 1 - RDMA Security Model Figure 1 - RDMA Security Model
4.1 Components 4.1 Components
The components shown in Figure 1 - RDMA Security Model are: The components shown in Figure 1 - RDMA Security Model are:
* RNIC Engine (RNIC) - the component that implements the * RDMA Network Interface Controller Engine (RNIC) - the
RDMA protocol and/or DDP protocol. component that implements the RDMA protocol and/or DDP
protocol.
* Privileged Resource Manager - the component responsible * Privileged Resource Manager - the component responsible
for managing and allocating resources associated with the for managing and allocating resources associated with the
RNIC Engine. The Resource Manager does not send or RNIC Engine. The Resource Manager does not send or
receive data. Note that whether the Resource Manager is receive data. Note that whether the Resource Manager is
an independent component, part of the RNIC, or part of an independent component, part of the RNIC, or part of
the ULP is implementation dependent. If a specific the ULP is implementation dependent. If a specific
implementation does not wish to address security issues implementation does not wish to address security issues
resolved by the Resource Manager, there may in fact be no resolved by the Resource Manager, there may in fact be no
resource manager at all. resource manager at all.
skipping to change at page 10, line 37 skipping to change at page 11, line 38
through the RNIC Interface, but it does not allow the through the RNIC Interface, but it does not allow the
Privileged ULP to directly consume RNIC Engine resources. Privileged ULP to directly consume RNIC Engine resources.
* Non-Privileged ULP - See Section 3 Introduction for a * Non-Privileged ULP - See Section 3 Introduction for a
definition of Non-Privileged ULP. definition of Non-Privileged ULP.
A design goal of the DDP and RDMAP protocols is to allow, under A design goal of the DDP and RDMAP protocols is to allow, under
constrained conditions, Non-Privileged ULP to send and receive constrained conditions, Non-Privileged ULP to send and receive
data directly to/from the RDMA Engine without Privileged Resource data directly to/from the RDMA Engine without Privileged Resource
Manager intervention - while ensuring that the host remains Manager intervention - while ensuring that the host remains
secure. Thus, one of the primary goals of this paper is to secure. Thus, one of the primary goals of this document is to
analyze this usage model for the enforcement that is required in analyze this usage model for the enforcement that is required in
the RNIC Engine to ensure the system remains secure. the RNIC Engine to ensure the system remains secure.
The host interfaces that could be exercised include: The host interfaces that could be exercised include:
* Privileged Control Interface - A Privileged Resource * Privileged Control Interface - A Privileged Resource
Manager uses the RNIC Interface to allocate and manage Manager uses the RNIC Interface to allocate and manage
RNIC Engine resources, control the state within the RNIC RNIC Engine resources, control the state within the RNIC
Engine, and monitor various events from the RNIC Engine. Engine, and monitor various events from the RNIC Engine.
It also uses this interface to act as a proxy for some It also uses this interface to act as a proxy for some
skipping to change at page 11, line 25 skipping to change at page 12, line 25
to a ULP data buffer. to a ULP data buffer.
* Figure 1 also shows the ability to load new firmware in * Figure 1 also shows the ability to load new firmware in
the RNIC Engine. Not all RNICs will support this, but it the RNIC Engine. Not all RNICs will support this, but it
is shown for completeness and is also reviewed under is shown for completeness and is also reviewed under
potential attacks. potential attacks.
If Internet control messages, such as ICMP, ARP, RIPv4, etc. are If Internet control messages, such as ICMP, ARP, RIPv4, etc. are
processed by the RNIC Engine, the threat analyses for those processed by the RNIC Engine, the threat analyses for those
protocols is also applicable, but outside the scope of this protocols is also applicable, but outside the scope of this
paper. document.
4.2 Resources 4.2 Resources
This section describes the primary resources in the RNIC Engine This section describes the primary resources in the RNIC Engine
that could be affected if under attack. For RDMAP, all of the that could be affected if under attack. For RDMAP, all of the
defined resources apply. For DDP, all of the resources except the defined resources apply. For DDP, all of the resources except the
RDMA Read Queue apply. RDMA Read Queue apply.
4.2.1 Stream Context Memory 4.2.1 Stream Context Memory
The state information for each Stream is maintained in memory, The state information for each Stream is maintained in memory,
which could be located in a number of places - on the NIC, inside which could be located in a number of places - on the NIC, inside
RAM attached to the NIC, in host memory, or in any combination of RAM attached to the NIC, in host memory, or in any combination of
the three, depending on the implementation. the three, depending on the implementation.
Stream Context Memory includes state associated with Data Stream Context Memory includes state associated with Data
Buffers. For Tagged Buffers, this includes how STag names, Data Buffers. For Tagged Buffers, this includes how STag names, Data
Buffers, and Page Translation Tables inter-relate. It also Buffers, and Page Translation Tables (see section 4.2.3 on page
includes the list of Untagged Data Buffers posted for reception 13) interrelate. It also includes the list of Untagged Data
of Untagged Messages (commonly called the Receive Queue), and a Buffers posted for reception of Untagged Messages (commonly
list of operations to perform to send data (commonly called the called the Receive Queue), and a list of operations to perform to
Send Queue). send data (commonly called the Send Queue).
4.2.2 Data Buffers 4.2.2 Data Buffers
There are two different ways to expose a data buffer; a buffer There are two different ways to expose a local ULP's data buffer;
can be exposed for receiving RDMAP Send Type Messages (a.k.a. DDP a buffer can be exposed for receiving RDMAP Send Type Messages
Untagged Messages) on DDP Queue zero or the buffer can be exposed (a.k.a. DDP Untagged Messages) on DDP Queue zero or the buffer
for remote access through STags (a.k.a. DDP Tagged Messages). can be exposed for remote access through STags (a.k.a. DDP Tagged
This distinction is important because the attacks and the Messages). This distinction is important because the attacks and
countermeasures used to protect against the attack are different the countermeasures used to protect against the attack are
depending on the method for exposing the buffer to the network. different depending on the method for exposing the buffer to the
network.
For the purposes of the security discussion, a single logical For the purposes of the security discussion, a single logical
Data Buffer is exposed with a single STag. Actual implementations Data Buffer is exposed with a single Stag on a given Stream.
may support scatter/gather capabilities to enable multiple Actual implementations may support scatter/gather capabilities to
physical data buffers to be accessed with a single STag, but from enable multiple physical data buffers to be accessed with a
a threat analysis perspective it is assumed that a single STag single STag, but from a threat analysis perspective it is assumed
enables access to a single logical Data Buffer. that a single STag enables access to a single logical Data
Buffer.
In any event, it is the responsibility of the RNIC to ensure that In any event, it is the responsibility of the Privileged Resource
no STag can be created that exposes memory that the consumer had Manager to ensure that no STag can be created that exposes memory
no authority to expose. that the consumer had no authority to expose.
4.2.3 Page Translation Tables 4.2.3 Page Translation Tables
Page Translation Tables are the structures used by the RNIC to be Page Translation Tables are the structures used by the RNIC to be
able to access ULP memory for data transfer operations. Even able to access ULP memory for data transfer operations. Even
though these structures are called "Page" Translation Tables, though these structures are called "Page" Translation Tables,
they may not reference a page at all - conceptually they are used they may not reference a page at all - conceptually they are used
to map a ULP address space representation of a buffer to the to map a ULP address space representation (e.g. a virtual
physical addresses that are used by the RNIC Engine to move data. address) of a buffer to the physical addresses that are used by
If on a specific system a mapping is not used, then a subset of the RNIC Engine to move data. If on a specific system a mapping
the attacks examined may be appropriate. Note that the Page is not used, then a subset of the attacks examined may be
Translation Table may or may not be a shared resource. appropriate. Note that the Page Translation Table may or may not
be a shared resource.
4.2.4 STag Namespace 4.2.4 STag Namespace
The DDP specification defines a 32-bit namespace for the STag. The DDP specification defines a 32-bit namespace for the STag.
Implementations may vary in terms of the actual number of STags Implementations may vary in terms of the actual number of STags
that are supported. In any case, this is a bounded resource that that are supported. In any case, this is a bounded resource that
can come under attack. Depending upon STag namespace allocation can come under attack. Depending upon STag namespace allocation
algorithms, the actual name space to attack may be significantly algorithms, the actual name space to attack may be significantly
less than 2^32. less than 2^32.
4.2.5 Completion Queues 4.2.5 Completion Queues
Completion Queues are used in this specification to conceptually Completion Queues are used in this document to conceptually
represent how the RNIC Engine notifies the ULP about the represent how the RNIC Engine notifies the ULP about the
completion of the transmission of data, or the completion of the completion of the transmission of data, or the completion of the
reception of data through the Data Transfer Interface. Because reception of data through the Data Transfer Interface. Because
there could be many transmissions or receptions in flight at any there could be many transmissions or receptions in flight at any
one time, completions are modeled as a queue rather than a single one time, completions are modeled as a queue rather than a single
event. An implementation may also use the Completion Queue to event. An implementation may also use the Completion Queue to
notify the ULP of other activities, for example, the completion notify the ULP of other activities, for example, the completion
of a mapping of an STag to a specific ULP buffer. Completion of a mapping of an STag to a specific ULP buffer. Completion
Queues may be shared by a group of Streams, or may be designated Queues may be shared by a group of Streams, or may be designated
to handle a specific Stream's traffic. to handle a specific Stream's traffic.
skipping to change at page 13, line 15 skipping to change at page 14, line 18
4.2.6 Asynchronous Event Queue 4.2.6 Asynchronous Event Queue
The Asynchronous Event Queue is a queue from the RNIC to the The Asynchronous Event Queue is a queue from the RNIC to the
Privileged Resource Manager of bounded size. It is used by the Privileged Resource Manager of bounded size. It is used by the
RNIC to notify the host of various events which might require RNIC to notify the host of various events which might require
management action, including protocol violations, Stream state management action, including protocol violations, Stream state
changes, local operation errors, low water marks on receive changes, local operation errors, low water marks on receive
queues, and possibly other events. queues, and possibly other events.
The Asynchronous Event Queue is a resource that can be attacked The Asynchronous Event Queue is a resource that can be attacked
because Remote or Local Peers can cause events to occur which because Remote or Local Peers and/or ULPs can cause events to
have the potential of overflowing the queue. occur which have the potential of overflowing the queue.
Note that an implementation is at liberty to implement the Note that an implementation is at liberty to implement the
functions of the Asynchronous Event Queue in a variety of ways, functions of the Asynchronous Event Queue in a variety of ways,
including multiple queues or even simple callbacks. All including multiple queues or even simple callbacks. All
vulnerabilities identified are intended to apply regardless of vulnerabilities identified are intended to apply regardless of
the implementation of the Asynchronous Event Queue. For example, the implementation of the Asynchronous Event Queue. For example,
a callback function is simply a very short queue. a callback function may be viewed as simply a very short queue.
4.2.7 RDMA Read Request Queue 4.2.7 RDMA Read Request Queue
The RDMA Read Request Queue is the memory that holds state The RDMA Read Request Queue is the memory that holds state
information for one or more RDMA Read Request Messages that have information for one or more RDMA Read Request Messages that have
arrived, but for which the RDMA Read Response Messages have not arrived, but for which the RDMA Read Response Messages have not
yet been completely sent. Because potentially more than one RDMA yet been completely sent. Because potentially more than one RDMA
Read Request can be outstanding at one time, the memory is Read Request can be outstanding at one time, the memory is
modeled as a queue of bounded size. modeled as a queue of bounded size. Some implementations may
enable sharing of a single RDMA Read Request Queue across
multiple Streams.
4.2.8 RNIC Interactions 4.2.8 RNIC Interactions
With RNIC resources and interfaces defined, it is now possible to With RNIC resources and interfaces defined, it is now possible to
examine the interactions supported by the generic RNIC functional examine the interactions supported by the generic RNIC functional
interfaces through each of the 3 interfaces - Privileged Control interfaces through each of the 3 interfaces - Privileged Control
Interface, Privileged Data Interface, and Non-Privileged Data Interface, Privileged Data Interface, and Non-Privileged Data
Interface. Interface.
4.2.8.1 Privileged Control Interface Semantics 4.2.8.1 Privileged Control Interface Semantics
Generically, the Privileged Control Interface controls the RNICs Generically, the Privileged Control Interface controls the RNICÆs
allocation, deallocation, and initialization of RNIC global allocation, deallocation, and initialization of RNIC global
resources. This includes allocation and deallocation of Stream resources. This includes allocation and deallocation of Stream
Context Memory, Page Translation Tables, STag names, Completion Context Memory, Page Translation Tables, STag names, Completion
Queues, RDMA Read Request Queues, and Asynchronous Event Queues. Queues, RDMA Read Request Queues, and Asynchronous Event Queues.
The Privileged Control Interface is also typically used for The Privileged Control Interface is also typically used for
managing Non-Privileged ULP resources for the Non-Privileged ULP managing Non-Privileged ULP resources for the Non-Privileged ULP
(and possibly for the Privileged ULP as well). This includes (and possibly for the Privileged ULP as well). This includes
initialization and removal of Page Translation Table resources, initialization and removal of Page Translation Table resources,
and managing RNIC events (possibly managing all events for the and managing RNIC events (possibly managing all events for the
skipping to change at page 14, line 15 skipping to change at page 15, line 19
4.2.8.2 Non-Privileged Data Interface Semantics 4.2.8.2 Non-Privileged Data Interface Semantics
The Non-Privileged Data Interface enables data transfer (transmit The Non-Privileged Data Interface enables data transfer (transmit
and receive) but does not allow initialization of the Page and receive) but does not allow initialization of the Page
Translation Table resources. However, once the Page Translation Translation Table resources. However, once the Page Translation
Table resources have been initialized, the interface may enable a Table resources have been initialized, the interface may enable a
specific STag mapping to be enabled and disabled by directly specific STag mapping to be enabled and disabled by directly
communicating with the RNIC, or create an STag mapping for a communicating with the RNIC, or create an STag mapping for a
buffer that has been previously initialized in the RNIC. buffer that has been previously initialized in the RNIC.
For RDMAP, transmitting data means sending RDMAP Send Type For RDMAP, ULP data can be sent by using RDMAP Send Type
Messages, RDMA Read Requests, and RDMA Writes. For data Messages, RDMA Read Responses, and RDMA Writes. ULP data
reception, for RDMAP it can receive Send Type Messages into reception through RDMAP can be done by receiving Send Type
buffers that have been posted on the Receive Queue or Shared Messages into buffers that have been posted on the Receive Queue
Receive Queue. It can also receive RDMA Write and RDMA Read or Shared Receive Queue. It can also be done by receiving RDMA
Response Messages into buffers that have previously been exposed Write and RDMA Read Response Messages into buffers that have
for external write access through advertisement of an STag. previously been exposed for external write access through
advertisement of an STag. Additionally, to cause ULP data to be
pulled (read) across the network, RDMAP uses an RDMA Read Request
Message (which only contains RDMAP control information necessary
to access the ULP buffer to be read), to cause an RDMA Read
Response Message to be generated that contains the ULP data.
For DDP, transmitting data means sending DDP Tagged or Untagged For DDP, transmitting data means sending DDP Tagged or Untagged
Messages. For data reception, for DDP it can receive Untagged Messages. For data reception, for DDP it can receive Untagged
Messages into buffers that have been posted on the Receive Queue Messages into buffers that have been posted on the Receive Queue
or Shared Receive Queue. It can also receive Tagged DDP Messages or Shared Receive Queue. It can also receive Tagged DDP Messages
into buffers that have previously been exposed for external write into buffers that have previously been exposed for external write
access through advertisement of an STag. access through advertisement of an STag.
Completion of data transmission or reception generally entails Completion of data transmission or reception generally entails
informing the ULP of the completed work by placing completion informing the ULP of the completed work by placing completion
skipping to change at page 14, line 46 skipping to change at page 15, line 55
The Privileged Data Interface semantics are a superset of the The Privileged Data Interface semantics are a superset of the
Non-Privileged Data Transfer semantics. The interface can do Non-Privileged Data Transfer semantics. The interface can do
everything defined in the prior section, as well as everything defined in the prior section, as well as
create/destroy buffer to STag mappings directly. This generally create/destroy buffer to STag mappings directly. This generally
entails initialization or clearing of Page Translation Table entails initialization or clearing of Page Translation Table
state in the RNIC. state in the RNIC.
4.2.9 Initialization of RNIC Data Structures for Data Transfer 4.2.9 Initialization of RNIC Data Structures for Data Transfer
Initialization of the mapping between an STag and a Data Buffer Initialization of the mapping between an STag and a Data Buffer
can be viewed in the abstract as two separate opertions: can be viewed in the abstract as two separate operations:
a. Initialization of the allocated Page Translation Table a. Initialization of the allocated Page Translation Table
entries with the location of the Data Buffer, and entries with the location of the Data Buffer, and
b. Initialization of a mapping from an allocated STag name b. Initialization of a mapping from an allocated STag name
to a set of Page Translation Table entry(s) or partial- to a set of Page Translation Table entry(s) or partial-
entries. entries.
Note that an implementation may not have a Page Translation Table Note that an implementation may not have a Page Translation Table
(i.e. it may support a direct mapping between an STag and a Data (i.e. it may support a direct mapping between an STag and a Data
skipping to change at page 15, line 17 skipping to change at page 16, line 27
Initialization of the contents of the Page Translation Table can Initialization of the contents of the Page Translation Table can
be done by either the Privileged ULP or by the Privileged be done by either the Privileged ULP or by the Privileged
Resource Manager as a proxy for the Non-Privileged ULP. By Resource Manager as a proxy for the Non-Privileged ULP. By
definition the Non-Privileged ULP is not trusted to directly definition the Non-Privileged ULP is not trusted to directly
manipulate the Page Translation Table. In general the concern is manipulate the Page Translation Table. In general the concern is
that the Non-Privileged ULP may try to maliciously initialize the that the Non-Privileged ULP may try to maliciously initialize the
Page Translation Table to access a buffer for which it does not Page Translation Table to access a buffer for which it does not
have permission. have permission.
The exact resource allocation algorithm for the Page Translation The exact resource allocation algorithm for the Page Translation
Table is outside the scope of this specification. It may be Table is outside the scope of this document. It may be allocated
allocated for a specific Data Buffer, or be allocated as a pooled for a specific Data Buffer, or be allocated as a pooled resource
resource to be consumed by potentially multiple Data Buffers, or to be consumed by potentially multiple Data Buffers, or be
be managed in some other way. This paper attempts to abstract managed in some other way. This document attempts to abstract
implementation dependent issues, and focus on higher level implementation dependent issues, and group them into higher level
security issues such as resource starvation and sharing of security issues such as resource starvation and sharing of
resources between Streams. resources between Streams.
The next issue is how an STag name is associated with a Data The next issue is how an STag name is associated with a Data
Buffer. For the case of an Untagged Data Buffer, there is no wire Buffer. For the case of an Untagged Data Buffer, there is no wire
visible mapping between an STag and the Data Buffer. Note that visible mapping between an STag and the Data Buffer. Note that
there may, in fact, be an STag which represents the buffer. there may, in fact, be an STag which represents the buffer, if an
However, because the STag by definition is not visible on the implementation chooses to internally represent Untagged Data
wire, this is a local host specific issue which should be Buffer using STags. However, because the STag by definition is
analyzed in the context of local host implementation specific not visible on the wire, this is a local host implementation
security analysis, and thus is outside the scope of this paper. specific issue which should be analyzed in the context of a local
host implementation specific security analysis, and thus is
outside the scope of this document.
For a Tagged Data Buffer, either the Privileged ULP, the Non- For a Tagged Data Buffer, either the Privileged ULP, the Non-
Privileged ULP, or the Privileged Resource Manager acting on Privileged ULP, or the Privileged Resource Manager acting on
behalf of the Non-Privileged Resource Manager may initialize a behalf of the Non-Privileged ULP may initialize a mapping from an
mapping from an STag to a Page Translation Table, or may have the STag to a Page Translation Table, or may have the ability to
ability to simply enable/disable an existing STag to Page simply enable/disable an existing STag to Page Translation Table
Translation Table mapping. There may also be multiple STag names mapping. There may also be multiple STag names which map to a
which map to a specific group of Page Translation Table entries specific group of Page Translation Table entries (or sub-
(or sub-entries). Specific security issues with this level of entries). Specific security issues with this level of flexibility
flexibility are examined in Section 7.3.3 Multiple STags to are examined in Section 7.3.3 Multiple STags to access the same
access the same buffer on page 25. buffer on page 27.
There are a variety of implementation options for initialization There are a variety of implementation options for initialization
of Page Translation Table entries and mapping an STag to a group of Page Translation Table entries and mapping an STag to a group
of Page Translation Table entries which have security of Page Translation Table entries which have security
repercussions. This includes support for separation of Mapping an repercussions. This includes support for separation of Mapping an
STag verses mapping a set of Page Translation Table entries, and STag versus mapping a set of Page Translation Table entries, and
support for ULPs directly manipulating STag to Page Translation support for ULPs directly manipulating STag to Page Translation
Table entry mappings (verses requiring access through the Table entry mappings (versus requiring access through the
Privileged Resource Manager). Privileged Resource Manager).
4.2.10 RNIC Data Transfer Interactions 4.2.10 RNIC Data Transfer Interactions
RNIC Data Transfer operations can be subdivided into send RNIC Data Transfer operations can be subdivided into send
operations and receive operations. operations and receive operations.
For send operations, there is typically a queue that enables the For send operations, there is typically a queue that enables the
ULP to post multiple operations to send data (referred to as the ULP to post multiple operation requests to send data (referred to
Send Queue). Depending upon the implementation, Data Buffers used as the Send Queue). Depending upon the implementation, Data
in the operations may or may not have Page Translation Table Buffers used in the operations may or may not have Page
entries associated with them, and may or may not have STags Translation Table entries associated with them, and may or may
associated with them. Because this is a local host specific not have STags associated with them. Because this is a local host
implementation issue rather than a protocol issue, the security specific implementation issue rather than a protocol issue, the
analysis of threats and mitigations is left to the host security analysis of threats and mitigations is left to the host
implementation. implementation.
Receive operations are different for Tagged Data Buffers verses Receive operations are different for Tagged Data Buffers versus
Untagged Data Buffers. If more than one Untagged Data Buffer can Untagged Data Buffers. If more than one Untagged Data Buffer can
be posted by the ULP, the DDP specification requires that they be be posted by the ULP, the DDP specification requires that they be
consumed in sequential order. Thus the most general consumed in sequential order. Thus the most general
implementation is that there is a sequential queue of receive implementation is that there is a sequential queue of receive
Untagged Data Buffers (Receive Queue). Some implementations may Untagged Data Buffers (Receive Queue). Some implementations may
also support sharing of the sequential queue between multiple also support sharing of the sequential queue between multiple
Streams. In this case defining "sequential" becomes non-trivial - Streams. In this case defining "sequential" becomes non-trivial -
in general the buffers for a single stream are consumed from the in general the buffers for a single Stream are consumed from the
queue in the order that they were placed on the queue, but there queue in the order that they were placed on the queue, but there
is no order guarantee between streams. is no consumption order guarantee between Streams.
For receive Tagged Data Buffers, at some time prior to data For receive Tagged Data Buffers, at some time prior to data
transfer, the mapping of the STag to specific Page Translation transfer, the mapping of the STag to specific Page Translation
Table entries (if present) and the mapping from the Page Table entries (if present) and the mapping from the Page
Translation Table entries to the Data Buffer must have been Translation Table entries to the Data Buffer must have been
initialized (see the prior section for interaction details). initialized (see section 4.2.9 for interaction details).
5 Trust and Resource Sharing 5 Trust and Resource Sharing
It is assumed that in general the Local and Remote Peer are It is assumed that in general the Local and Remote Peer are
untrusted, and thus attacks by either should have mitigations in untrusted, and thus attacks by either should have mitigations in
place. place.
A separate, but related issue is resource sharing between A separate, but related issue is resource sharing between
multiple streams. If local resources are not shared, the multiple Streams. If local resources are not shared, the
resources are dedicated on a per Stream basis. Resources are resources are dedicated on a per Stream basis. Resources are
defined in Section 4.2 Resources on page 10. The advantage of not defined in Section 4.2 Resources on page 12. The advantage of not
sharing resources between Streams is that it reduces the types of sharing resources between Streams is that it reduces the types of
attacks that are possible. The disadvantage is that ULPs might attacks that are possible. The disadvantage of not sharing
run out of resources. resources is that ULPs might run out of resources. Thus there can
be a strong incentive for sharing resources, if the security
issues associated with the sharing of resources can be mitigated.
It is assumed in this paper that the component that implements It is assumed in this document that the component that implements
the mechanism to control sharing of the RNIC Engine resources is the mechanism to control sharing of the RNIC Engine resources is
the Privileged Resource Manager. The RNIC Engine exposes its the Privileged Resource Manager. The RNIC Engine exposes its
resources through the RNIC Interface to the Privileged Resource resources through the RNIC Interface to the Privileged Resource
Manager. All Privileged and Non-Privileged ULPs request resources Manager. All Privileged and Non-Privileged ULPs request resources
from the Resource Manager (note that by definition both the Non- from the Resource Manager (note that by definition both the Non-
Privileged and the Privileged application might try to greedily Privileged and the Privileged application might try to greedily
consume resources, thus creating a potential Denial of Service consume resources, thus creating a potential Denial of Service
(DOS) attack. The Resource Manager implements resource management (DOS) attack). The Resource Manager implements resource
policies to ensure fair access to resources. The Resource Manager management policies to ensure fair access to resources. The
should be designed to take into account security attacks detailed Resource Manager should be designed to take into account security
in this specification. Note that for some systems the Privileged attacks detailed in this document. Note that for some systems the
Resource Manager may be implemented within the Privileged ULP. Privileged Resource Manager may be implemented within the
Privileged ULP.
All Non-Privileged ULP interactions with the RNIC Engine that All Non-Privileged ULP interactions with the RNIC Engine that
could affect other ULPs MUST be done using the Privileged could affect other ULPs MUST be done using the Privileged
Resource Manager as a proxy. All ULP resource allocation requests Resource Manager as a proxy. All ULP resource allocation requests
for scarce resources MUST also be done using a Privileged for scarce resources MUST also be done using a Privileged
Resource Manager. Resource Manager.
The sharing of resources across Streams should be under the The sharing of resources across Streams should be under the
control of the ULP, both in terms of the trust model the ULP control of the ULP, both in terms of the trust model the ULP
wishes to operate under, as well as the level of resource sharing wishes to operate under, as well as the level of resource sharing
the ULP wishes to give Local Peer processes. For more discussion the ULP wishes to give local processes. For more discussion on
on types of trust models which combine partial trust and sharing types of trust models which combine partial trust and sharing of
of resources, see Appendix C: Partial Trust Taxonomy on page 50. resources, see Appendix C: Partial Trust Taxonomy on page 52.
The Privileged Resource Manager MUST NOT assume different ULPs The Privileged Resource Manager MUST NOT assume different Streams
share Partial Mutual Trust unless there is a mechanism to ensure share Partial Mutual Trust unless there is a mechanism to ensure
that the ULPs do indeed share partial mutual trust. This can be that the Streams do indeed share Partial Mutual Trust. This can
done in several ways, including explicit notification from the be done in several ways, including explicit notification from the
ULP. ULP that owns the Streams.
6 Attacker Capabilities 6 Attacker Capabilities
An attackers capabilities delimit the types of attacks that An attackerÆs capabilities delimit the types of attacks that
attacker is able to launch. RDMAP and DDP require that the attacker is able to launch. RDMAP and DDP require that the
initial LLP Stream (and connection) be set up prior to initial LLP Stream (and connection) be set up prior to
transferring RDMAP/DDP Messages. Attackers with send only transferring RDMAP/DDP Messages. This requires at least one
capabilities must first guess the current LLP Stream parameters round-trip handshake to occur.
before they can attack RNIC resources (e.g. TCP sequence number).
Attackers with both send and receive capabilities have presumably If the attacker is not the Remote Peer that created the initial
setup a valid LLP Stream, and thus have a wider ability to attack connection, then the attacker's capabilities can be segmented
RNIC resources. into send only capabilities or send and receive capabilities.
Attacking with send only capabilities requires the attacker to
first guess the current LLP Stream parameters before they can
attack RNIC resources (e.g. TCP sequence number). If this class
of attacker also has receive capabilities, they are typically
referred to as a "man-in-the-middle" attacker, and they have a
much wider ability to attack RNIC resources. The breadth of
attack is essentially the same as that of an attacking Remote
Peer (i.e. the Remote Peer that setup the initial LLP Stream).
7 Attacks and Countermeasures 7 Attacks and Countermeasures
This section describes the attacks that are possible against the This section describes the attacks that are possible against the
RDMA system defined in Figure 1 - RDMA Security Model and the RDMA system defined in Figure 1 - RDMA Security Model and the
RNIC Engine resources defined in Section 4.2. The analysis RNIC Engine resources defined in Section 4.2. The analysis
includes a detailed description of each attack, what is being includes a detailed description of each attack, what is being
attacked, and a description of the countermeasures that can be attacked, and a description of the countermeasures that can be
taken to thwart the attack. taken to thwart the attack.
Note that connection setup and teardown is presumed to be done in Note that connection setup and teardown is presumed to be done in
stream mode (i.e. no RDMA encapsulation of the payload), so there stream mode (i.e. no RDMA encapsulation of the payload), so there
are no new attacks related to connection setup/teardown beyond are no new attacks related to connection setup/teardown beyond
what is already present in the LLP (e.g. TCP or SCTP). Note, what is already present in the LLP (e.g. TCP or SCTP). Note,
however, that RDMAP/DDP parameters may be exchanged in stream however, that RDMAP/DDP parameters may be exchanged in stream
mode, and if they are corrupted by an attacker unintended mode, and if they are corrupted by an attacker unintended
consequences will result. Therefore, any existing mitigations for consequences will result. Therefore, any existing mitigations for
LLP Spoofing, Tampering, Repudiation, Information Disclosure, LLP Spoofing, Tampering, Repudiation, Information Disclosure,
Denial of Service, or Elevation of Privilege continues to apply Denial of Service, or Elevation of Privilege continue to apply
(and is out of scope of this document). Thus the analysis in this (and are out of scope of this document). Thus the analysis in
section focuses on attacks that are present regardless of the LLP this section focuses on attacks that are present regardless of
Stream type. the LLP Stream type.
The attacks are classified into five categories: Spoofing, The attacks are classified into five categories: Spoofing,
Tampering, Information Disclosure, Denail of Service (DoS) Tampering, Information Disclosure, Denial of Service (DoS)
attacks, and Elevation of Privileges. Tampering is any attacks, and Elevation of Privileges. Tampering is any
modification of the legitimate traffic (machine internal or modification of the legitimate traffic (machine internal or
network). Spoofing attack is a special case of tempering; where network). Spoofing attack is a special case of tampering where
the attacker falsifies an identity of the Remote Peer (identity the attacker falsifies an identity of the Remote Peer (identity
can be an IP address, machine name, ULP level identity etc.). can be an IP address, machine name, ULP level identity etc.).
7.1 Tools for Countermeasures 7.1 Tools for Countermeasures
The tools described in this section are the primary mechanisms The tools described in this section are the primary mechanisms
that can be used to provide countermeasures to potential attacks. that can be used to provide countermeasures to potential attacks.
7.1.1 Protection Domain (PD) 7.1.1 Protection Domain (PD)
Protection Domains are associated with two of the resources of A Protection Domain (PD) is a local construct to the RDMA
concern, Stream Context Memory and STags associated with Page implementation, and never visible over the wire. Protection
Translation Table entries and data buffers. Protection Domains Domains are assigned to two of the resources of concern, Stream
are used mainly to ensure that an STag can only be used to access Context Memory and STags associated with Page Translation Table
the associated data buffer through Streams in the same Protection entries and data buffers. A correct implementation of a
Domain as that STag. Protection Domain requires that resources which belong to a given
Protection Domain can not be used on a resource belonging to
another Protection Domain, because Protection Domain membership
is checked by the RNIC prior to taking any action involving such
a resource. Protection Domains are therefore used to ensure that
an STag can only be used to access an associated data buffer on
one or more Streams that are associated with the same Protection
Domain as the specific STag.
If an implementation chooses to not share resources between If an implementation chooses to not share resources between
Streams, it is recommended that each Stream be associated with Streams, it is recommended that each Stream be associated with
its own, unique Protection Domain. If an implementation chooses its own, unique Protection Domain. If an implementation chooses
to allow resource sharing, it is recommended that Protection to allow resource sharing, it is recommended that Protection
Domain be limited to the number of Streams that have Partial Domain be limited to the collection of Streams that have Partial
Mutual Trust. Mutual Trust with each other.
Note that a ULP (either Privileged or Non-Privileged) can Note that a ULP (either Privileged or Non-Privileged) can
potentially have multiple Protection Domains. This could be used, potentially have multiple Protection Domains. This could be used,
for example, to ensure that multiple clients of a server do not for example, to ensure that multiple clients of a server do not
have the ability to corrupt each other. The server would allocate have the ability to corrupt each other. The server would allocate
a Protection Domain per client to ensure that resources covered a Protection Domain per client to ensure that resources covered
by the Protection Domain could not be used by another (untrusted) by the Protection Domain could not be used by another (untrusted)
client. client.
7.1.2 Limiting STag Scope 7.1.2 Limiting STag Scope
skipping to change at page 20, line 40 skipping to change at page 21, line 47
any Stream within a specific Protection Domain, and any Stream within a specific Protection Domain, and
is invalid if used on any Stream that is not a member is invalid if used on any Stream that is not a member
of the Protection Domain. of the Protection Domain.
* Single Stream scope. The STag is valid on a single * Single Stream scope. The STag is valid on a single
Stream, regardless of what the Stream association is Stream, regardless of what the Stream association is
to a Protection Domain. If used on any other Stream, to a Protection Domain. If used on any other Stream,
it is invalid. it is invalid.
* Limit the time an STag is valid. By Invalidating an * Limit the time an STag is valid. By Invalidating an
Advertised STag (e.g., revoking remote access to the advertised STag (e.g., revoking remote access to the
buffers described by an STag when done with the buffers described by an STag when done with the
transfer), an entire class of attacks can be eliminated. transfer), an entire class of attacks can be eliminated.
* Limit the buffer the STag can reference. Limiting the * Limit the buffer the STag can reference. Limiting the
scope of an STag access to *just* the intended ULP scope of an STag access to just the intended portion of
buffers to be exposed is critical to prevent certain the ULP buffers to be exposed is critical to prevent
forms of attacks. certain forms of attacks.
* Allocating and/or advertising STag numbers in an * Allocating and/or advertising STag numbers in an
unpredictable way. If STags are allocated/advertised unpredictable way. If STags are allocated/advertised
using an algorithm which makes it hard for the attacker using an algorithm which makes it hard for the attacker
to guess which STag(s) are currently in use, it makes it to guess which STag(s) are currently in use, it makes it
more difficult for an attacker to guess the correct more difficult for an attacker to guess the correct
value. As stated in the RDMAP specification [RDMAP], an value. As stated in the RDMAP specification [RDMAP], an
invalid STag will cause the RDMAP Stream to be invalid STag will cause the RDMAP Stream to be
terminated. For the case of [DDP], at a minimum it must terminated. For the case of [DDP], at a minimum it must
signal an error to the ULP, and commonly this will cause signal an error to the ULP. This permits the ULP to
the DDP stream to be terminated. detect such attempts, and take countermeasures. Commonly,
the ULP will cause the DDP Stream to be immediately
terminated.
7.1.3 Access Rights 7.1.3 Access Rights
Access Rights associated with a specific Advertised STag or Access Rights associated with a specific advertised STag or
RDMAP/DDP Stream provide another mechanism for ULPs to limit the RDMAP/DDP Stream provide another mechanism for ULPs to limit the
attack capabilities of the Remote Peer. The Local Peer can attack capabilities of the Remote Peer. The local ULP can control
control whether a data buffer is exposed for local only, or local whether a data buffer is exposed for local only, or local and
and remote access, and assign specific access privileges (read, remote access, and assign specific access privileges (read,
write, read and write) on a per stream basis. write, read and write) on a per Stream basis.
For DDP, when an STag is advertised, the Remote Peer is For DDP, when an STag is advertised, the Remote Peer is
presumably given write access rights to the data (otherwise there presumably given write access rights to the data (otherwise there
was not much point to the advertisement). For RDMAP, when a ULP was not much point to the advertisement). For RDMAP, when a ULP
advertises an STag, it can enable write-only, read-only, or both advertises an STag, it can enable write-only, read-only, or both
write and read access rights. write and read access rights.
Similarly, some ULPs may wish to provide a single buffer with Similarly, some ULPs may wish to provide a single buffer with
different access rights on a per-Stream or per-Stream basis. For different access rights on a per-Stream basis. For example, some
example, some Streams may have read-only access, some may have Streams may have read-only access, some may have remote read and
remote read and write access, while on other Streams only the write access, while on other Streams only the local ULP/Local
Local Peer is allowed access. Peer is allowed access.
7.1.4 Limiting the Scope of the Completion Queue 7.1.4 Limiting the Scope of the Completion Queue
Completions associated with sending and receiving data, or Completions associated with sending and receiving data, or
setting up buffers for sending and receiving data, could be setting up buffers for sending and receiving data, could be
accumulated in a shared Completion Queue for a group of RDMAP/DDP accumulated in a shared Completion Queue for a group of RDMAP/DDP
Streams, or a specific RDMAP/DDP Stream could have a dedicated Streams, or a specific RDMAP/DDP Stream could have a dedicated
Completion Queue. Limiting Completion Queue association to one, Completion Queue. Limiting Completion Queue association to one,
or a small number of RDMAP/DDP Streams can prevent several forms or a small number of RDMAP/DDP Streams can prevent several forms
of Denial of Service attacks. of Denial of Service attacks, by sharply limiting the scope of
the attackÆs effect.
7.1.5 Limiting the Scope of an Error 7.1.5 Limiting the Scope of an Error
To prevent a variety of attacks, it is important that an To prevent a variety of attacks, it is important that an
RDMAP/DDP implementation be robust in the face of errors. If an RDMAP/DDP implementation be robust in the face of errors. If an
error on a specific Stream can cause other unrelated Streams to error on a specific Stream can cause other unrelated Streams to
fail, then a broad class of attacks are enabled against the fail, then a broad class of attacks are enabled against the
implementation. implementation.
For example, an error on a specific RDMAP stream should not cause For example, an error on a specific RDMAP Stream should not cause
the RNIC to stop processing incoming packets, or corrupt a the RNIC to stop processing incoming packets, or corrupt a
receive queue for an unrelated stream. receive queue for an unrelated Stream.
7.2 Spoofing 7.2 Spoofing
Spoofing attacks can be launched by the Remote Peer, or by a Spoofing attacks can be launched by the Remote Peer, or by a
network based attacker. A network based spoofing attack applies network based attacker. A network based spoofing attack applies
to all Remote Peers. to all Remote Peers.
Because the RDMAP Stream requires an LLP Stream in the Because the RDMAP Stream requires an LLP Stream to be fully
ESTABLISHED state, certain types of traditional forms of wire initialized (e.g. for [TCP] it is in the ESTABLISHED state),
attacks do not apply -- an end-to-end handshake must have certain types of traditional forms of wire attacks do not apply -
occurred to establish the RDMAP Stream. So, the only form of - an end-to-end handshake must have occurred to establish the
spoofing that applies is one when a remote node can both send and RDMAP Stream. So, the only form of spoofing that applies is one
receive packets. Yet even with this limitation the Stream is when an attacker can both send and receive packets. Yet even with
still exposed to the following spoofing attacks. this limitation the Stream is still exposed to the following
spoofing attacks.
7.2.1 Impersonation 7.2.1 Impersonation
A network based attacker can impersonate a legal RDMA/DDP peer A network based attacker can impersonate a legal RDMAP/DDP Peer
(by spoofing a legal IP address), and establish an RDMA/DDP (by spoofing a legal IP address), and establish an RDMAP/DDP
Stream with the victim. End to end authentication (i.e. IPsec, Stream with the victim. End-to-end authentication (i.e. IPsec,
SSL or ULP authentication) provides protection against this SSL or ULP authentication) provides protection against this
attack. For additional information see Section 8, Security attack. For additional information see Section 8, Security
Services for RDMA and DDP, on page 38. Services for RDMAP and DDP, on page 40.
7.2.2 Stream Hijacking 7.2.2 Stream Hijacking
Stream hijacking happens when a network based attacker follows Stream hijacking happens when a network based attacker eavesdrops
the Stream establishment phase, and waits until the the LLP connection through the Stream establishment phase, and
authentication phase (if such a phase exists) is completed waits until the authentication phase (if such a phase exists) is
successfully. He can then spoof the IP address and re-direct the completed successfully. The attacker then spoofs the IP address
Stream from the victim to its own machine. For example, an and re-direct the Stream from the victim to its own machine. For
attacker can wait until an iSCSI authentication is completed example, an attacker can wait until an iSCSI authentication is
successfully, and hijack the iSCSI Stream. completed successfully, and then hijack the iSCSI Stream.
The best protection against this form of attack is end-to-end The best protection against this form of attack is end-to-end
integrity protection and authentication, such as IPsec (see integrity protection and authentication, such as IPsec (see
Section 8, Security Services for RDMA and DDP, on page 38), to Section 8, Security Services for RDMAP and DDP, on page 40), to
prevent spoofing. Another option is to provide physical security. prevent spoofing. Another option is to provide physical security.
Discussion of physical security is out of scope for this Discussion of physical security is out of scope for this
document. document.
Because the connection and/or Stream itself is established by the Because the connection and/or Stream itself is established by the
LLP, some LLPs are more difficult to hijack than others. Please LLP, some LLPs are more difficult to hijack than others. Please
see the relevant LLP documentation on security issues around see the relevant LLP documentation on security issues around
connection and/or Stream hijacking. connection and/or Stream hijacking.
7.2.3 Man in the Middle Attack 7.2.3 Man in the Middle Attack
skipping to change at page 22, line 56 skipping to change at page 24, line 17
If a network based attacker has the ability to delete, inject If a network based attacker has the ability to delete, inject
replay, or modify packets which will still be accepted by the LLP replay, or modify packets which will still be accepted by the LLP
(e.g., TCP sequence number is correct) then the Stream can be (e.g., TCP sequence number is correct) then the Stream can be
exposed to a man in the middle attack. One style of attack is for exposed to a man in the middle attack. One style of attack is for
the man-in-the-middle to send Tagged Messages (either RDMAP or the man-in-the-middle to send Tagged Messages (either RDMAP or
DDP). If it can discover a buffer that has been exposed for STag DDP). If it can discover a buffer that has been exposed for STag
enabled access, then the man-in-the-middle can use an RDMA Read enabled access, then the man-in-the-middle can use an RDMA Read
operation to read the contents of the associated data buffer, operation to read the contents of the associated data buffer,
perform an RDMA Write Operation to modify the contents of the perform an RDMA Write Operation to modify the contents of the
associated data buffer, or invalidate the STag to disable further associated data buffer, or invalidate the STag to disable further
access to the buffer. The only countermeasure for this form of access to the buffer.
attack is to either secure the RDMAP/DDP Stream (i.e. integrity
protect) or attempt to provide physical security to prevent man-
in-the-middle type attacks.
The best protection against this form of attack is end-to-end The best protection against this form of attack is end-to-end
integrity protection and authentication, such as IPsec (see integrity protection and authentication, such as IPsec (see
Section 8 Security Services for RDMA and DDP on page 38), to Section 8 Security Services for RDMAP and DDP on page 40), to
prevent spoofing or tampering. If Stream or session level prevent spoofing or tampering. If Stream or session level
authentication and integrity protection are not used, then a man- authentication and integrity protection are not used, then
in-the-middle attack can occur, enabling spoofing and tampering. physical protection must be employed, lest a man-in-the-middle
attack occur, enabling spoofing and tampering.
Because the connection/Stream itself is established by the LLP, Because the connection/Stream itself is established by the LLP,
some LLPs are more exposed to man-in-the-middle attack then some LLPs are more exposed to man-in-the-middle attack than
others. Please see the relevant LLP documentation on security others. Please see the relevant LLP documentation on security
issues around connection and/or Stream hijacking. issues around connection and/or Stream hijacking.
Another approach is to restrict access to only the local Another approach is to restrict access to only the local
subnet/link, and provide some mechanism to limit access, such as subnet/link, and provide some mechanism to limit access, such as
physical security or 802.1.x. This model is an extremely limited physical security or 802.1.x. This model is an extremely limited
deployment scenario, and will not be further examined here. deployment scenario, and will not be further examined here.
7.2.4 Using an STag on a Different Stream 7.2.4 Using an STag on a Different Stream
One style of attack from the Remote Peer is for it to attempt to One style of attack from the Remote Peer is for it to attempt to
use STag values that it is not authorized to use. Note that if use STag values that it is not authorized to use. Note that if
the Remote Peer sends an invalid STag to the Local Peer, per the the Remote Peer sends an invalid STag to the Local Peer, per the
DDP and RDMAP specifications, the Stream must be torn down. Thus DDP and RDMAP specifications, the Stream must be torn down. Thus
the threat exists if a STag has been enabled for Remote Access on the threat exists if an STag has been enabled for Remote Access
one Stream and a Remote Peer is able to use it on an unrelated on one Stream and a Remote Peer is able to use it on an unrelated
Stream. If the attack is successful, the attacker could Stream. If the attack is successful, the attacker could
potentially be able to perform either RDMA Read Operations to potentially be able to perform either RDMA Read Operations to
read the contents of the associated data buffer, perform RDMA read the contents of the associated data buffer, perform RDMA
Write Operations to modify the contents of the associated data Write Operations to modify the contents of the associated data
buffer, or to Invalidate the STag to disable further access to buffer, or to invalidate the STag to disable further access to
the buffer. the buffer.
An attempt by a Remote Peer to access a buffer with an STag on a An attempt by a Remote Peer to access a buffer with an STag on a
different Stream in the same Protection Domain may or may not be different Stream in the same Protection Domain may or may not be
an attack depending on whether resource sharing is intended (i.e. an attack depending on whether resource sharing is intended (i.e.
whether the Streams shared Partial Mutual Trust or not). For some whether the Streams shared Partial Mutual Trust or not). For some
ULPs using an STag on multiple Streams within the same Protection ULPs using an STag on multiple Streams within the same Protection
Domain could be desired behavior. For other ULPs attempting to Domain could be desired behavior. For other ULPs attempting to
use an STag on a different Stream could be considered to be an use an STag on a different Stream could be considered to be an
attack. Since this varies by ULP, a ULP typically would need to attack. Since this varies by ULP, a ULP typically would need to
be able to control the scope of the STag. be able to control the scope of the STag.
In the case where an implementation does not share resources In the case where an implementation does not share resources
between Streams (including STags), this attack can be defeated by between Streams (including STags), this attack can be defeated by
assigning each Stream to a different Protection Domain. Before assigning each Stream to a different Protection Domain. Before
allowing remote access to the buffer, the Protection Domain of allowing remote access to the buffer, the Protection Domain of
the Stream where the access attempt was made is matched against the Stream where the access attempt was made is matched against
the Protection Domain of the STag. If the Protection Domains do the Protection Domain of the STag. If the Protection Domains do
not match, access to the buffer is denied, an error is generated, not match, access to the buffer is denied, an error is generated,
and the RDMAP Stream associated with the attacking Stream should and the RDMAP Stream associated with the attacking Stream is
be terminated. terminated.
For implementations that share resources between multiple For implementations that share resources between multiple
Streams, it may not be practical to separate each Stream into its Streams, it may not be practical to separate each Stream into its
own Protection Domain. In this case, the ULP can still limit the own Protection Domain. In this case, the ULP can still limit the
scope of any of the STags to a single Stream (if it is enabling scope of any of the STags to a single Stream (if it is enabling
it for remote access). If the STag scope has been limited to a it for remote access). If the STag scope has been limited to a
single Stream, any attempt to use that STag on a different Stream single Stream, any attempt to use that STag on a different Stream
will result in an error, and the RDMA Stream should be will result in an error, and the RDMAP Stream is terminated.
terminated.
Thus for implementations that do not share STags between Streams, Thus for implementations that do not share STags between Streams,
each Stream MUST either be in a separate Protection Domain or the each Stream MUST either be in a separate Protection Domain or the
scope of an STag MUST be limited to a single Stream. scope of an STag MUST be limited to a single Stream.
An RNIC MUST ensure that a specific Stream in a specific
Protection Domain can not access an STag in a different
Protection Domain.
An RNIC MUST ensure that if an STag is limited in scope to a
single Stream, no other Stream can use the STag.
An additional issue may be unintended sharing of STags (i.e. a An additional issue may be unintended sharing of STags (i.e. a
bug in the ULP) or a bug in the Remote Peer which causes an off- bug in the ULP) or a bug in the Remote Peer which causes an off-
by-one STag to be used. For additional protection, an by-one STag to be used. For additional protection, an
implementation should allocate STags in such a fashion that it is implementation should allocate STags in such a fashion that it is
difficult to predict the next allocated STag number. Allocation difficult to predict the next allocated STag number, and also
methods which deterministically allocate the next STag should be ensure that STags are reused at as slow a rate as possible. Any
avoided (e.g. a method which always starts with STag equal to one allocation method which would lead to intentional or
and monotonically increases it for each new allocation, or a unintentional reuse of an STag by the peer should be avoided
method which always uses the same STag for each operation). (e.g. a method which always starts with a given STag and
monotonically increases it for each new allocation, or a method
which always uses the same STag for each operation).
7.3 Tampering 7.3 Tampering
A Remote Peer or a network based attacker can attempt to tamper A Remote Peer or a network based attacker can attempt to tamper
with the contents of data buffers on a Local Peer that have been with the contents of data buffers on a Local Peer that have been
enabled for remote write access. The types of tampering attacks enabled for remote write access. The types of tampering attacks
that are possible are outlined in the sections that follow. that are possible are outlined in the sections that follow.
7.3.1 Buffer Overrun - RDMA Write or Read Response 7.3.1 Buffer Overrun - RDMA Write or Read Response
This attack is an attempt by the Remote Peer to perform an RDMA This attack is an attempt by the Remote Peer to perform an RDMA
Write or RDMA Read Response to memory outside of the valid length Write or RDMA Read Response to memory outside of the valid length
range of the data buffer enabled for remote write access. This range of the data buffer enabled for remote write access. This
attack can occur even when no resources are shared across attack can occur even when no resources are shared across
Streams. This issue can also arise if the ULP has a bug. Streams. This issue can also arise if the ULP has a bug.
The countermeasure for this type of attack must be in the RNIC The countermeasure for this type of attack must be in the RNIC
implementation, using the STag. When the Local Peer specifies to implementation, leveraging the STag. When the local ULP specifies
the RNIC the base address and the number of bytes in the buffer to the RNIC the base address and the number of bytes in the
that it wishes to make accessible, the RNIC must ensure that the buffer that it wishes to make accessible, the RNIC must ensure
base and bounds check are applied to any access to the buffer that the base and bounds check are applied to any access to the
referenced by the STag before the STag is enabled for access. buffer referenced by the STag before the STag is enabled for
When an RDMA data transfer operation (which includes an STag) access. When an RDMA data transfer operation (which includes an
arrives on a Stream, a base and bounds byte granularity access STag) arrives on a Stream, a base and bounds byte granularity
check must be performed to ensure the operation accesses only access check must be performed to ensure the operation accesses
memory locations within the buffer described by that STag. only memory locations within the buffer described by that STag.
Thus an RNIC implementation MUST ensure that a Remote Peer is not Thus an RNIC implementation MUST ensure that a Remote Peer is not
able to access memory outside of the buffer specified when the able to access memory outside of the buffer specified when the
STag was enabled for remote access. STag was enabled for remote access.
7.3.2 Modifying a Buffer After Indication 7.3.2 Modifying a Buffer After Indication
This attack can occur if a Remote Peer attempts to modify the This attack can occur if a Remote Peer attempts to modify the
contents of an STag referenced buffer by performing an RDMA Write contents of an STag referenced buffer by performing an RDMA Write
or an RDMA Read Response after the Remote Peer has indicated to or an RDMA Read Response after the Remote Peer has indicated to
the Local Peer that the STag data buffer contents are ready for the Local Peer or local ULP (by a variety of means) that the STag
use. This attack can occur even when no resources are shared data buffer contents are ready for use. This attack can occur
across Streams. Note that a bug in a Remote Peer, or network even when no resources are shared across Streams. Note that a bug
based tampering, could also result in this problem. in a Remote Peer, or network based tampering, could also result
in this problem.
For example, assume the STag referenced buffer contains ULP For example, assume the STag referenced buffer contains ULP
control information as well as ULP payload, and the ULP sequence control information as well as ULP payload, and the ULP sequence
of operation is to first validate the control information and of operation is to first validate the control information and
then perform operations on the control information. If the Remote then perform operations on the control information. If the Remote
Peer can perform an additional RDMA Write or RDMA Read Response Peer can perform an additional RDMA Write or RDMA Read Response
(thus changing the buffer) after the validity checks have been (thus changing the buffer) after the validity checks have been
completed but before the control data is operated on, the Remote completed but before the control data is operated on, the Remote
Peer could force the ULP down operational paths that were never Peer could force the ULP down operational paths that were never
intended. intended.
The Local Peer can protect itself from this type of attack by The local ULP can protect itself from this type of attack by
revoking remote access when the original data transfer has revoking remote access when the original data transfer has
completed and before it validates the contents of the buffer. The completed and before it validates the contents of the buffer. The
Local Peer can either do this by explicitly revoking remote local ULP can either do this by explicitly revoking remote access
access rights for the STag when the Remote Peer indicates the rights for the STag when the Remote Peer indicates the operation
operation has completed, or by checking to make sure the Remote has completed, or by checking to make sure the Remote Peer
Peer Invalidated the STag through the RDMAP Invalidate invalidated the STag through the RDMAP Remote Invalidate
capability, and if it did not, the Local Peer then explicitly capability (see section 7.5.5 Remote Invalidate an STag Shared on
Multiple Streams on page 38 for a definition of Remote
Invalidate), and if it did not, the local ULP then explicitly
revokes the STag remote access rights. revokes the STag remote access rights.
The Local Peer SHOULD follow the above procedure to protect the The local ULP SHOULD follow the above procedure to protect the
buffer before it validates the contents of the buffer (or uses buffer before it validates the contents of the buffer (or uses
the buffer in any way). the buffer in any way).
An RNIC MUST ensure that network packets using the STag for a
previously advertised buffer can no longer modify the buffer
after the ULP revokes remote access rights for the specific STag.
7.3.3 Multiple STags to access the same buffer 7.3.3 Multiple STags to access the same buffer
See section 7.4.6 Using Multiple STags Which Alias to the Same See section 7.4.6 Using Multiple STags Which Alias to the Same
Buffer on page 27 for this analysis. Buffer on page 29 for this analysis.
7.3.4 Network based modification of buffer content 7.3.4 Network based modification of buffer content
This is actually a man in the middle attack - but only on the This is actually a man in the middle attack - but only on the
content of the buffer, as opposed to the man in the middle attack content of the buffer, as opposed to the man in the middle attack
presented above, where both the signaling and content can be presented above, where both the signaling and content can be
modified. See Section 7.2.3 Man in the Middle Attack on page 22. modified. See Section 7.2.3 Man in the Middle Attack on page 24.
7.4 Information Disclosure 7.4 Information Disclosure
The main potential source for information disclosure is through a The main potential source for information disclosure is through a
local buffer that has been enabled for remote access. If the local buffer that has been enabled for remote access. If the
buffer can be probed by a Remote Peer on another Stream, then buffer can be probed by a Remote Peer on another Stream, then
there is potential for information disclosure. there is potential for information disclosure.
The potential attacks that could result in unintended information The potential attacks that could result in unintended information
disclosure and countermeasures are detailed in the following disclosure and countermeasures are detailed in the following
skipping to change at page 26, line 33 skipping to change at page 28, line 5
If a buffer is being used for a combination of reads and writes If a buffer is being used for a combination of reads and writes
(either remote or local), and is exposed to the Remote Peer with (either remote or local), and is exposed to the Remote Peer with
at least remote read access rights, the Remote Peer may be able at least remote read access rights, the Remote Peer may be able
to examine the contents of the buffer before they are initialized to examine the contents of the buffer before they are initialized
with the correct data. In this situation, whatever contents were with the correct data. In this situation, whatever contents were
present in the buffer before the buffer is initialized can be present in the buffer before the buffer is initialized can be
viewed by the Remote Peer, if the Remote Peer performs an RDMA viewed by the Remote Peer, if the Remote Peer performs an RDMA
Read. Read.
Because of this, the Local Peer SHOULD ensure that no stale data Because of this, the local ULP SHOULD ensure that no stale data
is contained in the buffer before remote read access rights are is contained in the buffer before remote read access rights are
granted (this can be done by zeroing the contents of the memory, granted (this can be done by zeroing the contents of the memory,
for example). for example).
7.4.3 Accessing a Buffer After the Transfer 7.4.3 Accessing a Buffer After the Transfer
If the Remote Peer has remote read access to a buffer, and by If the Remote Peer has remote read access to a buffer, and by
some mechanism tells the Local Peer that the transfer has been some mechanism tells the local ULP that the transfer has been
completed, but the Local Peer does not disable remote access to completed, but the local ULP does not disable remote access to
the buffer before modifying the data, it is possible for the the buffer before modifying the data, it is possible for the
Remote Peer to retrieve the new data. Remote Peer to retrieve the new data.
This is similar to the attack defined in Section 7.3.2 Modifying This is similar to the attack defined in Section 7.3.2 Modifying
a Buffer After Indication on page 25. The same countermeasures a Buffer After Indication on page 26. The same countermeasures
apply. In addition, the Local Peer SHOULD grant remote read apply. In addition, the local ULP SHOULD grant remote read access
access rights only for the amount of time needed to retrieve the rights only for the amount of time needed to retrieve the data.
data.
7.4.4 Accessing Unintended Data With a Valid STag 7.4.4 Accessing Unintended Data With a Valid STag
If the ULP enables remote access to a buffer using an STag that If the ULP enables remote access to a buffer using an STag that
references the entire buffer, but intends only a portion of the references the entire buffer, but intends only a portion of the
buffer to be accessed, it is possible for the Remote Peer to buffer to be accessed, it is possible for the Remote Peer to
access the other parts of the buffer anyway. access the other parts of the buffer anyway.
To prevent this attack, the ULP SHOULD set the base and bounds of To prevent this attack, the ULP SHOULD set the base and bounds of
the buffer when the STag is initialized to expose only the data the buffer when the STag is initialized to expose only the data
skipping to change at page 27, line 30 skipping to change at page 28, line 53
The most obvious countermeasure for this attack is to not grant The most obvious countermeasure for this attack is to not grant
remote read access if the buffer is intended to be write-only. remote read access if the buffer is intended to be write-only.
Then the Remote Peer would not be able to retrieve data Then the Remote Peer would not be able to retrieve data
associated with the buffer. An attempt to do so would result in associated with the buffer. An attempt to do so would result in
an error and the RDMAP Stream associated with the Stream would be an error and the RDMAP Stream associated with the Stream would be
terminated. terminated.
Thus if a ULP only intends a buffer to be exposed for remote Thus if a ULP only intends a buffer to be exposed for remote
write access, it MUST set the access rights to the buffer to only write access, it MUST set the access rights to the buffer to only
enable remote write access. enable remote write access. Note that this requirement is not
meant to restrict the use of zero-length RDMA Reads. Zero-length
RDMA Reads do not expose ULP data. Because they are intended to
be used as a mechanism to ensure that all RDMA Writes have been
received, and do not even require a valid STag, their use is
permitted even if a buffer has only been enabled for write
access.
7.4.6 Using Multiple STags Which Alias to the Same Buffer 7.4.6 Using Multiple STags Which Alias to the Same Buffer
Multiple STags which alias to the same buffer at the same time Multiple STags which alias to the same buffer at the same time
can result in unintentional information disclosure if the STags can result in unintentional information disclosure if the STags
are used by different, mutually untrusted, Remote Peers. This are used by different, mutually untrusted, Remote Peers. This
model applies specifically to client/server communication, where model applies specifically to client/server communication, where
the server is communicating with multiple clients, each of which the server is communicating with multiple clients, each of which
do not mutually trust each other. do not mutually trust each other.
If only read access is enabled, then the Local Peer has complete If only read access is enabled, then the local ULP has complete
control over information disclosure. Thus a server which intended control over information disclosure. Thus a server which intended
to expose the same data (i.e. buffer) to multiple clients by to expose the same data (i.e. buffer) to multiple clients by
using multiple STags to the same buffer creates no new security using multiple STags to the same buffer creates no new security
issues beyond what has already been described in this document. issues beyond what has already been described in this document.
Note that if the server did not intend to expose the same data to Note that if the server did not intend to expose the same data to
the clients, it should use separate buffers for each client (and the clients, it should use separate buffers for each client (and
separate STags). separate STags).
When one STag has remote read access enabled and a different STag When one STag has remote read access enabled and a different STag
has remote write access enabled to the same buffer, it is has remote write access enabled to the same buffer, it is
possible for one Remote Peer to view the contents that have been possible for one Remote Peer to view the contents that have been
written by another Remote Peer. written by another Remote Peer.
If both STags have remote write access enabled and the two Remote If both STags have remote write access enabled and the two Remote
Peers do not mutually trust each other, it is possible for one Peers do not mutually trust each other, it is possible for one
Remote Peer to overwrite the contents that have been written by Remote Peer to overwrite the contents that have been written by
the other Remote Peer. the other Remote Peer.
Thus multiple Remote Peers which do not share Partial Mutual Thus a ULP with multiple Remote Peers which do not share Partial
Trust MUST NOT be granted write access to the same buffer through Mutual Trust MUST NOT grant write access to the same buffer
different STags. A buffer should be exposed to only one untrusted through different STags. A buffer should be exposed to only one
Remote Peer at a time to ensure that no information disclosure or untrusted Remote Peer at a time to ensure that no information
information tampering occurs between peers. disclosure or information tampering occurs between peers.
7.4.7 Remote Node Loading Firmware onto the RNIC 7.4.7 Remote Node Loading Firmware onto the RNIC
If the Remote Peer can cause firmware to be loaded onto the RNIC, If the Remote Peer can cause firmware to be loaded onto the RNIC,
there is an opportunity for information disclosure. See Elevation there is an opportunity for information disclosure. See Elevation
of Privilege in Section 7.6 for this analysis. of Privilege in Section 7.5.6 for this analysis.
7.4.8 Controlling Access to PTT & STag Mapping 7.4.8 Controlling Access to PTT & STag Mapping
If a Non-Privileged ULP is able to directly manipulate the RNIC If a Non-Privileged ULP is able to directly manipulate the RNIC
Page Translation Tables (which translate from an STag to a host Page Translation Tables (which translate from an STag to a host
address), it is possible that the Non-Privileged ULP could point address), it is possible that the Non-Privileged ULP could point
the Page Translation Table at an unrelated ULP’s buffers and the Page Translation Table at an unrelated Stream's or ULPÆs
thereby be able to gain access to information in the unrelated buffers and thereby be able to gain access to information of the
ULP. unrelated Stream/ULP.
As discussed in Section 4 Architectural Model on page 9, As discussed in Section 4 Architectural Model on page 10,
introduction of a Privileged Resource Manager to arbitrate the introduction of a Privileged Resource Manager to arbitrate the
mapping requests is an effective countermeasure. This enables the mapping requests is an effective countermeasure. This enables the
Privileged Resource Manager to ensure a ULP can only initialize Privileged Resource Manager to ensure a local ULP can only
the Page Translation Table (PTT)to point to its own buffers. initialize the Page Translation Table (PTT)to point to its own
buffers.
Thus if Non-Privileged ULPs are supported, the Privileged Thus if Non-Privileged ULPs are supported, the Privileged
Resource Manager MUST verify that the Non-Privileged ULP has the Resource Manager MUST verify that the Non-Privileged ULP has the
right to access a specific Data Buffer before allowing an STag right to access a specific Data Buffer before allowing an STag
for which the ULP has access rights to be associated with a for which the ULP has access rights to be associated with a
specific Data Buffer. This can be done when the Page Translation specific Data Buffer. This can be done when the Page Translation
Table is initialized to access the Data Buffer or when the STag Table is initialized to access the Data Buffer or when the STag
is initialized to point to a group of Page Translation Table is initialized to point to a group of Page Translation Table
entries, or both. entries, or both.
7.4.9 Network based eavesdropping 7.4.9 Network based eavesdropping
An attacker that is able to eavesdrop on the network can read the An attacker that is able to eavesdrop on the network can read the
content of all read and write access to the peer’s buffers. To content of all read and write accesses to a PeerÆs buffers. To
prevent information disclosure, the read/written data must be prevent information disclosure, the read/written data must be
encrypted. See also Section 7.2.3 Man in the Middle Attack on encrypted. See also Section 7.2.3 Man in the Middle Attack on
page 22. The encryption can be done either by the ULP, or by a page 24. The encryption can be done either by the ULP, or by a
protocol that provides security services to the LLP (e.g. IPsec protocol that provides security services to the LLP (e.g. IPsec
or SSL). Refer to section 8 for discussion of security services or SSL). Refer to section 8 for discussion of security services
for DDP/RDMA. for DDP/RDMA.
7.5 Denial of Service (DOS) 7.5 Denial of Service (DOS)
A DOS attack is one of the primary security risks of RDMAP. This A DOS attack is one of the primary security risks of RDMAP. This
is because RNIC resources are valuable and scarce, and many ULP is because RNIC resources are valuable and scarce, and many ULP
environments require communication with untrusted Remote Peers. environments require communication with untrusted Remote Peers.
If the remote ULP can be authenticated or encrypted, clearly, the If the remote ULP can be authenticated or encrypted, clearly, the
skipping to change at page 29, line 28 skipping to change at page 30, line 55
established, the attacker must be able to both send and receive established, the attacker must be able to both send and receive
messages over that connection/Stream, or be able to guess a valid messages over that connection/Stream, or be able to guess a valid
packet on an existing RDMAP Stream. packet on an existing RDMAP Stream.
This section outlines the potential attacks and the This section outlines the potential attacks and the
countermeasures available for dealing with each attack. countermeasures available for dealing with each attack.
7.5.1 RNIC Resource Consumption 7.5.1 RNIC Resource Consumption
This section covers attacks that fall into the general category This section covers attacks that fall into the general category
of a Local Peer attempting to unfairly allocate scarce (i.e. of a local ULP attempting to unfairly allocate scarce (i.e.
bounded) RNIC resources. The Local Peer may be attempting to bounded) RNIC resources. The local ULP may be attempting to
allocate resources on its own behalf, or on behalf of a Remote allocate resources on its own behalf, or on behalf of a Remote
Peer. Resources that fall into this category include: Protection Peer. Resources that fall into this category include: Protection
Domains, Stream Context Memory, Translation and Protection Domains, Stream Context Memory, Translation and Protection
Tables, and STag namespace. These can be attacks by currently Tables, and STag namespace. These can be due to attacks by
active Local Peers or ones that allocated resources earlier, but currently active local ULPs or ones that allocated resources
are now idle. earlier, but are now idle.
This type of attack can occur regardless of whether resources are This type of attack can occur regardless of whether or not
shared across Streams. resources are shared across Streams.
The allocation of all scarce resources MUST be placed under the The allocation of all scarce resources MUST be placed under the
control of a Privileged Resource Manager. This allows the control of a Privileged Resource Manager. This allows the
Privileged Resource Manager to: Privileged Resource Manager to:
* prevent a Local Peer from allocating more than its fair * prevent a local ULP from allocating more than its fair
share of resources. share of resources.
* detect if a Remote Peer is attempting to launch a DOS * detect if a Remote Peer is attempting to launch a DOS
attack by attempting to create an excessive number of attack by attempting to create an excessive number of
Streams (with associated resources) and take corrective Streams (with associated resources) and take corrective
action (such as refusing the request or applying network action (such as refusing the request or applying network
layer filters against the Remote Peer). layer filters against the Remote Peer).
This analysis assumes that the Resource Manager is responsible This analysis assumes that the Resource Manager is responsible
for handing out Protection Domains, and RNIC implementations will for handing out Protection Domains, and RNIC implementations will
provide enough Protection Domains to allow the Resource Manager provide enough Protection Domains to allow the Resource Manager
to be able to assign a unique Protection Domain for each to be able to assign a unique Protection Domain for each
unrelated, untrusted Local Peer (for a bounded, reasonable number unrelated, untrusted local ULP (for a bounded, reasonable number
of Local Peers). This analysis further assumes that the Resource of local ULPs). This analysis further assumes that the Resource
Manager implements policies to ensure that untrusted Local Peers Manager implements policies to ensure that untrusted local ULPs
are not able to consume all of the Protection Domains through a are not able to consume all of the Protection Domains through a
DOS attack. Note that Protection Domain consumption cannot result DOS attack. Note that Protection Domain consumption cannot result
from a DOS attack launched by a Remote Peer, unless a Local Peer from a DOS attack launched by a Remote Peer, unless a local ULP
is acting on the Remote Peer’s behalf. is acting on the Remote PeerÆs behalf.
7.5.2 Resource Consumption By Active ULPs 7.5.2 Resource Consumption By Active ULPs
This section describes DOS attacks from Local and Remote Peers This section describes DOS attacks from Local and Remote Peers
that are actively exchanging messages. Attacks on each RDMA NIC that are actively exchanging messages. Attacks on each RDMA NIC
resource are examined and specific countermeasures are resource are examined and specific countermeasures are
identified. Note that attacks on Stream Context Memory, Page identified. Note that attacks on Stream Context Memory, Page
Translation Tables, and STag namespace are covered in Section Translation Tables, and STag namespace are covered in Section
7.5.1 RNIC Resource Consumption, so are not included here. 7.5.1 RNIC Resource Consumption, so are not included here.
7.5.2.1 Multiple Streams Sharing Receive Buffers 7.5.2.1 Multiple Streams Sharing Receive Buffers
The Remote Peer can attempt to consume more than its fair share The Remote Peer can attempt to consume more than its fair share
of receive data buffers (Untagged DDP buffers or for RDMAP of receive data buffers (i.e. Untagged buffers for DDP are or
buffers consumed with Send Type Messages) if receive buffers are Send Type Messages for RDMAP) if receive buffers are shared
shared across multiple Streams. across multiple Streams.
If resources are not shared across multiple Streams, then this If resources are not shared across multiple Streams, then this
attack is not possible because the Remote Peer will not be able attack is not possible because the Remote Peer will not be able
to consume more buffers than were allocated to the Stream. The to consume more buffers than were allocated to the Stream. The
worst case scenario is that the Remote Peer can consume more worst case scenario is that the Remote Peer can consume more
receive buffers than the Local Peer allowed, resulting in no receive buffers than the local ULP allowed, resulting in no
buffers to be available, which could cause the Remote Peer’s buffers being available, which could cause the Remote PeerÆs
Stream to the Local Peer to be torn down, and all allocated Stream to the Local Peer to be torn down, and all allocated
resources to be released. resources to be released.
If local receive data buffers are shared among multiple Streams, If local receive data buffers are shared among multiple Streams,
then the Remote Peer can attempt to consume more than its fair then the Remote Peer can attempt to consume more than its fair
share of the receive buffers, causing a different Stream to be share of the receive buffers, causing a different Stream to be
short of receive buffers, thus possibly causing the other Stream short of receive buffers, thus possibly causing the other Stream
to be torn down. For example, if the Remote Peer sent enough one to be torn down. For example, if the Remote Peer sent enough one
byte Untagged Messages, they might be able to consume all local byte Untagged Messages, they might be able to consume all local
shared receive queue resources with little effort on their part. shared receive queue resources with little effort on their part.
skipping to change at page 31, line 6 skipping to change at page 32, line 33
Peer is attempting to use more than its fair share of resources Peer is attempting to use more than its fair share of resources
and terminate the Stream (causing the allocated resources to be and terminate the Stream (causing the allocated resources to be
released). However, if the Local Peer is sufficiently slow, it released). However, if the Local Peer is sufficiently slow, it
may be possible for the Remote Peer to still mount a denial of may be possible for the Remote Peer to still mount a denial of
service attack. One countermeasure that can protect against this service attack. One countermeasure that can protect against this
attack is implementing a low-water notification. The low-water attack is implementing a low-water notification. The low-water
notification alerts the ULP if the number of buffers in the notification alerts the ULP if the number of buffers in the
receive queue is less than a threshold. receive queue is less than a threshold.
If all of the following conditions are true, then the Local Peer If all of the following conditions are true, then the Local Peer
can size the amount of local receive buffers posted on the or local ULP can size the amount of local receive buffers posted
receive queue to ensure a DOS attack can be stopped. on the receive queue to ensure a DOS attack can be stopped.
* a low-water notification is enabled, and * a low-water notification is enabled, and
* the Local Peer is able to bound the amount of time that * the Local Peer is able to bound the amount of time that
it takes to replenish receive buffers, and it takes to replenish receive buffers, and
* the Local Peer maintains statistics to determine which * the Local Peer maintains statistics to determine which
Remote Peer is consuming buffers. Remote Peer is consuming buffers.
The above conditions enable the low-water notification to arrive The above conditions enable the low-water notification to arrive
before resources are depleted and thus the Local Peer can take before resources are depleted and thus the Local Peer or local
corrective action (e.g., terminate the Stream of the attacking ULP can take corrective action (e.g., terminate the Stream of the
Remote Peer). attacking Remote Peer).
A different, but similar attack is if the Remote Peer sends a A different, but similar attack is if the Remote Peer sends a
significant number of out-of-order packets and the RNIC has the significant number of out-of-order packets and the RNIC has the
ability to use the ULP buffer as a reassembly buffer. In this ability to use the ULP buffer (i.e. the Untagged Buffer for DDP
case the Remote Peer can consume a significant number of ULP or the buffer consumed by a Send Type Message for RDMAP) as a
buffers, but never send enough data to enable the ULP buffer to reassembly buffer. In this case the Remote Peer can consume a
be completed to the ULP. significant number of ULP buffers, but never send enough data to
enable the ULP buffer to be completed to the ULP.
An effective countermeasure is to create a high-water An effective countermeasure is to create a high-water
notification which alerts the ULP if there is more than a notification which alerts the ULP if there is more than a
specified number of receive buffers "in process" (partially specified number of receive buffers "in process" (partially
consumed, but not completed). The notification is generated when consumed, but not completed). The notification is generated when
more than the specified number of buffers are in process more than the specified number of buffers are in process
simultaneously on a specific Stream (i.e., packets have started simultaneously on a specific Stream (i.e., packets have started
to arrive for the buffer, but the buffer has not yet been to arrive for the buffer, but the buffer has not yet been
delivered to the ULP). delivered to the ULP).
A different countermeasure is for the RNIC Engine to provide the A different countermeasure is for the RNIC Engine to provide the
capability to limit the Remote Peers ability to consume receive capability to limit the Remote PeerÆs ability to consume receive
buffers on a per Stream basis. Unfortunately this requires a buffers on a per Stream basis. Unfortunately this requires a
large amount of state to be tracked in each RNIC on a per Stream large amount of state to be tracked in each RNIC on a per Stream
basis. basis.
Thus, if an RNIC Engine provides the ability to share receive Thus, if an RNIC Engine provides the ability to share receive
buffers across multiple Streams, the combination of the RNIC buffers across multiple Streams, the combination of the RNIC
Engine and the Privileged Resource Manager MUST be able to detect Engine and the Privileged Resource Manager MUST be able to detect
if the Remote Peer is attempting to consume more than its fair if the Remote Peer is attempting to consume more than its fair
share of resources so that the Local Peer can apply share of resources so that the Local Peer or local ULP can apply
countermeasures to detect and prevent the attack. countermeasures to detect and prevent the attack.
7.5.2.2 Local ULP Attacking a Shared CQ 7.5.2.2 Local ULP Attacking a Shared CQ
DOS attacks against a Shared Completion Queue (CQ) can be caused DOS attacks against a Shared Completion Queue (CQ) can be caused
by either the Local Peer's ULP or the Remote Peer if either by either the local ULP or the Remote Peer if either attempts to
attempts to cause more completions than its fair share of the cause more completions than its fair share of the number of
number of entries, thus potentially starving another unrelated entries, thus potentially starving another unrelated ULP such
ULP such that no Completion Queue entries are available. that no Completion Queue entries are available.
A Completion Queue entry can potentially be consumed by a A Completion Queue entry can potentially be maliciously consumed
completion from the Send Queue or a completion from the Receive by a completion from the Send Queue or a completion from the
Queue. In the former, the attacker is the Local Peer's ULP. In Receive Queue. In the former, the attacker is the local ULP. In
the later, the attacker is the Remote Peer. the latter, the attacker is the Remote Peer.
A form of attack can occur where the Local Peer ULPs can consume A form of attack can occur where the local ULPs can consume
resources on the CQ. A Local Peer ULP that is slow to free resources on the CQ. A local ULP that is slow to free resources
resources on the CQ by not reaping the completion status quickly on the CQ by not reaping the completion status quickly enough
enough could stall all other Local Peer ULPs attempting to use could stall all other local ULPs attempting to use that CQ.
that CQ.
One of two countermeasures can be used to avoid this kind of For these reasons, an RNIC MUST NOT enable sharing a CQ across
attack. The first is to only share a CQ between ULPs that share ULPs that do not share Partial Mutual Trust.
Partial Mutual Trust. The other is to use a trusted Local Peer to
act as a third party to free resources on the CQ and place the
status in intermediate storage until the untrusted ULP reaps the
status information. For these reasons, an RNIC MUST NOT enable
sharing a CQ across ULPs that do not share partial mutual trust.
7.5.2.3 Remote Peer Attacking a Shared CQ 7.5.2.3 Local or Remote Peer Attacking a Shared CQ
For an overview of the Shared CQ attack model, see Section For an overview of the shared CQ attack model, see Section
7.5.2.2. 7.5.2.2.
<TBD: add text that says if not shared there is no security The Remote Peer can attack a shared CQ by consuming more than its
threat). If you get a CQ overflow it MUST NOT affect any resource fair share of CQ entries by using one of the following methods:
outside the scope of the current Stream.>
The Remote Peer can attack a CQ by consuming more than its fair
share of CQ entries by using one of the following methods:
* The ULP protocol allows the Remote Peer to reserve a * The ULP protocol allows the Remote Peer to reserve a
specified number of CQ entries, possibly leaving specified number of CQ entries, possibly leaving
insufficient entries for other Streams that are sharing insufficient entries for other Streams that are sharing
the CQ. the CQ.
* If the Remote Peer or Local Peer (or both) can attack the * If the Remote Peer, Local Peer, or local ULP (or any
CQ by overwhelming the CQ with completions, then combination) can attack the CQ by overwhelming the CQ
completion processing on other Streams sharing that with completions, then completion processing on other
Completion Queue can be affected (e.g. the Completion Streams sharing that Completion Queue can be affected
Queue overflows and stops functioning). (e.g. the Completion Queue overflows and stops
functioning).
The first method of attack can be avoided if the ULP does not The first method of attack can be avoided if the ULP does not
allow a Remote Peer to reserve CQ entries or there is a trusted allow a Remote Peer to reserve CQ entries or there is a trusted
intermediary such as a Privileged Resource Manager. Unfortunately intermediary such as a Privileged Resource Manager. Unfortunately
it is often unrealistic to not allow a Remote Peer to reserve CQ it is often unrealistic to not allow a Remote Peer to reserve CQ
entries - particularly if the number of completion entries is entries - particularly if the number of completion entries is
dependent on other ULP negotiated parameters, such as the amount dependent on other ULP negotiated parameters, such as the amount
of buffering required by the ULP. Thus an implementation MUST of buffering required by the ULP. Thus an implementation MUST
implement a Privileged Resource Manager to control the allocation implement a Privileged Resource Manager to control the allocation
of CQ entries. See Section 4.1 Components on page 10 for a of CQ entries. See Section 4.1 Components on page 11 for a
definition of Privileged Resource Manager. definition of Privileged Resource Manager.
One way that a Local or Remote Peer can attempt to overwhelm a CQ One way that a Local or Remote Peer can attempt to overwhelm a CQ
with completions is by sending minimum length RDMAP/DDP Messages with completions is by sending minimum length RDMAP/DDP Messages
to cause as many completions (receive completions for the Remote to cause as many completions (receive completions for the Remote
Peer, send completions for the Local Peer) per second as Peer, send completions for the Local Peer) per second as
possible. If it is the Remote Peer attacking, and we assume that possible. If it is the Remote Peer attacking, and we assume that
the Local Peer does not run out of receive buffers (if they do, the Local Peer's receive queue(s) do not run out of receive
then this is a different attack, documented in Section 7.5.2.1 buffers (if they do, then this is a different attack, documented
Multiple Streams Sharing Receive Buffers on page 30), then it in Section 7.5.2.1 Multiple Streams Sharing Receive Buffers on
might be possible for the Remote Peer to consume more than its page 31), then it might be possible for the Remote Peer to
fair share of Completion Queue entries. Depending upon the CQ consume more than its fair share of Completion Queue entries.
implementation, this could either cause the CQ to overflow (if it Depending upon the CQ implementation, this could either cause the
is not large enough to handle all of the completions generated) CQ to overflow (if it is not large enough to handle all of the
or for another Stream to not be able to generate CQ entries (if completions generated) or for another Stream to not be able to
the RNIC had flow control on generation of CQ entries into the generate CQ entries (if the RNIC had flow control on generation
CQ). In either case, the CQ will stop functioning correctly and of CQ entries into the CQ). In either case, the CQ will stop
any Streams expecting completions on the CQ will stop functioning correctly and any Streams expecting completions on
functioning. the CQ will stop functioning.
This attack can occur regardless of whether all of the Streams This attack can occur regardless of whether all of the Streams
associated with the CQ are in the same Protection Domain or are associated with the CQ are in the same Protection Domain or are
in different Protection Domains - the key issue is that the in different Protection Domains - the key issue is that the
number of Completion Queue entries is less than the number of all number of Completion Queue entries is less than the number of all
outstanding operations that can cause a completion. outstanding operations that can cause a completion.
The Local Peer can protect itself from this type of attack using The Local Peer can protect itself from this type of attack using
either of the following methods: either of the following methods:
* Size the CQ to the appropriate level, as specified below * Size the CQ to the appropriate level, as specified below
(note that if the CQ currently exists, and it needs to be (note that if the CQ currently exists, and it needs to be
resized, resizing the CQ can fail, so the CQ resize resized, resizing the CQ is not required to succeed in
should be done before sizing the Send Queue and Receive all cases, so the CQ resize should be done before sizing
Queue on the Stream), OR the Send Queue and Receive Queue on the Stream), OR
* Grant fewer resources than the Remote Peer requested (not * Grant fewer resources than the Remote Peer requested (not
supplying the number of Receive Data Buffers requested). supplying the number of Receive Data Buffers requested).
The proper sizing of the CQ is dependent on whether the Local The proper sizing of the CQ is dependent on whether the local
Peer will post as many resources to the various queues as the ULP(s) will post as many resources to the various queues as the
size of the queue enables or not. If the Local Peer can be size of the queue enables or not. If the local ULP(s) can be
trusted to post a number of resources that is smaller than the trusted to post a number of resources that is smaller than the
size of the specific resources queue, then a correctly sized CQ size of the specific resourceÆs queue, then a correctly sized CQ
means that the CQ is large enough to hold completion status for means that the CQ is large enough to hold completion status for
all of the outstanding Data Buffers (both send and receive all of the outstanding Data Buffers (both send and receive
buffers), or: buffers), or:
CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ) CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ)
+ SUM(MaxPostedOnEachSRQ) + SUM(MaxPostedOnEachSRQ)
+ SUM(MaxPostedOnEachSQ) + SUM(MaxPostedOnEachSQ)
Where: Where:
skipping to change at page 34, line 23 skipping to change at page 35, line 38
specific Receive Queue. specific Receive Queue.
MaxPostedOnEachSRQ = the maximum number of requests which MaxPostedOnEachSRQ = the maximum number of requests which
can cause a completion that will be posted on a can cause a completion that will be posted on a
specific Shared Receive Queue. specific Shared Receive Queue.
MaxPostedOnEachSQ = the maximum number of requests which MaxPostedOnEachSQ = the maximum number of requests which
can cause a completion that will be posted on a can cause a completion that will be posted on a
specific Send Queue. specific Send Queue.
If the local peer must be able to completely fill the queues, or If the local ULP must be able to completely fill the queues, or
can not be trusted to observe a limit smaller than the queues, can not be trusted to observe a limit smaller than the queues,
then the CQ must be sized to accommodate the maximum number of then the CQ must be sized to accommodate the maximum number of
operations that it is possible to post at any one time. Thus the operations that it is possible to post at any one time. Thus the
equation becomes: equation becomes:
CQ_MIN_SIZE = SUM(SizeOfEachRQ) CQ_MIN_SIZE = SUM(SizeOfEachRQ)
+ SUM(SizeOfEachSRQ) + SUM(SizeOfEachSRQ)
+ SUM(SizeOfEachSQ) + SUM(SizeOfEachSQ)
Where: Where:
skipping to change at page 34, line 50 skipping to change at page 36, line 16
can cause a completion that can ever be posted can cause a completion that can ever be posted
on a specific Shared Receive Queue. on a specific Shared Receive Queue.
SizeOfEachSQ = the maximum number of requests which SizeOfEachSQ = the maximum number of requests which
can cause a completion that can ever be posted can cause a completion that can ever be posted
on a specific Send Queue. on a specific Send Queue.
Where MaxPosted*OnEach*Q and SizeOfEach*Q varies on a per Stream Where MaxPosted*OnEach*Q and SizeOfEach*Q varies on a per Stream
or per Shared Receive Queue basis. or per Shared Receive Queue basis.
If the ULP is sharing a CQ across multiple streams which do not If the ULP is sharing a CQ across multiple Streams which do not
share partial mutual trust, then the ULP MUST implement a share Partial Mutual Trust, then the ULP MUST implement a
mechanism to ensure that the Completion Queue can not overflow. mechanism to ensure that the Completion Queue can not overflow.
Note that it is possible to share CQs even if the Remote Peers Note that it is possible to share CQs even if the Remote Peers
accessing the CQs are untrusted if either of the above two accessing the CQs are untrusted if either of the above two
formulas are implemented. If the ULP can be trusted to not post formulas are implemented. If the ULP can be trusted to not post
more than MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and more than MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and
MaxPostedOnEachSQ, then the first formula applies. If the ULP can MaxPostedOnEachSQ, then the first formula applies. If the ULP can
not be trusted to obey the limit, then the second formula not be trusted to obey the limit, then the second formula
applies. applies.
7.5.2.4 Attacking the RDMA Read Request Queue 7.5.2.4 Attacking the RDMA Read Request Queue
If RDMA Read Request Queue resources are pooled across multiple If RDMA Read Request Queue resources are pooled across multiple
Streams, one attack is if the Local Peer attempts to unfairly Streams, one attack is if the local ULP attempts to unfairly
allocate RDMA Read Request Queue resources for its Streams. For allocate RDMA Read Request Queue resources for its Streams. For
example, the Local Peer attempts to allocate all available example, a local ULP attempts to allocate all available resources
resources on a specific RDMA Read Request Queue for its Streams, on a specific RDMA Read Request Queue for its Streams, thereby
thereby denying the resource to ULPs sharing the RDMA Read denying the resource to ULPs sharing the RDMA Read Request Queue.
Request Queue. The same type of argument applies even if the RDMA The same type of argument applies even if the RDMA Read Request
Read Request is not shared - but a Local Peer attempts to is not shared - but a local ULP attempts to allocate all of the
allocate all of the RNICs resource when the queue is created. RNIC's resources when the queue is created.
Thus access to interfaces that allocate RDMA Read Request Queue Thus access to interfaces that allocate RDMA Read Request Queue
entries MUST be restricted to a trusted Local Peer, such as a entries MUST be restricted to a trusted Local Peer, such as a
Privileged Resource Manager. The Privileged Resource Manager Privileged Resource Manager. The Privileged Resource Manager
SHOULD prevent a Local Peer from allocating more than its fair SHOULD prevent a local ULP from allocating more than its fair
share of resources. share of resources.
Another form of attack is if the Remote Peer sends more RDMA Read Another form of attack is if the Remote Peer sends more RDMA Read
Requests than the depth of the RDMA Read Request Queue at the Requests than the depth of the RDMA Read Request Queue at the
Local Peer. If the RDMA Read Request Queue is a shared resource, Local Peer. If the RDMA Read Request Queue is a shared resource,
this could corrupt the queue. If the queue is not shared, then this could corrupt the queue. If the queue is not shared, then
the worst case is that the current Stream is disabled. One the worst case is that the current Stream is no longer functional
approach to solving the shared RDMA Read Request Queue would be (e.g. torn down). One approach to solving the shared RDMA Read
to create thresholds, similar to those described in Section Request Queue would be to create thresholds, similar to those
7.5.2.1 Multiple Streams Sharing Receive Buffers on page 30. A described in Section 7.5.2.1 Multiple Streams Sharing Receive
simpler approach is to not share RDMA Read Request Queue Buffers on page 31. A simpler approach is to not share RDMA Read
resources amoung Streams or enforce hard limits of consumption Request Queue resources among Streams or enforce hard limits of
per Stream. Thus RDMA Read Request Queue resource consumption consumption per Stream. Thus RDMA Read Request Queue resource
MUST be controlled by the Privileged Resource Manager such that consumption MUST be controlled by the Privileged Resource Manager
RDMAP/DDP Streams which do not share Partial Mutual Trust do not such that RDMAP/DDP Streams which do not share Partial Mutual
share RDMA Read Request Queue resources. Trust do not share RDMA Read Request Queue resources.
If the issue is a bug in the Remote Peer’s implementation, and If the issue is a bug in the Remote PeerÆs implementation, but
not a malicious attack, the issue can be solved by requiring the not a malicious attack, the issue can be solved by requiring the
Remote Peers RNIC to throttle RDMA Read Requests. By properly Remote PeerÆs RNIC to throttle RDMA Read Requests. By properly
configuring the Stream at the Remote Peer through a trusted configuring the Stream at the Remote Peer through a trusted
agent, the RNIC can be made to not transmit RDMA Read Requests agent, the RNIC can be made to not transmit RDMA Read Requests
that exceed the depth of the RDMA Read Request Queue at the Local that exceed the depth of the RDMA Read Request Queue at the Local
Peer. If the Stream is correctly configured, and if the Remote Peer. If the Stream is correctly configured, and if the Remote
Peer submits more requests than the Local Peers RDMA Read Peer submits more requests than the Local PeerÆs RDMA Read
Request Queue can handle, the requests would be queued at the Request Queue can handle, the requests would be queued at the
Remote Peer’s RNIC until previous requests complete. If the Remote PeerÆs RNIC until previous requests complete. If the
Remote Peer’s Stream is not configured correctly, the RDMAP Remote PeerÆs Stream is not configured correctly, the RDMAP
Stream is terminated when more RDMA Read Requests arrive at the Stream is terminated when more RDMA Read Requests arrive at the
Local Peer than the Local Peer can handle (assuming the prior Local Peer than the Local Peer can handle (assuming the prior
paragraphs recommendation is implemented). Thus an RNIC paragraphÆs recommendation is implemented). Thus an RNIC
implementation SHOULD provide a mechanism to cap the number of implementation SHOULD provide a mechanism to cap the number of
outstanding RDMA Read Requests. The configuration of this limit outstanding RDMA Read Requests. The configuration of this limit
is outside the scope of this specification. is outside the scope of this document.
7.5.3 Resource Consumption by Idle ULPs 7.5.3 Resource Consumption by Idle ULPs
The simplest form of a DOS attack given a fixed amount of The simplest form of a DOS attack given a fixed amount of
resources is for the Remote Peer to create a RDMAP Stream to a resources is for the Remote Peer to create a RDMAP Stream to a
Local Peer, and request dedicated resources then do no actual Local Peer, and request dedicated resources then do no actual
work. This allows the Remote Peer to be very light weight (i.e. work. This allows the Remote Peer to be very light weight (i.e.
only negotiate resources, but do no data transfer) and consumes a only negotiate resources, but do no data transfer) and consumes a
disproportionate amount of resources in the server. disproportionate amount of resources at the Local Peer.
A general countermeasure for this style of attack is to monitor A general countermeasure for this style of attack is to monitor
active RDMAP Streams and if resources are getting low, reap the active RDMAP Streams and if resources are getting low, reap the
resources from RDMAP Streams that are not transferring data and resources from RDMAP Streams that are not transferring data and
possibly terminate the Stream. This would presumably be under possibly terminate the Stream. This would presumably be under
administrative control. administrative control.
Refer to Section 7.5.1 for the analysis and countermeasures for Refer to Section 7.5.1 for the analysis and countermeasures for
this style of attack on the following RNIC resources: Stream this style of attack on the following RNIC resources: Stream
Context Memory, Page Translation Tables and STag namespace. Context Memory, Page Translation Tables and STag namespace.
Note that some RNIC resources are not at risk of this type of Note that some RNIC resources are not at risk of this type of
attack from a Remote Peer because an attack requires the Remote attack from a Remote Peer because an attack requires the Remote
Peer to send messages in order to consume the resource. Receive Peer to send messages in order to consume the resource. Receive
Data Buffers, Completion Queue, and RDMA Read Request Queue Data Buffers, Completion Queue, and RDMA Read Request Queue
resources are examples. These resources are, however, at risk resources are examples. These resources are, however, at risk
from a Local Peer that attempts to allocate resources, then goes from a local ULP that attempts to allocate resources, then goes
idle. This could also be created if the ULP negotiates the idle. This could also be created if the ULP negotiates the
resource levels with the Remote Peer, which causes the Local Peer resource levels with the Remote Peer, which causes the Local Peer
to consume resources, however the Remote Peer never sends data to to consume resources, however the Remote Peer never sends data to
consume them. The general countermeasure described in this consume them. The general countermeasure described in this
section can be used to free resources allocated by an idle Local section can be used to free resources allocated by an idle Local
Peer. Peer.
7.5.4 Exercise of non-optimal code paths 7.5.4 Exercise of non-optimal code paths
Another form of DOS attack is to attempt to exercise data paths Another form of DOS attack is to attempt to exercise data paths
skipping to change at page 37, line 21 skipping to change at page 38, line 39
RDMAP Send with Invalidate or Send with SE and Invalidate RDMAP Send with Invalidate or Send with SE and Invalidate
Message. If the STag is only valid on the current Stream, then Message. If the STag is only valid on the current Stream, then
the only side effect is that the Remote Peer can no longer use the only side effect is that the Remote Peer can no longer use
the STag; thus there are no security issues. the STag; thus there are no security issues.
If the STag is valid across multiple Streams, then the Remote If the STag is valid across multiple Streams, then the Remote
Peer can prevent other Streams from using that STag by using the Peer can prevent other Streams from using that STag by using the
remote invalidate functionality. remote invalidate functionality.
Thus if RDDP Streams do not share Partial Mutual Trust (i.e. the Thus if RDDP Streams do not share Partial Mutual Trust (i.e. the
Remote Peer may attempt to invalidate the STag prematurely), the Remote Peer may attempt to remote invalidate the STag
ULP MUST NOT enable an STag which would be valid across multiple prematurely), the ULP MUST NOT enable an STag which would be
Streams. valid across multiple Streams.
7.5.6 Remote Peer attacking an Unshared CQ
The Remote Peer can attack an unshared CQ if the Local Peer does
not size the CQ correctly. For example, if the Local Peer enables
the CQ to handle completions of received buffers, and the receive
buffer queue is longer than the Completion Queue, then an
overflow can potentially occur. The effect on the attackerÆs
Stream is catastrophic. However if an RNIC does not have the
proper protections in place, then an attack to overflow the CQ
can also cause corruption and/or termination of an unrelated
Stream. Thus an RNIC MUST ensure that if a CQ overflows, any
Streams which do not use the CQ MUST remain unaffected.
7.6 Elevation of Privilege 7.6 Elevation of Privilege
The RDMAP/DDP Security Architecture explicitly differentiates The RDMAP/DDP Security Architecture explicitly differentiates
between three levels of privilege - Non-Privileged, Privileged, between three levels of privilege - Non-Privileged, Privileged,
and the Privileged Resource Manager. If a Non-Privileged ULP is and the Privileged Resource Manager. If a Non-Privileged ULP is
able to elevate its privilege level to a Privileged ULP, then able to elevate its privilege level to a Privileged ULP, then
mapping a physical address list to an STag can provide local and mapping a physical address list to an STag can provide local and
remote access to any physical address location on the node. If a remote access to any physical address location on the node. If a
Privileged Mode ULP is able to promote itself to be a Resource Privileged Mode ULP is able to promote itself to be a Resource
Manager, then it is possible for it to perform denial of service Manager, then it is possible for it to perform denial of service
type attacks where substantial amounts of local resources could type attacks where substantial amounts of local resources could
be consumed. be consumed.
In general, elevation of privilege is a local implementation In general, elevation of privilege is a local implementation
specific issue and thus outside the scope of this specification. specific issue and thus outside the scope of this document.
<TBD: make more general and include authentication etc>
There is one issue worth noting, however. If the RNIC There is one issue worth noting, however. If the RNIC
implementation, by some insecure mechanism (or implementation implementation, by some insecure mechanism (or implementation
defect), can enable a Remote Peer or un-trusted Local Peer to defect), can enable a Remote Peer or un-trusted local ULP to load
load firmware into the RNIC Engine, it is possible to use the firmware into the RNIC Engine, it is possible to use the RNIC to
RNIC to attack the host. Thus, an implementation MUST NOT enable attack the host. Thus, an RNIC implementation MUST NOT enable
firmware to be loaded on the RNIC Engine directly from a Remote firmware to be loaded on the RNIC Engine directly from an
Peer, unless the Remote Peer is properly authenticated (by a untrusted local ULP or Remote Peer, unless they are properly
mechanism outside the scope of this specification. The mechanism authenticated (by a mechanism outside the scope of this document.
presumably entails authenticating that the remote ULP has the The mechanism presumably entails authenticating that the remote
right to perform the update), and the update is done via a secure ULP has the right to perform the update), and the update is done
protocol, such as IPsec (See Section 8 Security Services for RDMA via a secure protocol, such as IPsec (See Section 8 Security
and DDP on page 38). Further, an implementation MUST NOT allow a Services for RDMAP and DDP on page 40).
Non-Privileged Local Peer to update firmware in the RNIC Engine.
8 Security Services for RDMA and DDP 8 Security Services for RDMAP and DDP
RDMA and DDP are used to control, read and write data buffers RDMAP and DDP are used to control, read and write data buffers
over IP networks. Therefore, the control and the data packets of over IP networks. Therefore, the control and the data packets of
these protocols are vulnerable to the spoofing, tampering and these protocols are vulnerable to the spoofing, tampering and
information disclosure attacks listed in Section 7. information disclosure attacks listed in Section 7.
Generally speaking, Stream confidentiality protects against Generally speaking, Stream confidentiality protects against
eavesdropping. Stream and/or session authentication and integrity eavesdropping. Stream and/or session authentication and integrity
protection is a counter measurement against various spoofing and protection is a counter measurement against various spoofing and
tampering attacks. The effectiveness of authentication and tampering attacks. The effectiveness of authentication and
integrity against a specific attack, depend on whether the integrity against a specific attack, depend on whether the
authentication is machine level authentication (as the one authentication is machine level authentication (as the one
skipping to change at page 39, line 36 skipping to change at page 41, line 36
authentication). authentication).
4. IPsec SA management (selector negotiation, options 4. IPsec SA management (selector negotiation, options
negotiation, create, delete, and rekeying). negotiation, create, delete, and rekeying).
Items 1 through 3 are accomplished in IKE Phase 1, while item 4 Items 1 through 3 are accomplished in IKE Phase 1, while item 4
is handled in IKE Phase 2. is handled in IKE Phase 2.
IKE phase 1 defines four authentication methods; three of them IKE phase 1 defines four authentication methods; three of them
require both sides to have certified signature or encryption require both sides to have certified signature or encryption
public keys; the forth require the side to exchange out-of-band a public keys; the fourth requires the side to exchange out-of-band
secret random string - called pre-shared-secret (PSS). a secret random string - called pre-shared-secret (PSS).
An IKE Phase 2 negotiation is performed to establish both an An IKE Phase 2 negotiation is performed to establish both an
inbound and an outbound IPsec SA. The traffic to be protected by inbound and an outbound IPsec SA. The traffic to be protected by
an IPsec SA is determined by a selector which has been proposed an IPsec SA is determined by a selector which has been proposed
by the IKE initiator and accepted by the IKE Responder. The IPsec by the IKE initiator and accepted by the IKE Responder. The IPsec
SA selector can be a "filter" or traffic classifier, defined as SA selector can be a "filter" or traffic classifier, defined as
the 5-tuple: <Source IP address, Destination IP address, the 5-tuple: <Source IP address, Destination IP address,
transport protocol (e.g. UDP/SCTP/TCP), Source port, Destination transport protocol (e.g. UDP/SCTP/TCP), Source port, Destination
port>. The successful establishment of a IKE Phase-2 SA results port>. The successful establishment of a IKE Phase-2 SA results
in the creation of two uni-directional IPsec SAs fully qualified in the creation of two uni-directional IPsec SAs fully qualified
by the tuple <Protocol (ESP/AH), destination address, SPI>. by the tuple <Protocol (ESP/AH), destination address, SPI>.
The session keys for each IPsec SA are derived from a master key, The session keys for each IPsec SA are derived from a master key,
typically via a MODP Diffie-Hellman computation. Rekeying of an typically via a MODP Diffie-Hellman computation. Rekeying of an
existing IPsec SA pair is accomplished by creating two new IPsec existing IPsec SA pair is accomplished by creating two new IPsec
SAs, making them active, and then optionally deleting the older SAs, making them active, and then optionally deleting the older
IPsec SA pair. Typically the new outbound SA is used immediately, IPsec SA pair. Typically the new outbound SA is used immediately,
and the old inbound SA is left active to receive packets for some and the old inbound SA is left active to receive packets for some
locally defined time, perhaps 30 seconds or 1 minute. Optionally, locally defined time, perhaps 30 seconds or 1 minute. Optionally,
rekeying can use Diffie-Helman for keying material generation. rekeying can use Diffie-Hellman for keying material generation.
8.1.2 Introduction to SSL Limitations on RDMAP 8.1.2 Introduction to SSL Limitations on RDMAP
SSL and TLS [RFC 2246] provide Stream authentication, integrity SSL and TLS [RFC 2246] provide Stream authentication, integrity
and confidentiality for TCP based ULPs. SSL supports one-way and confidentiality for TCP based ULPs. SSL supports one-way
(server only) or mutual certificates based authentication. (server only) or mutual certificates based authentication.
There are at least two limitations that make SSL underneath RDMAP There are at least two limitations that make SSL underneath RDMAP
less appropriate then IPsec for DDP/RDMA security: less appropriate than IPsec for DDP/RDMA security:
1. The maximum length supported by the TLS record layer protocol 1. The maximum length supported by the TLS record layer protocol
is 2^14 bytes - longer packets must be fragmented (as a is 2^14 bytes - longer packets must be fragmented (as a
comparison, the maximal length of an IPsec packet is comparison, the maximal length of an IPsec packet is
determined by the maximum length of an IP packet). determined by the maximum length of an IP packet).
2. SSL is a connection oriented protocol. If a stream cipher or 2. SSL is a connection oriented protocol. If a stream cipher or
block cipher in CBC mode is used for bulk encryption, then a block cipher in CBC mode is used for bulk encryption, then a
packet can be decrypted only after all the packets preceding packet can be decrypted only after all the packets preceding
it have already arrived. If SSL is used to protect DDP/RDMA it have already arrived. If SSL is used to protect DDP/RDMA
skipping to change at page 40, line 39 skipping to change at page 42, line 39
If SSL is layered on top of RDMAP or DDP, SSL does not protect If SSL is layered on top of RDMAP or DDP, SSL does not protect
the RDMAP and/or DDP headers. Thus a man-in-the-middle attack can the RDMAP and/or DDP headers. Thus a man-in-the-middle attack can
still occur by modifying the RDMAP/DDP header to incorrectly still occur by modifying the RDMAP/DDP header to incorrectly
place the data into the wrong buffer, thus effectively corrupting place the data into the wrong buffer, thus effectively corrupting
the data stream. the data stream.
8.1.3 ULPs Which Provide Security 8.1.3 ULPs Which Provide Security
ULPs which provide integrated security but wish to leverage ULPs which provide integrated security but wish to leverage
lower-layer protocol security should be aware of security lower-layer protocol security should be aware of security
concerns around correlating a specific channels security concerns around correlating a specific channelÆs security
mechanisms to the authentication performed by the ULP. See mechanisms to the authentication performed by the ULP. See
[NFSv4CHANNEL] for additional information on a promising approach [NFSv4CHANNEL] for additional information on a promising approach
called "channel binding". From [NFSv4CHANNEL]: called "channel binding". From [NFSv4CHANNEL]:
"The concept of channel bindings allows applications to "The concept of channel bindings allows applications to
prove that the end-points of two secure channels at prove that the end-points of two secure channels at
different network layers are the same by binding different network layers are the same by binding
authentication at one channel to the session protection at authentication at one channel to the session protection at
the other channel. The use of channel bindings allows the other channel. The use of channel bindings allows
applications to delegate session protection to lower layers, applications to delegate session protection to lower layers,
skipping to change at page 41, line 28 skipping to change at page 43, line 28
delete messages may be sent for idle SAs, as a means of keeping delete messages may be sent for idle SAs, as a means of keeping
the number of active Phase 2 SAs to a minimum. The receipt of an the number of active Phase 2 SAs to a minimum. The receipt of an
IKE Phase 2 delete message MUST NOT be interpreted as a reason IKE Phase 2 delete message MUST NOT be interpreted as a reason
for tearing down an DDP/RDMA Stream. Rather, it is preferable to for tearing down an DDP/RDMA Stream. Rather, it is preferable to
leave the Stream up, and if additional traffic is sent on it, to leave the Stream up, and if additional traffic is sent on it, to
bring up another IKE Phase 2 SA to protect it. This avoids the bring up another IKE Phase 2 SA to protect it. This avoids the
potential for continually bringing Streams up and down. potential for continually bringing Streams up and down.
Note that there are serious security issues if IPsec is not Note that there are serious security issues if IPsec is not
implemented end-to-end. For example, if IPsec is implemented as a implemented end-to-end. For example, if IPsec is implemented as a
tunnel in the middle of the network, any hosts between the peer tunnel in the middle of the network, any hosts between the Peer
and the IPsec tunneling device can freely attack the unprotected and the IPsec tunneling device can freely attack the unprotected
Stream. Stream.
9 Security considerations 9 Security considerations
This entire specification is focused on security considerations. This entire document is focused on security considerations.
10 References 10 References
10.1 Normative References 10.1 Normative References
[RFC2828] Shirley, R., "Internet Security Glossary", FYI 36, RFC [RFC2828] Shirley, R., "Internet Security Glossary", FYI 36, RFC
2828, May 2000. 2828, May 2000.
[DDP] Shah, H., J. Pinkerton, R.Recio, and P. Culley, "Direct [DDP] Shah, H., J. Pinkerton, R.Recio, and P. Culley, "Direct
Data Placement over Reliable Transports", Internet-Draft Data Placement over Reliable Transports", Internet-Draft Work
draft-ietf-rddp-ddp-01.txt, February 2003. in Progress draft-ietf-rddp-ddp-04.txt, December 2004.
[RDMAP] Recio, R., P. Culley, D. Garcia, J. Hilland, "An RDMA [RDMAP] Recio, R., P. Culley, D. Garcia, J. Hilland, "An RDMA
Protocol Specification", Internet-Draft draft-ietf-rddp- Protocol Specification", Internet-Draft Work in Progress
rdmap-01.txt, February 2003. draft-ietf-rddp-rdmap-03.txt, December 2004.
[RFC3723] Aboba B., et al, "Securing Block Storage Protocols over [RFC3723] Aboba B., et al, "Securing Block Storage Protocols over
IP", Internet draft (work in progress), RFC3723, April 2004. IP", Internet draft (work in progress), RFC3723, April 2004.
[SCTP] R. Stewart et al., "Stream Control Transmission Protocol", [SCTP] R. Stewart et al., "Stream Control Transmission Protocol",
RFC 2960, October 2000. RFC 2960, October 2000.
[TCP] Postel, J., "Transmission Control Protocol - DARPA Internet [TCP] Postel, J., "Transmission Control Protocol - DARPA Internet
Program Protocol Specification", RFC 793, September 1981. Program Protocol Specification", RFC 793, September 1981.
10.2 Informative References 10.2 Informative References
[IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor [IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor
Discovery Trust Models and threats", Internet-Draft draft- Discovery Trust Models and threats", Informational RFC,
ietf-send-psreq-01.txt, January 2003. RFC3756, May 2004.
[NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to [NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to
Secure Channels", Internet-Draft draft-ietf-nfsv4-channel- Secure Channels", Internet-Draft draft-ietf-nfsv4-channel-
bindings-02.txt, July 2004. bindings-02.txt, July 2004.
11 Appendix A: ULP Issues for RDDP Client/Server Protocols 11 Appendix A: ULP Issues for RDDP Client/Server Protocols
This section is a normative appendix to the specification that is This section is a normative appendix to the document that is
focused on client/server ULP implementation requirements to focused on client/server ULP implementation requirements to
ensure a secure server implementation. ensure a secure server implementation.
The prior sections outlined specific attacks and their The prior sections outlined specific attacks and their
countermeasures. This section summarizes the attacks and countermeasures. This section summarizes the attacks and
countermeasures that have been defined in the prior section which countermeasures that have been defined in the prior section which
are applicable to creation of a secure ULP (e.g. application) are applicable to creation of a secure ULP (e.g. application)
server. A ULP server is defined as a ULP which must be able to server. A ULP server is defined as a ULP which must be able to
communicate with many clients which do not trust each other and communicate with many clients which do not necessarily have a
ensure that each client can not attack another client through trust relationship with each other, and ensure that each client
server interactions. Further, the server may wish to use multiple can not attack another client through server interactions.
Streams to communicate with a specific client, and those Streams Further, the server may wish to use multiple Streams to
may share mutual trust. Note that this section assumes a communicate with a specific client, and those Streams may share
compliant RNIC and Privileged Resource Manager implementation - mutual trust. Note that this section assumes a compliant RNIC and
thus it focuses specifically on ULP server (e.g. application) Privileged Resource Manager implementation - thus it focuses
implementation issues. specifically on ULP server (e.g. application) implementation
issues.
All of the prior section's details on attacks and countermeasures All of the prior section's details on attacks and countermeasures
apply to the server. In some cases normative SHOULD statements apply to the server, thus requirements which are repeated in this
for the ULP in the main body of this document are made MUST section use non-normative "must", "should", "may". In some cases
statements for the ULP because the operating conditions can be normative SHOULD statements for the ULP from the main body of
refined to make the motives for a SHOULD inapplicable. If a prior this document are made MUST statements for the ULP server because
SHOULD is changed to a MUST in this section, it is explicitly the operating conditions can be refined to make the motives for a
noted. SHOULD inapplicable. If a prior SHOULD is changed to a MUST in
this section, it is explicitly noted and it uses upper-case
normative statements.
The following list summarizes the relevent attacks that clients The following list summarizes the relevant attacks that clients
can mount on the shared server, by re-stating the previous can mount on the shared server, by re-stating the previous
normative statements to be client/server specific: normative statements to be client/server specific. Note that each
client/server ULP may employ explicit RDMA operations (RDMA Read,
RDMA Write) in differing fashions. Therefore where appropriate,
"Local ULP", "Local Peer" and "Remote Peer" are used in place of
"server" or "client", in order to retain full generality of each
requirement.
* Spoofing * Spoofing
* Sections 7.2.1 to 7.2.3. For protection against many * Sections 7.2.1 to 7.2.3. For protection against many
forms of spoofing attacks, enable IPsec. forms of spoofing attacks, enable IPsec.
* Section 7.2.4 Using an STag on a Different Stream on * Section 7.2.4 Using an STag on a Different Stream on
page 23. To ensure that one client can not access page 24. To ensure that one client can not access
another client's data via use of the other client's another client's data via use of the other client's
STag, the server ULP MUST either scope an STag to a STag, the server ULP must either scope an STag to a
single Stream or use a Protection Domain per client. single Stream or use a unique Protection Domain per
If a single client has multiple streams that share client. If a single client has multiple Streams that
Partial Mutual Trust, then the STag can be shared share Partial Mutual Trust, then the STag can be
between the associated Streams by using a single shared between the associated Streams by using a
Protection Domain amoung the associated Streams (see single Protection Domain among the associated Streams
for additional issues). To prevent unintended sharing (see section 8.1.3 ULPs Which Provide Security on
of STags within the associated Streams, a server ULP page 42 for additional issues). To prevent unintended
SHOULD use STags in such a fashion that it is sharing of STags within the associated Streams, a
difficult to predict the next allocated STag number. server ULP should use STags in such a fashion that it
is difficult to predict the next allocated STag
number.
* Tampering * Tampering
* 7.3.2 Modifying a Buffer After Indication on page 25. * 7.3.2 Modifying a Buffer After Indication on page 26.
Before the server ULP operates on a buffer that was Before the local ULP operates on a buffer that was
written using an RDMA Write or RDMA Read, the server written by the Remote Peer using an RDMA Write or
MUST ensure the the buffer can no longer be modified RDMA Read, the local ULP MUST ensure the buffer can
by invalidating the STag for remote access (note that no longer be modified, by invalidating the STag for
this is stronger than the SHOULD in section 7.3.2). remote access (note that this is stronger than the
This can either be done explicitly by revoking remote SHOULD in section 7.3.2). This can either be done
access rights for the STag when the client indicates explicitly by revoking remote access rights for the
the operation has completed, or by checking to make STag when the Remote Peer indicates the operation has
sure the client Invalidated the STag through the completed, or by checking to make sure the Remote
RDMAP Invalidate capability, and if it did not, the Peer Invalidated the STag through the RDMAP
Local Peer then explicitly revokes the STag remote Invalidate capability, and if it did not, the local
access rights. ULP then explicitly revoking the STag remote access
rights.
* Information Disclosure * Information Disclosure
* 7.4.2 Using RDMA Read to Access Stale Data on page * 7.4.2 Using RDMA Read to Access Stale Data on page
26. A server ULP MUST (this is stronger than the 27. In a general purpose server environment there is
no compelling rationale to not require a buffer to be
initialized before remote read is enabled (and an
enormous down side of unintentionally sharing data).
Thus a local ULP MUST (this is stronger than the
SHOULD in section 7.4.2) ensure that no stale data is SHOULD in section 7.4.2) ensure that no stale data is
contained in a buffer before remote read access contained in a buffer before remote read access
rights are granted to a client (this can be done by rights are granted to a Remote Peer (this can be done
zeroing the contents of the memory, for example). by zeroing the contents of the memory, for example).
* 7.4.3 Accessing a Buffer After the Transfer on page * 7.4.3 Accessing a Buffer After the Transfer on page
26. This mitigation is already covered by section 28. This mitigation is already covered by section
7.3.2 (above). 7.3.2 (above).
* 7.4.4 Accessing Unintended Data With a Valid STag on * 7.4.4 Accessing Unintended Data With a Valid STag on
page 26. The ULP server MUST set the base and bounds page 28. The ULP must set the base and bounds of the
of the buffer when the STag is initialized to expose buffer when the STag is initialized to expose only
only the data to be retrieved. the data to be retrieved.
* 7.4.5 RDMA Read into an RDMA Write Buffer on page 27. * 7.4.5 RDMA Read into an RDMA Write Buffer on page 28.
If a server only intends a buffer to be exposed for If a peer only intends a buffer to be exposed for
remote write access, it MUST set the access rights to remote write access, it must set the access rights to
the buffer to only enable remote write access. the buffer to only enable remote write access.
* 7.4.6 Using Multiple STags Which Alias to the Same * 7.4.6 Using Multiple STags Which Alias to the Same
Buffer on page 27. The requirement in section 7.2.4 Buffer on page 29. The requirement in section 7.2.4
(above) mitigates this attack. A buffer is exposed to (above) mitigates this attack. A server buffer is
only one client at a time to ensure that no exposed to only one client at a time to ensure that
information disclosure or information tampering no information disclosure or information tampering
occurs between peers. occurs between peers.
* 7.4.9 Network based eavesdropping on page 28. Enable * 7.4.9 Network based eavesdropping on page 30.
IPsec if this threat is a concern. Confidentiality services should be enabled by the ULP
if this threat is a concern.
* Denial of Service * Denial of Service
* 7.5.2.1 Multiple Streams Sharing Receive Buffers on * 7.5.2.1 Multiple Streams Sharing Receive Buffers on
page 30. ULP memory footprint size can be important page 31. ULP memory footprint size can be important
for some server ULPs. If a server ULP is expecting for some server ULPs. If a server ULP is expecting
significant network traffic from multiple clients, significant network traffic from multiple clients,
using a receive buffer queue per Stream where there using a receive buffer queue per Stream where there
is a large number of Streams can consume substantial is a large number of Streams can consume substantial
amounts of memory. Thus a receive queue that can be amounts of memory. Thus a receive queue that can be
shared by multiple Streams is attractive. shared by multiple Streams is attractive.
However, because of the attacks outlined in this However, because of the attacks outlined in this
section, sharing a single receive queue between section, sharing a single receive queue between
multiple clients MUST only be done if the mechanism multiple clients must only be done if a mechanism is
to ensure one client can't consume too many receive in place to ensure one client cannot consume receive
buffers is enabled. For multiple Streams within a buffers in excess of its limits, as defined by each
single client ULP (which presumably shared partial ULP. For multiple Streams within a single client ULP
mutual trust) this added overhead does not have to be (which presumably shared Partial Mutual Trust) this
enabled. added overhead may be avoided.
* 7.5.2.2 Local ULP Attacking a Shared CQ on page 31.
<TBD>The normative mitigations were
* RNIC MUST NOT enable sharing a CQ across Streams * 7.5.2.2 Local ULP Attacking a Shared CQ on page 33.
that belong to different Protection Domains. The normative RNIC mitigations require the RNIC to
not enable sharing of a CQ if the local ULPs do not
share Partial Mutual Trust. Thus while the ULP is not
allowed to enable this feature in an unsafe mode, if
the two local ULPs share Partial Tutual Trust, they
must behave in the following manner:
* A ULP SHOULD NOT share a CQ between Streams which 1) The sizing of the completion queue is based on the
do not share Partial Mutual Trust. size of the receive queue and send queues as
documented in 7.5.2.3 Local or Remote Peer Attacking
a Shared CQ on page 33.
Because the attack is a local server ULP attacking 2) The local ULP ensures that CQ entries are reaped
another server ULP, frequently enough to adhere to section 7.5.2.3's
rules.
* 7.5.2.3 Remote Peer Attacking a Shared CQ on page 32. * 7.5.2.3 Local or Remote Peer Attacking a Shared CQ on
There are two mitigations specified in this section - page 33. There are two mitigations specified in this
one requires a worst-case size of the CQ, and can be section - one requires a worst-case size of the CQ,
implemented entirely within the Privileged Resource and can be implemented entirely within the Privileged
Manager. The second approach requires cooperation Resource Manager. The second approach requires
with the local ULP server (to not post too many cooperation with the local ULP server (to not post
buffers), and enables a smaller CQ to be used. too many buffers), and enables a smaller CQ to be
used.
In some server environments, partial trust of the In some server environments, partial trust of the
server ULP (but not the clients) is acceptable, thus server ULP (but not the clients) is acceptable, thus
the smaller CQ fully mitigates the remote attacker. the smaller CQ fully mitigates the remote attacker.
In other environments, the local server ULP could In other environments, the local server ULP could
also contain untrusted elements which can attack the also contain untrusted elements which can attack the
local machine (or have bugs). In those environments, local machine (or have bugs). In those environments,
the worst-case size of the CQ must be used. the worst-case size of the CQ must be used.
* 7.5.2.4 The section requires a Privileged Resource * 7.5.2.4 The section requires a serverÆs Privileged
Manager to not enable sharing of RDMA Read Request Resource Manager to not allow sharing of RDMA Read
Queues across multiple Streams that do not share Request Queues across multiple Streams that do not
partial mutual trust. However, because the server ULP share Partial Mutual Trust, for a ULP which performs
knows best which of its Streams share partial mutual RDMA Read operations to server buffers. However,
trust, this requirement can be reflected back to the because the server ULP knows best which of its
ULP. The ULP (i.e. server) requirement is that it Streams share Partial Mutual Trust, this requirement
MUST NOT request RDMA Read Request Queues to be can be reflected back to the ULP. The ULP (i.e.
shared between ULPs which do not have partial mutual server) requirement in this case is that it MUST NOT
trust. allow RDMA Read Request Queues to be shared between
ULPs which do not have Partial Mutual Trust.
* 7.5.5 Remote Invalidate an STag Shared on Multiple * 7.5.5 Remote Invalidate an STag Shared on Multiple
Streams on page 37. This mitigation is already Streams on page 38. This mitigation is already
covered by section 7.3.2 (above). covered by section 7.3.2 (above).
12 Appendix B: Summary of RNIC and ULP Implementation Requirements 12 Appendix B: Summary of RNIC and ULP Implementation Requirements
This appendix is informative.
Below is a summary of implementation requirements for the RNIC: Below is a summary of implementation requirements for the RNIC:
* 5 Trust and Resource Sharing
* 7.2.4 Using an STag on a Different Stream
* 7.3.1 Buffer Overrun - RDMA Write or Read Response * 7.3.1 Buffer Overrun - RDMA Write or Read Response
* 7.3.2 Modifying a Buffer After Indication
* 7.4.8 Controlling Access to PTT & STag Mapping * 7.4.8 Controlling Access to PTT & STag Mapping
* 7.5.1 RNIC Resource Consumption * 7.5.1 RNIC Resource Consumption
* 7.5.2.1 Multiple Streams Sharing Receive Buffers * 7.5.2.1 Multiple Streams Sharing Receive Buffers
* 7.5.2.2 Local ULP Attacking a Shared CQ * 7.5.2.2 Local ULP Attacking a Shared CQ
* 7.5.2.3 Remote Peer Attacking a Shared CQ * 7.5.2.3 Local or Remote Peer Attacking a Shared CQ
* 7.5.2.4 Attacking the RDMA Read Request Queue * 7.5.2.4 Attacking the RDMA Read Request Queue
* 7.5.4 Exercise of non-optimal code paths * 7.5.6 Remote Peer attacking an Unshared CQ on page 38.
* 7.6 Elevation of Privilege * 7.6 Elevation of Privilege 39
* 8.2 Requirements for IPsec Encapsulation of DDP * 8.2 Requirements for IPsec Encapsulation of DDP
Below is a summary of implementation requirements for the ULP Below is a summary of implementation requirements for the ULP
above the RNIC: above the RNIC:
* 7.2.4 Using an STag on a Different Stream * 7.2.4 Using an STag on a Different Stream
* 7.3.2 Modifying a Buffer After Indication * 7.3.2 Modifying a Buffer After Indication
skipping to change at page 48, line 49 skipping to change at page 51, line 4
* 7.4.4 Accessing Unintended Data With a Valid STag * 7.4.4 Accessing Unintended Data With a Valid STag
* 7.4.5 RDMA Read into an RDMA Write Buffer * 7.4.5 RDMA Read into an RDMA Write Buffer
* 7.4.6 Using Multiple STags Which Alias to the Same Buffer * 7.4.6 Using Multiple STags Which Alias to the Same Buffer
* 7.4.9 Network based eavesdropping * 7.4.9 Network based eavesdropping
* 7.5.2.2 Local ULP Attacking a Shared CQ * 7.5.2.2 Local ULP Attacking a Shared CQ
* 7.5.5 Remote Invalidate an STag Shared on Multiple * 7.5.5 Remote Invalidate an STag Shared on Multiple
Streams Streams
13 Appendix C: Partial Trust Taxonomy 13 Appendix C: Partial Trust Taxonomy
This appendix is informative.
Partial Trust is defined as when one party is willing to assume Partial Trust is defined as when one party is willing to assume
that another party will refrain from a specific attack or set of that another party will refrain from a specific attack or set of
attacks, the parties are said to be in a state of Partial Trust. attacks, the parties are said to be in a state of Partial Trust.
Note that the partially trusted peer may attempt a different set Note that the partially trusted peer may attempt a different set
of attacks. This may be appropriate for many ULPs where any of attacks. This may be appropriate for many ULPs where any
adverse effects of the betrayal is easily confined and does not adverse effects of the betrayal is easily confined and does not
place other clients or ULPs at risk. place other clients or ULPs at risk.
The Trust Models described in this section have three primary The Trust Models described in this section have three primary
distinguishing characteristics. The Trust Model refers to a Local distinguishing characteristics. The Trust Model refers to a local
Peer and Remote Peer, which are the local and remote ULP ULP and Remote Peer, which are intended to be the local and
instances communicating via RDMA/DDP. remote ULP instances communicating via RDMA/DDP.
* Local Resource Sharing (yes/no) - When local resources * Local Resource Sharing (yes/no) - When local resources
are shared, they are shared across a grouping of are shared, they are shared across a grouping of
RDMAP/DDP Streams. If local resources are not shared, the RDMAP/DDP Streams. If local resources are not shared, the
resources are dedicated on a per Stream basis. Resources resources are dedicated on a per Stream basis. Resources
are defined in Section 4.2 - Resources on page 11. The are defined in Section 4.2 - Resources on page 12. The
advantage of not sharing resources between Streams is advantage of not sharing resources between Streams is
that it reduces the types of attacks that are possible. that it reduces the types of attacks that are possible.
The disadvantage is that ULPs might run out of resources. The disadvantage is that ULPs might run out of resources.
* Local Partial Trust (yes/no) - Local Partial Trust is * Local Partial Trust (yes/no) - Local Partial Trust is
determined based on whether the local grouping of determined based on whether the local grouping of
RDMAP/DDP Streams (which typically equates to one ULP or RDMAP/DDP Streams (which typically equates to one ULP or
group of ULPs) mutually trust each other to not perform a group of ULPs) mutually trust each other to not perform a
specific set of attacks. specific set of attacks.
* Remote Partial Trust (yes/no) - The Remote Partial Trust * Remote Partial Trust (yes/no) - The Remote Partial Trust
level is determined based on whether the Local Peer of a level is determined based on whether the local ULP of a
specific RDMAP/DDP Stream partially trusts the Remote specific RDMAP/DDP Stream partially trusts the Remote
Peer of the Stream (see the definition of Partial Trust Peer of the Stream (see the definition of Partial Trust
in Section 3 Introduction). in Section 3 Introduction).
Not all of the combinations of the trust characteristics are Not all of the combinations of the trust characteristics are
expected to be used by ULPs. This paper specifically analyzes expected to be used by ULPs. This document specifically analyzes
five ULP Trust Models that are expected to be in common use. The five ULP Trust Models that are expected to be in common use. The
Trust Models are as follows: Trust Models are as follows:
* NS-NT - Non-Shared Local Resources, no Local Trust, no * NS-NT - Non-Shared Local Resources, no Local Trust, no
Remote Trust - typically a server ULP that wants to run Remote Trust - typically a server ULP that wants to run
in the safest mode possible. All attack mitigations are in the safest mode possible. All attack mitigations are
in place to ensure robust operation. in place to ensure robust operation.
* NS-RT - Non-Shared Local Resources, no Local Trust, * NS-RT - Non-Shared Local Resources, no Local Trust,
Remote Partial Trust - typically a peer-to-peer ULP, Remote Partial Trust - typically a peer-to-peer ULP,
which has, by some method outside of the scope of this which has, by some method outside of the scope of this
specification, authenticated the Remote Peer. Note that document, authenticated the Remote Peer. Note that unless
unless some form of key based authentication is used on a some form of key based authentication is used on a per
per RDMA/DDP Stream basis, it may not be possible be RDMA/DDP Stream basis, it may not be possible be possible
possible for man-in-the-middle attacks to occur. See for man-in-the-middle attacks to occur. See section 8,
section 8, Security Services for RDMA and DDP on page 38. Security Services for RDMAP and DDP on page 40.
* S-NT - Shared Local Resources, no Local Trust, no Remote * S-NT - Shared Local Resources, no Local Trust, no Remote
Trust - typically a server ULP that runs in an untrusted Trust - typically a server ULP that runs in an untrusted
environment where the amount of resources required is environment where the amount of resources required is
either too large or too dynamic to dedicate for each either too large or too dynamic to dedicate for each
RDMAP/DDP Stream. RDMAP/DDP Stream.
* S-LT - Shared Local Resources, Local Partial Trust, no * S-LT - Shared Local Resources, Local Partial Trust, no
Remote Trust - typically a ULP, which provides a session Remote Trust - typically a ULP, which provides a session
layer and uses multiple Streams, to provide additional layer and uses multiple Streams, to provide additional
skipping to change at page 51, line 36 skipping to change at page 53, line 38
performance requirements, the application typically performance requirements, the application typically
authenticates with all of its peers and then runs in a authenticates with all of its peers and then runs in a
highly trusted environment. The application peers are all highly trusted environment. The application peers are all
in a single application fault domain and depend on one in a single application fault domain and depend on one
another to be well-behaved when accessing data another to be well-behaved when accessing data
structures. If a trusted Remote Peer has an structures. If a trusted Remote Peer has an
implementation defect that results in poor behavior, the implementation defect that results in poor behavior, the
entire application could be corrupted. entire application could be corrupted.
Models NS-NT and S-NT above are typical for Internet networking - Models NS-NT and S-NT above are typical for Internet networking -
neither Local Peers nor the Remote Peer is trusted. Sometimes neither local ULPs nor the Remote Peer is trusted. Sometimes
optimizations can be done that enable sharing of Page Translation optimizations can be done that enable sharing of Page Translation
Tables across multiple Local Peers, thus Model S-LT can be Tables across multiple local ULPs, thus Model S-LT can be
advantageous. Model S-T is typically used when resource scaling advantageous. Model S-T is typically used when resource scaling
across a large parallel ULP makes it infeasible to use any other across a large parallel ULP makes it infeasible to use any other
model. Resource scaling issues can either be due to performance model. Resource scaling issues can either be due to performance
around scaling or because there simply are not enough resources. around scaling or because there simply are not enough resources.
Model NS-RT is probably the least likely model to be used, but is Model NS-RT is probably the least likely model to be used, but is
presented for completeness. presented for completeness.
14 Authors Addresses 14 AuthorÆs Addresses
James Pinkerton James Pinkerton
Microsoft Corporation Microsoft Corporation
One Microsoft Way One Microsoft Way
Redmond, WA. 98052 USA Redmond, WA. 98052 USA
Phone: +1 (425) 705-5442 Phone: +1 (425) 705-5442
Email: jpink@windows.microsoft.com Email: jpink@windows.microsoft.com
Ellen Deleganes Ellen Deleganes
Intel Corporation Intel Corporation
skipping to change at page 54, line 7 skipping to change at page 56, line 7
Bernard Aboba Bernard Aboba
Microsoft Corporation Microsoft Corporation
One Microsoft Way One Microsoft Way
Redmond, WA. 98052 USA Redmond, WA. 98052 USA
Phone: +1 (425) 706-6606 Phone: +1 (425) 706-6606
Email: bernarda@windows.microsoft.com Email: bernarda@windows.microsoft.com
16 Full Copyright Statement 16 Full Copyright Statement
Copyright (C) The Internet Society (2001). All Rights Reserved. Copyright (C) The Internet Society (2004).
This document and translations of it may be copied and furnished This document is subject to the rights, licenses and restrictions
to others, and derivative works that comment on or otherwise contained in BCP 78 and except as set forth therein, the authors
explain it or assist in its implementation may be prepared, retain all their rights.
copied, published and distributed, in whole or in part, without
restriction of any kind, provided that the above copyright notice
and this paragraph are included on all such copies and derivative
works. However, this document itself may not be modified in any
way, such as by removing the copyright notice or references to
the Internet Society or other Internet organizations, except as
needed for the purpose of developing Internet standards in which
case the procedures for copyrights defined in the Internet
Standards process must be followed, or as required to translate
it into languages other than English.
The limited permissions granted above are perpetual and will not This document and the information contained herein are provided
be revoked by the Internet Society or its successors or assigns. on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE.
This document and the information contained herein is provided on Intellectual Property
an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR The IETF takes no position regarding the validity or scope of any
IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE Intellectual Property Rights or other rights that might be
OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY claimed to pertain to the implementation or use of the technology
IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR described in this document or the extent to which any license
PURPOSE. under such rights might or might not be available; nor does it
represent that it has made any independent effort to identify any
such rights. Information on the procedures with respect to
rights in RFC documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the
use of such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR
repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention
any copyrights, patents or patent applications, or other
proprietary rights that may cover technology that may be required
to implement this standard. Please address the information to
the IETF at ietf-ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is currently provided by the Funding for the RFC Editor function is currently provided by the
Internet Society. Internet Society.
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/