draft-ietf-rddp-security-04.txt   draft-ietf-rddp-security-05.txt 
Internet Draft James Pinkerton Internet Draft James Pinkerton
draft-ietf-rddp-security-04.txt Microsoft Corporation draft-ietf-rddp-security-05.txt Microsoft Corporation
Category: Standards Track Ellen Deleganes Category: Standards Track Ellen Deleganes
Expires: February, 2005 Intel Corporation Expires: February, 2005 Intel Corporation
Sara Bitan Sara Bitan
Microsoft Corporation Microsoft Corporation
August 2004 August 2004
DDP/RDMAP Security DDP/RDMAP Security
1 Status of this Memo 1 Status of this Memo
skipping to change at page 2, line ? skipping to change at page 2, line ?
disclosure, denial of service, and elevation of privilege. disclosure, denial of service, and elevation of privilege.
Finally, the document concludes with a summary of security Finally, the document concludes with a summary of security
services for DDP and RDMAP, such as IPsec. services for DDP and RDMAP, such as IPsec.
J. Pinkerton, et al. Expires February 2005 1 J. Pinkerton, et al. Expires February 2005 1
Table of Contents Table of Contents
1 Status of this Memo.........................................1 1 Status of this Memo.........................................1
2 Abstract....................................................1 2 Abstract....................................................1
2.1 Revision History............................................3 2.1 Revision History............................................3
2.1.1 Changes from -02 to -03 version............................3 2.1.1 Changes from -04 to -05 version............................3
2.1.2 Changes from the -01 to the -02 version....................5 2.1.2 Changes from -03 to -04 version............................4
2.1.3 Changes from the -00 to -01 version........................5 2.1.3 Changes from -02 to -03 version............................4
3 Introduction................................................6 2.1.4 Changes from the -01 to the -02 version....................5
4 Architectural Model.........................................8 2.1.5 Changes from the -00 to -01 version........................5
4.1 Components..................................................9 3 Introduction................................................7
4 Architectural Model.........................................9
4.1 Components.................................................10
4.2 Resources..................................................11 4.2 Resources..................................................11
4.2.1 Stream Context Memory.....................................11 4.2.1 Stream Context Memory.....................................11
4.2.2 Data Buffers..............................................11 4.2.2 Data Buffers..............................................11
4.2.3 Page Translation Tables...................................11 4.2.3 Page Translation Tables...................................12
4.2.4 STag Namespace............................................12 4.2.4 STag Namespace............................................12
4.2.5 Completion Queues.........................................12 4.2.5 Completion Queues.........................................12
4.2.6 Asynchronous Event Queue..................................12 4.2.6 Asynchronous Event Queue..................................13
4.2.7 RDMA Read Request Queue...................................13 4.2.7 RDMA Read Request Queue...................................13
4.2.8 RNIC Interactions.........................................13 4.2.8 RNIC Interactions.........................................13
4.2.8.1 Privileged Control Interface Semantics................13 4.2.8.1 Privileged Control Interface Semantics................13
4.2.8.2 Non-Privileged Data Interface Semantics...............13 4.2.8.2 Non-Privileged Data Interface Semantics...............14
4.2.8.3 Privileged Data Interface Semantics...................14 4.2.8.3 Privileged Data Interface Semantics...................14
4.2.9 Initialization of RNIC Data Structures for Data Transfer..14 4.2.9 Initialization of RNIC Data Structures for Data Transfer..14
4.2.10 RNIC Data Transfer Interactions.........................15 4.2.10 RNIC Data Transfer Interactions.........................16
5 Trust and Resource Sharing.................................17 5 Trust and Resource Sharing.................................17
6 Attacker Capabilities......................................18 6 Attacker Capabilities......................................18
7 Attacks and Countermeasures................................19 7 Attacks and Countermeasures................................19
7.1 Tools for Countermeasures..................................19 7.1 Tools for Countermeasures..................................19
7.1.1 Protection Domain (PD)....................................19 7.1.1 Protection Domain (PD)....................................19
7.1.2 Limiting STag Scope.......................................20 7.1.2 Limiting STag Scope.......................................20
7.1.3 Access Rights.............................................21 7.1.3 Access Rights.............................................21
7.1.4 Limiting the Scope of the Completion Queue................21 7.1.4 Limiting the Scope of the Completion Queue................21
7.1.5 Limiting the Scope of an Error............................21 7.1.5 Limiting the Scope of an Error............................21
7.2 Spoofing...................................................21 7.2 Spoofing...................................................21
skipping to change at page 3, line 8 skipping to change at page 3, line 10
7.4.2 Using RDMA Read to Access Stale Data......................26 7.4.2 Using RDMA Read to Access Stale Data......................26
7.4.3 Accessing a Buffer After the Transfer.....................26 7.4.3 Accessing a Buffer After the Transfer.....................26
7.4.4 Accessing Unintended Data With a Valid STag...............26 7.4.4 Accessing Unintended Data With a Valid STag...............26
7.4.5 RDMA Read into an RDMA Write Buffer.......................27 7.4.5 RDMA Read into an RDMA Write Buffer.......................27
7.4.6 Using Multiple STags Which Alias to the Same Buffer.......27 7.4.6 Using Multiple STags Which Alias to the Same Buffer.......27
7.4.7 Remote Node Loading Firmware onto the RNIC................28 7.4.7 Remote Node Loading Firmware onto the RNIC................28
7.4.8 Controlling Access to PTT & STag Mapping..................28 7.4.8 Controlling Access to PTT & STag Mapping..................28
7.4.9 Network based eavesdropping...............................28 7.4.9 Network based eavesdropping...............................28
7.5 Denial of Service (DOS)....................................29 7.5 Denial of Service (DOS)....................................29
7.5.1 RNIC Resource Consumption.................................29 7.5.1 RNIC Resource Consumption.................................29
7.5.2 Resource Consumption By Active Applications...............30 7.5.2 Resource Consumption By Active ULPs.......................30
7.5.2.1 Multiple Streams Sharing Receive Buffers..............30 7.5.2.1 Multiple Streams Sharing Receive Buffers..............30
7.5.2.2 Local Peer Attacking a Shared CQ......................31 7.5.2.2 Local ULP Attacking a Shared CQ.......................31
7.5.2.3 Remote Peer Attacking a Shared CQ.....................32 7.5.2.3 Remote Peer Attacking a Shared CQ.....................32
7.5.2.4 Attacking the RDMA Read Request Queue.................35 7.5.2.4 Attacking the RDMA Read Request Queue.................35
7.5.3 Resource Consumption by Idle Applications.................36 7.5.3 Resource Consumption by Idle ULPs.........................36
7.5.4 Exercise of non-optimal code paths........................36 7.5.4 Exercise of non-optimal code paths........................36
7.5.5 Remote Invalidate an STag Shared on Multiple Streams......37 7.5.5 Remote Invalidate an STag Shared on Multiple Streams......37
7.6 Elevation of Privilege.....................................37 7.6 Elevation of Privilege.....................................37
8 Security Services for RDMA and DDP.........................38 8 Security Services for RDMA and DDP.........................38
8.1 Introduction to Security Options...........................38 8.1 Introduction to Security Options...........................38
8.1.1 Introduction to IPsec.....................................38 8.1.1 Introduction to IPsec.....................................38
8.1.2 Introduction to SSL Limitations on RDMAP..................40 8.1.2 Introduction to SSL Limitations on RDMAP..................40
8.1.3 Applications Which Provide Security.......................40 8.1.3 ULPs Which Provide Security...............................40
8.2 Requirements for IPsec Encapsulation of DDP................40 8.2 Requirements for IPsec Encapsulation of DDP................41
9 Security considerations....................................42 9 Security considerations....................................42
10 References.................................................43 10 References.................................................43
10.1 Normative References......................................43 10.1 Normative References......................................43
10.2 Informative References....................................43 10.2 Informative References....................................43
11 Appendix A: Implementing Client/Server Protocols...........44 11 Appendix A: ULP Issues for RDDP Client/Server Protocols....44
12 Appendix B: Summary Table of Attacks.......................48 12 Appendix B: Summary of RNIC and ULP Implementation
Requirements.....................................................48
13 Appendix C: Partial Trust Taxonomy.........................50 13 Appendix C: Partial Trust Taxonomy.........................50
14 AuthorÆs Addresses.........................................52 14 Authors Addresses.........................................52
15 Acknowledgments............................................53 15 Acknowledgments............................................53
16 Full Copyright Statement...................................54 16 Full Copyright Statement...................................54
Table of Figures Table of Figures
Figure 1 - RDMA Security Model....................................9 Figure 1 - RDMA Security Model...................................10
2.1 Revision History 2.1 Revision History
2.1.1 Changes from -03 to -04 version 2.1.1 Changes from -04 to -05 version
* Small modifications to normative statements per phone
call review.
* 4.1 - Moved MUST statement from Privileged Resource
Manager to section 5. Also added additional normative
statements around resource sharing and assumptions of
who trusts whom.
* 7.2.4 - changed last paragraph SHOULD to should.
* 7.4.4 - changed last paragraph MUST to SHOULD.
* 7.5.2.2 - clarified it is the ULP at issue, and
removed reference to Protection Domain - key issue is
whether they share partial mutual trust.
* 7.5.2.4 - remove MUST statement at the end of the 3rd
paragraph - it was replaced with a more general MUST
in section <TBD>. Also changed the cap on the number
of outstanding RDMA Read Requests at the sender to a
SHOULD (from MUST).
* 8.1 - first paragraph after enumerated list. Change
MAY to may. It is a ULP issue.
* Removed "application" from the document and replaced it
with "ULP". In some cases also changed "Local Peer" to
ULP to clarify what the text meant.
2.1.2 Changes from -03 to -04 version
* Removed "issues" section because all issues have been * Removed "issues" section because all issues have been
resolved. resolved.
* Completed section "Applications Which Provide Security" * Completed section "ULPs Which Provide Security" by
by providing a cross reference to channel bindings. providing a cross reference to channel bindings.
* Substantial rewrite of Section 11 Appendix A: * Substantial rewrite of Section 11 Appendix A: ULP Issues
Implementing Client/Server Protocols. Retargeted it to for RDDP Client/Server Protocols. Retargeted it to focus
focus on server application requirements, rather than on server application requirements, rather than RNIC
RNIC requirements. requirements.
* Changed "IPSec" to "IPsec" everywhere to match the RFC. * Changed "IPSec" to "IPsec" everywhere to match the RFC.
* Added new ULP requirement in section 7.5.2.4 Attacking * Added new ULP requirement in section 7.5.2.4 Attacking
the RDMA Read Request Queue. the RDMA Read Request Queue.
* Reviesed Sectio 12 Appendix B: Summary of RNIC and ULP * Reviesed Sectio 12 Appendix B: Summary of RNIC and ULP
Implementation Requirements slightly to add one ULP Implementation Requirements slightly to add one ULP
requirement and one RNIC requirement which is stated in requirement and one RNIC requirement which is stated in
the document but was missed in this summary. the document but was missed in this summary.
2.1.2 Changes from -02 to -03 version 2.1.3 Changes from -02 to -03 version
* ID changed from Informational to Standards Track. This * ID changed from Informational to Standards Track. This
caused previous RECOMMENDATIONS to be categorized into caused previous RECOMMENDATIONS to be categorized into
the categories of MUST, SHOULD, MAY, RECOMMENDED, and in the categories of MUST, SHOULD, MAY, RECOMMENDED, and in
one case, "recommended". one case, "recommended".
* Completed Appendix B: Summary of Attacks to provide a * Completed Appendix B: Summary of Attacks to provide a
summary of implementation requirements for applications summary of implementation requirements for applications
using RDDP and for RNICs in Appendix B: Summary of using RDDP and for RNICs in Appendix B: Summary of
Attacks. Attacks.
skipping to change at page 5, line 5 skipping to change at page 5, line 38
* Changed section 8.2 to normative xref to IPS Security, * Changed section 8.2 to normative xref to IPS Security,
plus comment on the value of end-to-end IPsec. plus comment on the value of end-to-end IPsec.
* Added clarifying example on STag invalidation (e.g. One- * Added clarifying example on STag invalidation (e.g. One-
Shot STag discussion). Shot STag discussion).
* Added clarifying text on why SSL is a bad idea. * Added clarifying text on why SSL is a bad idea.
* Normative statement on mitigation for Shared RQ. * Normative statement on mitigation for Shared RQ.
2.1.3 Changes from the -01 to the -02 version 2.1.4 Changes from the -01 to the -02 version
Minimal - some typos, deleted some text previously marked for Minimal - some typos, deleted some text previously marked for
deletion. deletion.
2.1.4 Changes from the -00 to -01 version 2.1.5 Changes from the -00 to -01 version
* Added two pages to the architectural model to describe * Added two pages to the architectural model to describe
the Asynchronous Event Queue, and the types of the Asynchronous Event Queue, and the types of
interactions that can occur between the RNIC and the interactions that can occur between the RNIC and the
modules above it. modules above it.
* Addressed Mike Krauses comments submitted on 12/8/2003 * Addressed Mike Krauses comments submitted on 12/8/2003
* Moved "Trust Models" from the body of the document to an * Moved "Trust Models" from the body of the document to an
appendix. Removed references to it throughout the appendix. Removed references to it throughout the
skipping to change at page 6, line 11 skipping to change at page 7, line 11
* Globally tried to change "connection" to "Stream". In * Globally tried to change "connection" to "Stream". In
some cases it can be either a connection or stream. some cases it can be either a connection or stream.
3 Introduction 3 Introduction
RDMA enables new levels of flexibility when communicating between RDMA enables new levels of flexibility when communicating between
two parties compared to current conventional networking practice two parties compared to current conventional networking practice
(e.g. a stream-based model or datagram model). This flexibility (e.g. a stream-based model or datagram model). This flexibility
brings new security issues that must be carefully understood when brings new security issues that must be carefully understood when
designing application protocols utilizing RDMA and when designing Upper Layer Protocols (ULPs) utilizing RDMA and when
implementing RDMA-aware NICs (RNICs). Note that for the purposes implementing RDMA-aware NICs (RNICs). Note that for the purposes
of this security analysis, an RNIC may implement RDMAP and DDP, of this security analysis, an RNIC may implement RDMAP and DDP,
or just DDP. or just DDP. Also, a ULP may be an application or it may be a
middleware library.
The specification first develops an architectural model that is The specification first develops an architectural model that is
relevant for the security analysis - it details components, relevant for the security analysis - it details components,
resources, and system properties that may be attacked in Section resources, and system properties that may be attacked in Section
4. 4.
It then defines what resources a ULP may share locally across It then defines what resources a ULP may share locally across
Streams and what resources the ULP may share with the Remote Peer Streams and what resources the ULP may share with the Remote Peer
across Streams in Section 5. Intentional sharing of resources across Streams in Section 5. Intentional sharing of resources
between multiple Streams may imply some level of trust between between multiple Streams may imply some level of trust between
the Streams. However, some types of resource sharing have the Streams. However, some types of resource sharing have
unmitigated security attacks which would mandate not sharing a unmitigated security attacks which would mandate not sharing a
specific type of resource unless there is some level of trust specific type of resource unless there is some level of trust
between the Streams sharing resources. Partial Mutual Trust is between the Streams sharing resources. Partial Mutual Trust is
defined to address this concept: defined to address this concept:
Partial Mutual Trust - a collection of RDMAP/DDP Streams, Partial Mutual Trust - a collection of RDMAP/DDP Streams,
which represent the local and remote end points of the which represent the local and remote end points of the
Stream, are willing to assume that the Streams from the Stream, are willing to assume that the Streams from the
collection will not perform malicious attacks against any of collection will not perform malicious attacks against any of
the Streams in the collection. Applications have explicit the Streams in the collection. ULPs have explicit control of
control of which collection of endpoints is in the which collection of endpoints is in the collection through
collection through tools discussed in Section 7.1 Tools for tools discussed in Section 7.1 Tools for Countermeasures on
Countermeasures on page 19. page 19.
An untrusted peer relationship is appropriate when an application An untrusted peer relationship is appropriate when a ULP wishes
wishes to ensure that it will be robust and uncompromised even in to ensure that it will be robust and uncompromised even in the
the face of a deliberate attack by its peer. For example, a face of a deliberate attack by its peer. For example, a single
single application that concurrently supports multiple unrelated ULP that concurrently supports multiple unrelated Streams (e.g. a
Streams (e.g. a server) would presumably treat each of its peers server) would presumably treat each of its peers as an untrusted
as an untrusted peer. For a collection of Streams which share peer. For a collection of Streams which share Partial Mutual
Partial Mutual Trust, the assumption is that any Stream not in Trust, the assumption is that any Stream not in the collection is
the collection is untrusted. For the untrusted peer, a brief list untrusted. For the untrusted peer, a brief list of capabilities
of capabilities is enumerated in Section 6. is enumerated in Section 6.
The rest of the specification is focused on analyzing attacks. The rest of the specification is focused on analyzing attacks.
First, the tools for mitigating attacks are listed (Section 7.1), First, the tools for mitigating attacks are listed (Section 7.1),
and then a series of attacks on components, resources, or system and then a series of attacks on components, resources, or system
properties is enumerated in the rest of Section 7. For each properties is enumerated in the rest of Section 7. For each
attack, possible countermeasures are reviewed. If all recommended attack, possible countermeasures are reviewed. If all recommended
mitigations are in place the implemented usage models, the mitigations are in place the implemented usage models, the
RDMAP/DDP protocol can be shown to not expose any new security RDMAP/DDP protocol can be shown to not expose any new security
vulnerabilities. vulnerabilities.
Applications within a host are divided into two categories - ULPs within a host are divided into two categories - Privileged
Privileged and Non-Privileged. Both application types can send and Non-Privileged. Both ULP types can send and receive data and
and receive data and request resources. The key differences request resources. The key differences between the two are:
between the two are:
The Privileged Application is trusted by the system to not The Privileged ULP is trusted by the system to not
maliciously attack the operating environment, but it is not maliciously attack the operating environment, but it is not
trusted to optimize resource allocation globally. For trusted to optimize resource allocation globally. For
example, the Privileged Application could be a kernel example, the Privileged ULP could be a kernel ULP, thus the
application, thus the kernel presumably has in some way kernel presumably has in some way vetted the ULP before
vetted the application before allowing it to execute. allowing it to execute.
A Non-Privileged ApplicationÆs capabilities are a logical A Non-Privileged ULP's capabilities are a logical sub-set of
sub-set of the Privileged ApplicationÆs. It is assumed by the Privileged ULP's. It is assumed by the local system that
the local system that a Non-Privileged Application is a Non-Privileged ULP is untrusted. All Non-Privileged ULP
untrusted. All Non-Privileged Application interactions with interactions with the RNIC Engine that could affect other
the RNIC Engine that could affect other applications need to ULPs need to be done through a trusted intermediary that can
be done through a trusted intermediary that can verify the verify the Non-Privileged ULP requests.
Non-Privileged Application requests.
4 Architectural Model 4 Architectural Model
This section describes an RDMA architectural reference model that This section describes an RDMA architectural reference model that
is used as security issues are examined. It introduces the is used as security issues are examined. It introduces the
components of the model, the resources that can be attacked, the components of the model, the resources that can be attacked, the
types of interactions possible between components and resources, types of interactions possible between components and resources,
and the system properties which must be preserved. and the system properties which must be preserved.
Figure 1 shows the components comprising the architecture and the Figure 1 shows the components comprising the architecture and the
interfaces where potential security attacks could be launched. interfaces where potential security attacks could be launched.
External attacks can be injected into the system from an External attacks can be injected into the system from a ULP that
application that sits above the RNIC Interface or from the sits above the RNIC Interface or from the network.
network.
The intent here is to describe high level components and The intent here is to describe high level components and
capabilities which affect threat analysis, and not focus on capabilities which affect threat analysis, and not focus on
specific implementation options. Also note that the architectural specific implementation options. Also note that the architectural
model is an abstraction, and an actual implementation may choose model is an abstraction, and an actual implementation may choose
to subdivide its components along different boundary lines than to subdivide its components along different boundary lines than
defined here. For example, the Privileged Resource Manager may be defined here. For example, the Privileged Resource Manager may be
partially or completely encapsulated in the Privileged partially or completely encapsulated in the Privileged ULP.
Application. Regardless, it is expected that the security Regardless, it is expected that the security analysis of the
analysis of the potential threats and countermeasures still potential threats and countermeasures still apply.
apply.
+-------------+ +-------------+
| Privileged | | Privileged |
| Resource | | Resource |
Admin<-+>| Manager | App Control Interface Admin<-+>| Manager | ULP Control Interface
| | |<------+-------------------+ | | |<------+-------------------+
| +-------------+ | | | +-------------+ | |
| ^ v v | ^ v v
| | +-------------+ +-----------------+ | | +-------------+ +-----------------+
|---------------->| Privileged | | Non-Privileged | |---------------->| Privileged | | Non-Privileged |
| | Application | | Application | | | ULP | | ULP |
| +-------------+ +-----------------+ | +-------------+ +-----------------+
| ^ ^ | ^ ^
|Privileged |Privileged |Non-Privileged |Privileged |Privileged |Non-Privileged
|Control |Data |Data |Control |Data |Data
|Interface |Interface |Interface |Interface |Interface |Interface
RNIC | | | RNIC | | |
Interface v v v Interface v v v
================================================================= =================================================================
+--------------------------------------+ +--------------------------------------+
skipping to change at page 9, line 48 skipping to change at page 10, line 18
The components shown in Figure 1 - RDMA Security Model are: The components shown in Figure 1 - RDMA Security Model are:
* RNIC Engine (RNIC) - the component that implements the * RNIC Engine (RNIC) - the component that implements the
RDMA protocol and/or DDP protocol. RDMA protocol and/or DDP protocol.
* Privileged Resource Manager - the component responsible * Privileged Resource Manager - the component responsible
for managing and allocating resources associated with the for managing and allocating resources associated with the
RNIC Engine. The Resource Manager does not send or RNIC Engine. The Resource Manager does not send or
receive data. Note that whether the Resource Manager is receive data. Note that whether the Resource Manager is
an independent component, part of the RNIC, or part of an independent component, part of the RNIC, or part of
the application is implementation dependent. If a the ULP is implementation dependent. If a specific
specific implementation does not wish to address security implementation does not wish to address security issues
issues resolved by the Resource Manager, there may in resolved by the Resource Manager, there may in fact be no
fact be no resource manager at all. resource manager at all.
* Privileged Application - See Section 3 Introduction for a * Privileged ULP - See Section 3 Introduction for a
definition of Privileged Application. The local host definition of Privileged ULP. The local host
infrastructure can enable the Privileged Application to infrastructure can enable the Privileged ULP to map a
map a data buffer directly from the RNIC Engine to the data buffer directly from the RNIC Engine to the host
host through the RNIC Interface, but it does not allow through the RNIC Interface, but it does not allow the
the Privileged Application to directly consume RNIC Privileged ULP to directly consume RNIC Engine resources.
Engine resources.
* Non-Privileged Application - See Section 3 Introduction * Non-Privileged ULP - See Section 3 Introduction for a
for a definition of Non-Privileged Application. All Non- definition of Non-Privileged ULP.
Privileged Application interactions with the RNIC Engine
that could affect other applications MUST be done using
the Privileged Resource Manager as a proxy.
A design goal of the DDP and RDMAP protocols is to allow, under A design goal of the DDP and RDMAP protocols is to allow, under
constrained conditions, Non-Privileged applications to send and constrained conditions, Non-Privileged ULP to send and receive
receive data directly to/from the RDMA Engine without Privileged data directly to/from the RDMA Engine without Privileged Resource
Resource Manager intervention - while ensuring that the host Manager intervention - while ensuring that the host remains
remains secure. Thus, one of the primary goals of this paper is secure. Thus, one of the primary goals of this paper is to
to analyze this usage model for the enforcement that is required analyze this usage model for the enforcement that is required in
in the RNIC Engine to ensure the system remains secure. the RNIC Engine to ensure the system remains secure.
The host interfaces that could be exercised include: The host interfaces that could be exercised include:
* Privileged Control Interface - A Privileged Resource * Privileged Control Interface - A Privileged Resource
Manager uses the RNIC Interface to allocate and manage Manager uses the RNIC Interface to allocate and manage
RNIC Engine resources, control the state within the RNIC RNIC Engine resources, control the state within the RNIC
Engine, and monitor various events from the RNIC Engine. Engine, and monitor various events from the RNIC Engine.
It also uses this interface to act as a proxy for some It also uses this interface to act as a proxy for some
operations that a Non-Privileged Application may require operations that a Non-Privileged ULP may require (after
(after performing appropriate countermeasures). performing appropriate countermeasures).
* Application Control Interface - An application uses this * ULP Control Interface - An ULP uses this interface to the
interface to the Privileged Resource Manager to allocate Privileged Resource Manager to allocate RNIC Engine
RNIC Engine resources. The Privileged Resource Manager resources. The Privileged Resource Manager implements
implements countermeasures to ensure that if the Non- countermeasures to ensure that if the Non-Privileged ULP
Privileged Application launches an attack it can prevent launches an attack it can prevent the attack from
the attack from affecting other applications. affecting other ULPs.
* Non-Privileged Data Transfer Interface - A Non-Privileged * Non-Privileged Data Transfer Interface - A Non-Privileged
Application uses this interface to initiate and to check ULP uses this interface to initiate and to check the
the status of data transfer operations. status of data transfer operations.
* Privileged Data Transfer Interface - A superset of the * Privileged Data Transfer Interface - A superset of the
functionality provided by the Non-Privileged Data functionality provided by the Non-Privileged Data
Transfer Interface. The application is allowed to Transfer Interface. The ULP is allowed to directly
directly manipulate RNIC Engine mapping resources to map manipulate RNIC Engine mapping resources to map an STag
an STag to an application data buffer. to a ULP data buffer.
* Figure 1 also shows the ability to load new firmware in * Figure 1 also shows the ability to load new firmware in
the RNIC Engine. Not all RNICs will support this, but it the RNIC Engine. Not all RNICs will support this, but it
is shown for completeness and is also reviewed under is shown for completeness and is also reviewed under
potential attacks. potential attacks.
If Internet control messages, such as ICMP, ARP, RIPv4, etc. are If Internet control messages, such as ICMP, ARP, RIPv4, etc. are
processed by the RNIC Engine, the threat analyses for those processed by the RNIC Engine, the threat analyses for those
protocols is also applicable, but outside the scope of this protocols is also applicable, but outside the scope of this
paper. paper.
skipping to change at page 11, line 56 skipping to change at page 12, line 21
a threat analysis perspective it is assumed that a single STag a threat analysis perspective it is assumed that a single STag
enables access to a single logical Data Buffer. enables access to a single logical Data Buffer.
In any event, it is the responsibility of the RNIC to ensure that In any event, it is the responsibility of the RNIC to ensure that
no STag can be created that exposes memory that the consumer had no STag can be created that exposes memory that the consumer had
no authority to expose. no authority to expose.
4.2.3 Page Translation Tables 4.2.3 Page Translation Tables
Page Translation Tables are the structures used by the RNIC to be Page Translation Tables are the structures used by the RNIC to be
able to access application memory for data transfer operations. able to access ULP memory for data transfer operations. Even
though these structures are called "Page" Translation Tables,
Even though these structures are called "Page" Translation they may not reference a page at all - conceptually they are used
Tables, they may not reference a page at all - conceptually they to map a ULP address space representation of a buffer to the
are used to map an application address space representation of a physical addresses that are used by the RNIC Engine to move data.
buffer to the physical addresses that are used by the RNIC Engine If on a specific system a mapping is not used, then a subset of
to move data. If on a specific system a mapping is not used, then the attacks examined may be appropriate. Note that the Page
a subset of the attacks examined may be appropriate. Note that Translation Table may or may not be a shared resource.
the Page Translation Table may or may not be a shared resource.
4.2.4 STag Namespace 4.2.4 STag Namespace
The DDP specification defines a 32-bit namespace for the STag. The DDP specification defines a 32-bit namespace for the STag.
Implementations may vary in terms of the actual number of STags Implementations may vary in terms of the actual number of STags
that are supported. In any case, this is a bounded resource that that are supported. In any case, this is a bounded resource that
can come under attack. Depending upon STag namespace allocation can come under attack. Depending upon STag namespace allocation
algorithms, the actual name space to attack may be significantly algorithms, the actual name space to attack may be significantly
less than 2^32. less than 2^32.
4.2.5 Completion Queues 4.2.5 Completion Queues
Completion Queues are used in this specification to conceptually Completion Queues are used in this specification to conceptually
represent how the RNIC Engine notifies the Application about the represent how the RNIC Engine notifies the ULP about the
completion of the transmission of data, or the completion of the completion of the transmission of data, or the completion of the
reception of data through the Data Transfer Interface. Because reception of data through the Data Transfer Interface. Because
there could be many transmissions or receptions in flight at any there could be many transmissions or receptions in flight at any
one time, completions are modeled as a queue rather than a single one time, completions are modeled as a queue rather than a single
event. An implementation may also use the Completion Queue to event. An implementation may also use the Completion Queue to
notify the application of other activities, for example, the notify the ULP of other activities, for example, the completion
completion of a mapping of an STag to a specific application of a mapping of an STag to a specific ULP buffer. Completion
buffer. Completion Queues may be shared by a group of Streams, or Queues may be shared by a group of Streams, or may be designated
may be designated to handle a specific Stream's traffic. to handle a specific Stream's traffic.
Some implementations may allow this queue to be manipulated Some implementations may allow this queue to be manipulated
directly by both Non-Privileged and Privileged applications. directly by both Non-Privileged and Privileged ULPs.
4.2.6 Asynchronous Event Queue 4.2.6 Asynchronous Event Queue
The Asynchronous Event Queue is a queue from the RNIC to the The Asynchronous Event Queue is a queue from the RNIC to the
Privileged Resource Manager of bounded size. It is used by the Privileged Resource Manager of bounded size. It is used by the
RNIC to notify the host of various events which might require RNIC to notify the host of various events which might require
management action, including protocol violations, Stream state management action, including protocol violations, Stream state
changes, local operation errors, low water marks on receive changes, local operation errors, low water marks on receive
queues, and possibly other events. queues, and possibly other events.
skipping to change at page 13, line 26 skipping to change at page 13, line 44
4.2.8 RNIC Interactions 4.2.8 RNIC Interactions
With RNIC resources and interfaces defined, it is now possible to With RNIC resources and interfaces defined, it is now possible to
examine the interactions supported by the generic RNIC functional examine the interactions supported by the generic RNIC functional
interfaces through each of the 3 interfaces - Privileged Control interfaces through each of the 3 interfaces - Privileged Control
Interface, Privileged Data Interface, and Non-Privileged Data Interface, Privileged Data Interface, and Non-Privileged Data
Interface. Interface.
4.2.8.1 Privileged Control Interface Semantics 4.2.8.1 Privileged Control Interface Semantics
Generically, the Privileged Control Interface controls the RNICÆs Generically, the Privileged Control Interface controls the RNICs
allocation, deallocation, and initialization of RNIC global allocation, deallocation, and initialization of RNIC global
resources. This includes allocation and deallocation of Stream resources. This includes allocation and deallocation of Stream
Context Memory, Page Translation Tables, STag names, Completion Context Memory, Page Translation Tables, STag names, Completion
Queues, RDMA Read Request Queues, and Asynchronous Event Queues. Queues, RDMA Read Request Queues, and Asynchronous Event Queues.
The Privileged Control Interface is also typically used for The Privileged Control Interface is also typically used for
managing Non-Privileged Application resources for the Non- managing Non-Privileged ULP resources for the Non-Privileged ULP
Privileged Application (and possibly for the Privileged (and possibly for the Privileged ULP as well). This includes
Application as well). This includes initialization and removal of initialization and removal of Page Translation Table resources,
Page Translation Table resources, and managing RNIC events and managing RNIC events (possibly managing all events for the
(possibly managing all events for the Asynchronous Event Queue). Asynchronous Event Queue).
4.2.8.2 Non-Privileged Data Interface Semantics 4.2.8.2 Non-Privileged Data Interface Semantics
The Non-Privileged Data Interface enables data transfer (transmit The Non-Privileged Data Interface enables data transfer (transmit
and receive) but does not allow initialization of the Page and receive) but does not allow initialization of the Page
Translation Table resources. However, once the Page Translation Translation Table resources. However, once the Page Translation
Table resources have been initialized, the interface may enable a Table resources have been initialized, the interface may enable a
specific STag mapping to be enabled and disabled by directly specific STag mapping to be enabled and disabled by directly
communicating with the RNIC, or create an STag mapping for a communicating with the RNIC, or create an STag mapping for a
buffer that has been previously initialized in the RNIC. buffer that has been previously initialized in the RNIC.
skipping to change at page 14, line 13 skipping to change at page 14, line 31
for external write access through advertisement of an STag. for external write access through advertisement of an STag.
For DDP, transmitting data means sending DDP Tagged or Untagged For DDP, transmitting data means sending DDP Tagged or Untagged
Messages. For data reception, for DDP it can receive Untagged Messages. For data reception, for DDP it can receive Untagged
Messages into buffers that have been posted on the Receive Queue Messages into buffers that have been posted on the Receive Queue
or Shared Receive Queue. It can also receive Tagged DDP Messages or Shared Receive Queue. It can also receive Tagged DDP Messages
into buffers that have previously been exposed for external write into buffers that have previously been exposed for external write
access through advertisement of an STag. access through advertisement of an STag.
Completion of data transmission or reception generally entails Completion of data transmission or reception generally entails
informing the application of the completed work by placing informing the ULP of the completed work by placing completion
completion information on the Completion Queue. information on the Completion Queue.
4.2.8.3 Privileged Data Interface Semantics 4.2.8.3 Privileged Data Interface Semantics
The Privileged Data Interface semantics are a superset of the The Privileged Data Interface semantics are a superset of the
Non-Privileged Data Transfer semantics. The interface can do Non-Privileged Data Transfer semantics. The interface can do
everything defined in the prior section, as well as everything defined in the prior section, as well as
create/destroy buffer to STag mappings directly. This generally create/destroy buffer to STag mappings directly. This generally
entails initialization or clearing of Page Translation Table entails initialization or clearing of Page Translation Table
state in the RNIC. state in the RNIC.
skipping to change at page 14, line 43 skipping to change at page 15, line 8
b. Initialization of a mapping from an allocated STag name b. Initialization of a mapping from an allocated STag name
to a set of Page Translation Table entry(s) or partial- to a set of Page Translation Table entry(s) or partial-
entries. entries.
Note that an implementation may not have a Page Translation Table Note that an implementation may not have a Page Translation Table
(i.e. it may support a direct mapping between an STag and a Data (i.e. it may support a direct mapping between an STag and a Data
Buffer). In this case threats and mitigations associated with the Buffer). In this case threats and mitigations associated with the
Page Translation Table are not relevant. Page Translation Table are not relevant.
Initialization of the contents of the Page Translation Table can Initialization of the contents of the Page Translation Table can
be done by either the Privileged Application or by the Privileged be done by either the Privileged ULP or by the Privileged
Resource Manager as a proxy for the Non-Privileged Application. Resource Manager as a proxy for the Non-Privileged ULP. By
By definition the Non-Privileged Application is not trusted to definition the Non-Privileged ULP is not trusted to directly
directly manipulate the Page Translation Table. In general the manipulate the Page Translation Table. In general the concern is
concern is that the Non-Privileged application may try to that the Non-Privileged ULP may try to maliciously initialize the
maliciously initialize the Page Translation Table to access a Page Translation Table to access a buffer for which it does not
buffer for which it does not have permission. have permission.
The exact resource allocation algorithm for the Page Translation The exact resource allocation algorithm for the Page Translation
Table is outside the scope of this specification. It may be Table is outside the scope of this specification. It may be
allocated for a specific Data Buffer, or be allocated as a pooled allocated for a specific Data Buffer, or be allocated as a pooled
resource to be consumed by potentially multiple Data Buffers, or resource to be consumed by potentially multiple Data Buffers, or
be managed in some other way. This paper attempts to abstract be managed in some other way. This paper attempts to abstract
implementation dependent issues, and focus on higher level implementation dependent issues, and focus on higher level
security issues such as resource starvation and sharing of security issues such as resource starvation and sharing of
resources between Streams. resources between Streams.
The next issue is how an STag name is associated with a Data The next issue is how an STag name is associated with a Data
Buffer. For the case of an Untagged Data Buffer, there is no wire Buffer. For the case of an Untagged Data Buffer, there is no wire
visible mapping between an STag and the Data Buffer. Note that visible mapping between an STag and the Data Buffer. Note that
there may, in fact, be an STag which represents the buffer. there may, in fact, be an STag which represents the buffer.
However, because the STag by definition is not visible on the However, because the STag by definition is not visible on the
wire, this is a local host specific issue which should be wire, this is a local host specific issue which should be
analyzed in the context of local host implementation specific analyzed in the context of local host implementation specific
security analysis, and thus is outside the scope of this paper. security analysis, and thus is outside the scope of this paper.
For a Tagged Data Buffer, either the Privileged Application, the For a Tagged Data Buffer, either the Privileged ULP, the Non-
Non-Privileged Application, or the Privileged Resource Manager Privileged ULP, or the Privileged Resource Manager acting on
acting on behalf of the Non-Privileged Resource Manager may behalf of the Non-Privileged Resource Manager may initialize a
initialize a mapping from an STag to a Page Translation Table, or mapping from an STag to a Page Translation Table, or may have the
may have the ability to simply enable/disable an existing STag to ability to simply enable/disable an existing STag to Page
Page Translation Table mapping. There may also be multiple STag Translation Table mapping. There may also be multiple STag names
names which map to a specific group of Page Translation Table which map to a specific group of Page Translation Table entries
entries (or sub-entries). Specific security issues with this (or sub-entries). Specific security issues with this level of
level of flexibility are examined in Section 7.3.3 Multiple STags flexibility are examined in Section 7.3.3 Multiple STags to
to access the same buffer on page 25. access the same buffer on page 25.
There are a variety of implementation options for initialization There are a variety of implementation options for initialization
of Page Translation Table entries and mapping an STag to a group of Page Translation Table entries and mapping an STag to a group
of Page Translation Table entries which have security of Page Translation Table entries which have security
repercussions. This includes support for separation of Mapping an repercussions. This includes support for separation of Mapping an
STag verses mapping a set of Page Translation Table entries, and STag verses mapping a set of Page Translation Table entries, and
support for Applications directly manipulating STag to Page support for ULPs directly manipulating STag to Page Translation
Translation Table entry mappings (verses requiring access through Table entry mappings (verses requiring access through the
the Privileged Resource Manager). Privileged Resource Manager).
4.2.10 RNIC Data Transfer Interactions 4.2.10 RNIC Data Transfer Interactions
RNIC Data Transfer operations can be subdivided into send RNIC Data Transfer operations can be subdivided into send
operations and receive operations. operations and receive operations.
For send operations, there is typically a queue that enables the For send operations, there is typically a queue that enables the
Application to post multiple operations to send data (referred to ULP to post multiple operations to send data (referred to as the
as the Send Queue). Depending upon the implementation, Data Send Queue). Depending upon the implementation, Data Buffers used
Buffers used in the operations may or may not have Page in the operations may or may not have Page Translation Table
Translation Table entries associated with them, and may or may entries associated with them, and may or may not have STags
not have STags associated with them. Because this is a local host associated with them. Because this is a local host specific
specific implementation issue rather than a protocol issue, the implementation issue rather than a protocol issue, the security
security analysis of threats and mitigations is left to the host analysis of threats and mitigations is left to the host
implementation. implementation.
Receive operations are different for Tagged Data Buffers verses Receive operations are different for Tagged Data Buffers verses
Untagged Data Buffers. If more than one Untagged Data Buffer can Untagged Data Buffers. If more than one Untagged Data Buffer can
be posted by the Application, the DDP specification requires that be posted by the ULP, the DDP specification requires that they be
they be consumed in sequential order. Thus the most general consumed in sequential order. Thus the most general
implementation is that there is a sequential queue of receive implementation is that there is a sequential queue of receive
Untagged Data Buffers (Receive Queue). Some implementations may Untagged Data Buffers (Receive Queue). Some implementations may
also support sharing of the sequential queue between multiple also support sharing of the sequential queue between multiple
Streams. In this case defining "sequential" becomes non-trivial - Streams. In this case defining "sequential" becomes non-trivial -
in general the buffers for a single stream are consumed from the in general the buffers for a single stream are consumed from the
queue in the order that they were placed on the queue, but there queue in the order that they were placed on the queue, but there
is no order guarantee between streams. is no order guarantee between streams.
For receive Tagged Data Buffers, at some time prior to data For receive Tagged Data Buffers, at some time prior to data
transfer, the mapping of the STag to specific Page Translation transfer, the mapping of the STag to specific Page Translation
skipping to change at page 17, line 14 skipping to change at page 17, line 14
5 Trust and Resource Sharing 5 Trust and Resource Sharing
It is assumed that in general the Local and Remote Peer are It is assumed that in general the Local and Remote Peer are
untrusted, and thus attacks by either should have mitigations in untrusted, and thus attacks by either should have mitigations in
place. place.
A separate, but related issue is resource sharing between A separate, but related issue is resource sharing between
multiple streams. If local resources are not shared, the multiple streams. If local resources are not shared, the
resources are dedicated on a per Stream basis. Resources are resources are dedicated on a per Stream basis. Resources are
defined in Section 4.2 - Resources on page 10. The advantage of defined in Section 4.2 Resources on page 10. The advantage of not
not sharing resources between Streams is that it reduces the sharing resources between Streams is that it reduces the types of
types of attacks that are possible. The disadvantage is that attacks that are possible. The disadvantage is that ULPs might
applications might run out of resources. run out of resources.
It is assumed in this paper that the component that implements It is assumed in this paper that the component that implements
the mechanism to control sharing of the RNIC Engine resources is the mechanism to control sharing of the RNIC Engine resources is
the Privileged Resource Manager. The RNIC Engine exposes its the Privileged Resource Manager. The RNIC Engine exposes its
resources through the RNIC Interface to the Privileged Resource resources through the RNIC Interface to the Privileged Resource
Manager. All Privileged and Non-Privileged applications request Manager. All Privileged and Non-Privileged ULPs request resources
resources from the Resource Manager. The Resource Manager from the Resource Manager (note that by definition both the Non-
implements resource management policies to ensure fair access to Privileged and the Privileged application might try to greedily
resources. The Resource Manager should be designed to take into consume resources, thus creating a potential Denial of Service
account security attacks detailed in this specification. Note (DOS) attack. The Resource Manager implements resource management
that for some systems the Privileged Resource Manager may be policies to ensure fair access to resources. The Resource Manager
implemented within the Privileged Application. should be designed to take into account security attacks detailed
in this specification. Note that for some systems the Privileged
Resource Manager may be implemented within the Privileged ULP.
All Non-Privileged ULP interactions with the RNIC Engine that
could affect other ULPs MUST be done using the Privileged
Resource Manager as a proxy. All ULP resource allocation requests
for scarce resources MUST also be done using a Privileged
Resource Manager.
The sharing of resources across Streams should be under the The sharing of resources across Streams should be under the
control of the application, both in terms of the trust model the control of the ULP, both in terms of the trust model the ULP
application wishes to operate under, as well as the level of wishes to operate under, as well as the level of resource sharing
resource sharing the application wishes to give Local Peer the ULP wishes to give Local Peer processes. For more discussion
processes. For more discussion on types of trust models which on types of trust models which combine partial trust and sharing
combine partial trust and sharing of resources, see Appendix C: of resources, see Appendix C: Partial Trust Taxonomy on page 50.
Partial Trust Taxonomy on page 50.
The Privileged Resource Manager MUST NOT assume different ULPs
share Partial Mutual Trust unless there is a mechanism to ensure
that the ULPs do indeed share partial mutual trust. This can be
done in several ways, including explicit notification from the
ULP.
6 Attacker Capabilities 6 Attacker Capabilities
An attackerÆs capabilities delimit the types of attacks that An attackers capabilities delimit the types of attacks that
attacker is able to launch. RDMAP and DDP require that the attacker is able to launch. RDMAP and DDP require that the
initial LLP Stream (and connection) be set up prior to initial LLP Stream (and connection) be set up prior to
transferring RDMAP/DDP Messages. Attackers with send only transferring RDMAP/DDP Messages. Attackers with send only
capabilities must first guess the current LLP Stream parameters capabilities must first guess the current LLP Stream parameters
before they can attack RNIC resources (e.g. TCP sequence number). before they can attack RNIC resources (e.g. TCP sequence number).
Attackers with both send and receive capabilities have presumably Attackers with both send and receive capabilities have presumably
setup a valid LLP Stream, and thus have a wider ability to attack setup a valid LLP Stream, and thus have a wider ability to attack
RNIC resources. RNIC resources.
7 Attacks and Countermeasures 7 Attacks and Countermeasures
skipping to change at page 20, line 5 skipping to change at page 20, line 5
the associated data buffer through Streams in the same Protection the associated data buffer through Streams in the same Protection
Domain as that STag. Domain as that STag.
If an implementation chooses to not share resources between If an implementation chooses to not share resources between
Streams, it is recommended that each Stream be associated with Streams, it is recommended that each Stream be associated with
its own, unique Protection Domain. If an implementation chooses its own, unique Protection Domain. If an implementation chooses
to allow resource sharing, it is recommended that Protection to allow resource sharing, it is recommended that Protection
Domain be limited to the number of Streams that have Partial Domain be limited to the number of Streams that have Partial
Mutual Trust. Mutual Trust.
Note that an application (either Privileged or Non-Privileged) Note that a ULP (either Privileged or Non-Privileged) can
can potentially have multiple Protection Domains. This could be potentially have multiple Protection Domains. This could be used,
used, for example, to ensure that multiple clients of a server do for example, to ensure that multiple clients of a server do not
not have the ability to corrupt each other. The server would have the ability to corrupt each other. The server would allocate
allocate a Protection Domain per client to ensure that resources a Protection Domain per client to ensure that resources covered
covered by the Protection Domain could not be used by another by the Protection Domain could not be used by another (untrusted)
(untrusted) client. client.
7.1.2 Limiting STag Scope 7.1.2 Limiting STag Scope
The key to protecting a local data buffer is to limit the scope The key to protecting a local data buffer is to limit the scope
of its STag to the level appropriate for the Streams which share of its STag to the level appropriate for the Streams which share
Partial Mutual Trust. The scope of the STag can be measured in Partial Mutual Trust. The scope of the STag can be measured in
multiple ways. multiple ways.
* Number of Connections and/or Streams on which the STag is * Number of Connections and/or Streams on which the STag is
valid. One way to limit the scope of the STag is to limit valid. One way to limit the scope of the STag is to limit
skipping to change at page 20, line 45 skipping to change at page 20, line 45
Stream, regardless of what the Stream association is Stream, regardless of what the Stream association is
to a Protection Domain. If used on any other Stream, to a Protection Domain. If used on any other Stream,
it is invalid. it is invalid.
* Limit the time an STag is valid. By Invalidating an * Limit the time an STag is valid. By Invalidating an
Advertised STag (e.g., revoking remote access to the Advertised STag (e.g., revoking remote access to the
buffers described by an STag when done with the buffers described by an STag when done with the
transfer), an entire class of attacks can be eliminated. transfer), an entire class of attacks can be eliminated.
* Limit the buffer the STag can reference. Limiting the * Limit the buffer the STag can reference. Limiting the
scope of an STag access to *just* the intended scope of an STag access to *just* the intended ULP
application buffers to be exposed is critical to prevent buffers to be exposed is critical to prevent certain
certain forms of attacks. forms of attacks.
* Allocating and/or advertising STag numbers in an * Allocating and/or advertising STag numbers in an
unpredictable way. If STags are allocated/advertised unpredictable way. If STags are allocated/advertised
using an algorithm which makes it hard for the attacker using an algorithm which makes it hard for the attacker
to guess which STag(s) are currently in use, it makes it to guess which STag(s) are currently in use, it makes it
more difficult for an attacker to guess the correct more difficult for an attacker to guess the correct
value. As stated in the RDMAP specification [RDMAP], an value. As stated in the RDMAP specification [RDMAP], an
invalid STag will cause the RDMAP Stream to be invalid STag will cause the RDMAP Stream to be
terminated. For the case of [DDP], at a minimum it must terminated. For the case of [DDP], at a minimum it must
signal an error to the ULP, and commonly this will cause signal an error to the ULP, and commonly this will cause
the DDP stream to be terminated. the DDP stream to be terminated.
7.1.3 Access Rights 7.1.3 Access Rights
Access Rights associated with a specific Advertised STag or Access Rights associated with a specific Advertised STag or
RDMAP/DDP Stream provide another mechanism for applications to RDMAP/DDP Stream provide another mechanism for ULPs to limit the
limit the attack capabilities of the Remote Peer. The Local Peer attack capabilities of the Remote Peer. The Local Peer can
can control whether a data buffer is exposed for local only, or control whether a data buffer is exposed for local only, or local
local and remote access, and assign specific access privileges and remote access, and assign specific access privileges (read,
(read, write, read and write) on a per stream basis. write, read and write) on a per stream basis.
For DDP, when an STag is advertised, the Remote Peer is For DDP, when an STag is advertised, the Remote Peer is
presumably given write access rights to the data (otherwise there presumably given write access rights to the data (otherwise there
was not much point to the advertisement). For RDMAP, when an was not much point to the advertisement). For RDMAP, when a ULP
application advertises an STag, it can enable write-only, read- advertises an STag, it can enable write-only, read-only, or both
only, or both write and read access rights. write and read access rights.
Similarly, some applications may wish to provide a single buffer Similarly, some ULPs may wish to provide a single buffer with
with different access rights on a per-Stream or per-Stream basis. different access rights on a per-Stream or per-Stream basis. For
For example, some Streams may have read-only access, some may example, some Streams may have read-only access, some may have
have remote read and write access, while on other Streams only remote read and write access, while on other Streams only the
the Local Peer is allowed access. Local Peer is allowed access.
7.1.4 Limiting the Scope of the Completion Queue 7.1.4 Limiting the Scope of the Completion Queue
Completions associated with sending and receiving data, or Completions associated with sending and receiving data, or
setting up buffers for sending and receiving data, could be setting up buffers for sending and receiving data, could be
accumulated in a shared Completion Queue for a group of RDMAP/DDP accumulated in a shared Completion Queue for a group of RDMAP/DDP
Streams, or a specific RDMAP/DDP Stream could have a dedicated Streams, or a specific RDMAP/DDP Stream could have a dedicated
Completion Queue. Limiting Completion Queue association to one, Completion Queue. Limiting Completion Queue association to one,
or a small number of RDMAP/DDP Streams can prevent several forms or a small number of RDMAP/DDP Streams can prevent several forms
of Denial of Service attacks. of Denial of Service attacks.
skipping to change at page 23, line 44 skipping to change at page 23, line 44
potentially be able to perform either RDMA Read Operations to potentially be able to perform either RDMA Read Operations to
read the contents of the associated data buffer, perform RDMA read the contents of the associated data buffer, perform RDMA
Write Operations to modify the contents of the associated data Write Operations to modify the contents of the associated data
buffer, or to Invalidate the STag to disable further access to buffer, or to Invalidate the STag to disable further access to
the buffer. the buffer.
An attempt by a Remote Peer to access a buffer with an STag on a An attempt by a Remote Peer to access a buffer with an STag on a
different Stream in the same Protection Domain may or may not be different Stream in the same Protection Domain may or may not be
an attack depending on whether resource sharing is intended (i.e. an attack depending on whether resource sharing is intended (i.e.
whether the Streams shared Partial Mutual Trust or not). For some whether the Streams shared Partial Mutual Trust or not). For some
applications using an STag on multiple Streams within the same ULPs using an STag on multiple Streams within the same Protection
Protection Domain could be desired behavior. For other Domain could be desired behavior. For other ULPs attempting to
applications attempting to use an STag on a different Stream use an STag on a different Stream could be considered to be an
could be considered to be an attack. Since this varies by attack. Since this varies by ULP, a ULP typically would need to
application, an application typically would need to be able to be able to control the scope of the STag.
control the scope of the STag.
In the case where an implementation does not share resources In the case where an implementation does not share resources
between Streams (including STags), this attack can be defeated by between Streams (including STags), this attack can be defeated by
assigning each Stream to a different Protection Domain. Before assigning each Stream to a different Protection Domain. Before
allowing remote access to the buffer, the Protection Domain of allowing remote access to the buffer, the Protection Domain of
the Stream where the access attempt was made is matched against the Stream where the access attempt was made is matched against
the Protection Domain of the STag. If the Protection Domains do the Protection Domain of the STag. If the Protection Domains do
not match, access to the buffer is denied, an error is generated, not match, access to the buffer is denied, an error is generated,
and the RDMAP Stream associated with the attacking Stream should and the RDMAP Stream associated with the attacking Stream should
be terminated. be terminated.
For implementations that share resources between multiple For implementations that share resources between multiple
Streams, it may not be practical to separate each Stream into its Streams, it may not be practical to separate each Stream into its
own Protection Domain. In this case, the application can still own Protection Domain. In this case, the ULP can still limit the
limit the scope of any of the STags to a single Stream (if it is scope of any of the STags to a single Stream (if it is enabling
enabling it for remote access). If the STag scope has been it for remote access). If the STag scope has been limited to a
limited to a single Stream, any attempt to use that STag on a single Stream, any attempt to use that STag on a different Stream
different Stream will result in an error, and the RDMA Stream will result in an error, and the RDMA Stream should be
should be terminated. terminated.
Thus for implementations that do not share STags between Streams, Thus for implementations that do not share STags between Streams,
each Stream MUST either be in a separate Protection Domain or the each Stream MUST either be in a separate Protection Domain or the
scope of an STag be limited to a single Stream. scope of an STag MUST be limited to a single Stream.
An additional issue may be unintended sharing of STags (i.e. a An additional issue may be unintended sharing of STags (i.e. a
bug in the application) or a bug in the Remote Peer which causes bug in the ULP) or a bug in the Remote Peer which causes an off-
an off-by-one STag to be used. For additional protection, an by-one STag to be used. For additional protection, an
implementation SHOULD allocate STags in such a fashion that it is implementation should allocate STags in such a fashion that it is
difficult to predict the next allocated STag number. Allocation difficult to predict the next allocated STag number. Allocation
methods which deterministically allocate the next STag should be methods which deterministically allocate the next STag should be
avoided (e.g. a method which always starts with STag equal to one avoided (e.g. a method which always starts with STag equal to one
and monotonically increases it for each new allocation, or a and monotonically increases it for each new allocation, or a
method which always uses the same STag for each operation). method which always uses the same STag for each operation).
7.3 Tampering 7.3 Tampering
A Remote Peer or a network based attacker can attempt to tamper A Remote Peer or a network based attacker can attempt to tamper
with the contents of data buffers on a Local Peer that have been with the contents of data buffers on a Local Peer that have been
enabled for remote write access. The types of tampering attacks enabled for remote write access. The types of tampering attacks
that are possible are outlined in the sections that follow. that are possible are outlined in the sections that follow.
7.3.1 Buffer Overrun - RDMA Write or Read Response 7.3.1 Buffer Overrun - RDMA Write or Read Response
This attack is an attempt by the Remote Peer to perform an RDMA This attack is an attempt by the Remote Peer to perform an RDMA
Write or RDMA Read Response to memory outside of the valid length Write or RDMA Read Response to memory outside of the valid length
range of the data buffer enabled for remote write access. This range of the data buffer enabled for remote write access. This
attack can occur even when no resources are shared across attack can occur even when no resources are shared across
Streams. This issue can also arise if the application has a bug. Streams. This issue can also arise if the ULP has a bug.
The countermeasure for this type of attack must be in the RNIC The countermeasure for this type of attack must be in the RNIC
implementation, using the STag. When the Local Peer specifies to implementation, using the STag. When the Local Peer specifies to
the RNIC the base address and the number of bytes in the buffer the RNIC the base address and the number of bytes in the buffer
that it wishes to make accessible, the RNIC must ensure that the that it wishes to make accessible, the RNIC must ensure that the
base and bounds check are applied to any access to the buffer base and bounds check are applied to any access to the buffer
referenced by the STag before the STag is enabled for access. referenced by the STag before the STag is enabled for access.
When an RDMA data transfer operation (which includes an STag) When an RDMA data transfer operation (which includes an STag)
arrives on a Stream, a base and bounds byte granularity access arrives on a Stream, a base and bounds byte granularity access
check must be performed to ensure the operation accesses only check must be performed to ensure the operation accesses only
skipping to change at page 26, line 54 skipping to change at page 26, line 54
Remote Peer to retrieve the new data. Remote Peer to retrieve the new data.
This is similar to the attack defined in Section 7.3.2 Modifying This is similar to the attack defined in Section 7.3.2 Modifying
a Buffer After Indication on page 25. The same countermeasures a Buffer After Indication on page 25. The same countermeasures
apply. In addition, the Local Peer SHOULD grant remote read apply. In addition, the Local Peer SHOULD grant remote read
access rights only for the amount of time needed to retrieve the access rights only for the amount of time needed to retrieve the
data. data.
7.4.4 Accessing Unintended Data With a Valid STag 7.4.4 Accessing Unintended Data With a Valid STag
If the Local Peer enables remote access to a buffer using an STag If the ULP enables remote access to a buffer using an STag that
that references the entire buffer, but intends only a portion of references the entire buffer, but intends only a portion of the
the buffer to be accessed, it is possible for the Remote Peer to buffer to be accessed, it is possible for the Remote Peer to
access the other parts of the buffer anyway. access the other parts of the buffer anyway.
To prevent this attack, the Local Peer MUST set the base and To prevent this attack, the ULP SHOULD set the base and bounds of
bounds of the buffer when the STag is initialized to expose only the buffer when the STag is initialized to expose only the data
the data to be retrieved. to be retrieved.
7.4.5 RDMA Read into an RDMA Write Buffer 7.4.5 RDMA Read into an RDMA Write Buffer
One form of disclosure can occur if the access rights on the One form of disclosure can occur if the access rights on the
buffer enabled remote read, when only remote write access was buffer enabled remote read, when only remote write access was
intended. If the buffer contained application data, or data from intended. If the buffer contained ULP data, or data from a
a transfer on an unrelated Stream, the Remote Peer could retrieve transfer on an unrelated Stream, the Remote Peer could retrieve
the data through an RDMA Read operation. the data through an RDMA Read operation. Note that an RNIC
implementation is not required to support STags that have both
read and write access.
The most obvious countermeasure for this attack is to not grant The most obvious countermeasure for this attack is to not grant
remote read access if the buffer is intended to be write-only. remote read access if the buffer is intended to be write-only.
Then the Remote Peer would not be able to retrieve data Then the Remote Peer would not be able to retrieve data
associated with the buffer. An attempt to do so would result in associated with the buffer. An attempt to do so would result in
an error and the RDMAP Stream associated with the Stream would be an error and the RDMAP Stream associated with the Stream would be
terminated. terminated.
Thus if an application only intends a buffer to be exposed for Thus if a ULP only intends a buffer to be exposed for remote
remote write access, it MUST set the access rights to the buffer write access, it MUST set the access rights to the buffer to only
to only enable remote write access. enable remote write access.
7.4.6 Using Multiple STags Which Alias to the Same Buffer 7.4.6 Using Multiple STags Which Alias to the Same Buffer
Multiple STags which alias to the same buffer at the same time Multiple STags which alias to the same buffer at the same time
can result in unintentional information disclosure if the STags can result in unintentional information disclosure if the STags
are used by different, mutually untrusted, Remote Peers. This are used by different, mutually untrusted, Remote Peers. This
model applies specifically to client/server communication, where model applies specifically to client/server communication, where
the server is communicating with multiple clients, each of which the server is communicating with multiple clients, each of which
do not mutually trust each other. do not mutually trust each other.
skipping to change at page 28, line 21 skipping to change at page 28, line 24
information tampering occurs between peers. information tampering occurs between peers.
7.4.7 Remote Node Loading Firmware onto the RNIC 7.4.7 Remote Node Loading Firmware onto the RNIC
If the Remote Peer can cause firmware to be loaded onto the RNIC, If the Remote Peer can cause firmware to be loaded onto the RNIC,
there is an opportunity for information disclosure. See Elevation there is an opportunity for information disclosure. See Elevation
of Privilege in Section 7.6 for this analysis. of Privilege in Section 7.6 for this analysis.
7.4.8 Controlling Access to PTT & STag Mapping 7.4.8 Controlling Access to PTT & STag Mapping
If a Non-Privileged application is able to directly manipulate If a Non-Privileged ULP is able to directly manipulate the RNIC
the RNIC Page Translation Tables (which translate from an STag to Page Translation Tables (which translate from an STag to a host
a host address), it is possible that the Non-Privileged address), it is possible that the Non-Privileged ULP could point
application could point the Page Translation Table at an the Page Translation Table at an unrelated ULP’s buffers and
unrelated applicationÆs buffers and thereby be able to gain thereby be able to gain access to information in the unrelated
access to information in the unrelated application. ULP.
As discussed in Section 4 Architectural Model on page 8, As discussed in Section 4 Architectural Model on page 9,
introduction of a Privileged Resource Manager to arbitrate the introduction of a Privileged Resource Manager to arbitrate the
mapping requests is an effective countermeasure. This enables the mapping requests is an effective countermeasure. This enables the
Privileged Resource Manager to ensure an application can only Privileged Resource Manager to ensure a ULP can only initialize
initialize the Page Translation Table (PTT)to point to its own the Page Translation Table (PTT)to point to its own buffers.
buffers.
Thus if Non-Privileged applications are supported, the Privileged Thus if Non-Privileged ULPs are supported, the Privileged
Resource Manager MUST verify that the Non-Privileged application Resource Manager MUST verify that the Non-Privileged ULP has the
has the right to access a specific Data Buffer before allowing an right to access a specific Data Buffer before allowing an STag
STag for which the application has access rights to be associated for which the ULP has access rights to be associated with a
with a specific Data Buffer. This can be done when the Page specific Data Buffer. This can be done when the Page Translation
Translation Table is initialized to access the Data Buffer or Table is initialized to access the Data Buffer or when the STag
when the STag is initialized to point to a group of Page is initialized to point to a group of Page Translation Table
Translation Table entries, or both. entries, or both.
7.4.9 Network based eavesdropping 7.4.9 Network based eavesdropping
An attacker that is able to eavesdrop on the network can read the An attacker that is able to eavesdrop on the network can read the
content of all read and write access to the peerÆs buffers. To content of all read and write access to the peers buffers. To
prevent information disclosure, the read/written data must be prevent information disclosure, the read/written data must be
encrypted. See also Section 7.2.3 Man in the Middle Attack on encrypted. See also Section 7.2.3 Man in the Middle Attack on
page 22. The encryption can be done either by the ULP, or by a page 22. The encryption can be done either by the ULP, or by a
protocol that provides security services to the LLP (e.g. IPsec protocol that provides security services to the LLP (e.g. IPsec
or SSL). Refer to section 8 for discussion of security services or SSL). Refer to section 8 for discussion of security services
for DDP/RDMA. for DDP/RDMA.
7.5 Denial of Service (DOS) 7.5 Denial of Service (DOS)
A DOS attack is one of the primary security risks of RDMAP. This A DOS attack is one of the primary security risks of RDMAP. This
is because RNIC resources are valuable and scarce, and many is because RNIC resources are valuable and scarce, and many ULP
application environments require communication with untrusted environments require communication with untrusted Remote Peers.
Remote Peers. If the remote application can be authenticated or If the remote ULP can be authenticated or encrypted, clearly, the
encrypted, clearly, the DOS profile can be reduced. For the DOS profile can be reduced. For the purposes of this analysis, it
purposes of this analysis, it is assumed that the RNIC must be is assumed that the RNIC must be able to operate in untrusted
able to operate in untrusted environments, which are open to DOS environments, which are open to DOS style attacks.
style attacks.
Denial of service attacks against RNIC resources are not the Denial of service attacks against RNIC resources are not the
typical unknown party spraying packets at a random host (such as typical unknown party spraying packets at a random host (such as
a TCP SYN attack). Because the connection/Stream must be fully a TCP SYN attack). Because the connection/Stream must be fully
established, the attacker must be able to both send and receive established, the attacker must be able to both send and receive
messages over that connection/Stream, or be able to guess a valid messages over that connection/Stream, or be able to guess a valid
packet on an existing RDMAP Stream. packet on an existing RDMAP Stream.
This section outlines the potential attacks and the This section outlines the potential attacks and the
countermeasures available for dealing with each attack. countermeasures available for dealing with each attack.
skipping to change at page 29, line 50 skipping to change at page 29, line 49
The allocation of all scarce resources MUST be placed under the The allocation of all scarce resources MUST be placed under the
control of a Privileged Resource Manager. This allows the control of a Privileged Resource Manager. This allows the
Privileged Resource Manager to: Privileged Resource Manager to:
* prevent a Local Peer from allocating more than its fair * prevent a Local Peer from allocating more than its fair
share of resources. share of resources.
* detect if a Remote Peer is attempting to launch a DOS * detect if a Remote Peer is attempting to launch a DOS
attack by attempting to create an excessive number of attack by attempting to create an excessive number of
Streams and take corrective action (such as refusing the Streams (with associated resources) and take corrective
request or applying network layer filters against the action (such as refusing the request or applying network
Remote Peer). layer filters against the Remote Peer).
This analysis assumes that the Resource Manager is responsible This analysis assumes that the Resource Manager is responsible
for handing out Protection Domains, and RNIC implementations will for handing out Protection Domains, and RNIC implementations will
provide enough Protection Domains to allow the Resource Manager provide enough Protection Domains to allow the Resource Manager
to be able to assign a unique Protection Domain for each to be able to assign a unique Protection Domain for each
unrelated, untrusted Local Peer (for a bounded, reasonable number unrelated, untrusted Local Peer (for a bounded, reasonable number
of Local Peers). This analysis further assumes that the Resource of Local Peers). This analysis further assumes that the Resource
Manager implements policies to ensure that untrusted Local Peers Manager implements policies to ensure that untrusted Local Peers
are not able to consume all of the Protection Domains through a are not able to consume all of the Protection Domains through a
DOS attack. Note that Protection Domain consumption cannot result DOS attack. Note that Protection Domain consumption cannot result
from a DOS attack launched by a Remote Peer, unless a Local Peer from a DOS attack launched by a Remote Peer, unless a Local Peer
is acting on the Remote PeerÆs behalf. is acting on the Remote Peers behalf.
7.5.2 Resource Consumption By Active Applications 7.5.2 Resource Consumption By Active ULPs
This section describes DOS attacks from Local and Remote Peers This section describes DOS attacks from Local and Remote Peers
that are actively exchanging messages. Attacks on each RDMA NIC that are actively exchanging messages. Attacks on each RDMA NIC
resource are examined and specific countermeasures are resource are examined and specific countermeasures are
identified. Note that attacks on Stream Context Memory, Page identified. Note that attacks on Stream Context Memory, Page
Translation Tables, and STag namespace are covered in Section Translation Tables, and STag namespace are covered in Section
7.5.1 RNIC Resource Consumption, so are not included here. 7.5.1 RNIC Resource Consumption, so are not included here.
7.5.2.1 Multiple Streams Sharing Receive Buffers 7.5.2.1 Multiple Streams Sharing Receive Buffers
The Remote Peer can attempt to consume more than its fair share The Remote Peer can attempt to consume more than its fair share
of receive data buffers (Untagged DDP buffers or for RDMAP of receive data buffers (Untagged DDP buffers or for RDMAP
buffers consumed with Send Type Messages) if receive buffers are buffers consumed with Send Type Messages) if receive buffers are
shared across multiple Streams. shared across multiple Streams.
If resources are not shared across multiple Streams, then this If resources are not shared across multiple Streams, then this
attack is not possible because the Remote Peer will not be able attack is not possible because the Remote Peer will not be able
to consume more buffers than were allocated to the Stream. The to consume more buffers than were allocated to the Stream. The
worst case scenario is that the Remote Peer can consume more worst case scenario is that the Remote Peer can consume more
receive buffers than the Local Peer allowed, resulting in no receive buffers than the Local Peer allowed, resulting in no
buffers to be available, which could cause the Remote PeerÆs buffers to be available, which could cause the Remote Peers
Stream to the Local Peer to be torn down, and all allocated Stream to the Local Peer to be torn down, and all allocated
resources to be released. resources to be released.
If local receive data buffers are shared among multiple Streams, If local receive data buffers are shared among multiple Streams,
then the Remote Peer can attempt to consume more than its fair then the Remote Peer can attempt to consume more than its fair
share of the receive buffers, causing a different Stream to be share of the receive buffers, causing a different Stream to be
short of receive buffers, thus possibly causing the other Stream short of receive buffers, thus possibly causing the other Stream
to be torn down. For example, if the Remote Peer sent enough one to be torn down. For example, if the Remote Peer sent enough one
byte Untagged Messages, they might be able to consume all local byte Untagged Messages, they might be able to consume all local
shared receive queue resources with little effort on their part. shared receive queue resources with little effort on their part.
One method the Local Peer could use is to recognize that a Remote One method the Local Peer could use is to recognize that a Remote
Peer is attempting to use more than its fair share of resources Peer is attempting to use more than its fair share of resources
and terminate the Stream (causing the allocated resources to be and terminate the Stream (causing the allocated resources to be
released). However, if the Local Peer is sufficiently slow, it released). However, if the Local Peer is sufficiently slow, it
may be possible for the Remote Peer to still mount a denial of may be possible for the Remote Peer to still mount a denial of
service attack. One countermeasure that can protect against this service attack. One countermeasure that can protect against this
attack is implementing a low-water notification. The low-water attack is implementing a low-water notification. The low-water
notification alerts the application if the number of buffers in notification alerts the ULP if the number of buffers in the
the receive queue is less than a threshold. receive queue is less than a threshold.
If all of the following conditions are true, then the Local Peer If all of the following conditions are true, then the Local Peer
can size the amount of local receive buffers posted on the can size the amount of local receive buffers posted on the
receive queue to ensure a DOS attack can be stopped. receive queue to ensure a DOS attack can be stopped.
* a low-water notification is enabled, and * a low-water notification is enabled, and
* the Local Peer is able to bound the amount of time that * the Local Peer is able to bound the amount of time that
it takes to replenish receive buffers, and it takes to replenish receive buffers, and
* the Local Peer maintains statistics to determine which * the Local Peer maintains statistics to determine which
Remote Peer is consuming buffers. Remote Peer is consuming buffers.
The above conditions enable the low-water notification to arrive The above conditions enable the low-water notification to arrive
before resources are depleted and thus the Local Peer can take before resources are depleted and thus the Local Peer can take
corrective action (e.g., terminate the Stream of the attacking corrective action (e.g., terminate the Stream of the attacking
Remote Peer). Remote Peer).
A different, but similar attack is if the Remote Peer sends a A different, but similar attack is if the Remote Peer sends a
significant number of out-of-order packets and the RNIC has the significant number of out-of-order packets and the RNIC has the
ability to use the application buffer as a reassembly buffer. In ability to use the ULP buffer as a reassembly buffer. In this
this case the Remote Peer can consume a significant number of case the Remote Peer can consume a significant number of ULP
application buffers, but never send enough data to enable the buffers, but never send enough data to enable the ULP buffer to
application buffer to be completed to the application. be completed to the ULP.
An effective countermeasure is to create a high-water An effective countermeasure is to create a high-water
notification which alerts the application if there is more than a notification which alerts the ULP if there is more than a
specified number of receive buffers "in process" (partially specified number of receive buffers "in process" (partially
consumed, but not completed). The notification is generated when consumed, but not completed). The notification is generated when
more than the specified number of buffers are in process more than the specified number of buffers are in process
simultaneously on a specific Stream (i.e., packets have started simultaneously on a specific Stream (i.e., packets have started
to arrive for the buffer, but the buffer has not yet been to arrive for the buffer, but the buffer has not yet been
delivered to the ULP). delivered to the ULP).
A different countermeasure is for the RNIC Engine to provide the A different countermeasure is for the RNIC Engine to provide the
capability to limit the Remote PeerÆs ability to consume receive capability to limit the Remote Peers ability to consume receive
buffers on a per Stream basis. Unfortunately this requires a buffers on a per Stream basis. Unfortunately this requires a
large amount of state to be tracked in each RNIC on a per Stream large amount of state to be tracked in each RNIC on a per Stream
basis. basis.
Thus, if an RNIC Engine provides the ability to share receive Thus, if an RNIC Engine provides the ability to share receive
buffers across multiple Streams, the combination of the RNIC buffers across multiple Streams, the combination of the RNIC
Engine and the Privileged Resource Manager MUST be able to detect Engine and the Privileged Resource Manager MUST be able to detect
if the Remote Peer is attempting to consume more than its fair if the Remote Peer is attempting to consume more than its fair
share of resources so that the Local Peer can apply share of resources so that the Local Peer can apply
countermeasures to detect and prevent the attack. countermeasures to detect and prevent the attack.
7.5.2.2 Local Peer Attacking a Shared CQ 7.5.2.2 Local ULP Attacking a Shared CQ
DOS attacks against a Shared Completion Queue (CQ) can be caused DOS attacks against a Shared Completion Queue (CQ) can be caused
by either the Local Peer or the Remote Peer if either attempts to by either the Local Peer's ULP or the Remote Peer if either
cause more completions than its fair share of the number of attempts to cause more completions than its fair share of the
entries, thus potentially starving another unrelated Stream such number of entries, thus potentially starving another unrelated
that no Completion Queue entries are available. ULP such that no Completion Queue entries are available.
A Completion Queue entry can potentially be consumed by a A Completion Queue entry can potentially be consumed by a
completion from the Send Queue or a Receive Queue completion. In completion from the Send Queue or a completion from the Receive
the former, the attacker is the Local Peer. In the later, the Queue. In the former, the attacker is the Local Peer's ULP. In
attacker is the Remote Peer. the later, the attacker is the Remote Peer.
A form of attack can occur where the Local Peers can consume A form of attack can occur where the Local Peer ULPs can consume
resources on the CQ. A Local Peer that is slow to free resources resources on the CQ. A Local Peer ULP that is slow to free
on the CQ by not reaping the completion status quickly enough resources on the CQ by not reaping the completion status quickly
could stall all other Local Peers attempting to use that CQ. enough could stall all other Local Peer ULPs attempting to use
that CQ.
One of two countermeasures can be used to avoid this kind of One of two countermeasures can be used to avoid this kind of
attack. The first is to only share a CQ between Streams that attack. The first is to only share a CQ between ULPs that share
share Partial Mutual Trust (i.e. Streams within the same Partial Mutual Trust. The other is to use a trusted Local Peer to
Protection Domain). The other is to use a trusted Local Peer to
act as a third party to free resources on the CQ and place the act as a third party to free resources on the CQ and place the
status in intermediate storage until the untrusted Local Peer status in intermediate storage until the untrusted ULP reaps the
reaps the status information. For these reasons, an RNIC MUST NOT status information. For these reasons, an RNIC MUST NOT enable
enable sharing a CQ across Streams that belong to different sharing a CQ across ULPs that do not share partial mutual trust.
Protection Domains. Additionally, an application SHOULD NOT share
a CQ between Streams which do not share Partial Mutual Trust.
7.5.2.3 Remote Peer Attacking a Shared CQ 7.5.2.3 Remote Peer Attacking a Shared CQ
For an overview of the Shared CQ attack model, see Section For an overview of the Shared CQ attack model, see Section
7.5.2.2. 7.5.2.2.
<TBD: add text that says if not shared there is no security
threat). If you get a CQ overflow it MUST NOT affect any resource
outside the scope of the current Stream.>
The Remote Peer can attack a CQ by consuming more than its fair The Remote Peer can attack a CQ by consuming more than its fair
share of CQ entries by using one of the following methods: share of CQ entries by using one of the following methods:
* The ULP protocol allows the Remote Peer to reserve a * The ULP protocol allows the Remote Peer to reserve a
specified number of CQ entries, possibly leaving specified number of CQ entries, possibly leaving
insufficient entries for other Streams that are sharing insufficient entries for other Streams that are sharing
the CQ. the CQ.
* If the Remote Peer or Local Peer (or both) can attack the * If the Remote Peer or Local Peer (or both) can attack the
CQ by overwhelming the CQ with completions, then CQ by overwhelming the CQ with completions, then
skipping to change at page 33, line 4 skipping to change at page 33, line 5
Queue overflows and stops functioning). Queue overflows and stops functioning).
The first method of attack can be avoided if the ULP does not The first method of attack can be avoided if the ULP does not
allow a Remote Peer to reserve CQ entries or there is a trusted allow a Remote Peer to reserve CQ entries or there is a trusted
intermediary such as a Privileged Resource Manager. Unfortunately intermediary such as a Privileged Resource Manager. Unfortunately
it is often unrealistic to not allow a Remote Peer to reserve CQ it is often unrealistic to not allow a Remote Peer to reserve CQ
entries - particularly if the number of completion entries is entries - particularly if the number of completion entries is
dependent on other ULP negotiated parameters, such as the amount dependent on other ULP negotiated parameters, such as the amount
of buffering required by the ULP. Thus an implementation MUST of buffering required by the ULP. Thus an implementation MUST
implement a Privileged Resource Manager to control the allocation implement a Privileged Resource Manager to control the allocation
of CQ entries. See Section 4.1 Components on page 9 for a of CQ entries. See Section 4.1 Components on page 10 for a
definition of Privileged Resource Manager. definition of Privileged Resource Manager.
One way that a Local or Remote Peer can attempt to overwhelm a CQ One way that a Local or Remote Peer can attempt to overwhelm a CQ
with completions is by sending minimum length RDMAP/DDP Messages with completions is by sending minimum length RDMAP/DDP Messages
to cause as many completions (receive completions for the Remote to cause as many completions (receive completions for the Remote
Peer, send completions for the Local Peer) per second as Peer, send completions for the Local Peer) per second as
possible. If it is the Remote Peer attacking, and we assume that possible. If it is the Remote Peer attacking, and we assume that
the Local Peer does not run out of receive buffers (if they do, the Local Peer does not run out of receive buffers (if they do,
then this is a different attack, documented in Section 7.5.2.1 then this is a different attack, documented in Section 7.5.2.1
Multiple Streams Sharing Receive Buffers on page 30), then it Multiple Streams Sharing Receive Buffers on page 30), then it
skipping to change at page 33, line 47 skipping to change at page 33, line 48
should be done before sizing the Send Queue and Receive should be done before sizing the Send Queue and Receive
Queue on the Stream), OR Queue on the Stream), OR
* Grant fewer resources than the Remote Peer requested (not * Grant fewer resources than the Remote Peer requested (not
supplying the number of Receive Data Buffers requested). supplying the number of Receive Data Buffers requested).
The proper sizing of the CQ is dependent on whether the Local The proper sizing of the CQ is dependent on whether the Local
Peer will post as many resources to the various queues as the Peer will post as many resources to the various queues as the
size of the queue enables or not. If the Local Peer can be size of the queue enables or not. If the Local Peer can be
trusted to post a number of resources that is smaller than the trusted to post a number of resources that is smaller than the
size of the specific resourceÆs queue, then a correctly sized CQ size of the specific resources queue, then a correctly sized CQ
means that the CQ is large enough to hold completion status for means that the CQ is large enough to hold completion status for
all of the outstanding Data Buffers (both send and receive all of the outstanding Data Buffers (both send and receive
buffers), or: buffers), or:
CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ) CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ)
+ SUM(MaxPostedOnEachSRQ) + SUM(MaxPostedOnEachSRQ)
+ SUM(MaxPostedOnEachSQ) + SUM(MaxPostedOnEachSQ)
Where: Where:
skipping to change at page 34, line 50 skipping to change at page 34, line 50
can cause a completion that can ever be posted can cause a completion that can ever be posted
on a specific Shared Receive Queue. on a specific Shared Receive Queue.
SizeOfEachSQ = the maximum number of requests which SizeOfEachSQ = the maximum number of requests which
can cause a completion that can ever be posted can cause a completion that can ever be posted
on a specific Send Queue. on a specific Send Queue.
Where MaxPosted*OnEach*Q and SizeOfEach*Q varies on a per Stream Where MaxPosted*OnEach*Q and SizeOfEach*Q varies on a per Stream
or per Shared Receive Queue basis. or per Shared Receive Queue basis.
The Local Peer MUST implement a mechanism to ensure that the If the ULP is sharing a CQ across multiple streams which do not
Completion Queue can not overflow. Note that it is possible to share partial mutual trust, then the ULP MUST implement a
share CQs even if the Remote Peers accessing the CQs are mechanism to ensure that the Completion Queue can not overflow.
untrusted if either of the above two formulas are implemented. If Note that it is possible to share CQs even if the Remote Peers
the Local Peer can be trusted to not post more than accessing the CQs are untrusted if either of the above two
MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and MaxPostedOnEachSQ, formulas are implemented. If the ULP can be trusted to not post
then the first formula applies. If the Local Peer can not be more than MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and
trusted to obey the limit, then the second formula applies. MaxPostedOnEachSQ, then the first formula applies. If the ULP can
not be trusted to obey the limit, then the second formula
applies.
7.5.2.4 Attacking the RDMA Read Request Queue 7.5.2.4 Attacking the RDMA Read Request Queue
If RDMA Read Request Queue resources are pooled across multiple If RDMA Read Request Queue resources are pooled across multiple
Streams, one attack is if the Local Peer attempts to unfairly Streams, one attack is if the Local Peer attempts to unfairly
allocate RDMA Read Request Queue resources for its Streams. For allocate RDMA Read Request Queue resources for its Streams. For
example, the Local Peer attempts to allocate all available example, the Local Peer attempts to allocate all available
resources on a specific RDMA Read Request Queue for its Streams, resources on a specific RDMA Read Request Queue for its Streams,
thereby denying the resource to applications sharing the RDMA thereby denying the resource to ULPs sharing the RDMA Read
Read Request Queue. The same type of argument applies even if the Request Queue. The same type of argument applies even if the RDMA
RDMA Read Request is not shared - but a Local Peer attempts to Read Request is not shared - but a Local Peer attempts to
allocate all of the RNICs resource when the queue is created. allocate all of the RNICs resource when the queue is created.
Thus access to interfaces that allocate RDMA Read Request Queue Thus access to interfaces that allocate RDMA Read Request Queue
entries MUST be restricted to a trusted Local Peer, such as a entries MUST be restricted to a trusted Local Peer, such as a
Privileged Resource Manager. The Privileged Resource Manager Privileged Resource Manager. The Privileged Resource Manager
SHOULD prevent a Local Peer from allocating more than its fair SHOULD prevent a Local Peer from allocating more than its fair
share of resources. share of resources.
Another form of attack is if the Remote Peer sends more RDMA Read Another form of attack is if the Remote Peer sends more RDMA Read
Requests than the depth of the RDMA Read Request Queue at the Requests than the depth of the RDMA Read Request Queue at the
skipping to change at page 35, line 38 skipping to change at page 35, line 39
this could corrupt the queue. If the queue is not shared, then this could corrupt the queue. If the queue is not shared, then
the worst case is that the current Stream is disabled. One the worst case is that the current Stream is disabled. One
approach to solving the shared RDMA Read Request Queue would be approach to solving the shared RDMA Read Request Queue would be
to create thresholds, similar to those described in Section to create thresholds, similar to those described in Section
7.5.2.1 Multiple Streams Sharing Receive Buffers on page 30. A 7.5.2.1 Multiple Streams Sharing Receive Buffers on page 30. A
simpler approach is to not share RDMA Read Request Queue simpler approach is to not share RDMA Read Request Queue
resources amoung Streams or enforce hard limits of consumption resources amoung Streams or enforce hard limits of consumption
per Stream. Thus RDMA Read Request Queue resource consumption per Stream. Thus RDMA Read Request Queue resource consumption
MUST be controlled by the Privileged Resource Manager such that MUST be controlled by the Privileged Resource Manager such that
RDMAP/DDP Streams which do not share Partial Mutual Trust do not RDMAP/DDP Streams which do not share Partial Mutual Trust do not
share RDMA Read Request Queue resources. A ULP SHOULD indicate to share RDMA Read Request Queue resources.
the Privileged Resource Manager when allocating a RDMA Read
Request Queue whether or not it shares partial mutual trust with
any other Stream(s).
If the issue is a bug in the Remote PeerÆs implementation, and If the issue is a bug in the Remote Peers implementation, and
not a malicious attack, the issue can be solved by requiring the not a malicious attack, the issue can be solved by requiring the
Remote PeerÆs RNIC to throttle RDMA Read Requests. By properly Remote Peers RNIC to throttle RDMA Read Requests. By properly
configuring the Stream at the Remote Peer through a trusted configuring the Stream at the Remote Peer through a trusted
agent, the RNIC can be made to not transmit RDMA Read Requests agent, the RNIC can be made to not transmit RDMA Read Requests
that exceed the depth of the RDMA Read Request Queue at the Local that exceed the depth of the RDMA Read Request Queue at the Local
Peer. If the Stream is correctly configured, and if the Remote Peer. If the Stream is correctly configured, and if the Remote
Peer submits more requests than the Local PeerÆs RDMA Read Peer submits more requests than the Local Peers RDMA Read
Request Queue can handle, the requests would be queued at the Request Queue can handle, the requests would be queued at the
Remote PeerÆs RNIC until previous requests complete. If the Remote Peer’s RNIC until previous requests complete. If the
Remote PeerÆs Stream is not configured correctly, the RDMAP Remote Peer’s Stream is not configured correctly, the RDMAP
Stream is terminated when more RDMA Read Requests arrive at the Stream is terminated when more RDMA Read Requests arrive at the
Local Peer than the Local Peer can handle (assuming the prior Local Peer than the Local Peer can handle (assuming the prior
paragraphÆs recommendation is implemented). Thus an RNIC paragraph’s recommendation is implemented). Thus an RNIC
implementation MUST provide a mechanism to cap the number of implementation SHOULD provide a mechanism to cap the number of
outstanding RDMA Read Requests. outstanding RDMA Read Requests. The configuration of this limit
is outside the scope of this specification.
7.5.3 Resource Consumption by Idle Applications 7.5.3 Resource Consumption by Idle ULPs
The simplest form of a DOS attack given a fixed amount of The simplest form of a DOS attack given a fixed amount of
resources is for the Remote Peer to create a RDMAP Stream to a resources is for the Remote Peer to create a RDMAP Stream to a
Local Peer, and request dedicated resources then do no actual Local Peer, and request dedicated resources then do no actual
work. This allows the Remote Peer to be very light weight (i.e. work. This allows the Remote Peer to be very light weight (i.e.
only negotiate resources, but do no data transfer) and consumes a only negotiate resources, but do no data transfer) and consumes a
disproportionate amount of resources in the server. disproportionate amount of resources in the server.
A general countermeasure for this style of attack is to monitor A general countermeasure for this style of attack is to monitor
active RDMAP Streams and if resources are getting low, reap the active RDMAP Streams and if resources are getting low, reap the
skipping to change at page 37, line 23 skipping to change at page 37, line 22
Message. If the STag is only valid on the current Stream, then Message. If the STag is only valid on the current Stream, then
the only side effect is that the Remote Peer can no longer use the only side effect is that the Remote Peer can no longer use
the STag; thus there are no security issues. the STag; thus there are no security issues.
If the STag is valid across multiple Streams, then the Remote If the STag is valid across multiple Streams, then the Remote
Peer can prevent other Streams from using that STag by using the Peer can prevent other Streams from using that STag by using the
remote invalidate functionality. remote invalidate functionality.
Thus if RDDP Streams do not share Partial Mutual Trust (i.e. the Thus if RDDP Streams do not share Partial Mutual Trust (i.e. the
Remote Peer may attempt to invalidate the STag prematurely), the Remote Peer may attempt to invalidate the STag prematurely), the
application MUST NOT allow an STag to be valid across multiple ULP MUST NOT enable an STag which would be valid across multiple
Streams. Streams.
7.6 Elevation of Privilege 7.6 Elevation of Privilege
The RDMAP/DDP Security Architecture explicitly differentiates The RDMAP/DDP Security Architecture explicitly differentiates
between three levels of privilege - Non-Privileged, Privileged, between three levels of privilege - Non-Privileged, Privileged,
and the Privileged Resource Manager. If a Non-Privileged and the Privileged Resource Manager. If a Non-Privileged ULP is
Application is able to elevate its privilege level to a able to elevate its privilege level to a Privileged ULP, then
Privileged Application, then mapping a physical address list to mapping a physical address list to an STag can provide local and
an STag can provide local and remote access to any physical remote access to any physical address location on the node. If a
address location on the node. If a Privileged Mode Application is Privileged Mode ULP is able to promote itself to be a Resource
able to promote itself to be a Resource Manager, then it is Manager, then it is possible for it to perform denial of service
possible for it to perform denial of service type attacks where type attacks where substantial amounts of local resources could
substantial amounts of local resources could be consumed. be consumed.
In general, elevation of privilege is a local implementation In general, elevation of privilege is a local implementation
specific issue and thus outside the scope of this specification. specific issue and thus outside the scope of this specification.
<TBD: make more general and include authentication etc>
There is one issue worth noting, however. If the RNIC There is one issue worth noting, however. If the RNIC
implementation, by some insecure mechanism (or implementation implementation, by some insecure mechanism (or implementation
defect), can enable a Remote Peer or un-trusted Local Peer to defect), can enable a Remote Peer or un-trusted Local Peer to
load firmware into the RNIC Engine, it is possible to use the load firmware into the RNIC Engine, it is possible to use the
RNIC to attack the host. Thus, an implementation MUST NOT enable RNIC to attack the host. Thus, an implementation MUST NOT enable
firmware to be loaded on the RNIC Engine directly from a Remote firmware to be loaded on the RNIC Engine directly from a Remote
Peer, unless the Remote Peer is properly authenticated (by a Peer, unless the Remote Peer is properly authenticated (by a
mechanism outside the scope of this specification. The mechanism mechanism outside the scope of this specification. The mechanism
presumably entails authenticating that the remote application has presumably entails authenticating that the remote ULP has the
the right to perform the update), and the update is done via a right to perform the update), and the update is done via a secure
secure protocol, such as IPsec (See Section 8 Security Services protocol, such as IPsec (See Section 8 Security Services for RDMA
for RDMA and DDP on page 38). Further, an implementation MUST NOT and DDP on page 38). Further, an implementation MUST NOT allow a
allow a Non-Privileged Local Peer to update firmware in the RNIC Non-Privileged Local Peer to update firmware in the RNIC Engine.
Engine.
8 Security Services for RDMA and DDP 8 Security Services for RDMA and DDP
RDMA and DDP are used to control, read and write data buffers RDMA and DDP are used to control, read and write data buffers
over IP networks. Therefore, the control and the data packets of over IP networks. Therefore, the control and the data packets of
these protocols are vulnerable to the spoofing, tampering and these protocols are vulnerable to the spoofing, tampering and
information disclosure attacks listed in Section 7. information disclosure attacks listed in Section 7.
Generally speaking, Stream confidentiality protects against Generally speaking, Stream confidentiality protects against
eavesdropping. Stream and/or session authentication and integrity eavesdropping. Stream and/or session authentication and integrity
skipping to change at page 38, line 42 skipping to change at page 38, line 42
3. Per-packet integrity - protects against tampering done by 3. Per-packet integrity - protects against tampering done by
network based modification of buffer content (section 7.3.4) network based modification of buffer content (section 7.3.4)
4. Packet sequencing - protects against replay attacks, which is 4. Packet sequencing - protects against replay attacks, which is
a special case of the above tampering attack. a special case of the above tampering attack.
If an RDMAP/DDP Stream may be subject to impersonation attacks, If an RDMAP/DDP Stream may be subject to impersonation attacks,
or Stream hijacking attacks, it is recommended that the Stream be or Stream hijacking attacks, it is recommended that the Stream be
authenticated, integrity protected, and protected from replay authenticated, integrity protected, and protected from replay
attacks; it MAY use confidentiality protection to protect from attacks; it may use confidentiality protection to protect from
eavesdropping (in case the RDMAP/DDP Stream traverses a public eavesdropping (in case the RDMAP/DDP Stream traverses a public
network). network).
Both IPsec and SSL are capable of providing the above security Both IPsec and SSL are capable of providing the above security
services for IP and TCP traffic respectively. ULP protocols are services for IP and TCP traffic respectively. ULP protocols are
able to provide only part of the above security services. The able to provide only part of the above security services. The
next sections describe the different security options. next sections describe the different security options.
8.1.1 Introduction to IPsec 8.1.1 Introduction to IPsec
skipping to change at page 40, line 10 skipping to change at page 40, line 10
existing IPsec SA pair is accomplished by creating two new IPsec existing IPsec SA pair is accomplished by creating two new IPsec
SAs, making them active, and then optionally deleting the older SAs, making them active, and then optionally deleting the older
IPsec SA pair. Typically the new outbound SA is used immediately, IPsec SA pair. Typically the new outbound SA is used immediately,
and the old inbound SA is left active to receive packets for some and the old inbound SA is left active to receive packets for some
locally defined time, perhaps 30 seconds or 1 minute. Optionally, locally defined time, perhaps 30 seconds or 1 minute. Optionally,
rekeying can use Diffie-Helman for keying material generation. rekeying can use Diffie-Helman for keying material generation.
8.1.2 Introduction to SSL Limitations on RDMAP 8.1.2 Introduction to SSL Limitations on RDMAP
SSL and TLS [RFC 2246] provide Stream authentication, integrity SSL and TLS [RFC 2246] provide Stream authentication, integrity
and confidentiality for TCP based applications. SSL supports one- and confidentiality for TCP based ULPs. SSL supports one-way
way (server only) or mutual certificates based authentication. (server only) or mutual certificates based authentication.
There are at least two limitations that make SSL underneath RDMAP There are at least two limitations that make SSL underneath RDMAP
less appropriate then IPsec for DDP/RDMA security: less appropriate then IPsec for DDP/RDMA security:
1. The maximum length supported by the TLS record layer protocol 1. The maximum length supported by the TLS record layer protocol
is 2^14 bytes - longer packets must be fragmented (as a is 2^14 bytes - longer packets must be fragmented (as a
comparison, the maximal length of an IPsec packet is comparison, the maximal length of an IPsec packet is
determined by the maximum length of an IP packet). determined by the maximum length of an IP packet).
2. SSL is a connection oriented protocol. If a stream cipher or 2. SSL is a connection oriented protocol. If a stream cipher or
skipping to change at page 40, line 35 skipping to change at page 40, line 35
traffic, then SSL must gather all out-of-order packets before traffic, then SSL must gather all out-of-order packets before
RDMAP/DDP can place them into the ULP buffer, which might RDMAP/DDP can place them into the ULP buffer, which might
cause a significant decrease in its efficiency. cause a significant decrease in its efficiency.
If SSL is layered on top of RDMAP or DDP, SSL does not protect If SSL is layered on top of RDMAP or DDP, SSL does not protect
the RDMAP and/or DDP headers. Thus a man-in-the-middle attack can the RDMAP and/or DDP headers. Thus a man-in-the-middle attack can
still occur by modifying the RDMAP/DDP header to incorrectly still occur by modifying the RDMAP/DDP header to incorrectly
place the data into the wrong buffer, thus effectively corrupting place the data into the wrong buffer, thus effectively corrupting
the data stream. the data stream.
8.1.3 Applications Which Provide Security 8.1.3 ULPs Which Provide Security
Applications which provide integrated security but wish to ULPs which provide integrated security but wish to leverage
leverage lower-layer protocol security should be aware of lower-layer protocol security should be aware of security
security concerns around correlating a specific channelÆs concerns around correlating a specific channel’s security
security mechanisms to the authentication performed by the mechanisms to the authentication performed by the ULP. See
application. See [NFSv4CHANNEL] for additional information on a [NFSv4CHANNEL] for additional information on a promising approach
promising approach called "channel binding". From [NFSv4CHANNEL]: called "channel binding". From [NFSv4CHANNEL]:
The concept of channel bindings allows applications to prove "The concept of channel bindings allows applications to
that the end-points of two secure channels at different prove that the end-points of two secure channels at
network layers are the same by binding authentication at one different network layers are the same by binding
channel to the session protection at the other channel. The authentication at one channel to the session protection at
use of channel bindings allows applications to delegate the other channel. The use of channel bindings allows
session protection to lower layers, which may significantly applications to delegate session protection to lower layers,
improve performance for some applications. which may significantly improve performance for some
applications."
8.2 Requirements for IPsec Encapsulation of DDP 8.2 Requirements for IPsec Encapsulation of DDP
The IP Storage working group has spent significant time and The IP Storage working group has spent significant time and
effort to define the normative IPsec requirements for IP Storage effort to define the normative IPsec requirements for IP Storage
[RFC3723]. Portions of that specification are applicable to a [RFC3723]. Portions of that specification are applicable to a
wide variety of protocols, including the RDDP protocol suite. In wide variety of protocols, including the RDDP protocol suite. In
order to not replicate this effort, an RNIC implementation MUST order to not replicate this effort, an RNIC implementation MUST
follow the requirements defined in RFC3723 Section 2.3 and follow the requirements defined in RFC3723 Section 2.3 and
Section 5, including the associated normative references for Section 5, including the associated normative references for
skipping to change at page 44, line 5 skipping to change at page 44, line 5
10.2 Informative References 10.2 Informative References
[IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor [IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor
Discovery Trust Models and threats", Internet-Draft draft- Discovery Trust Models and threats", Internet-Draft draft-
ietf-send-psreq-01.txt, January 2003. ietf-send-psreq-01.txt, January 2003.
[NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to [NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to
Secure Channels", Internet-Draft draft-ietf-nfsv4-channel- Secure Channels", Internet-Draft draft-ietf-nfsv4-channel-
bindings-02.txt, July 2004. bindings-02.txt, July 2004.
11 Appendix A: Implementing Client/Server Protocols 11 Appendix A: ULP Issues for RDDP Client/Server Protocols
This section is a normative appendix to the specification that is This section is a normative appendix to the specification that is
focused on client/server application implementation requirements focused on client/server ULP implementation requirements to
to ensure a secure server implementation. ensure a secure server implementation.
The prior sections outlined specific attacks and their The prior sections outlined specific attacks and their
countermeasures. This section summarizes the attacks and countermeasures. This section summarizes the attacks and
countermeasures that have been defined in the prior section which countermeasures that have been defined in the prior section which
are applicable to creation of a secure application server. An are applicable to creation of a secure ULP (e.g. application)
application server is defined as an application which must be server. A ULP server is defined as a ULP which must be able to
able to communicate with many clients which do not trust each communicate with many clients which do not trust each other and
other and ensure that each client can not attack another client ensure that each client can not attack another client through
through server interactions. Further, the server may wish to use server interactions. Further, the server may wish to use multiple
multiple Streams to communicate with a specific client, and those Streams to communicate with a specific client, and those Streams
Streams may share mutual trust. Note that this section assumes a may share mutual trust. Note that this section assumes a
compliant RNIC and Privileged Resource Manager implementation - compliant RNIC and Privileged Resource Manager implementation -
thus it focuses specifically on application server (i.e. ULP) thus it focuses specifically on ULP server (e.g. application)
implementation issues. implementation issues.
All of the prior section's details on attacks and countermeasures All of the prior section's details on attacks and countermeasures
apply to the server. In some cases normative SHOULD statements apply to the server. In some cases normative SHOULD statements
for the ULP in the main body of this document are made MUST for the ULP in the main body of this document are made MUST
statements for the ULP because the operating conditions can be statements for the ULP because the operating conditions can be
refined to make the motives for a SHOULD inapplicable. If a prior refined to make the motives for a SHOULD inapplicable. If a prior
SHOULD is changed to a MUST in this section, it is explicitly SHOULD is changed to a MUST in this section, it is explicitly
noted. noted.
skipping to change at page 44, line 45 skipping to change at page 44, line 45
normative statements to be client/server specific: normative statements to be client/server specific:
* Spoofing * Spoofing
* Sections 7.2.1 to 7.2.3. For protection against many * Sections 7.2.1 to 7.2.3. For protection against many
forms of spoofing attacks, enable IPsec. forms of spoofing attacks, enable IPsec.
* Section 7.2.4 Using an STag on a Different Stream on * Section 7.2.4 Using an STag on a Different Stream on
page 23. To ensure that one client can not access page 23. To ensure that one client can not access
another client's data via use of the other client's another client's data via use of the other client's
STag, the server application MUST either scope an STag, the server ULP MUST either scope an STag to a
STag to a single Stream or use a Protection Domain single Stream or use a Protection Domain per client.
per client. If a single client has multiple streams If a single client has multiple streams that share
that share Partial Mutual Trust, then the STag can be Partial Mutual Trust, then the STag can be shared
shared between the associated Streams by using a between the associated Streams by using a single
single Protection Domain amoung the associated Protection Domain amoung the associated Streams (see
Streams (see section 8.1.3 Applications Which Provide for additional issues). To prevent unintended sharing
Security on page 40 for additional issues). To of STags within the associated Streams, a server ULP
prevent unintended sharing of STags within the SHOULD use STags in such a fashion that it is
associated Streams, a server application SHOULD use difficult to predict the next allocated STag number.
STags in such a fashion that it is difficult to
predict the next allocated STag number.
* Tampering * Tampering
* 7.3.2 Modifying a Buffer After Indication on page 25. * 7.3.2 Modifying a Buffer After Indication on page 25.
Before the server application operates on a buffer Before the server ULP operates on a buffer that was
that was written using an RDMA Write or RDMA Read, written using an RDMA Write or RDMA Read, the server
the server MUST ensure the the buffer can no longer MUST ensure the the buffer can no longer be modified
be modified by invalidating the STag for remote by invalidating the STag for remote access (note that
access (note that this is stronger than the SHOULD in this is stronger than the SHOULD in section 7.3.2).
section 7.3.2). This can either be done explicitly by This can either be done explicitly by revoking remote
revoking remote access rights for the STag when the access rights for the STag when the client indicates
client indicates the operation has completed, or by the operation has completed, or by checking to make
checking to make sure the client Invalidated the STag sure the client Invalidated the STag through the
through the RDMAP Invalidate capability, and if it RDMAP Invalidate capability, and if it did not, the
did not, the Local Peer then explicitly revokes the Local Peer then explicitly revokes the STag remote
STag remote access rights. access rights.
* Information Disclosure * Information Disclosure
* 7.4.2 Using RDMA Read to Access Stale Data on page * 7.4.2 Using RDMA Read to Access Stale Data on page
26. A server application MUST (this is stronger than 26. A server ULP MUST (this is stronger than the
the SHOULD in section 7.4.2) ensure that no stale SHOULD in section 7.4.2) ensure that no stale data is
data is contained in a buffer before remote read contained in a buffer before remote read access
access rights are granted to a client (this can be rights are granted to a client (this can be done by
done by zeroing the contents of the memory, for zeroing the contents of the memory, for example).
example).
* 7.4.3 Accessing a Buffer After the Transfer on page * 7.4.3 Accessing a Buffer After the Transfer on page
26. This mitigation is already covered by section 26. This mitigation is already covered by section
7.3.2 (above). 7.3.2 (above).
* 7.4.4 Accessing Unintended Data With a Valid STag on * 7.4.4 Accessing Unintended Data With a Valid STag on
page 26. The application server MUST set the base and page 26. The ULP server MUST set the base and bounds
bounds of the buffer when the STag is initialized to of the buffer when the STag is initialized to expose
expose only the data to be retrieved. only the data to be retrieved.
* 7.4.5 RDMA Read into an RDMA Write Buffer on page 27. * 7.4.5 RDMA Read into an RDMA Write Buffer on page 27.
If a server only intends a buffer to be exposed for If a server only intends a buffer to be exposed for
remote write access, it MUST set the access rights to remote write access, it MUST set the access rights to
the buffer to only enable remote write access. the buffer to only enable remote write access.
* 7.4.6 Using Multiple STags Which Alias to the Same * 7.4.6 Using Multiple STags Which Alias to the Same
Buffer on page 27. The requirement in section 7.2.4 Buffer on page 27. The requirement in section 7.2.4
(above) mitigates this attack. A buffer is exposed to (above) mitigates this attack. A buffer is exposed to
only one client at a time to ensure that no only one client at a time to ensure that no
information disclosure or information tampering information disclosure or information tampering
occurs between peers. occurs between peers.
* 7.4.9 Network based eavesdropping on page 28. Enable * 7.4.9 Network based eavesdropping on page 28. Enable
IPsec if this threat is a concern. IPsec if this threat is a concern.
* Denial of Service * Denial of Service
* 7.5.2.1 Multiple Streams Sharing Receive Buffers on * 7.5.2.1 Multiple Streams Sharing Receive Buffers on
page 30. Application memory footprint size can be page 30. ULP memory footprint size can be important
important for some server applications. If a server for some server ULPs. If a server ULP is expecting
application is expecting significant network traffic significant network traffic from multiple clients,
from multiple clients, using a receive buffer queue using a receive buffer queue per Stream where there
per Stream where there is a large number of Streams is a large number of Streams can consume substantial
can consume substantial amounts of memory. Thus a amounts of memory. Thus a receive queue that can be
receive queue that can be shared by multiple Streams shared by multiple Streams is attractive.
is attractive.
However, because of the attacks outlined in this However, because of the attacks outlined in this
section, sharing a single receive queue between section, sharing a single receive queue between
multiple clients MUST ONLY be done if the mechanism multiple clients MUST only be done if the mechanism
to ensure one client can't consume too many receive to ensure one client can't consume too many receive
buffers is enabled. For multiple Streams within a buffers is enabled. For multiple Streams within a
single client application (which presumably shared single client ULP (which presumably shared partial
partial mutual trust) this added overhead does not mutual trust) this added overhead does not have to be
have to be enabled. enabled.
* 7.5.2.2 Local Peer Attacking a Shared CQ on page 31. * 7.5.2.2 Local ULP Attacking a Shared CQ on page 31.
<TBD>The normative mitigations were <TBD>The normative mitigations were
* RNIC MUST NOT enable sharing a CQ across Streams * RNIC MUST NOT enable sharing a CQ across Streams
that belong to different Protection Domains. that belong to different Protection Domains.
* An application SHOULD NOT share a CQ between * A ULP SHOULD NOT share a CQ between Streams which
Streams which do not share Partial Mutual Trust. do not share Partial Mutual Trust.
Because the attack is a local server application Because the attack is a local server ULP attacking
attacking another server application, another server ULP,
* 7.5.2.3 Remote Peer Attacking a Shared CQ on page 32. * 7.5.2.3 Remote Peer Attacking a Shared CQ on page 32.
There are two mitigations specified in this section - There are two mitigations specified in this section -
one requires a worst-case size of the CQ, and can be one requires a worst-case size of the CQ, and can be
implemented entirely within the Privileged Resource implemented entirely within the Privileged Resource
Manager. The second approach requires cooperation Manager. The second approach requires cooperation
with the local application server (to not post too with the local ULP server (to not post too many
many buffers), and enables a smaller CQ to be used. buffers), and enables a smaller CQ to be used.
In some server environments, partial trust of the In some server environments, partial trust of the
server application (but not the clients) is server ULP (but not the clients) is acceptable, thus
acceptable, thus the smaller CQ fully mitigates the the smaller CQ fully mitigates the remote attacker.
remote attacker. In other environments, the local In other environments, the local server ULP could
server application could also contain untrusted also contain untrusted elements which can attack the
elements which can attack the local machine (or have local machine (or have bugs). In those environments,
bugs). In those environments, the worst-case size of the worst-case size of the CQ must be used.
the CQ must be used.
* 7.5.2.4 The section requires a Privileged Resource * 7.5.2.4 The section requires a Privileged Resource
Manager to not enable sharing of RDMA Read Request Manager to not enable sharing of RDMA Read Request
Queues across multiple Streams that do not share Queues across multiple Streams that do not share
partial mutual trust. However, because the server partial mutual trust. However, because the server ULP
application knows best which of its Streams share knows best which of its Streams share partial mutual
partial mutual trust, this requirement can be trust, this requirement can be reflected back to the
reflected back to the application. The ULP (i.e. ULP. The ULP (i.e. server) requirement is that it
server) requirement is that it MUST NOT request RDMA MUST NOT request RDMA Read Request Queues to be
Read Request Queues to be shared between applications shared between ULPs which do not have partial mutual
which do not have partial mutual trust. trust.
* 7.5.5 Remote Invalidate an STag Shared on Multiple * 7.5.5 Remote Invalidate an STag Shared on Multiple
Streams on page 37. This mitigation is already Streams on page 37. This mitigation is already
covered by section 7.3.2 (above). covered by section 7.3.2 (above).
12 Appendix B: Summary of RNIC and ULP Implementation Requirements 12 Appendix B: Summary of RNIC and ULP Implementation Requirements
Below is a summary of implementation requirements for the RNIC: Below is a summary of implementation requirements for the RNIC:
* 7.3.1 Buffer Overrun - RDMA Write or Read Response * 7.3.1 Buffer Overrun - RDMA Write or Read Response
* 7.4.8 Controlling Access to PTT & STag Mapping * 7.4.8 Controlling Access to PTT & STag Mapping
* 7.5.1 RNIC Resource Consumption * 7.5.1 RNIC Resource Consumption
* 7.5.2.1 Multiple Streams Sharing Receive Buffers * 7.5.2.1 Multiple Streams Sharing Receive Buffers
* 7.5.2.2 Local Peer Attacking a Shared CQ * 7.5.2.2 Local ULP Attacking a Shared CQ
* 7.5.2.3 Remote Peer Attacking a Shared CQ * 7.5.2.3 Remote Peer Attacking a Shared CQ
* 7.5.2.4 Attacking the RDMA Read Request Queue * 7.5.2.4 Attacking the RDMA Read Request Queue
* 7.5.4 Exercise of non-optimal code paths * 7.5.4 Exercise of non-optimal code paths
* 7.6 Elevation of Privilege * 7.6 Elevation of Privilege
* 8.2 Requirements for IPsec Encapsulation of DDP * 8.2 Requirements for IPsec Encapsulation of DDP
Below is a summary of implementation requirements for the Below is a summary of implementation requirements for the ULP
application above the RNIC: above the RNIC:
* 7.2.4 Using an STag on a Different Stream * 7.2.4 Using an STag on a Different Stream
* 7.3.2 Modifying a Buffer After Indication * 7.3.2 Modifying a Buffer After Indication
* 7.4.2 Using RDMA Read to Access Stale Data * 7.4.2 Using RDMA Read to Access Stale Data
* 7.4.3 Accessing a Buffer After the Transfer * 7.4.3 Accessing a Buffer After the Transfer
* 7.4.4 Accessing Unintended Data With a Valid STag * 7.4.4 Accessing Unintended Data With a Valid STag
* 7.4.5 RDMA Read into an RDMA Write Buffer * 7.4.5 RDMA Read into an RDMA Write Buffer
* 7.4.6 Using Multiple STags Which Alias to the Same Buffer * 7.4.6 Using Multiple STags Which Alias to the Same Buffer
* 7.4.9 Network based eavesdropping * 7.4.9 Network based eavesdropping
* 7.5.2.2 Local Peer Attacking a Shared CQ * 7.5.2.2 Local ULP Attacking a Shared CQ
* 7.5.5 Remote Invalidate an STag Shared on Multiple * 7.5.5 Remote Invalidate an STag Shared on Multiple
Streams Streams
13 Appendix C: Partial Trust Taxonomy 13 Appendix C: Partial Trust Taxonomy
Partial Trust is defined as when one party is willing to assume Partial Trust is defined as when one party is willing to assume
that another party will refrain from a specific attack or set of that another party will refrain from a specific attack or set of
attacks, the parties are said to be in a state of Partial Trust. attacks, the parties are said to be in a state of Partial Trust.
Note that the partially trusted peer may attempt a different set Note that the partially trusted peer may attempt a different set
of attacks. This may be appropriate for many applications where of attacks. This may be appropriate for many ULPs where any
any adverse effects of the betrayal is easily confined and does adverse effects of the betrayal is easily confined and does not
not place other clients or applications at risk. place other clients or ULPs at risk.
The Trust Models described in this section have three primary The Trust Models described in this section have three primary
distinguishing characteristics. The Trust Model refers to a Local distinguishing characteristics. The Trust Model refers to a Local
Peer and Remote Peer, which are the local and remote application Peer and Remote Peer, which are the local and remote ULP
instances communicating via RDMA/DDP. instances communicating via RDMA/DDP.
* Local Resource Sharing (yes/no) - When local resources * Local Resource Sharing (yes/no) - When local resources
are shared, they are shared across a grouping of are shared, they are shared across a grouping of
RDMAP/DDP Streams. If local resources are not shared, the RDMAP/DDP Streams. If local resources are not shared, the
resources are dedicated on a per Stream basis. Resources resources are dedicated on a per Stream basis. Resources
are defined in Section 4.2 - Resources on page 11. The are defined in Section 4.2 - Resources on page 11. The
advantage of not sharing resources between Streams is advantage of not sharing resources between Streams is
that it reduces the types of attacks that are possible. that it reduces the types of attacks that are possible.
The disadvantage is that applications might run out of The disadvantage is that ULPs might run out of resources.
resources.
* Local Partial Trust (yes/no) - Local Partial Trust is * Local Partial Trust (yes/no) - Local Partial Trust is
determined based on whether the local grouping of determined based on whether the local grouping of
RDMAP/DDP Streams (which typically equates to one RDMAP/DDP Streams (which typically equates to one ULP or
application or group of applications) mutually trust each group of ULPs) mutually trust each other to not perform a
other to not perform a specific set of attacks. specific set of attacks.
* Remote Partial Trust (yes/no) - The Remote Partial Trust * Remote Partial Trust (yes/no) - The Remote Partial Trust
level is determined based on whether the Local Peer of a level is determined based on whether the Local Peer of a
specific RDMAP/DDP Stream partially trusts the Remote specific RDMAP/DDP Stream partially trusts the Remote
Peer of the Stream (see the definition of Partial Trust Peer of the Stream (see the definition of Partial Trust
in Section 3 Introduction). in Section 3 Introduction).
Not all of the combinations of the trust characteristics are Not all of the combinations of the trust characteristics are
expected to be used by applications. This paper specifically expected to be used by ULPs. This paper specifically analyzes
analyzes five application Trust Models that are expected to be in five ULP Trust Models that are expected to be in common use. The
common use. The Trust Models are as follows: Trust Models are as follows:
* NS-NT - Non-Shared Local Resources, no Local Trust, no * NS-NT - Non-Shared Local Resources, no Local Trust, no
Remote Trust - typically a server application that wants Remote Trust - typically a server ULP that wants to run
to run in the safest mode possible. All attack in the safest mode possible. All attack mitigations are
mitigations are in place to ensure robust operation. in place to ensure robust operation.
* NS-RT - Non-Shared Local Resources, no Local Trust, * NS-RT - Non-Shared Local Resources, no Local Trust,
Remote Partial Trust - typically a peer-to-peer Remote Partial Trust - typically a peer-to-peer ULP,
application, which has, by some method outside of the which has, by some method outside of the scope of this
scope of this specification, authenticated the Remote specification, authenticated the Remote Peer. Note that
Peer. Note that unless some form of key based unless some form of key based authentication is used on a
authentication is used on a per RDMA/DDP Stream basis, it per RDMA/DDP Stream basis, it may not be possible be
may not be possible be possible for man-in-the-middle possible for man-in-the-middle attacks to occur. See
attacks to occur. See section 8, Security Services for section 8, Security Services for RDMA and DDP on page 38.
RDMA and DDP on page 38.
* S-NT - Shared Local Resources, no Local Trust, no Remote * S-NT - Shared Local Resources, no Local Trust, no Remote
Trust - typically a server application that runs in an Trust - typically a server ULP that runs in an untrusted
untrusted environment where the amount of resources environment where the amount of resources required is
required is either too large or too dynamic to dedicate either too large or too dynamic to dedicate for each
for each RDMAP/DDP Stream. RDMAP/DDP Stream.
* S-LT - Shared Local Resources, Local Partial Trust, no * S-LT - Shared Local Resources, Local Partial Trust, no
Remote Trust - typically an application, which provides a Remote Trust - typically a ULP, which provides a session
session layer and uses multiple Streams, to provide layer and uses multiple Streams, to provide additional
additional throughput or fail-over capabilities. All of throughput or fail-over capabilities. All of the Streams
the Streams within the local application partially trust within the local ULP partially trust each other, but do
each other, but do not trust the Remote Peer. This trust not trust the Remote Peer. This trust model may be
model may be appropriate for embedded environments. appropriate for embedded environments.
* S-T - Shared Local Resources, Local Partial Trust, Remote * S-T - Shared Local Resources, Local Partial Trust, Remote
Partial Trust - typically a distributed application, such Partial Trust - typically a distributed application, such
as a distributed database application or a High as a distributed database application or a High
Performance Computer (HPC) application, which is intended Performance Computer (HPC) application, which is intended
to run on a cluster. Due to extreme resource and to run on a cluster. Due to extreme resource and
performance requirements, the application typically performance requirements, the application typically
authenticates with all of its peers and then runs in a authenticates with all of its peers and then runs in a
highly trusted environment. The application peers are all highly trusted environment. The application peers are all
in a single application fault domain and depend on one in a single application fault domain and depend on one
another to be well-behaved when accessing data another to be well-behaved when accessing data
structures. If a trusted Remote Peer has an structures. If a trusted Remote Peer has an
implementation defect that results in poor behavior, the implementation defect that results in poor behavior, the
entire application could be corrupted. entire application could be corrupted.
Models NS-NT and S-NT above are typical for Internet networking - Models NS-NT and S-NT above are typical for Internet networking -
neither Local Peers nor the Remote Peer is trusted. Sometimes neither Local Peers nor the Remote Peer is trusted. Sometimes
optimizations can be done that enable sharing of Page Translation optimizations can be done that enable sharing of Page Translation
Tables across multiple Local Peers, thus Model S-LT can be Tables across multiple Local Peers, thus Model S-LT can be
advantageous. Model S-T is typically used when resource scaling advantageous. Model S-T is typically used when resource scaling
across a large parallel application makes it infeasible to use across a large parallel ULP makes it infeasible to use any other
any other model. Resource scaling issues can either be due to model. Resource scaling issues can either be due to performance
performance around scaling or because there simply are not enough around scaling or because there simply are not enough resources.
resources. Model NS-RT is probably the least likely model to be Model NS-RT is probably the least likely model to be used, but is
used, but is presented for completeness. presented for completeness.
14 AuthorÆs Addresses 14 Authors Addresses
James Pinkerton James Pinkerton
Microsoft Corporation Microsoft Corporation
One Microsoft Way One Microsoft Way
Redmond, WA. 98052 USA Redmond, WA. 98052 USA
Phone: +1 (425) 705-5442 Phone: +1 (425) 705-5442
Email: jpink@windows.microsoft.com Email: jpink@windows.microsoft.com
Ellen Deleganes Ellen Deleganes
Intel Corporation Intel Corporation
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/