Internet Draft James Pinkerton
Document: draft-ietf-rddp-security-00.txtdraft-ietf-rddp-security-01.txt Microsoft Corporation Expires: April,September, 2004 Ellen Deleganes Intel Corporation Allyn Romanow Cisco Systems Bernard AbobaSara Bitan Microsoft Corporation October 2003February 2004 DDP/RDMAP Security 1 Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-DraftsInternet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 2 Abstract This document analyzes security issues around implementation and use of the Direct Data Placement Protocol(DDP) and Remote Direct Memory Access Protocol (RDMAP). It first defines an architectural model for an RDMA Network Interface Card (RNIC), which can implement DDP or RDMAP and DDP. The model includes a definition of resources that can be attacked. This document then introduces various Trust Models between a local peer and a remote peer and the tools that can be used to create countermeasures against attacks. Finally, thedocument reviews various attacks against the resources defined in the architectural model and the countermeasures tothat can be used against them, groupingto protect the attackssystem. Attacks are grouped into spoofing, tampering, information disclosure, denial of service, and elevation of privilege. Finally, the document concludes with a summary of security services for RDDP, such as IPSec. J. Pinkerton, et al. Expires September 2004 1 Table of Contents 1 Status of this Memo.........................................1 2 Abstract....................................................1 2.1 Issues......................................................3 2.2 Revision History............................................4 2.2.1 Changes from the -00 to -01 version........................4 3 Introduction................................................5Introduction................................................6 4 Architectural Model.........................................7Model.........................................8 4.1 Components..................................................8Components..................................................9 4.2 Resources..................................................10Resources..................................................11 4.2.1 ConnectionStream Context Memory.................................10Memory.....................................11 4.2.2 Data Buffers..............................................10Buffers..............................................11 4.2.3 Page Translation Tables...................................11 4.2.4 STag Namespace............................................11Namespace............................................12 4.2.5 Completion Queues.........................................11Queues.........................................12 4.2.6 Asynchronous Event Queue..................................12 4.2.7 RDMA Read Request Queue...................................11 4.2.7 RDMA Asynchronous Event Queue.............................11Queue...................................13 4.2.8 RNIC Interactions.........................................13 220.127.116.11 Privileged Control Interactions.................................12Interface Semantics................13 18.104.22.168 Non-Privileged Data Interface Semantics...............13 22.214.171.124 Privileged Data Interface Semantics...................14 4.2.9 Initialization of RNIC Data Structures for Data Transfer..12Transfer..14 4.2.10 RNIC Data Transfer Interactions.........................13 4.3 System Properties..........................................14Interactions.........................15 5 Trust Models...............................................15and Resource Sharing.................................17 6 Attacker Capabilities......................................17Capabilities......................................18 7 Attacks and Countermeasures................................18Countermeasures................................19 7.1 Tools for Countermeasures..................................18Countermeasures..................................19 7.1.1 Protection Domain (PD)....................................18(PD)....................................19 7.1.2 Limiting STag Scope.......................................19Scope.......................................20 7.1.3 Access Rights.............................................19Rights.............................................21 7.1.4 Limiting the Scope of the Completion Queue................20Queue................21 7.1.5 Limiting the Scope of an Error............................20Error............................21 7.2 Spoofing...................................................20Spoofing...................................................21 7.2.1 Connection Hijacking......................................20Impersonation.............................................22 7.2.2 Stream Hijacking..........................................22 7.2.3 Man in the Middle Attack..................................22 7.2.4 Using an STag on a different connection...................21Different Stream.......................23 7.3 Tampering..................................................22Tampering..................................................24 7.3.1 Buffer Overrun - RDMA Write or Read Response to Memory Outside the Buffer..22Response..............24 7.3.2 Modifying a Buffer After Indicating Contents Are Ready....23Indication.......................25 7.3.3 UsingMultiple StagsSTags to access the same buffer............23buffer..................25 7.3.4 Network based modification of buffer content..............25 7.4 Information Disclosure.....................................23Disclosure.....................................25 7.4.1 Probing memory outside of the buffer bounds...............24bounds...............26 7.4.2 Using RDMA Read to Access Stale Data......................24Data......................26 7.4.3 Accessing a buffer afterBuffer After the transfer is over.............24Transfer.....................26 7.4.4 Accessing data withinUnintended Data With a valid STag that was unintended....24Valid STag...............26 7.4.5 UsingRDMA Read on a buffer meant only forinto an RDMA Write.....25Write Buffer.......................27 7.4.6 Using Multiple StagsSTags to access the same buffer............25Access One Buffer.................27 7.4.7 Remote node loading firmwareNode Loading Firmware onto the RNIC................26RNIC................28 7.4.8 Controlling Access to Page Translation Table andPTT & STag Mapping 26Mapping..................28 7.4.9 Network based eaves dropping..............................28 7.5 Denial of Service (DOS)....................................26(DOS)....................................28 7.5.1 RNIC Resource Consumption.................................27Consumption.................................29 7.5.2 Resource Consumption By Active Applications...............27Applications...............30 126.96.36.199 Multiple Streams Sharing Receive Data Buffers..................................28Buffers..............30 188.8.131.52 Completion Queue (CQ) Resource Consumption............29Local Peer Attacking a Shared CQ......................31 184.108.40.206 Remote Peer Attacking a Shared CQ.....................32 220.127.116.11 RDMA Read Request Queue...............................31Queue...............................34 7.5.3 Resource Consumption by Idle Applications.................32Applications.................35 7.5.4 Exercise of non-optimal code paths........................32paths........................35 7.5.5 Remote Invalidation ofRI an STag Shared Acrosson Multiple Connections......................................................32Streams.....................36 7.5.6 Remote Peer Consumes too manyUntagged Receive Buffers....33Buffers.............36 7.6 Elevation of Privilege.....................................33 7.6.1 Loading Firmware into the RNIC............................34Privilege.....................................36 8 Security Services for RDDP.................................35RDDP.................................38 8.1 Introduction to Security Options...........................35Options...........................38 8.1.1 Introduction to IPsec.....................................35IPsec.....................................39 8.1.2 Introduction to SSL Limitations on RDMAP..................36RDMAP..................40 8.1.3 Applications Which Provide Security.......................36 8.1.4 Authentication Only.......................................36 8.1.5 Privacy...................................................36Security.......................40 8.2 Recommendations for IPsec Encapsulation of RDDP............36RDDP............40 8.2.1 Transforms................................................41 8.2.2 IPsec modes...............................................41 8.2.3 IKE.......................................................41 8.2.4 Security Policy Configuration.............................43 9 Summary Table of Attacks and Trust Models..................38Security considerations....................................45 10 References.................................................40References.................................................46 10.1 Normative References......................................40References......................................46 10.2 Informative References....................................40References....................................47 11 AuthorÆs Addresses.........................................41Appendix A: Implementing Client/Server Protocols...........48 12 Acknowledgments............................................42Appendix B: Summary Table of Attacks.......................51 12.1 Spoofing..................................................52 12.2 Tampering.................................................52 12.3 Information Disclosure....................................52 12.4 Denial of Service.........................................52 13 Appendix C: Partial Trust Taxonomy.........................54 14 AuthorÆs Addresses.........................................56 15 Acknowledgments............................................57 16 Full Copyright Statement...................................43Statement...................................58 Table of Figures Figure 1 - RDMA Security Model....................................8Model....................................9 Figure 2 - Summary Attacks and Trust Model Table.................39Table.................53 2.1 Issues This section is temporary and will go away when all issues have been resolved. Note: this is far from a complete list of issues; as more are raised, they will be added to this list until some sort of consensus is reached. They are in the order found in the specification. Issue: Discuss issues in allowing the non-privileged consumer<TBD û remove this section: this section was deleted because it was a duplicate of Section 18.104.22.168 Multiple Streams Sharing Receive Buffers on page 30) Thus comments on this section were added to control mapping the Page Translation Table and Stag..............26that section.>..........................................36 Issue: NeedThe spec currently makes specific IPsec SHOULD recommendations. Should this be relaxed to analyzenot be normative, since the case of sharing a queue of Untagged receive buffers across multiple connections, and that the Remote Peer can mount a denial of service attack........................33 Issue: Security Services sectionprotocol is just a placeholder for now........35transport protocol, not an application protocol?............................................38 Issue: Guidance for application protocols like NFS which implement security.........................................................36security <TBD>.........................................40 Issue: IPsec recommendationsI think we should refer to IPS security considerations. Most of the issues discussed there are relevant for RDMAP/DDP.......................36RDDP/RDMA as well (exceptions are the discussion on user certificates).<TBD>..45 Issue: Finish Summary table of Attacks/Trust Models..............38Models <TBD>........51 2.2 Revision History 2.2.1 Changes from the -00 to -01 version * Added two pages to the architectural model to describe the Asynchronous Event Queue, and the types of interactions that can occur between the RNIC and the modules above it. * Addressed Mike Krauses comments submitted on 12/8/2003 * Moved "Trust Models" from the body of the document to an appendix. Removed references to it throughout the document (including use of "partial trust". Document now assumes Remote Peer is untrusted. Thus the key issue is whether local resources are shared, and what the resource is. * Misc cleanup throughout the document. * The Summary of Attacks at the end of the document is now an Appendix. It also now provides a summary. Cleared change bars because became unreadable. Also shortened section names for attacks to fit in table. * Added a new concept of "Partial Mutual Trust" between a collection of Streams to better characterize a set of attacks in a client/server environment. * Filled in Security Services for RDDP section (almost all is new, except IPsec overview). * Globally tried to change "connection" to "Stream". In some cases it can be either a connection or stream. 3 Introduction RDMA enables new levels of flexibility when communicating between two parties compared to current conventional networking practice (e.g. a stream-based model or datagram model). This flexibility brings new security issues that must be carefully understood when designing application protocols utilizing RDMA and when implementing RDMA-aware NICs (RNICs). Note that for the purposes of this security analysis, an RNIC may implement RDMAP and DDP, or just DDP. The specification first develops an architectural model that is relevant for the security analysis - it details components, resources, and system properties that may be attacked. The specificationattacked in Section 4. It then defines Partial Trust Models. Partial Trustwhat resources a ULP may share locally across Streams and what resources the ULP may share with the Remote Peer across Streams in Section 5. In general, intentional sharing of resources between multiple Streams implies a trust model between the Streams. This is defined as: Partial Mutual Trust - When one party isû a collection of RDMAP/DDP Streams, which represent the local and remote end points of the Stream, are willing to assume that another party will refrainthe Streams from a specific attack or setthe collection will not perform malicious attacks against any of attacks,the parties are said to beStreams in a state of Partial Trust. Note thatthe partially trusted peer may attempt a different setcollection. Applications have explicit control of attacks. This may be appropriate for many applications where any adverse effectswhich collection of the betrayalendpoints is easily confined and does not place other clients or applications at risk.in the collection through tools discussed in Section 7.1 Tools for Countermeasures on page 19. An Untrusted Peeruntrusted peer relationship is appropriate when an application wishes to ensure that it will be robust and uncompromised even in the face of a deliberate attack by its peer. For example, a single application that concurrently supports multiple unrelated sessions (e.g. a server) would presumably treat each of its peers as an Untrusted Peer.untrusted peer. For a collection of Streams which share Partial Mutual Trust, the assumption is that any Stream not in the collection is untrusted. For the Untrusteduntrusted peer, a brief list of capabilities is enumerated.enumerated in Section 6. The rest of the specification is focused on analyzing attacks. First, the tools for mitigating attacks are listed,listed (Section 7.1), and then a series of attacks on components, resources, or system properties is enumerated.enumerated in the rest of Section 7. For each attack, possible countermeasures are reviewed. If all recommended mitigations are in place the implemented usage models, the RDMAP/DDP protocol can be shown to not expose any new security vulnerabilities. Applications within a host are divided into two categories - Privileged and Non-Privileged. Both application types can send and receive data and request resources. The key differences between the two are: The Privileged Application is Partially Trusted. It is assumed thattrusted by the Privileged Application willsystem to not intentionallymaliciously attack the system (e.g.,operating environment, but it is not trusted to optimize resource allocation globally. For example, the Privileged Application could be a kernel application), althoughapplication, thus the kernel presumably has in some way vetted the application before allowing it may be greedy for resources.to execute. A Non-Privileged ApplicationÆs capabilities are a logical sub- setsub-set of the Privileged ApplicationÆs. It is assumed by the local host infrastructuresystem that a Non-Privileged Application is Untrusted.untrusted. All Non-Privileged Application interactions with the RNIC Engine that could affect other applications need to be done through a Trustedtrusted intermediary that can verify the Non- PrivilegedNon-Privileged Application requests. 4 Architectural Model This section describes an RDMA architectural reference model that is used as security issues are examined. It introduces the components of the model, the resources that can be attacked, the types of interactions possible between components and resources, and the system properties, which should be preserved when under attack. Figure 1 shows the components comprising the architecture and the interfaces where potential security attacks could be launched. External attacks can be injected into the system from an application that sits above the RI or from the Internet. The intent here is to describe high level components and capabilities which affect threat analysis, and not focus on specific implementation options. Also note that the architectural model is an abstraction, and an actual implementation may choose to subdivide its components along different boundary lines than defined here. For example, the Privileged Resource Manager may be partially or completely encapsulated in the Privileged Application. Regardless, it is expected that the security analysis of the potential threats and countermeasures still apply. +-------------+ Request Proxy Interface| Privileged |<--------------------------------+| Resource| Resource | Admin<-->|Admin<-+>| Manager | App Control Interface | | |<------+-------------------+ | +-------------+ | | | ^ v v v| | +-------------+ +-----------------+ | ||---------------->| Privileged | | Non-Privileged | | | Application | | Application | | +-------------+ +-----------------+ | ^ ^ |Privileged |Privileged |Non-Privileged |Control |Data |Data |Interface |Interface |Interface RNIC | | | Interface(RI) v v v ================================================================= +-----------------------------------------++--------------------------------------+ | | | RNIC Engine | <------<-- Firmware | | +-----------------------------------------++--------------------------------------+ ^ | v Internet Figure 1 - RDMA Security Model 4.1 Components The components shown in Figure 1 - RDMA Security Model are: * RNIC Engine - the component that implements the RDMA protocol and/or DDP protocol. * Privileged Resource Manager - the component responsible for managing and allocating resources associated with the RNIC Engine. The Resource Manager does not send or receive data. Note that whether the Resource Manager is an independent component, part of the RNIC, or part of the application is implementation dependent. If a specific implementation does not wish to address security issues resolved by the Resource Manager, there may in fact be no resource manager at all. * Privileged Application - See Section 3 Introduction for a definition of Privileged Application. The local host infrastructure can enable the Privileged Application to map a data buffer directly from the RNIC Engine to the host through the RNIC Interface, but it does not allow the Privileged Application to directly consume RNIC Engine resources. * Non-Privileged Application - See Section 3 Introduction for a definition of Non-Privileged Application. All Non- Privileged Application interactions with the RNIC Engine that could affect other applications MUST be done using the Privileged Resource Manager as a proxy. A design goal of the DDP and RDMAP protocols is to allow, under constrained conditions, Non-Privileged applications to send and receive data directly to/from the RDMA Engine without Privileged Resource Manager intervention - while ensuring that the host remains secure. Thus, one of the primary goals of this paper is to analyze this usage model for the enforcement that is required in the RNIC Engine to ensure the system remains secure. The host interfaces that could be exercised include: * Privileged Control Interface - A Privileged Resource Manager uses the RI to allocate and manage RNIC Engine resources, control the state within the RNIC Engine, and monitor various events from the RNIC Engine. It also uses this interface to act as a proxy for some operations that a Non-Privileged Application may require (after performing appropriate countermeasures). * Application Control Interface û An application uses this interface to the Privileged Resource Manager to allocate RNIC Engine resources. The Privileged Resource Manager implements countermeasures to ensure that if the Non- Privileged Application launches an attack it can prevent the attack from affecting other applications. * Non-Privileged Data Transfer Interface - A Non-Privileged Application uses this interface to initiate and to check the status of data transfer operations. * Privileged Data Transfer Interface - A superset of the functionality provided by the Non-Privileged Data Transfer Interface. The application is allowed to directly manipulate RNIC Engine mapping resources to map an STag to an application data buffer. * Request Proxy Interface - a Non-Privileged Application uses this interface to control RNIC Engine resources that could affect other applications - such as manipulating the RNIC Engine's mapping of an STag to an application data buffer. The Privileged Resource Manager implements countermeasures to ensure that if the Non-Privileged Application launches an attack it can prevent the attack from affecting other applications. *Figure 1 also shows the ability to load new firmware in the RNIC Engine. Not all RNICs will support this, but it is shown for completeness and is also reviewed under potential attacks. If Internet control messages, such as ICMP, ARP, RIPv4, etc. are processed by the RNIC Engine, the threat analyses for those protocols is also applicable, but outside the scope of this paper. 4.2 Resources This section describes the primary resources in the RNIC Engine that could be affected if under attack. For RDMAP, all of the defined resources apply. For DDP, all of the resources except the RDMA Read Queue apply. 4.2.1 ConnectionStream Context Memory The state information for each connectionStream is maintained in memory, which could be located in a number of places - on the NIC, inside RAM attached to the NIC, in host memory, or in any combination of the three, depending on the implementation. ConnectionStream Context Memory includes state associated with Data Buffers. For Tagged Buffers, this includes how STag names, Data Buffers, and Page Translation Tables inter-relate. It also includes the FIFO list of Untagged Data Buffers posted for reception of Untagged Messages (referred to in some contexts as the Receive Queue), and a list of operations to perform to send data (referred to in some contexts as the Send Queue). 4.2.2 Data Buffers There are two different ways to expose a data buffer; a buffer can be exposed for receiving RDMAP Send Type Messages (a.k.a. DDP Untagged Messages) on DDP Queue zero or the buffer can be exposed for remote access through STags (a.k.a. DDP Tagged Messages). This distinction is important because the attacks and the countermeasures used to protect against the attack are different depending on the method for exposing the buffer to the Internet. For the purposes of the security discussion, a single logical Data Buffer is exposed with a single STag. Actual implementations may support scatter/gather capabilities to enable multiple physical data buffers to be accessed with a single STag, but from a threat analysis perspective it is assumed that a single STag enables access to a single logical Data Buffer. In any event, it is the responsibility of the RI to ensure that no STag can be created that exposes memory that the consumer had no authority to expose. 4.2.3 Page Translation Tables Page Translation Tables are the structures used by the RNIC to be able to access application memory for data transfer operations. Even though these structures are called "Page" Translation Tables, they may not reference a page at all - conceptually they are used to map an application address space representation of a buffer to the physical addresses that are used by the RNIC Engine to move data. If on a specific system, a mapping is not used, then a subset of the attacks examined may be appropriate. 4.2.4 STag Namespace The DDP specification defines a 32-bit namespace for the STag. Implementations may vary in terms of the actual number of STags that are supported. In any case, this is a bounded resource that can come under attack. Depending upon StagSTag namespace allocation algorithms, the actual name space to attack may be significantly less than 2^32. 4.2.5 Completion Queues Completion Queues are used in this specification to conceptually represent how the RNIC Engine notifies the Application ofabout the completion of the transmission of data, or the completion of the reception of data through the Data Transfer Interface. Because there could be many transmissions or receptions in flight at any one time, completions are modeled as a queue rather than a single event. An implementation may also use the Completion Queue to notify the application of other activities, for example, the completion of a mapping of an STag to a specific application buffer. 4.2.6 RDMA Read Request Queue The RDMA Read Request Queue is the memory holding state information for oneCompletion Queues may be shared by a group of Streams, or more RDMA Read Request Messages that have arrived, but for which the RDMA Read Response Messages have not yet been completely sent. 4.2.7 RDMAmay be designated to handle a specific Stream's traffic. Some implementations may allow this queue to be manipulated directly by both Non-Privileged and Privileged applications. 4.2.6 Asynchronous Event Queue The Asynchronous Event Queue is a queue from the RNIC to the Privileged Resource Manager of bounded size. It is used by the RNIC to notify the host of various events which might require management action, including protocol violations, connectionStream state changes, local operation errors, low water marks on receive queues, and possibly other events. The RDMAAsynchronous Event Queue is a resource that can be attacked because Remote or Local Peers can cause events to occur which have the potential of overflowing the queue. Note that an implementation is at liberty to implement the functions of the RDMAAsynchronous Event Queue in a variety of ways, including multiple queues or even simple callbacks. All vulnerabilities identified for a singleare intended to apply regardless of the implementation of the Asynchronous Event Queue apply to specific- purpose subsets. AQueue. For example, a callback function is simply a very short queue. 4.2.7 RDMA Read Request Queue The RDMA Read Request Queue is the memory that holds state information for one or more RDMA Read Request Messages that have arrived, but for which the RDMA Read Response Messages have not yet been completely sent. Because potentially more than one RDMA Read Request can be outstanding at one time, the memory is modeled as a queue of bounded size. 4.2.8 RNIC ControlInteractions With RNIC resources and interfaces defined, it is now possible to examine the interactions supported by the generic RNIC functional interfaces through each of the 3 interfaces - Privileged Control Interface, Privileged Data Interface, and Non-Privileged Data Interface. 22.214.171.124 Privileged Control Interface Semantics Generically, the Privileged Control Interface controls the RNICÆs allocation, deallocation, and initialization of RNIC global resources. This includes allocation and deallocation of ConnectionStream Context Memory, Page Translation Tables, STag names, Completion Queues, andRDMA Read Request Queues. Event Queue InitializationQueues, and removal of Page Translation Table resources. 4.2.9 Initialization of RNIC Data StructuresAsynchronous Event Queues. The Privileged Control Interface is also typically used for managing Non-Privileged Application resources for Data Transfer Initialization ofthe mapping between an STag and a Data Buffer can be viewed inNon- Privileged Application (and possibly for the abstractPrivileged Application as two separate opertions: a. Initialization of the allocated Page Translation Table entries with the location of the Data Buffer,well). This includes initialization and b. Initialization of a mapping from an allocated STag name to a setremoval of Page Translation Table entry(s) or partial-entries. Note that an implementation may not have a Page Translation Table (i.e. it may support a direct mapping between an STag and a Data Buffer). In this case threatsresources, and mitigations associated with the Page Translation Table are not relevent. Initialization of the contents of the Page Translation Table can be done by either the Privileged Application or by the Privileged Resource Manager as a proxymanaging RNIC events (possibly managing all events for the Asynchronous Event Queue). 126.96.36.199 Non-Privileged Application. By definition theData Interface Semantics The Non-Privileged Application isData Interface enables data transfer (transmit and receive) but does not trusted to directly manipulate the Page Translation Table. In general the concern is that the Non-Privileged application may try to maliciously initializeallow initialization of the Page Translation Table to access a buffer for which it does not have permission. The exact resource allocation algorithm forresources. However, once the Page Translation Table is outsideresources have been initialized, the scope of this specification. Itinterface may be allocated forenable a specific Data Buffer, or be allocated as a pooled resourceSTag mapping to be consumedenabled and disabled by potentially multiple Data Buffers,directly communicating with the RNIC, or be managedcreate an STag mapping for a buffer that has been previously initialized in some other way. This paper attempts to abstract implementation dependent issues,the RNIC. For RDMAP, transmitting data means sending RDMAP Send Type Messages, RDMA Read Requests, and focusRDMA Writes. For data reception, for RDMAP it can receive Send Type Messages into buffers that have been posted on higher level security issues such as resource starvationthe Receive Queue or Shared Receive Queue. It can also receive RDMA Write and sharingRDMA Read Response Messages into buffers that have previously been exposed for external write access through advertisement of resources between connections. The next issue is howan STag name is associated with a Data Buffer.STag. For DDP, transmitting data means sending DDP Tagged or Untagged Messages. For data reception, for DDP it can receive Untagged Messages into buffers that have been posted on the caseReceive Queue or Shared Receive Queue. It can also receive Tagged DDP Messages into buffers that have previously been exposed for external write access through advertisement of an UntaggedSTag. Completion of data transmission or reception generally entails informing the application of the completed work by placing completion information on the Completion Queue. 188.8.131.52 Privileged Data Buffer, there is no wire visibleInterface Semantics The Privileged Data Interface semantics are a superset of the Non-Privileged Data Transfer semantics. The interface can do everything defined in the prior section, as well as create/destroy buffer to STag mappings directly. This generally entails initialization or clearing of Page Translation Table state in the RNIC. 4.2.9 Initialization of RNIC Data Structures for Data Transfer Initialization of the mapping between an STag nameand a Data Buffer. Note that there may, in fact, be a mapping that is not visible from the wire, but this is a local host specific issue which shouldBuffer can be analyzedviewed in the context of local host implementation specific security analysis, and thus is outside the scopeabstract as two separate opertions: a. Initialization of this paper. For a Tagged Data Buffer, eitherthe Privileged Application, the Non-Privileged Application, orallocated Page Translation Table entries with the Privileged Resource Manager acting on behalflocation of the Non-Privileged Resource Manager may initializeData Buffer, and b. Initialization of a mapping from an allocated STag name to a set of Page Translation Table,Table entry(s) or may have the ability to simply enable/disablepartial- entries. Note that an existing STag to Page Translation Table mapping. Thereimplementation may also be multiple STag names which map tonot have a specific group ofPage Translation Table entries (or sub-entries). Specific security issues with this level of flexibility are examined later. There are(i.e. it may support a variety of implementation options for initialization of Page Translation Table entries anddirect mapping between an STag toand a group ofData Buffer). In this case threats and mitigations associated with the Page Translation Table entries which have security repercussions. This includes support for separationare not relevant. Initialization of Mapping an STag verses mapping a setthe contents of the Page Translation Table entries, and support for Applications directly manipulating STag to Page Translation Table entry mappings (verses requiring access throughcan be done by either the Privileged Application or by the Privileged Resource Manager). 4.2.10 RNIC Data Transfer Interactions RNIC Data Transfer operations can be subdivided into send operations and receive operations. For send operations, there is typicallyManager as a queue that enablesproxy for the Non-Privileged Application. By definition the Non-Privileged Application is not trusted to post multiple operations. Depending upon the implementation, Data Buffers used indirectly manipulate the operations may or may not havePage Translation Table entries associated with them, and may orTable. In general the concern is that the Non-Privileged application may try to maliciously initialize the Page Translation Table to access a buffer for which it does not have STags associated with them. Because this is a local host specific implementation issue rather than a protocol issue,permission. The exact resource allocation algorithm for the security analysis of threats and mitigationsPage Translation Table is left tooutside the host implementation. Receive operations are differentscope of this specification. It may be allocated for Tagged Data Buffers verses Untagged Data Buffers. If more than one Untaggeda specific Data Buffer canBuffer, or be posted by the Application, the DDP specification requires that theyallocated as a pooled resource to be consumed by potentially multiple Data Buffers, or be managed in FIFO order. Thus the most generalsome other way. This paper attempts to abstract implementation dependent issues, and focus on higher level security issues such as resource starvation and sharing of resources between Streams. The next issue is that therehow an STag name is associated with a FIFO queueData Buffer. For the case of receivean Untagged Data Buffers. Some implementations may also support sharing of the FIFO queueBuffer, there is no wire visible mapping between multiple connections. In this case defining ææFIFOÆÆ becomes non- trivial - in general the buffers foran STag name and a single stream are consumed from the queueData Buffer. Note that there may, in the orderfact, be a mapping that they were placed onis not visible from the queue,wire, but therethis is no order guarantee between streams. For receive Tagged Data Buffers, at some time prior to data transfer,a local host specific issue which should be analyzed in the mappingcontext of local host implementation specific security analysis, and thus is outside the scope of this paper. For a Tagged Data Buffer, either the Privileged Application, the Non-Privileged Application, or the Privileged Resource Manager acting on behalf of the Non-Privileged Resource Manager may initialize a mapping from an STag to a Page Translation Table, or may have the ability to simply enable/disable an existing STag to Page Translation Table mapping. There may also be multiple STag names which map to a specific group of Page Translation Table entries (or sub-entries). Specific security issues with this level of flexibility are examined later. There are a variety of implementation options for initialization of Page Translation Table entries (if present)and themapping from thean STag to a group of Page Translation Table entries which have security repercussions. This includes support for separation of Mapping an STag verses mapping a set of Page Translation Table entries, and support for Applications directly manipulating STag to Page Translation Table entry mappings (verses requiring access through the Privileged Resource Manager). 4.2.10 RNIC Data Buffer must have been initialized (see the prior section for interaction details). 4.3 System Properties System properties thatTransfer Interactions RNIC Data Transfer operations can be attacked included system integrity, system stability (liveness, large fluctuations in performance),subdivided into send operations and confidentiality. 5 Trust Models The Trust Models described in this section have three primary distinguishing characteristics. * Local Resource Sharing (yes/no) - When local resources are shared, they are shared acrossreceive operations. For send operations, there is typically a grouping of RDMAP/DDP Streams. If local resources are not shared,queue that enables the resources are dedicated on a per Stream basis. Resources are defined in Section 4.2 - Resources on page 10. The advantage of not sharing resources between Streams is that it reducesApplication to post multiple operations. Depending upon the types of attacks that are possible. The disadvantage is that applications might run out of resources. * Local Partial Trust (yes/no) - Local Partial Trust is determined based on whetherimplementation, Data Buffers used in the local grouping of RDMAP/DDP Streams (which typically equates to one applicationoperations may or group of applications) mutually trust each other tomay not perform a specific set of attacks. * Remote Partial Trust (yes/no) - The Remote Partial Trust levelhave Page Translation Table entries associated with them, and may or may not have STags associated with them. Because this is determined based on whether the Local Peer ofa local host specific RDMAP/DDP Stream partially trustsimplementation issue rather than a protocol issue, the Remote Peersecurity analysis of threats and mitigations is left to the Stream (seehost implementation. Receive operations are different for Tagged Data Buffers verses Untagged Data Buffers. If more than one Untagged Data Buffer can be posted by the definition of Partial Trust in Section 3 Introduction). It is assumed in this paper thatApplication, the componentDDP specification requires that implementsthey be consumed in FIFO order. Thus the mechanism to controlmost general implementation is that there is a FIFO queue of receive Untagged Data Buffers. Some implementations may also support sharing of RNIC Engine resources is the Privileged Resource Manager. The RNIC Engine exposes its resources throughthe RIFIFO queue between multiple Streams. In this case defining "FIFO" becomes non-trivial - in general the buffers for a single stream are consumed from the queue in the order that they were placed on the queue, but there is no order guarantee between streams. For receive Tagged Data Buffers, at some time prior to data transfer, the mapping of the STag to specific Page Translation Table entries (if present) and the mapping from the Page Translation Table entries to the Data Buffer must have been initialized (see the prior section for interaction details). 5 Trust and Resource Sharing It is assumed that in general the Local and Remote Peer are untrusted, and thus attacks by either should have mitigations in place. A separate, but related issue is resource sharing between multiple streams. If local resources are not shared, the resources are dedicated on a per Stream basis. Resources are defined in Section 4.2 - Resources on page 10. The advantage of not sharing resources between Streams is that it reduces the types of attacks that are possible. The disadvantage is that applications might run out of resources. It is assumed in this paper that the component that implements the mechanism to control sharing of RNIC Engine resources is the Privileged Resource Manager. The RNIC Engine exposes its resources through the RI to the Privileged Resource Manager. All Privileged and Non-Privileged applications request resources from the Resource Manager. The Resource Manager implements resource management policies to ensure fair access to resources. The Resource Manager should be designed to take into account security attacks detailed in this specification. Note that for some systems the Privileged Resource Manager may be implemented within the Privileged Application. The sharing of resources across connectionsStreams should be under the control of the application, both in terms of the Trust Modeltrust model the application wishes to operate under, as well as the level of resource sharing the application wishes to give Local Peer processes. Not all of the combinationsFor more discussion on types of thetrust characteristics are expected to be used by applications. This paper specifically analyzes five applicationmodels which combine partial trust and sharing of resources, see Appendix C: Partial Trust ModelsTaxonomy on page 54. 6 Attacker Capabilities An attackerÆs capabilities delimit the types of attacks that are expectedattacker is able to launch. RDMAP and DDP require that the initial LLP Stream (and connection) be in common use. The Trust Modelsset up prior to transferring RDMAP/DDP Messages. For the attacker to actively generate an RDMAP/DDP protocol attack, it must have the capability to both send and receive messages. Attackers with send only capabilities should be addressed by the LLP, not by RDMAP/DDP. 7 Attacks and Countermeasures This section describes the attacks that are as follows: 1. NS-NT - Non-Shared Local Resources, no Local Trust, no Remote Trustpossible against the RDMA system defined in Figure 1 - typically a server application that wants to runRDMA Security Model and the RNIC Engine resources defined in Section 4.2. The analysis includes a mode that has the least numberdetailed description of potential attacks. 2. NS-RT - Non-Shared Local Resources, no Local Trust, Remote Partial Trust - typicallyeach attack, what is being attacked, and a peer-to-peer application, which has, by some method outsidedescription of the scope of this specification, authenticatedcountermeasures that can be taken to thwart the Remote Peer.attack. Note that unless some form of key based authenticationconnection setup and teardown is used on a per packet basis, it may not be possiblepresumed to bindbe done in stream mode (i.e. no RDMA encapsulation of the authentication resultpayload), so there are no new attacks related to connection setup/teardown beyond what is already present in the RDMA packet. 3. S-NT - Shared Local Resources, no Local Trust, no Remote Trust - typically a server application that runs in an untrusted environment where the amount of resources required is either too largeLLP (e.g. TCP or too dynamic to dedicate for each RDMAP/DDP Stream. 4. S-LT - Shared Local Resources, Local Partial Trust, no Remote Trust - typically an application, which provides a session layer and uses multiple Streams, to provide additional throughputSCTP). Consequently, any existing analysis of Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, or fail-over capabilities. AllElevation of Privilege continues to apply. Thus, the Streams withinanalysis in this section focuses on attacks that are present regardless of the local application partially trust each other, but do not trustLLP Stream type. The attacks are classified into five categories: Spoofing, Tampering, Information Disclosure, Denail of Service (DoS) attacks, and Elevation of Privileges. Tampering is any modification of the remote peer. 5. S-T - Shared Local Resources, Local Partial Trust, Remote Partial Trust - typically a distributed application, such as a distributed database applicationlegitimate traffic (machine internal or a High Performance Computer (HPC) application, whichnetwork). Spoofing attack is intended to run ona cluster. Due to extreme resource and performance requirements,special case of tempering; where the application typically authenticates with allattacker falsifies an identity of its peers and then runs in a highly trusted environment. The application peers are all in a single application fault domain and depend on one another to be well-behaved when accessing data structures. If a trustedthe Remote Peer has(identity can be an implementation defect that resultsIP address, machine name, ULP level identity etc.). 7.1 Tools for Countermeasures The tools described in poor behavior,this section are the entire application couldprimary mechanisms that can be corrupted. Models NS-NT and S-NT aboveused to provide countermeasures to potential attacks. 7.1.1 Protection Domain (PD) Protection Domains are typical for Internet networking - neither Local Peers norassociated with two of the Remote Peer is trusted. Sometimes optimizations can be done that enable sharingresources of concern, Stream Context Memory and STags associated with Page Translation Tables across multiple Local Peers, thus Model S-LT can be advantageous. Model S-T is typicallyTable entries and data buffers. Protection Domains are used when resource scaling across a large parallel application makes it infeasiblemainly to use any other model. Resource scaling issuesensure that an STag can eitheronly be dueused to performance around scaling or because there simply are not enough resources. Model NS-RT is probablyaccess the least likely model to be used, but is presented for completeness. 6 Attacker Capabilities An attackerÆs capabilities delimitassociated data buffer through Streams in the types of attackssame Protection Domain as that attacker is ableSTag. If an implementation chooses to launch. RDMAP and DDP requirenot share resources between Streams, it is recommended that the initial LLPeach Stream (and connection)be set up prior to transferring RDMAP/DDP Messages. For the attacker to actively generateassociated with its own, unique Protection Domain. If an RDMAP/DDP protocol attack, it must have the capabilityimplementation chooses to both send and receive messages. Attackers with send only capabilities shouldallow resource sharing, it is recommended that Protection Domain be addressed by the LLP, not by RDMAP/DDP. 7 Attacks and Countermeasures This section describeslimited to the attacksnumber of Streams that are possible against the RDMA system defined in Figure 1 - RDMA Security Model andhave Partial Mutual Trust. Note that an application (either Privileged or Non-Privileged) can potentially have multiple Protection Domains. This could be used, for example, to ensure that multiple clients of a server do not have the RNIC Engine resources defined in Section 4.2.ability to corrupt each other. The analysis includesserver would allocate a detailed description of each attack, the Trust ModelsProtection Domain per client to ensure that resources covered by the attack appliesProtection Domain could not be used by another (untrusted) client. 7.1.2 Limiting STag Scope The key to (see Section 5 forprotecting a description oflocal data buffer is to limit the Trust Models), and a descriptionscope of its STag to the countermeasureslevel appropriate tofor the Trust Model(s) thatStreams which share Partial Mutual Trust. The scope of the STag can be taken to thwartmeasured in multiple ways. * Number of Connections and/or Streams on which the attack. Note that, connection setup and teardownSTag is presumedvalid. One way to be done in stream mode (i.e. no RDMA encapsulationlimit the scope of the payload), so thereSTag is to limit the connections and/or Streams that are no new attacks relatedallowed to connection setup/teardown beyond what is already presentuse the STag. As noted in the LLP (e.g. TCP or SCTP). Consequently, any existing analysis of Spoofing, Tampering, Repudiation, Information Disclosure, Denialprevious section, use of Service, or ElevationProtection Domains appropriately can limit the scope of Privilege continues to apply. Thus,the STag. The analysis presented in this section focuses on attacks that are present regardlessdocument assumes two mechanisms for limiting the scope of Streams for which the LLP Stream type. 7.1 Tools for Countermeasures The tools described in this section are the primary mechanisms that can be used to provide countermeasures to potential attacks. 7.1.1STag is valid: * Protection Domain (PD)scope. The STag is valid if used on any Stream within a specific Protection Domains are associated with two of the resources of concern, connection context memory and STags associated with Page Translation Table entriesDomain, and data buffers. Protection Domains areis invalid if used mainly to ensureon any Stream that an STag can only be used to access the associated data buffer through connections inis not a member of the sameProtection Domain as that STag. For the Trust Models that are defined to have non-shared resources (Trust Models NS-NT and NS-RT), itDomain. * Single Stream scope. The STag is recommended that eachvalid on a single Stream, regardless of what the Stream be associated with its own, uniqueassociation is to a Protection Domain. For those Trust Models where resources are shared (Trust Models S-NT, S-LT and S-T),If used on any other Stream, it is recommended that Protection Domain be limitedinvalid. * Limit the time an STag is valid. By Invalidating an Advertised STag (e.g., revoking remote access to the number of Streams that sharebuffers described by an STag when done with the same Trust Model. Note thattransfer), an application (either Privileged or Non-Privileged)entire class of attacks can potentially have multiple Protection Domains. This couldbe used, for example, to ensure that multiple clients of a server do not haveeliminated. * Limit the ability to corrupt each other. Thisbuffer the STag can apply to any combinationreference. Limiting the scope of the Trust Models, because Partial Trust does not imply complete trust. 7.1.2 Limiting STag Scope The key to protecting a local data buffer is to limit the scope of its STag to the level appropriate for the Trust Model. The scope of the STag can be measured in multiple ways. * Number of Connections and/or Streams on which the STag is valid. One way to limit the scope of the STag is to limit the connections and/or Streams that are allowed to use the STag. As noted in the previous section, use of Protection Domains appropriately can limit the scope of the STag. It is also possible to create an STag that is valid only on a single connection, even in the case where several connections are associated with the Protection Domain of the STag. * Limit the time an STag is valid. By Invalidating an Advertised STag (e.g., revoking remote access to the buffers described by an STag when done with the transfer), an entire class of attacks can be eliminated. * Limit the buffer the STag can reference. Limiting the scope of an STag access to *just*an STag access to *just* the intended application buffers to be exposed is critical to prevent certain forms of attacks. * Allocating StagSTag numbers in an unpredictable way. If STags are allocated using an algorithm which makes it hard for the Remote Peer to guess which STag(s) are currently in use, it makes it more difficult for an attacker to guess the correct value. As stated in the RDMAP specification [RDMAP], an invalid STag will cause the RDMAP Stream to be terminated. For the case of [DDP], at a minimum it must signal an error to the ULP, and commonly this will cause the DDP stream to be terminated. 7.1.3 Access Rights Access Rights associated with a specific Advertised STag or RDMAP/DDP Stream provide another mechanism for applications to limit the attack capabilities of the Remote Peer. The Local Peer can control whether a data buffer is exposed for local only, or local and remote access, and assign specific access privileges (read, write, read and write).write) on a per stream or session basis. For DDP, when an STag is advertised, the Remote Peer is presumably given write access rights to the data (otherwise there was not much point to the advertisement). For RDMAP, when an application advertises an STag, it can enable write-only, read-only,read- only, or both write and read access rights. Similarly, some applications may wish to provide a single buffer with different access rights on a per-connectionper-Stream or per-Stream basis. For example, some connectionsStreams may have read-only access, some may have remote read and write access, while on other connectionsStreams only the Local Peer is allowed access. 7.1.4 Limiting the Scope of the Completion Queue Completions associated with sending and receiving data, or setting up buffers for sending and receiving data, could be accumulated in a shared Completion Queue for a group of RDMAP/DDP Streams, or a specific RDMAP/DDP Stream could have a dedicated Completion Queue. Limiting Completion Queue association to one, or a small number of RDMAP/DDP Streams can prevent several forms of Denial of Service attacks. 7.1.5 Limiting the Scope of an Error To prevent a variety of attacks, it is important that an RDMAP/DDP implementation be robust in the face of errors. If an error on a specific Stream can cause other unrelated Streams to fail, then a broad class of attacks are enabled against the implementation. 7.2 Spoofing Because theFor example, an error on a specific RDMAP Stream is only offloaded if it is in thestream should not cause the RNIC to stop processing incoming packets, or corrupt a receive queue for an unrelated stream. 7.2 Spoofing Spoofing attacks can be launched by the Remote Peer, or by a network based attacker. A network based spoofing attack applies to all Remote Peers. Because the RDMAP Stream is only offloaded if it is in the ESTABLISHED state, certain types of traditional forms of wire attacks do not apply -- an end-to-end handshake must have occurred to establish the RDMAP Stream. So, the only form of spoofing that applies is one when a remote node can both send and receive packets. Yet even with this limitation the Stream is still exposed to the following spoofing attacks. 7.2.1 ConnectionImpersonation A network based attacker can impersonate a legal RDMA/RDDP peer (by spoofing a legal IP address), and establish an RDMA/RDDP Stream with the victim. End to end authentication (i.e. IPsec, SSL or ULP authentication) provides protection against this attack. For additional information see Section 8, Security Services for RDDP, on page 38. 7.2.2 Stream Hijacking Stream Hijacking happens when a network based attacker follows the session establishment phase, and waits until the authentication phase (if such a phase exists) is completed successfully. He can then spoof the IP address and re-direct the Stream from the victim to its own machine. For example, an attacker can wait until an iSCSI authentication is completed successfully, and hijack the iSCSI Stream. The best protection against this form of attack is end-to-end session level integrity protection and authentication, such as IPsec (see Section 8, Security Services for RDDP, on page 38), to prevent spoofing. Another option is to provide physical security. Discussion of physical security is out of scope for this document. Because the connection and/or Stream itself is established by the LLP, some LLPs are more difficult to hijack than others. Please see the relevant LLP documentation on security issues around connection and/or Stream hijacking <TBD: references for SCTP and TCP on connection hijacking>. 7.2.3 Man in the Middle Attack If a man-in-the-middlenetwork based attacker has the ability to delete, inject replay, or modify packets which will still be accepted by the LLP (e.g., TCP sequence number is correct) then the connectionStream can essentiallybe hijacked. Oneexposed to a man in the middle attack. One style of attack is for the man-in-the-middle to send Tagged Messages (either RDMAP or DDP). If it can discover a buffer that has been exposed for STag enabled access, then the man-in-the-middle can use an RDMA Read operation to read the contents of the associated data buffer, perform an RDMA Write Operation to modify the contents of the associated data buffer, or invalidate the STag to disable further access to the buffer. The only countermeasure for this form of attack is to either secure the RDMAP/DDP Stream (i.e. integrity protect) or attempt to provide physical security to prevent man-in-the-middleman- in-the-middle type access.attacks. The best protection against this form of attack is end-to-end integrity protection and authentication, such as IPsec (see Section 8 Security Services for RDDP on page 35),38), to prevent spoofing or tampering. If Stream or session level authentication isand integrity protection are not used, then a man-in-the-middle attack can occur, enabling spoofing, tampering,spoofing and repudiation.tampering. Because the connectionconnection/Stream itself is established by the LLP, some LLPs are more difficultexposed to hijack thanman-in-the-middle attack then others. Please see the relevant LLP documentation on security issues around connection and/or Stream hijacking <TBD: references for SCTP and TCP on connection hijacking>. Another approach is to restrict access to only the local subnet/link, and provide some mechanism to limit access, such as physical security or 802.1.x. This model is an extremely limited deployment scenario, and will not be further examined here. 184.108.40.206.4 Using an STag on a different connectionDifferent Stream One style of attack from the Remote Peer is for it to attempt to use STag values that it is not authorized to use. Note that,that if the Remote Peer sends an invalid STag to the Local Peer, per the DDP and RDMAP specifications, the connectionStream must be torn down. Thus,Thus the threat exists if a STag has been enabled for Remote Access on one connectionStream and a Remote Peer is able to use it on an unrelated connection.Stream. If the attack is successful, the attacker could potentially be able to perform either RDMA Read Operations to read the contents of the associated data buffer, perform RDMA Write Operations to modify the contents of the associated data buffer, or to Invalidate the STag to disable further access to the buffer. An attempt by a Remote Peer to access a buffer with an STag on a different connectionStream in the same Protection Domain may or may not be an attack depending on whether resource sharing is intended (i.e. whether the Streams shared Partial Mutual Trust Model employed by the application.or not). For Trust Model S-T, where resources are shared between connections, and both Local and Remote Peers are Trusted,some applications using an STag on multiple connectionsStreams within the same Protection Domain is allowed, andcould be desired behavior. For theother four Trust Models where the Remote Peer is not Trusted, and/or resources are not intended to be shared,applications attempting to use an STag on a different connectionStream could be considered to be an attack. Since this varies by application, an application typically would need to be able to control the scope of the STag. In the case where the Trust Model is defined with no sharedan implementation does not share resources between connections (Trust Models NS-NT and NS-RT),Streams (including STags), this attack can be defeated by assigning each connectionStream to a different Protection Domain. Before allowing remote access to the buffer, the Protection Domain of the connectionStream where the access attempt was made is matched against the Protection Domain of the STag. If the Protection Domains do not match, access to the buffer is denied, an error is generated, and the RDMAP Stream associated with the attacking connectionStream should be terminated. Thus, for Trust Models NS-NT and NS-RT, it is RECOMMENDED that each connection be in a separate Protection Domain.For Trust Models S-NT and S-LT, whereimplementations that share resources are shared, but the Remote Peers are Untrusted,between multiple Streams, it may not be practical to separate each connectionStream into its own Protection Domain. In this case, the application can still limit the scope of any of the STags to a single Stream (if it is enabling it for remote access to a single connection.access). If the STag scope has been limited to a single connection,Stream, any attempt to use that STag on a different connectionStream will result in an error, and the RDMA Stream associated with that connectionshould be terminated. Thus,Thus for Trust Models S-NT and S-LT,implementations that do not share STags between Streams it is RECOMMENDED that the scope of an STageither each Stream be limited toin a single connection. For Trust Models S-NT and S-LT (Untrusted Remote Peers), if it is not possible to useseparate Protection DomainsDomain or to limitthe scope of an STag be limited to a single connection, it is RECOMMENDED that STag allocators selectStream. An additional issue may be unintended sharing of STags (i.e. a bug in the application) or a bug in the Remote Peer which causes an off-by-one STag using an algorithm which makesto be used. For additional protection, it is RECOMMENDED that the allocation of STags be done in such a fashion that it is difficult to guesspredict the next allocated STag number. This approach is good practice in general.Allocation methods which always start with the same number (e.g. zero) after Stream initialization or simplydeterministically allocate the next STag in a monotonically increasing namespaceshould be avoided.avoided (e.g. a method which always starts with STag equal to one and monotonically increases it for each new allocation). 7.3 Tampering A Remote Peer or a network based attacker can attempt to tamper with the contents of data buffers on a Local Peer that have been enabled for remote write access. The types of tampering attacks that are possible are outlined in the sections that follow. 7.3.1 Buffer Overrun - RDMA Write or Read Response to Memory Outside the BufferThis attack is an attempt by the Remote Peer to perform an RDMA Write or RDMA Read Response to memory outside of the valid length range of the data buffer enabled for remote write access. This attack applies primarily to Trust Models with Untrusted Remote Peers (NS-NT, S-NT and S-LT), andcan occur even when no resources are shared across connections.Streams. This issue can also arise for Trust Models NS-RT and S-T, which assume remote Partial Trust,if the application has a bug. Thus it is RECOMMENDED that all Trust Models ensure this countermeasures are in place against this form of attack.The countermeasure for this type of attack must be in the RNIC implementation, using the STag. When the Local Peer specifies to the RI,RI the base address and the number of bytes in the buffer that it wishes to make accessible, the RI must ensure that the base and bounds check are applied to any access to the buffer referenced by the STag before the STag is enabled for access. When an RDMA data transfer operation (which includes an STag) arrives on a connection,Stream, a base and bounds byte granularity access check must be performed to ensure the operation accesses only memory locations within the buffer described by that STag. Thus, it is RECOMMENDED that an RI implementation ensure that a Remote Peer, regardless of Trust Model,Peer will not be able to access memory outside of the buffer specified when the STag was enabled for remote access. 7.3.2 Modifying a Buffer After Indicating Contents Are ReadyIndication This attack occurs if a Remote Peer attempts to modify the contents by performing an RDMA Write or an RDMA Read Response after it had indicated to the Local Peer that the data buffer contents were ready for use. This attack applies primarily to the Trust Models where the Remote Peers are not Trusted (Trust Models NS-NT, S-NT and S-LT), andcan occur even when no resources are shared across connections.Streams. Note that, an error on the part ofthat a bug in a TrustedRemote PeerPeer, or network based tampering, could also result in this problem. The Local Peer can protect itself from this type of attack by revoking remote access when the original data transfer has completed and before it validates the contents of the buffer. The Local Peer can either do this by explicitly revoking remote access rights for the STag when the Remote Peer indicates the operation has completed, or by checking to make sure the Remote Peer Invalidated the STag through the RDMAP Invalidate capability, and if it did not, the Local Peer then explicitly revokes the STag remote access rights. It is RECOMMENDED that the Local Peer follow the above procedure for Trust Models NS-NT, S-NT, and S-LTto protect the buffer. The Local Peer MAY also wish to use this procedure for Trust Models NS-RT and S-T to protect itself from unintended tampering due to an error inbuffer before it validates the Remote Peer.contents of the buffer (or uses the buffer in any way). 7.3.3 UsingMultiple StagsSTags to access the same buffer See section 7.4.6 on page 2527 for this analysis. 7.4 Information Disclosure The main potential source for information disclosure7.3.4 Network based modification of buffer content This is throughactually a local buffer that has been enabled for remote access. Ifman in the buffer can be probed bymiddle attack û but only on the content of the buffer, as opposed to the man in the middle attack presented above, where both the signaling and content can be modified. See Section 7.2.3 Man in the Middle Attack on page 22. 7.4 Information Disclosure The main potential source for information disclosure is through a local buffer that has been enabled for remote access. If the buffer can be probed by a Remote Peer on another connection,Stream, then there is potential for information disclosure. Information disclosure attacks mainly apply to the Trust Models that include Untrusted Remote Peers (Trust Models NS-NT, S-NT, and S-LT as defined in Section 5). Trusted Remote Peers are assumed not to purposely attempt such attacks - any attempt is assumed to be due to an error or other unexpected failure in the Remote Peer.The potential attacks that could result in unintended information disclosure and countermeasures are as follows:detailed in the following sections. 7.4.1 Probing memory outside of the buffer bounds This is essentially the same attack as described in Section 7.3.1, except an RDMA Read Request is used to mount the attack. The same countermeasure applies. 7.4.2 Using RDMA Read to Access Stale Data If a buffer is being used for a combination of reads and writes (either remote or local), and is exposed to the Remote Peer with at least remote read access rights, the Remote Peer may be able to examine the contents of the buffer before they are initialized with the correct data. In this situation, whatever contents were present in the buffer before the buffer is initialized can be viewed by the Remote Peer, if the Remote Peer performs an RDMA Read. This threat applies to Trust Models NS-NT, S-NT, and S-LT.Because of this, it is RECOMMENDED that the Local Peer ensure that no stale data is contained in the buffer whenbefore remote read access rights are initiallygranted (this can be done by zeroing the contents of the memory, for example). 7.4.3 Accessing a buffer afterBuffer After the transfer is overTransfer If the Remote Peer has remote read access to a buffer, and by some mechanism tells the Local Peer that the transfer has been completed, but the Local Peer does not disable remote access to the buffer before modifying the data, it is possible for the Remote Peer to retrieve the new data. This is similar to the attack defined in Section 7.3.2 Modifying a Buffer After Indicating Contents Are ReadyIndicati on page 23.25. The same countermeasures apply. In addition, it is RECOMMENDED that the Local Peer should grant remote read access rights only for the amount of time needed to retrieve the data. 7.4.4 Accessing data withinUnintended Data With a validValid STag that was unintendedIf the Local Peer enables remote access to a buffer using an STag that references the entire buffer, but intends only a portion of the buffer to be accessed, it is possible for the Remote Peer to access the other parts of the buffer anyway. This threat applies to Trust Models NS-NT, S-NT, and S-LT.To prevent this attack, it is RECOMMENDED that the Local Peer set the base and bounds of the buffer when the STag is initialized to expose only the data to be retrieved. 7.4.5 UsingRDMA Read on a buffer meant only forinto an RDMA Write Buffer One form of disclosure can occur if the access rights on the buffer were set forenabled remote read, when only remote write access was intended. This attack applies primarily to Trust Models with Untrusted Remote Peers (NS-NT, S-NT and S-LT).If the buffer contained application data, or data from a transfer on an unrelated connection,Stream, the Remote Peer could retrieve the data through an RDMA Read operation. The most obvious countermeasure for this attack is to not grant remote read access if the buffer is intended to be write-only. TheThen the Remote Peer would not be able to retrieve data associated with the buffer. An attempt to do so would result in an error and the RDMAP Stream associated with the connectionStream would be terminated. Thus, it is RECOMMENDED that if an application only intends a buffer to be exposed for remote write access, it set the access rights to the buffer to only enable remote write access. 7.4.6 Using Multiple StagsSTags to access the same bufferAccess One Buffer Multiple STags accessing the same buffer at the same time can result in unintentional information disclosure if the STags are used by differentdifferent, mutually untrusted, Remote Peers. Because an RDMA implementation could allow an STag to have read, write, or read and write access associated with an Stag, it is possibleThis model applies specifically to have unintended information disclosure ifclient/server communication, where the Remote Peersserver is communicating with multiple clients, each of which do not share the same Trust Model.mutually trust each other. If only read access is enabled, then the Local Peer has complete control over theinformation disclosure anddisclosure. Thus a server which intended to expose the same data (i.e. buffer) to multiple clients by using multiple StagsSTags to the same buffer creates no new security issues. Whenissues beyond what has already been described in this document. Note that if the server did not intend to expose the same data to the clients, it should use separate buffers for each client (and separate STags). When one STag has remote read access enabled and a different STag has remote write access enabled to the same buffer, it is possible for one connectionRemote Peer to view the contents that have been written by another Remote Peer. If both StagsSTags have remote write access enabled and the two Remote Peers do not mutually trust each other, it is possible for one Remote Peer to overwrite the contents that have been written by the other Remote Peer. For Trust Models NS-NT, S-NT, S-LTThus it is RECOMMENDED that multiple Remote Peers which do not share Partial Mutual Trust not be granted write access to the same buffer through different STags at the same time.STags. A buffer should be exposed to only one Untrusteduntrusted Remote Peer at a time to ensure that no information disclosure or information tampering occurs between peers. 7.4.7 Remote node loading firmwareNode Loading Firmware onto the RNIC If the Remote Peer can cause firmware to be loaded onto the RNIC, there is an opportunity for information disclosure. See Elevation of Privilege in Section 7.6 for this analysis. 7.4.8 Controlling Access to PTT & STag Mapping If a Non-Privileged application is able to directly manipulate the RNIC Page Translation Tables (which translate from an STag to a host address), it is possible that the Non-Privileged application could point the Page Translation Table at an unrelated applicationÆs buffers and STag Mapping Issue: Discuss issuesthereby be able to gain access to information in allowingthe non-privileged consumerunrelated application. As discussed in Section 4 Architectural Model on page 8, introduction of a Privileged Resource Manager to controlarbitrate the mapping requests is an effective countermeasure. This enables the Privileged Resource Manager to ensure an application can only initialize the Page Translation Table and Stag. It(PTT)to point to its own buffers. Thus it is RECOMMENDED that the Privileged Resource Manager verify that the Non-Privileged application has the right to access a specific Data Buffer before allowing an STag for which the application has access rights to be associated with a specific Data Buffer. This can be done when the Page Translation Table is initialized to access the Data Buffer or when the STag is initialized to point to a group of Page Translation Table entries, or both. 7.4.9 Network based eaves dropping An attacker, eaves dropping the network, can read the content of all read and write access to the peerÆs buffers. To prevent information disclosure, the read/written data must be encrypted. The encryption can be done either by the ULP, or by a protocol that provides security services to the LLP (e.g. IPsec or SSL). Refer to section 8 for discussion of security services for RDDP/RDMA. 7.5 Denial of Service (DOS) A DOS attack is one of the primary security risks of RDMAP. This is because RNIC resources are valuable and scarce, and many application environments require communication with Untrusteduntrusted Remote Peers. If the remote application can be authenticated or encrypted, clearly, the DOS profile can be reduced. For the purposes of this analysis, it is assumed that the RNIC must be able to operate in Untrusteduntrusted environments, which are open to DOS style attacks. Denial of service attacks against RI resources are not the typical unknown party spraying packets at a random host (such as a TCP SYN attack). Because the connectionconnection/Stream must be fully established, the attacker must be able to both send and receive messages over that connection,connection/Stream, or be able to guess a valid packet on an existing RDMAP Stream. This section outlines the potential attacks and the countermeasures available for dealing with each attack. For each attack, the Trust Model that it applies to is described.7.5.1 RNIC Resource Consumption This section covers attacks that fall into the general category of a Local Peer attempting to unfairly allocate scarce RNIC resources. The Local Peer may be attempting to allocate resources on its own behalf, or on behalf of a Remote Peer. Resources that fall into this category include: Protection Domains, ConnectionStream Context Memory, Translation and Protection Tables, and STag namespace. These can be attacks by currently active Local Peers or ones that allocated resources earlier, but are now idle. These attacks generally apply to any Trust Model that includes Untrusted Local Peers (Trust Models NS-NT, NS-RT and S-NT).This type of attack can occur even whenregardless of whether resources are notshared across connections.Streams. It is RECOMMENDED that the allocation of all scarce resources be placed under the control of a Privileged Resource Manager. This allows the Privileged Resource Manager to: * prevent a Local Peer from allocating more than its fair share of resources, andresources. * detect if a Remote Peer is attempting to launch a DOS attack by attempting to create an excessive number of connectionsStreams and take corrective action (such as refusing the request or applying network layer filters against the Remote Peer). Note that, placing scarce resource management under the control of a Resource Manager also prevents other Trusted Local Peers from attempting to allocate more than their fair share of resources.This analysis assumes that the Resource Manager is responsible for handing out Protection Domains, and RNIC implementations will provide enough Protection Domains to allow the Resource Manager to be able to assign a unique Protection Domain for each unrelated, Untrusteduntrusted Local Peer (for a bounded, reasonable number of Local Peers). This analysis further assumes that the Resource Manager implements policies to ensure that Untrusteduntrusted Local Peers are not able to consume all of the Protection Domains through a DOS attack. Note that Protection Domain consumption cannot result from a DOS attack launched by a Remote Peer, unless a Local Peer is acting on the Remote PeerÆs behalf. 7.5.2 Resource Consumption By Active Applications This section describes DOS attacks from Local and Remote Peers that are actively exchanging messages. Attacks on each RDMA NIC resource are examined, including the Trust Models that apply,examined and thespecific countermeasures.countermeasures are identified. Note that,that attacks on ConnectionStream Context Memory, Page Translation Tables, and STag namespace are covered in Section 7.5.1 RNIC Resource Consumption, so are not included here. 220.127.116.11 Multiple Streams Sharing Receive DataBuffers The Remote Peer can attempt to consume more than its fair share of receive data buffers (Untagged DDP buffers or for RDMAP buffers consumed with Send Type Messages).Messages) if receive buffers are shared across multiple Streams. If resources are not shared across multiple connections (Trust Models NS-NT, NS-RT),Streams, then this attack is not possible because the Remote Peer will not be able to consume more buffers than were allocated to the Stream. The worst case scenario is that the Remote Peer can consume more receive buffers than the Local Peer allowed, resulting in no buffers to be available, which wouldcould cause the Remote PeerÆs connectionStream to the Local Peer to be torn down. If local receive data buffers are shared among multiple Streams and the Remote Peer is not Trusted (Trust Models S-NT, S-LT),Streams, then the Remote Peer can attempt to consume more than its fair share of the receive buffers, causing a different Stream to be short of receive buffers, thus possibly causing the other Stream to be torn down. For example, if the Remote Peer sent enough one byte Untagged Messages, they might be able to consume all local shared receive queue resources with little effort on their part. One method the Local Peer could use is to recognize that a Remote Peer is attempting to use more than its fair share of resources and terminate the Stream. However, if the Local Peer is sufficiently slow, it may be possible for the Remote Peer to still mount a denial of service attack. An RNIC Engine implementation that enables a more robustOne countermeasure is onethat provides high andcan protect against this attack is implementing a low-water notifications to enablenotification. The low-water notification alerts the application if the number of buffers in the receive queue is less than a threshold. If all of the following conditions are true, then the Local Peer can size the amount of local receive buffers posted on the receive queue to detect and preventensure a DOS attacks against shared data buffers. Ifattack can be stopped. * a low-water notification is enabled, and * the Local Peer is able to bound the amount of time that it takes to replenish receive buffers, and * the Local Peer maintains statistics to determine which Remote Peer is consuming buffers, then the Local Peer can size the amount of local receive buffers posted on the receive queue such thatbuffers. The above conditions enable the low-water notification canto arrive before resources are depleted and thus the Local Peer can take corrective action can be taken(e.g., terminate the Stream of the attacking Remote Peer). Enabling the high-water notification can helpA different, but similar attack is if the Local Peer detect aRemote Peer that is launching an attack by sendingsends a large number of out- of-order packets. The notification is generated when more than the specifiedsignificant number of buffers are in process simultaneously on a Stream (i.e.,out-of-order packets have started to arrive forand the buffer, butRNIC has the ability to use the application buffer as a reassembly buffer. In this case the Remote Peer can consume a significant number of application buffers, but never send enough data to enable the application buffer to be completed to the application. An effective countermeasure is to create a high-water notification which alerts the application if there is more than a specified number of receive buffers "in process" (partially consumed, but not completed). The notification is generated when more than the specified number of buffers are in process simultaneously on a specific Stream (i.e., packets have started to arrive for the buffer, but the buffer has not yet been delivered to the ULP). A different countermeasure is for the RNIC Engine to provide the capability to limit the Remote PeerÆs ability to consume receive buffers on a per Stream basis. Unfortunately this requires a large amount of state to be tracked in each RNIC on a per RNICStream basis. Thus, if an RNIC Engine provides the ability to share receive buffers across multiple Streams, it is RECOMMENDED that it enable the Local Peer to detect if the Remote Peer is attempting to consume more than its fair share of resources so that the application can apply countermeasures to detect and prevent the attack. 18.104.22.168 Completion Queue (CQ) Resource ConsumptionLocal Peer Attacking a Shared CQ DOS attacks against thea Shared Completion Queue (CQ) can be caused by either the Local Peer or the Remote Peer if either attempts to cause more completions than its fair share,share of the number of entries, thus potentially starving another unrelated Stream such that no Completion Queue entries are available. A Completion Queue entry can potentially be consumed by a completion from the send queue or a receive completion. In the former, the attacker is the Local Peer (Trust Models S-NT).Peer. In the later, the attacker is the Remote Peer (S-NT, S-LT). The potential attacks and the countermeasures for each are described in the subsections that follow. 22.214.171.124.1 Local Peer Attacking a Shared CQPeer. A form of attack can occur for Trust Models NS-NT, NS-RT, and S-NT,where the Local Peers are Untrusted, and Local Peerscan consume resources on the CQ. Sharing a CQ across connections that belong to different Protection Domains is NOT RECOMMENDED in cases where any of the Local Peers are Untrusted.A Local Peer that is slow to free resources on the CQ by not reaping the completion status quickquickly enough could stall all other Local Peers attempting to use that CQ. One of two countermeasures can be used to avoid this kind of attack. The first is to useonly share a differentCQ per Untrusted Local Peer.between Streams that share Partial Mutual Trust. The other is to use a Trustedtrusted Local Peer to act as a third party to free resources on the CQ and place the status in intermediate storage until the Untrusteduntrusted Local Peer reaps the status information. 126.96.36.199.2For these reason, sharing a CQ across Streams that belong to different Protection Domains is NOT RECOMMENDED. 188.8.131.52 Remote Peer Attacking a Shared CQ For an overview of the Shared CQ attack model, see Section 184.108.40.206. The Remote Peer can attack a CQ by consuming more than its fair share of CQ entries by using one of two methods. The first method can only be used ifthe following methods: * The ULP protocol allows the Remote Peer to reserve a specified number of CQ entries, possibly leaving insufficient entries for other connectionsStreams that are sharing the CQ. The other method is if* If the Remote Peer or Local Peer (or both) can attack the CQ by overwhelming the CQ with completions, which can affectthen completion processing on other Streams sharing that connection. The first method of attackCompletion Queue can be affected (e.g. the Completion Queue overflows and stops functioning). The first method of attack can be avoided if the ULP does not allow a Remote Peer to reserve CQ entries. Thisentries or there is RECOMMENDED particularly for Trust Models S-NT and S-LT, with shared resources and Untrusted Remote Peers. Ifa Local Peer allows this type of resource allocation, andtrusted intermediary such as a Privileged Resource Manager. Unfortunately it has any Untrustedis often unrealistic to not allow a Remote Peers, then the LocalPeer to reserve CQ entries û particularly if the number of completion entries is dependent on other ULP negotiated parameters, such as the amount of buffering required by the ULP. Thus it is RECOMMENDED that an implementation require a Privileged Resource Manager to control the allocation of CQ be isolated to connections within a single Protection Domain.entries. One way that a Local or Remote Peer can attempt to overwhelm itsa CQ with completions is by sending minimum length RDMAP/DDP Messages to cause as many completions (receive completions for the Remote Peer, send completions for the Local Peer) per second as possible. AssumingIf it is the Remote Peer attacking, and we assume that the Local Peer does not run out of receive buffers (if they do, then this is a different attack, documented in Section 220.127.116.11 Multiple Streams Sharing Receive DataBuffers on page 28),30), then it might be possible for the Remote Peer to consume more than its fair share of Completion Queue entries. Depending upon the CQ implementation, this could either cause the CQ to overflow (if it is not large enough to handle all of the completions generated) or for another Stream to not be able to generate CQ entries (if the RNIC had flow control on generation of CQ entries into the CQ). In either case, the CQ will stop functioning correctly and any connectionsStreams expecting completions on the CQ will stop functioning. This attack can occur regardless of whether all of the connectionsStreams associated with the CQ are in the same Protection Domain or are in different Protection Domains. Because this attack assumesDomains û the key issue is that the number of Completion Queue entries is less than the number of all outstanding operations that can cause a shared local resource and an Untrusted Remote Peer, Trust Models S-NT, S-LT apply.completion. The Local Peer can protect itself from this type of attack using either of the following methods: * Resize the CQ to the appropriate level(note that resizing the CQ can fail, so the CQ resize should be done before sizing the buffersSend Queue and Receive Queue on the connection),Stream), OR * Grant fewer resources than the Remote Peer requested (not supplying the number of Receive Data Buffers requested). The proper sizing of the CQ is dependent on whether the Trust Model. ForLocal Peer will post as many resources to the various queues as the size of the queue enables or not. If the Trust Model described in Section 5, with TrustedLocal Peers and Untrusted Remote Peers (Trust Model S-LT),Peer can be trusted to post a number of resources that is smaller than the size of the specific resourceÆs queue, then a correctly sized CQ means that the CQ is large enough to hold completion status for all of the outstanding ReceiveData Buffers,Buffers (both send and receive buffers), or: CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ) + SUM(MaxPostedOnEachS-RQ) + SUM(MaxPostedOnEachSQ) If the Trust Model assumes neitherlocal peer must be able to completely fill the Local Peer nor the Remote Peer is trusted (Trust Model S-NTqueues, or S-LT),can not be trusted to observe a limit smaller than the queues, then the CQ must be sized to accommodate the maximum number of operations or Receive Data Buffersthat it is possible to post at any one time, thustime. Thus the equation becomes: CQ_MIN_SIZE = SUM(SizeOfEachRQ) + SUM(SizeOfEachS-RQ) + SUM(SizeOfEachSQ) Where MaxPosted*OnEach*Q and SizeOfEach*Q varies on a per connectionStream or per Shared Receive Queue basis. 18.104.22.168 RDMA Read RequestIt is RECOMMENDED that the Local Peer implement a mechanism to ensure that the Completion Queue Two typescan not overflow. Note that it is possible to share CQs even if the Remote Peers accessing the CQs are untrusted if either of attacksthe above two formulas are possible against resources associated withimplemented. If the Local Peer can be trusted to not post more than MaxPostedOnEachRQ, MaxPostedOnEachS-RQ, and MaxPostedOnEachSQ, then the first formula applies. If the Local Peer can not be trusted to obey the limit, then the second formula applies. 22.214.171.124 RDMA Read Request Queues. One style of attack can only occur when theQueue If RDMA Read Request Queue resources are pooled across multiple connections. ThisStreams, one attack occurs when an Untrustedis if the Local Peer attempts to unfairly allocate RDMA Read Request Queue resources for its connections. ItStreams. For example, the Local Peer attempts to allocate all available resources on a specific RDMA Read Request Queue for its Streams, thereby denying the resource to applications sharing the RDMA Read Request Queue. The same type of argument applies even if the RDMA Read Request is not shared û but a Local Peer attempts to allocate all of the RNICs resource when the queue is created. Thus it is RECOMMENDED that access to interfaces that allocate RDMA Read Request Queue entries be restricted to a Trustedtrusted Local Peer, such as a Privileged Resource Manager. The Privileged Resource Manager should prevent a Local Peer from allocating more than its fair share of resources. Another form of attack is if the Remote Peer sendingsends more RDMA Read Requests than the depth of the RDMA Read Request Queue at the Local Peer. This attack can be prevented by properly configuring the connection when the connection is established. The Remote PeerÆs end of the connection should be configured by a trusted agent such that the RNIC will not transmit RDMA Read Requests that exceed the depth ofIf the RDMA Read Request Queue atis a shared resource, this could corrupt the Local Peer.queue. If the connectionqueue is not shared, then the worst case is that the current Stream is disabled. One approach to solving the shared RDMA Read Request Queue would be to create thresholds, similar to those described in Section 126.96.36.199 Multiple Streams Sharing Receive Buffers on page 30. A simpler approach is to not share RDMA Read Request Queue resources amoung Streams or enforce hard limits of consumption per Stream. Thus it is RECOMMENDED that RDMA Read Request Queue resource consumption be controlled such that RDMAP/DDP Streams which do not share Partial Mutual Trust do not share RDMA Read Request Queue resources. If the issue is a bug in the Remote PeerÆs implementation, and not a malicious attack, the issue can be solved by requiring the Remote PeerÆs RNIC to throttle RDMA Read Requests. By properly configuring the Stream at the Remote Peer through a trusted agent, the RNIC can be made to not transmit RDMA Read Requests that exceed the depth of the RDMA Read Request Queue at the Local Peer. If the Stream is correctly configured, and if the Remote Peer submits more requests than the Local PeerÆs RDMA Read Request Queue can handle, the requests willwould be queued at the Remote PeerÆs connectionRNIC until previous requests complete. If the Remote PeerÆs connectionStream is not configured correctly, the RDMAP Stream for that connectionis terminated when more RDMA Read Requests arrive at the Local Peer than the Local Peer can handle. Thus,handle (assuming the Remote Peerprior paragraphÆs recommendation is able to only affect its own connection.implemented). 7.5.3 Resource Consumption by Idle Applications The simplest form of a DOS attack given a fixed amount of resources is for the Remote Peer to create a RDMAP Stream to a Local Peer, and request dedicated resources then do no actual work. This allows the Remote Peer to be very light weight (i.e. only negotiate resources, but do no data transfer) and consumes a disproportionate amount of resources in the server. A general countermeasure for this style of attack is to monitor active RDMAP Streams and if resources are getting low, reap the resources from RDMAP Streams that are not transferring data and possibly terminate the connection.Stream. This needs towould presumably be under administrative control, and demonstrates the need for a MIB for RDMAP so this condition can be detected and acted upon.control. Refer to Section 7.5.1 for the analysis and countermeasures for this style of attack on the following RNIC resources: ConnectionStream Context Memory, Page Translation Tables and STag namespace. Note that some RNIC resources are not at risk of this type of attack from a Remote Peer because an attack requires the Remote Peer to send messages in order to consume the resource. Receive Data Buffers, Completion Queue, and RDMA Read Request Queue resources are examples. These resources are, however, at risk formfrom a Local Peer that attempts to allocate resources, then goes idle. This could also be created if the ULP negotiates the resource levels with the Remote Peer, which causes the Local Peer to consume resources, however the Remote Peer never sends data to consume them. The general countermeasure described in this section can be used to free resources allocated by an idle Local Peer. 7.5.4 Exercise of non-optimal code paths Another form of DOS attack is to attempt to exercise data paths that can consume a disproportionate amount of resources. An example might be if error cases are handled on a ææslow pathÆÆ"slow path" (consuming either host or RNIC computational resources), and an attacker generates excessive numbers of errors in an attempt to consume these resources. It is RECOMMENDED that an implementation provide the ability to detect the above condition and allow an administrator to act, including potentially administratively tearing down the RDMAP Stream associated with the connectionStream exercising data paths consuming a disproportionate amount of resources. 7.5.5 Remote Invalidation ofRI an STag Shared Acrosson Multiple ConnectionsStreams If a Local Peer has enabled an STag for remote access, the Remote Peer could attempt to remote invalidate (RI) the STag by using the RDMAP Send with Invalidate or Send with SE and Invalidate Message. If the STag is only valid on the current connection (NS-NT or NS-RT, S-NT),Stream, then the only side effect is that the Remote Peer can no longer use the STag,STag; thus there are no security issues. If the STag is valid across multiple connections,Streams, then the Remote Peer can prevent other connectionsStreams from using that STag by using the remote invalidate functionality. Thus forif RDDP Streams do not share Partial Mutual Trust Models where(i.e. the Remote Peer may attempt to invalidate the STag prematurely, the application SHOULDprematurely), it is NOT RECOMMENDED that the application allow an STag to be valid across multiple connections.Streams. 7.5.6 Remote Peer Consumes too manyUntagged Receive Buffers Issue: Need to analyze the case of sharing a queue of Untagged receive buffers across multiple connections, and that the Remote Peer can mount<TBD û remove this section: this section was deleted because it was a denialduplicate of service attack. Below are some notes. Many ways to attack here. If receive queue is not shared, itÆs a simple queue overflow attack on a dedicated resource. Make sure when the queue is empty and a DDP segment arrives nothing bad happens. For a shared receive queue, one node attacks with single byte Untagged Messages to consume large UntaggedSection 188.8.131.52 Multiple Streams Sharing Receive Buffers (this maximizes packet arrival rate). One node provides DDP segments out of orderon page 30) Thus comments on this section were added to consume out-of-order resources (this is only possible if out-of- order placement is supported within a merged TCP/SCTP and DDP implementation).that section.> 7.6 Elevation of Privilege The RDMAP/DDP Security Architecture explicitly differentiates between three levels of privilege - Non-Privileged, Privileged, and the Privileged Resource Manager. If a Non-Privileged Application is able to elevate its privilege level to a Privileged Application, then mapping a physical address list to an STag can provide local and remote access to any physical address location on the node. If a Privileged Mode Application is able to promote itself to be a Resource Manager, then it is possible for it to perform denial of service type attacks where substantial amounts of local resources could be consumed. There is only one mechanism discovered to date, other than implementation defects, which would potentially allow anIn general, elevation of privilege. 7.6.1 Loading Firmware intoprivilege is a local implementation specific issue and thus outside the RNICscope of this specification. There is one issue worth noting, however. If the RI implementation, by some insecure mechanism (or implementation defect), can enable a Remote Peer or un-trusted Local Peer to load firmware into the RNIC Engine, it is possible to use the RNIC to attack the host. Thus, it is RECOMMENDED that an implementation not enable firmware to be loaded on the RNIC Engine directly from a Remote Peer, unless the Remote Peer is properly authenticated (by a mechanism outside the scope of this specification), and the update is done via a secure protocol, such as IPsec (See Section 8 Security Services for RDDP on page 35).38). It is RECOMMENDED that an implementation not allow a Non- Privileged Local Peer to update firmware in the RNIC Engine. 8 Security Services for RDDP Issue: Security Services section is a placeholderThe spec currently took the IPSec requirements for now. 8.1 IntroductioniSCSI and made them a SHOULD recommendation. A different approach would be to Security Options 8.1.1 Introductionsimply outline the issues in this section, but leave IPSec implementation requirements to IPsec IPsecbe specified by ULP/Application requirements. The argument here is that RDDP is a transport, and security requirements û particularly authentication and confidentiality requirements, are dictated by application concerns, not transport protocol suite which isconcerns. Which approach should be taken? RDMA and RDDP are used to secure communication atcontrol, read and write data buffers over IP networks. Therefore, the network layercontrol and the data packets of these protocols are vulnerable to the spoofing, tampering and information disclosure attacks listed in Section 7. Generally speaking, session confidentiality protects against eaves dropping. Session authentication and integrity protection is a counter measurement against various spoofing and tampering attacks. The effectiveness of authentication and integrity against a specific attack, depend on whether the authentication is machine level authentication (as the one provided by IPsec and SSL), or ULP authentication. 8.1 Introduction to Security Options The following security services can be applied to an RDDP/RDMA session: 1. Session confidentiality - protects against eaves dropping (section 7.4.9). 2. Per-packet data source authentication - protects against the following spoofing attacks: network based impersonation (section 7.2.1), Stream hijacking (section 7.2.2), and man in the middle (section 7.2.3). 3. Per-packet integrity - protects against tampering done by network based modification of buffer content (section 7.3.4) 4. Packet sequencing - protects against replay attacks, which is a special case of the above tampering attack. If RDDP/RDMA session may be subject to impersonation attacks, or Stream hijacking attacks, it is RECOMMENDED that the session be authenticated, integrity protected, and protected from replay attacks; it MAY use confidentiality protection to protect from eaves dropping (in case the RDDP/RDMA session traverses a public network). Both IPsec and SSL are capable of providing the above security services for IP and TCP traffic respectively. ULP protocols are able to provide only part of the above security services. The next sections describe the different security options. 8.1.1 Introduction to IPsec IPsec is a protocol suite which is used to secure communication at the network layer between two peers. The IPsec protocol suite is specified within the IP Security Architecture [RFC2401], IKE [RFC2409], IPsec Authentication Header (AH) [RFC2402] and IPsec Encapsulating Security Payload (ESP) [RFC2406] documents. IKE is the key management protocol while AH and ESP are used to protect IP traffic. An IPsec SA is a one-way security association, uniquely identified by the 3-tuple: Security Parameter Index (SPI), protocol (ESP) and destination IP. The parameters for an IPsec security association are typically established by a key management protocol. These include the encapsulation mode, encapsulation type, session keys and SPI values. IKE is a two phase negotiation protocol based on the modular exchange of messages defined by ISAKMP [RFC2408],and the IP Security Domain of Interpretation (DOI) [RFC2407]. IKE has two phases, and accomplishes the following functions: 1. Protected cipher suite and options negotiation - using keyed MACs and encryption and anti-replay mechanismsmechanisms. 2. Master key generation - such asvia MODPDiffie-Hellman calculationscalculations. 3. Authentication of end-points (usually machine level authentication). 4. IPsec SA management (selector negotiation, options negotiation, create, delete, and rekeying)rekeying). Items 1 through 3 are accomplished in IKE Phase 1, while item 4 is handled in IKE Phase 2. IKE phase 1 defines four authentication methods; three of them require both sides to have certified signature or encryption public keys; the forth require the side to exchange out-of-band a secret random string û called pre-shared-secret (PSS). An IKE Phase 2 negotiation is performed to establish both an inbound and an outbound IPsec SA. The traffic to be protected by an IPsec SA is determined by a selector which has been proposed by the IKE initiator and accepted by the IKE Responder. In IPsec transport mode, theThe IPsec SA selector can be a "filter" or traffic classifier, defined as the 5-tuple: <Source IP address, Destination IP address, transport protocol (UDP/SCTP/TCP),(e.g. UDP/SCTP/TCP), Source port, Destination port>. The successful establishment of a IKE Phase-2 SA results in the creation of two uni-directional IPsec SAs fully qualified by the tuple <Protocol (ESP/AH), destination address, SPI>. The session keys for each IPsec SA are derived from a master key, typically via a MODP Diffie-Hellman computation. Rekeying of an existing IPsec SA pair is accomplished by creating two new IPsec SAs, making them active, and then optionally deleting the older IPsec SA pair. Typically the new outbound SA is used immediately, and the old inbound SA is left active to receive packets for some locally defined time, perhaps 30 seconds or 1 minute. Optionally, rekeying can use Diffie-Helman for keying material generation. 8.1.2 Introduction to SSL Limitations on RDMAP 8.1.3 Applications Which Provide Security Issue: GuidanceSSL and TLS [RFC 2246] provide session authentication, integrity and confidentiality for applicationTCP based applications. SSL supports one- way (server only) or mutual certificates based authentication. There are at least two limitations that make SSL less appropriate then IPsec for RDDP/RDMA security: 1. The maximum length supported by the TLS record layer protocol is 2^14 bytes, longer packets must be fragmented (as a comparison, the maximal length of an IPsec packet, is determined by the maximum length of an IP packet). 2. SSL is a connection oriented protocol. If a stream cipher or block cipher in CBC mode is used for bulk encryption, then a packet can be decrypted only after all the packets preceding it have already arrived. If SSL is used to protect RDDP/RDMA traffic, then RDDP/RDMA must gather all out-of-order packets before placing them into the ULP buffer, which might cause a significant decrease in its efficiency. 8.1.3 Applications Which Provide Security Issue: Guidance for application protocols like NFS which implement security. 8.1.4 Authentication Only 8.1.5 Privacysecurity <TBD>. 8.2 Recommendations for IPsec Encapsulation of RDDP Issue: IPsec recommendations for RDMAP/DDP This is work thatSince iSCSI is stillexpected to be done. Hopefully this wonÆt be terribly complex. One possible thoughtone of the ULPs running on top of RDDP, the approach: a. Userecommendations in this section follow the lines of [IPSSEC]. 8.2.1 Transforms All RDDP/RDMA security compliant implementations SHOULD support IPsec ESP with authentication[RFC2406] to provide security for both control packets and data packets, as well as the replay protection mechanisms of IPsec. When ESP is utilized, per-packet data origin authentication, integrity and replay protection. b. Use IKE for key management. c. (optionally) use a non-null transform for encryption. This shouldprotection MUST be something other than DES. 9 Summary Table of Attacksused. To provide confidentiality with ESP, ESP with 3DES in CBC mode [RFC2451] SHOULD be supported, and Trust Models Issue: Finish Summary table of Attacks/Trust Models <editor: This section is under construction,AES in Counter mode, as described in [AESCTR], SHOULD be supported. To provide data origin authentication and willintegrity with ESP, HMAC-SHA1 [RFC2404] SHOULD be completedsupported, and AES in a future version of this document> Rows are the attack (grouped into categories) Columns are the: * Sec - Section the attack is discussed * Attack Name - short nameCBC MAC mode with XCBC extensions [AESXCBC] SHOULD be supported. DES in CBC mode SHOULD NOT be used due to its inherent weakness. ESP with NULL encryption SHOULD be supported for authentication. 8.2.2 IPsec modes Conformant IP RDDP/RDMA security implementations SHOULD support ESP [RFC2406] in tunnel mode and MAY implement IPsec with ESP in transport mode. 8.2.3 IKE Conformant RDDP/RDMA security implementations SHOULD support IKE [RFC2409] for peer authentication, negotiation of security associations, and key management, using the attack * Threat - threat type (DOS, etc) * Columns labeled 1-5 areIPsec DOI [RFC2407]. Manual keying MUST NOT be used since it does not provide the Trust Model number (see section 5 Trust Models on page 15). Each entry hasnecessary rekeying support. Conformant RDDP/RDMA security implementations SHOULD support peer authentication using a value of +, -,pre-shared secret, and R (research). +-------+--------------------------+-------+---+---+---+---+---+ | Sec | Attack Name |Threat | 1 | 2 | 3 | 4 | 5 | +-------+--------------------------+-------+---+---+---+---+---+ | 7.2.1 | STag use on different | Spoof | | | | | | | | connectionMAY support certificate-based peer authentication using digital signatures. Peer authentication using the public key encryption methods outlined in same PD | | | | | | | +-------+--------------------------+-------+---+---+---+---+---+ | 7.3.1 | Memory write outsideIKE's sections 5.2 and 5.3 [RFC2409] SHOULD NOT be used. Conformant RDDP/RDMA security implementations SHOULD support IKE Main Mode and Aggressive Mode. IKE Main Mode with pre-shared key authentication SHOULD NOT be used when either of | Tamper| | | | | | | | buffer range | | | | | | | | 7.3.2 | Modify Buffer after | Tamper| | | | | | | | contents ready | | | | | | | +-------+--------------------------+-------+---+---+---+---+---+ | 7.4.1 | Probe memory outsidethe peers uses a dynamically assigned IP address. While Main Mode with pre-shared key authentication offers good security in many cases, situations where dynamically assigned addresses are used force use of | ID | | | | | | | | buffer bounds | | | | | | | | 7.4.2 | Access stale data | ID | | | | | | | 7.4.3 | Access buffer after | ID | | | | | | | | transfer over | | | | | | | | 7.4.4 | Unintended data access | ID | | | | | | | | using valid STag | | | | | | | | 7.4.5 | Using RDMA Read ona | ID | | | | | | | | buffer meant only for | | | | | | | | | RDMA Write | | | | | | | | 7.4.6 | Using multiple STagsgroup pre-shared key, which is vulnerable to | ID | | | | | | | |man-in-the-middle attack. Since IKE Aggressive mode with pre-shared secret authentication is exposed to off-line dictionary attack if it is used then the selected pre-shared secrets must be random (or pseudo-random) strings not shorter than 128 bits. When digital signatures are used for authentication, either IKE Main Mode or IKE Aggressive Mode MAY be used. In all cases, access to locally stored secret information (pre-shared key, or private key for digital signing) must be suitably restricted, since compromise of the same buffer | | | | | | | | 7.4.7 | Remote node loading | ID | | | | | | | | firmware onto RNIC | | | | | | | +-------+--------------------------+-------+---+---+---+---+---+ | 7.5.1 | RNIC resource consumption| DOS | | | | | | | 7.5.2 | Resource consumption by | DOS | | | | | | | | active processes | | | | | | | | 7.5.3 | Resource consumptionsecret information nullifies the security properties of the IKE/IPsec protocols. When digital signatures are used to achieve authentication, an IKE negotiator SHOULD use IKE Certificate Request Payload(s) to specify the certificate authority (or authorities) that are trusted in accordance with its local policy. IKE negotiators SHOULD check the pertinent Certificate Revocation List (CRL) before accepting a PKI certificate for use in IKE's authentication procedures. The IPsec DOI [RFC2407] provides for several types of identification data. Within IKE Phase 1, for use within the IDii and IDir payloads, conformant RDDP/RDMA security implementations SHOULD support the ID_IPV4_ADDR, ID_IPV6_ADDR (if the protocol stack supports IPv6) and ID_FQDN Identity Payloads. Identities other than ID_IPV4_ADDR and ID_IPV6_ADDR (such as ID_FQDN) SHOULD be employed in situations where Aggressive mode is utilized along with pre-shared keys and IP addresses are dynamically assigned. The IP Subnet, IP Address Range, ID_DER_ASN1_DN, ID_DER_ASN1_GN, and ID_USER_FQDN formats SHOULD NOT be used for RDDP/RDMA protocol security; The ID_KEY_ID Identity Payload MUST NOT be used. As described in [RFC2407], within Phase 1 the ID port and protocol fields MUST be set to zero or to UDP port 500. Also, as noted in [RFC2407]: When an IKE exchange is authenticated using certificates (of any format), any ID's used for input to local policy decisions SHOULD be contained in the certificate used in the authentication of the exchange. The Phase 2 Quick Mode exchanges used by | DOS | | | | | | | | idle processes | | | | | | | | 7.5.4 | Non-optimal code paths | DOS | | | | | | +-------+--------------------------+-------+---+---+---+---+---+ | 7.6.1 | Loading firmware onto | Elev | | | | | | | | RNIC | | | | | | | +-------+--------------------------+-------+---+---+---+---+---+ FigureRDDP/RDMA protocol implementations SHOULD explicitly carry the Identity Payload fields (IDci and IDcr). Each Phase 2 - Summary AttacksIDci and Trust Model Table 10 References 10.1 Normative References [RFC2828] Shirley, R., "Internet Security Glossary", FYI 36, RFC 2828, May 2000. [DDP] Shah, H., J. Pinkerton, R.Recio,IDcr Payload SHOULD carry a single IP address (ID_IPV4_ADDR, ID_IPV6_ADDR) and P. Culley, "Direct Data Placement over Reliable Transports", Internet-Draft draft-ietf- rddp-ddp-01.txt, February 2003. [RDMAP] Recio, R., P. Culley, D. Garcia, J. Hilland, "An RDMA Protocol Specification", Internet-Draft draft-ietf-rddp-rdmap- 01.txt, February 2003. [SEC-CONS] Rescorla, E., B. Korver, IAB, "GuidelinesSHOULD NOT use the IP Subnet or IP Address Range formats. Other ID payload formats MUST NOT be used. To support iSCSI PFS requirements [IPSSEC}, conformant RDDP/RDMA security implementation SHOULD support PFS in the rekeying process (i.e. in the Quick Mode exchange). Since IPsec acceleration hardware may only be able to handle a limited number of active IKE Phase 2 SAs, Phase 2 delete messages may be sent for Writing RFC Textidle SAs, as a means of keeping the number of active Phase 2 SAs to a minimum. The receipt of an IKE Phase 2 delete message MUST NOT be interpreted as a reason for tearing down an RDDP/RDMA Stream. Rather, it is preferable to leave the Stream up, and if additional traffic is sent on it, to bring up another IKE Phase 2 SA to protect it. This avoids the potential for continually bringing Streams up and down. 8.2.4 Security Policy Configuration One of the goals of this specification is to enable a high level of interoperability without requiring extensive configuration. This section provides guidelines on setting of IKE parameters so as to enhance the probability of a successful negotiation. It also describes how information on security policy configuration can be provided so as to further enhance the chances of success. To enhance the prospects for interoperability, some of the actions to consider include:  Transform restriction. Since support for 3DES-CBC and HMAC- SHA1 is required of all implementations, offering these transforms enhances the probability of a successful negotiation. If AES-CTR [AESCTR] with XCBC-MAC [AESXCBC] is supported, this transform combination will typically be preferred, with 3DES- CBC/HMAC-SHA1 as a secondary offer.  Group Restriction. If 3DES-CBC/HMAC-SHA1 is offered, and DH groups are offered, then it is recommended that a DH group of at least 1024 bits be offered along with it. If AES-CTR/XCBC-MAC is the preferred offer, and DH groups are offered, then it is recommended that a DH group of at least 2048 bits be offered along with it, as noted in [KeyLen]. If perfect forward secrecy is required in Quick Mode, then it is recommended that the QM PFS DH group be the same as the IKE Phase 1 DH group. This reduces the total number of combinations, enhancing the chances for interoperability.  Key lifetimes. If a key lifetime is offered that is longer than desired, then rather than causing the IKE negotiation to fail, it is recommended that the Responder consider the offered lifetime as a maximum, and accept it. The key can then use a lesser value for the lifetime, and utilize a Lifetime Notify in order to inform the other peer of lifetime expiration. Even when the above advice is taken, it still may be useful to be able to provide additional configuration information in order to enhance the chances of success, and it is useful to be able to manage security configuration regardless of the scale of the deployment. For example, it may be desirable to configure the security policy of an RDDP/RDMA device. This can be done manually or automatically via a security policy distribution mechanism. Alternatively, if the ULP supports a distribution mechanism such as iSCSI with iSNS or SLPv2, those mechanism can be used to supply security policy. If an IP block storage endpoint can obtain the required security policy by other means (manually, or automatically via a security policy distribution mechanism) then it need not request this information through the ULP specific mechanism. However, if the required security policy configuration is not available via other mechanisms, those mechanismm can be used. It may also be helpful to obtain information about the preferences of the peer prior to initiating IKE. While it is generally possible to negotiate security parameters within IKE, there are situations in which incompatible parameters can cause the IKE negotiation to fail. The following information can be provided via ULP specific or other mechanisms:  IPsec or cleartext support. The minimum piece of peer configuration required is whether an RDDP/RDMA endpoint requires IPsec or cleartext. This cannot be determined from the IKE negotiation alone without risking a long timeout, which is highly undesirable for the RDMA/DDP protocol.  Perfect Forward Secrecy (PFS) support. It is helpful to know whether a peer allows PFS, since an IKE Phase 2 Quick Mode can fail if an initiator proposes PFS to a Responder that does not allow it.  Preference for tunnel mode. While it is legal to propose both transport and tunnel mode within the same offer, not all IKE implementations will support this. As a result, it is useful to know whether a peer prefers tunnel mode or transport mode, so that it is possible to negotiate the preferred mode on the first try.  Main Mode and Aggressive Mode support. Since the IKE negotiation can fail if a mode is proposed to a peer that doesn't allow it, it is helpful to know which modes a peer allows, so that an allowed mode can be negotiated on the first try. Since iSNS or SLPv2 can be used to distribute IPsec security policy and configuration information for use with IP block storage protocols and RDDP/RDMA, these discovery protocols would constitute a 'weak link' were they not secured at least as well as the protocols whose security they configure. Since the major vulnerability is packet modification and replay, when iSNS or SLPv2 are used to distribute security policy or configuration information, at a minimum, per-packet data origin authentication, integrity and replay protection MUST be used to protect the discovery protocol. 9 Security considerations Issue: I think we should refer to IPS security considerations. Most of the issues discussed there are relevant for RDDP/RDMA as well (exceptions are the discussion on user certificates).<TBD> 10 References 10.1 Normative References [RFC2828] Shirley, R., "Internet Security Glossary", FYI 36, RFC 2828, May 2000. [DDP] Shah, H., J. Pinkerton, R.Recio, and P. Culley, "Direct Data Placement over Reliable Transports", Internet-Draft draft-ietf-rddp-ddp-01.txt, February 2003. [RDMAP] Recio, R., P. Culley, D. Garcia, J. Hilland, "An RDMA Protocol Specification", Internet-Draft draft-ietf-rddp- rdmap-01.txt, February 2003. [SEC-CONS] Rescorla, E., B. Korver, IAB, "Guidelines for Writing RFC Text on Security Considerations", Internet-Draft draft- ab-sec-cons-03.txt, January 2003. [RFC2246] T. Dierks, C. Allen, "The TLS Protocol Version 1.0", RFC 2246, January 1999. [RFC2401] Atkinson, R. and Kent, S., "Security Architecture for the Internet Protocol", RFC 2401, November 1998 [RFC2402] Kent, S., Atkinson, R., "IP Authentication Header", RFC 2402, November 1998 [RFC2404] Madson, C., Glenn, R., "The Use of HMAC-SHA-1-96 within ESP and AH", RFC 2404, November 1998 [RFC2406] Kent, S., Atkinson, R., "IP Encapsulating Security Payload (ESP)", RFC 2406, November 1998 [RFC2407] Piper, D., "The Internet IP Security Domain of Interpretation of ISAKMP", RFC 2407, November 1998 [RFC2408] Maughan, D., Schertler, M., Schneider, M., Turner, J., "Internet Security Association and Key Management Protocol (ISAKMP), RFC 2408, November 1998 [RFC2409] Harkins, D., Carrel, D., "The Internet Key Exchange (IKE)", RFC 2409, November 1998 [AESCTR] Housley, R., "Using AES Counter Mode With IPsec ESP",Internet draft (work in progress), draft-ietf-ipsec- ciph-aes-ctr-05.txt, July 2003 [KeyLen] Orman, H., Hoffman, P., "Determining Strengths For Public Keys Used For Exchanging Symmetric Keys", Internet draft (work in progress), draft-orman-public-key-lengths- 07.txt, January 2004 [AESXCBC] Frankel, S., Herbert, H., "The AES-XCBC-MAC-96 Algorithm and Its Use with IPsec", Internet draft (work in progress), draft-ietf-ipsec-ciph-aes-xcbc-mac-02.txt, June 2002 [IPSSEC] Aboba B., et al, "Securing Block Storage Protocols over IP", Internet draft (work in progress), draft-ietf-ips- security-19.txt, January 2003 [SCTP] R. Stewart et al., "Stream Control Transmission Protocol", RFC 2960, October 2000. [TCP] Postel, J., "Transmission Control Protocol - DARPA Internet Program Protocol Specification", RFC 793, September 1981. 10.2 Informative References [IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor Discovery trust modelsTrust Models and threats", Internet- Draft draft-ietf-send-psreq-01.txt, January 2003. 11 Appendix A: Implementing Client/Server Protocols The prior sections outlined specific attacks and their countermeasures. This section summarizes the attacks and countermeasures defined in the prior section which are applicable to creation of a secure application server. An application server is defined as an application which must be able to communicate with many clients which do not trust each other and ensure that each client can not attack another client through server interactions. Further, the server may wish to use multiple Streams to communicate with a specific client, and those Streams may share mutual trust. All of the prior section's details on attacks and countermeasures to protect a single Stream apply to the server. This section focuses on security issues where multiple clients are talking with a single server. The following list summarizes the relevent attacks that clients can mount on the shared server, by re-stating the previous RECOMMENDations to be client/server specific (the following are just restatements of the prior RECOMMENDations): * Spoofing * Section 7.2.4 Using an STag on a Different on page 23. To ensure that one client can not access another client's data via use of their STag, it is RECOMMENDED that the server either scope an STag to a single Stream or use a Protection Domain per client, or a combination of the two approaches. * Tampering * 7.3.3 Multiple STags to access the same buffer on page 25. See the following bullet's discussion of Section 7.4.6. * Information Disclosure * 7.4.2 Using RDMA Read to Access Stale Data on page 26. It is RECOMMENDED that the server ensure that no stale data is contained in a buffer before remote read access rights are granted to a client (this can be done by zeroing the contents of the memory, for example). * 7.4.5 RDMA Read into an RDMA Write Buffer on page 27. It is RECOMMENDED that if a server only intends a buffer to be exposed for remote write access, it set the access rights to the buffer to only enable remote write access. * 7.4.6 Using Multiple STags to Access One Buffer on page 27. It is RECOMMENDED that separate clients not be granted write access to the same buffer through different STags. A buffer should be exposed to only one client at a time to ensure that no information disclosure or information tampering occurs between peers. * Denial of Service * 7.5.1 RNIC Resource Consumption on page 29. It is RECOMMENDED that the server place the allocation of all scarce resources be placed under the control of a Privileged Resource Manager. * 184.108.40.206 Multiple Streams Sharing Receive Buffers on page 30. If an RNIC Engine provides the ability to share receive buffers across multiple Streams, it is RECOMMENDED that it enable the server to detect if the client is attempting to consume more than its fair share of resources so that the server can apply countermeasures to detect and prevent the attack. * 220.127.116.11 Local Peer Attacking a Shared CQ on page 31. Sharing a CQ across Streams that belong to different Protection Domains is NOT RECOMMENDED. * 18.104.22.168 Remote Peer Attacking a Shared CQ on page 32. If a server allows the client to influence CQ entry resource allocation, then it is RECOMMENDED that the CQ be isolated to Streams within a single Protection Domain (i.e. streams that share Partial Mutual Trust). It is RECOMMENDED that the Local Peer implement a mechanism to ensure that the Completion Queue can not overflow. * 22.214.171.124 RDMA Read Request Queue on page 34. It is RECOMMENDED that access to interfaces that allocate RDMA Read Request Queue entries be restricted to a trusted Local Peer, such as a Privileged Resource Manager. It is RECOMMENDED that RDMA Read Request Queue resource consumption be controlled such that RDMAP/DDP Streams which do not share Partial Mutual Trust do not share RDMA Read Request Queue resources. * 7.5.3 Resource Consumption by Idle Applications on page 35. Refer to Section 7.5.1. * 7.5.5 RI an STag Shared on Multiple Streams on page 36. If RDDP Streams do not share Partial Mutual Trust (i.e. the client may attempt to invalidate the STag prematurely), it is NOT RECOMMENDED that the server allow an STag to be valid across multiple Streams. 12 Appendix B: Summary Table of Attacks Issue: Finish Summary table of Attacks/Trust Models <TBD> <editor: This section is under construction, and will be completed in a future version of this document> Rows are the attack (grouped into categories) Columns are the: * Sec - Section the attack is discussed * Attack Name - short name for the attack * Threat - threat type (Spoof (Spoofing), Tamp (Tampering), ID (Information Disclosure), and DOS (Denial of Service)) * SH û Does the threat assume there are shared resources (yes/no/NA û not applicable)? * TR û Does the threat assume there is Partial Mutual Trust between Streams (MT), no trust between Streams (NT), or is this parameter not applicable (NA)? 12.1 Spoofing +--------+---------------------------------------------+-----+--+--+ | Sec | Attack Name |Sh|TR| +--------+---------------------------------------------+-----+--+--+ | 7.2.1 | Impersonation |NA|NA| +--------+---------------------------------------------+-----+--+--+ | 7.2.2 | Stream Hijacking |NA|NA| +--------+---------------------------------------------+-----+--+--+ | 7.2.3 | Man in the Middle Attack |NA|NA| +--------+---------------------------------------------+-----+--+--+ | 7.2.4 | Using an STag on a Different |Y |NT| +--------+---------------------------------------------+-----+--+--+ 12.2 Tampering +--------+---------------------------------------------+-----+--+--+ | 7.3.1 | Buffer Overrun - RDMA Write or Read Response|NA|NT| +--------+---------------------------------------------+-----+--+--+ | 7.3.2 | Modifying a Buffer After Indication |NA|NT| +--------+---------------------------------------------+-----+--+--+ | 7.3.3 | Multiple STags to access the same buffer |Y |NT| +--------+---------------------------------------------+-----+--+--+ | 7.3.4 | Network based modification of buffer content|NA|NA| +--------+---------------------------------------------+-----+--+--+ 12.3 Information Disclosure +--------+---------------------------------------------+-----+--+--+ | 7.4.1 | Probing memory outside of the buffer bounds |NA|NT| +--------+---------------------------------------------+-----+--+--+ | 7.4.2 | Using RDMA Read to Access Stale Data | +--------+---------------------------------------------+-----+--+--+ | 7.4.3 | Accessing a Buffer After the Transfer | +--------+---------------------------------------------+-----+--+--+ | 7.4.4 | Accessing Unintended Data With a Valid STag | +--------+---------------------------------------------+-----+--+--+ | 7.4.5 | RDMA Read into an RDMA Write Buffer | +--------+---------------------------------------------+-----+--+--+ | 7.4.6 | Using Multiple STags to Access One Buffer | +--------+---------------------------------------------+-----+--+--+ | 7.4.7 | Remote Node Loading Firmware onto the RNIC | +--------+---------------------------------------------+-----+--+--+ | 7.4.8 | Controlling Access to PTT & STag Mapping | +--------+---------------------------------------------+-----+--+--+ | 7.4.9 | Network based eaves dropping | +--------+---------------------------------------------+-----+--+--+ 12.4 Denial of Service +--------+---------------------------------------------+-----+--+--+ | 7.5.1 | RNIC Resource Consumption | +--------+---------------------------------------------+-----+--+--+ | 126.96.36.199| Multiple Streams Sharing Receive Buffers | +--------+---------------------------------------------+-----+--+--+ | 188.8.131.52| Local Peer Attacking a Shared CQ |Error! Reference source not found. +--------+---------------------------------------------+-----+--+--+ | 184.108.40.206| Remote Peer Attacking a Shared CQ | +--------+---------------------------------------------+-----+--+--+ | 220.127.116.11| RDMA Read Request Queue | +--------+---------------------------------------------+-----+--+--+ | 7.5.3 | Resource Consumption by Idle Applications | +--------+---------------------------------------------+-----+--+--+ | 7.5.4 | Exercise of non-optimal code paths | +--------+---------------------------------------------+-----+--+--+ | 7.5.5 | RI an STag Shared on Multiple Streams | +--------+---------------------------------------------+-----+--+--+ | 7.5.6 | Remote Peer Consumes Untagged Receive Buffers| +--------+---------------------------------------------+-----+--+--+ Figure 2 - Summary Attacks and Trust Model Table 13 Appendix C: Partial Trust Taxonomy Partial Trust is defined as when one party is willing to assume that another party will refrain from a specific attack or set of attacks, the parties are said to be in a state of Partial Trust. Note that the partially trusted peer may attempt a different set of attacks. This may be appropriate for many applications where any adverse effects of the betrayal is easily confined and does not place other clients or applications at risk. The Trust Models described in this section have three primary distinguishing characteristics. The Trust Model refers to a Local Peer and Remote Peer, which are the local and remote application instances communicating via RDMA/DDP. * Local Resource Sharing (yes/no) - When local resources are shared, they are shared across a grouping of RDMAP/DDP Streams. If local resources are not shared, the resources are dedicated on a per Stream basis. Resources are defined in Section 4.2 - Resources on page 11. The advantage of not sharing resources between Streams is that it reduces the types of attacks that are possible. The disadvantage is that applications might run out of resources. * Local Partial Trust (yes/no) - Local Partial Trust is determined based on whether the local grouping of RDMAP/DDP Streams (which typically equates to one application or group of applications) mutually trust each other to not perform a specific set of attacks. * Remote Partial Trust (yes/no) - The Remote Partial Trust level is determined based on whether the Local Peer of a specific RDMAP/DDP Stream partially trusts the Remote Peer of the Stream (see the definition of Partial Trust in Section 3 Introduction). Not all of the combinations of the trust characteristics are expected to be used by applications. This paper specifically analyzes five application Trust Models that are expected to be in common use. The Trust Models are as follows: 1. NS-NT - Non-Shared Local Resources, no Local Trust, no Remote Trust - typically a server application that wants to run in the safest mode possible. All attack mitigations are in place to ensure robust operation. 2. NS-RT - Non-Shared Local Resources, no Local Trust, Remote Partial Trust - typically a peer-to-peer application, which has, by some method outside of the scope of this specification, authenticated the Remote Peer. Note that unless some form of key based authentication is used on a per RDMA/DDP session basis, it may not be possible be possible for man-in-the-middle attacks to occur. See section 8, Security Considerations", Internet-Draft draft-ab- sec-cons-03.txt, January 2003. [RFC2401] Atkinson, R. and Kent, S., "Security ArchitectureServices for RDDP on page 38. 3. S-NT - Shared Local Resources, no Local Trust, no Remote Trust - typically a server application that runs in an untrusted environment where the Internet Protocol", RFC 2401, November 1998 [RFC2402] Kent, S., Atkinson, R., "IP Authentication Header", RFC 2402, November 1998 [RFC2404] Madson, C., Glenn, R., "The Useamount of HMAC-SHA-1-96 within ESPresources required is either too large or too dynamic to dedicate for each RDMAP/DDP Stream. 4. S-LT - Shared Local Resources, Local Partial Trust, no Remote Trust - typically an application, which provides a session layer and AH", RFC 2404, November 1998 [RFC2406] Kent, S., Atkinson, R., "IP Encapsulating Security Payload (ESP)", RFC 2406, November 1998 [RFC2407] Piper, D., "The Internet IP Security Domainuses multiple Streams, to provide additional throughput or fail-over capabilities. All of Interpretationthe Streams within the local application partially trust each other, but do not trust the Remote Peer. This trust model may be appropriate for embedded environments. 5. S-T - Shared Local Resources, Local Partial Trust, Remote Partial Trust - typically a distributed application, such as a distributed database application or a High Performance Computer (HPC) application, which is intended to run on a cluster. Due to extreme resource and performance requirements, the application typically authenticates with all of ISAKMP", RFC 2407, November 1998 [RFC2408] Maughan, D., Schertler, M., Schneider, M., Turner, J., "Internet Security Associationits peers and Key Management Protocol (ISAKMP), RFC 2408, November 1998 [RFC2409] Harkins, D., Carrel, D., "The Internet Key Exchange (IKE)", RFC 2409, November 1998 10.2 Informative References [IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor Discovery trust modelsTrustthen runs in a highly trusted environment. The application peers are all in a single application fault domain and depend on one another to be well-behaved when accessing data structures. If a trusted Remote Peer has an implementation defect that results in poor behavior, the entire application could be corrupted. Models NS-NT and threats", Internet-Draft draft-ietf-send-psreq-01.txt, January 2003. 11S-NT above are typical for Internet networking - neither Local Peers nor the Remote Peer is trusted. Sometimes optimizations can be done that enable sharing of Page Translation Tables across multiple Local Peers, thus Model S-LT can be advantageous. Model S-T is typically used when resource scaling across a large parallel application makes it infeasible to use any other model. Resource scaling issues can either be due to performance around scaling or because there simply are not enough resources. Model NS-RT is probably the least likely model to be used, but is presented for completeness. 14 AuthorÆs Addresses James Pinkerton Microsoft Corporation One Microsoft Way Redmond, WA. 98052 USA Phone: +1 (425) 705-5442 Email: email@example.com Ellen Deleganes Intel Corporation MS JF5-355 2111 NE 25th Ave. Hillsboro, OR 97124 USA Phone: +1 (503) 712-4173 Email: firstname.lastname@example.org Allyn Romanow Cisco Systems 170 W Tasman Drive San Jose, CA 95134 USA Phone: +1 408 525 8836 Email: email@example.com Bernard AbobaSara Bitan Microsoft Corporation One Microsoft Way Redmond, WA. 98052 USA Phone: +1 (425) 706-6606Email: firstname.lastname@example.org email@example.com 15 Acknowledgments Catherine Meadows Naval Research Laboratory Code 5543 Washington, DC 20375 Email: firstname.lastname@example.org Patricia Thaler Agilent Technologies, Inc. 1101 Creekside Ridge Drive, #100 M/S-RG10 Roseville, CA 95678 Phone: +1-916-788-5662 email: email@example.com James Livingston NEC Solutions (America), Inc. 7525 166th Ave. N.E., Suite D210 Redmond, WA 98052-7811 Phone: +1 (425) 897-2033 Email: firstname.lastname@example.org John Carrier Adaptec, Inc. 691 S. Milpitas Blvd. Milpitas, CA 95035 USA Phone: +1 (360) 378-8526 Email: email@example.com Caitlin Bestler Email: firstname.lastname@example.org 13Bernard Aboba Microsoft Corporation One Microsoft Way Redmond, WA. 98052 USA Phone: +1 (425) 706-6606 Email: email@example.com 16 Full Copyright Statement Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Funding for the RFC Editor function is currently provided by the Internet Society.