draft-ietf-nfsv4-rpcrdma-cm-pvt-data-04.txt   draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.txt 
Network File System Version 4 C. Lever Network File System Version 4 C. Lever
Internet-Draft Oracle Internet-Draft Oracle
Intended status: Informational June 13, 2019 Updates: 8166 (if approved) November 8, 2019
Expires: December 15, 2019 Intended status: Standards Track
Expires: May 11, 2020
RDMA Connection Manager Private Data For RPC-Over-RDMA Version 1 RDMA Connection Manager Private Data For RPC-Over-RDMA Version 1
draft-ietf-nfsv4-rpcrdma-cm-pvt-data-04 draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05
Abstract Abstract
This document specifies the format of RDMA-CM Private Data exchanged This document specifies the format of RDMA-CM Private Data exchanged
between RPC-over-RDMA version 1 peers as part of establishing a between RPC-over-RDMA version 1 peers as part of establishing a
connection. Such private data is used to indicate peer support for connection. The addition of the private data payload specified in
remote invalidation and larger-than-default inline thresholds. The this document is an optional extension that does not alter the RPC-
addition of the private data payload specified in this document is an over-RDMA version 1 protocol. This document updates RFC 8166.
OPTIONAL extension. The RPC-over-RDMA version 1 protocol does not
require the payload to be present.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on December 15, 2019. This Internet-Draft will expire on May 11, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 16 skipping to change at page 2, line 16
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Requirements Language . . . . . . . . . . . . . . . . . . . . 3 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 3
3. Advertised Transport Properties . . . . . . . . . . . . . . . 3 3. Advertised Transport Properties . . . . . . . . . . . . . . . 3
3.1. Inline Threshold Size . . . . . . . . . . . . . . . . . . 3 3.1. Inline Threshold Size . . . . . . . . . . . . . . . . . . 3
3.2. Remote Invalidation . . . . . . . . . . . . . . . . . . . 4 3.2. Remote Invalidation . . . . . . . . . . . . . . . . . . . 4
4. Private Data Message Format . . . . . . . . . . . . . . . . . 5 4. Private Data Message Format . . . . . . . . . . . . . . . . . 5
4.1. Interoperability Considerations . . . . . . . . . . . . . 6 4.1. Interoperability Considerations . . . . . . . . . . . . . 6
4.1.1. Amongst RPC-over-RDMA Version 1 Implementations . . . 7
4.1.2. Amongst Implementations of Other Upper-Layer
Protocols . . . . . . . . . . . . . . . . . . . . . . 7
5. Updating the Message Format . . . . . . . . . . . . . . . . . 7 5. Updating the Message Format . . . . . . . . . . . . . . . . . 7
5.1. Feature Support Flags . . . . . . . . . . . . . . . . . . 7 5.1. Feature Support Flags . . . . . . . . . . . . . . . . . . 8
5.2. Inline Threshold Values . . . . . . . . . . . . . . . . . 8 5.2. Inline Threshold Values . . . . . . . . . . . . . . . . . 8
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8
6.1. Guidance for Designated Experts . . . . . . . . . . . . . 9
7. Security Considerations . . . . . . . . . . . . . . . . . . . 9 7. Security Considerations . . . . . . . . . . . . . . . . . . . 9
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 10
8.1. Normative References . . . . . . . . . . . . . . . . . . 9 8.1. Normative References . . . . . . . . . . . . . . . . . . 10
8.2. Informative References . . . . . . . . . . . . . . . . . 9 8.2. Informative References . . . . . . . . . . . . . . . . . 10
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 10 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 11
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 10 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 12
1. Introduction 1. Introduction
The RPC-over-RDMA version 1 transport protocol [RFC8166] enables The RPC-over-RDMA version 1 transport protocol [RFC8166] enables
payload data transfer using Remote Direct Memory Access (RDMA) for payload data transfer using Remote Direct Memory Access (RDMA) for
upper layer protocols based on Remote Procedure Calls (RPC) upper-layer protocols based on Remote Procedure Calls (RPC)
[RFC5531]. The terms "Remote Direct Memory Access" (RDMA) and [RFC5531]. The terms "Remote Direct Memory Access" (RDMA) and
"Direct Data Placement" (DDP) are introduced in [RFC5040]. "Direct Data Placement" (DDP) are introduced in [RFC5040].
The two most immediate shortcomings of RPC-over-RDMA version 1 are: The two most immediate shortcomings of RPC-over-RDMA version 1 are:
o Setting up an RDMA data transfer (via RDMA Read or Write) can be o Setting up an RDMA data transfer (via RDMA Read or Write) can be
costly. The small default size of messages transmitted using RDMA costly. The small default size of messages transmitted using RDMA
Send forces the use of RDMA Read or Write operations even for Send forces the use of RDMA Read or Write operations even for
relatively small messages and data payloads. relatively small messages and data payloads.
The original specification of RPC-over-RDMA version 1 provided an The original specification of RPC-over-RDMA version 1 provided an
skipping to change at page 3, line 11 skipping to change at page 3, line 18
RPC-over-RDMA version 1 has no means of extending its XDR definition RPC-over-RDMA version 1 has no means of extending its XDR definition
in such a way that interoperability with existing implementations is in such a way that interoperability with existing implementations is
preserved. As a result, an out-of-band mechanism is needed to help preserved. As a result, an out-of-band mechanism is needed to help
relieve these constraints for existing RPC-over-RDMA version 1 relieve these constraints for existing RPC-over-RDMA version 1
implementations. implementations.
This document specifies a simple, non-XDR-based message format This document specifies a simple, non-XDR-based message format
designed to be passed between RPC-over-RDMA version 1 peers at the designed to be passed between RPC-over-RDMA version 1 peers at the
time each RDMA transport connection is first established. The time each RDMA transport connection is first established. The
purpose of such a message exchange is to enable the connecting peers mechanism assumes that the underlying RDMA transport has a private
to indicate support for transport properties that are not defined in data field that is passed between peers at connection time, such as
the base RPC-over-RDMA version 1 protocol defined in [RFC8166]. is present in the iWARP protocol (described in Section 7.1 of
[RFC5044]) or the InfiniBand Connection Manager [IBA].
To enable current RPC-over-RDMA version 1 implementations to
interoperate with implementations that support the private message
format described in this document, implementation of the private data
message is OPTIONAL. When the private data message has been
successfully exchanged, peers may choose to perform extended RDMA
semantics. However, the private message format does not alter the
XDR definition specified in [RFC8166].
The message format is intended to be further extensible within the The message format is intended to be further extensible within the
normal scope of such IETF work (see Section 5 for further details). normal scope of such IETF work (see Section 5 for further details).
Section 6 of the current document defines an IANA registry for this Section 6 of the current document defines an IANA registry for this
purpose. In addition, interoperation between implementations of RPC- purpose. In addition, interoperation between implementations of RPC-
over-RDMA version 1 that present this message format to peers and over-RDMA version 1 that present this message format to peers and
those that do not recognize this message format is guaranteed. those that do not recognize this message format is guaranteed.
2. Requirements Language 2. Requirements Language
skipping to change at page 5, line 17 skipping to change at page 5, line 33
There are some NFS/RDMA client implementations whose STags are always There are some NFS/RDMA client implementations whose STags are always
safe to invalidate remotely. For such clients, indicating to the safe to invalidate remotely. For such clients, indicating to the
responder that remote invalidation is always safe can allow such responder that remote invalidation is always safe can allow such
invalidation without the need for additional protocol to be defined. invalidation without the need for additional protocol to be defined.
4. Private Data Message Format 4. Private Data Message Format
With an InfiniBand lower layer, for example, RDMA connection setup With an InfiniBand lower layer, for example, RDMA connection setup
uses a Connection Manager when establishing a Reliable Connection uses a Connection Manager when establishing a Reliable Connection
[IBARCH]. When an RPC-over-RDMA version 1 transport connection is [IBA]. When an RPC-over-RDMA version 1 transport connection is
established, the client (which actively establishes connections) and established, the client (which actively establishes connections) and
the server (which passively accepts connections) populate the CM the server (which passively accepts connections) populate the CM
Private Data field exchanged as part of CM connection establishment. Private Data field exchanged as part of CM connection establishment.
The transport properties exchanged via this mechanism are fixed for The transport properties exchanged via this mechanism are fixed for
the life of the connection. Each new connection presents an the life of the connection. Each new connection presents an
opportunity for a fresh exchange. An implementation of the extension opportunity for a fresh exchange. An implementation of the extension
described in this document MUST be prepared for the settings to described in this document MUST be prepared for the settings to
change upon a reconnection. change upon a reconnection.
skipping to change at page 5, line 49 skipping to change at page 6, line 18
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Format Identifier | | Format Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version | Flags | Send Size | Receive Size | | Version | Flags | Send Size | Receive Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Format Identifier: This field contains a fixed 32-bit value that Format Identifier: This field contains a fixed 32-bit value that
identifies the content of the Private Data field as an RPC-over- identifies the content of the Private Data field as an RPC-over-
RDMA version 1 CM Private Data message. The value of this field RDMA version 1 CM Private Data message. In RPC-over-RDMA version
is always 0xf6ab0e18, in network byte order. The use of this 1 Private Data, the value of this field is always 0xf6ab0e18, in
field is further expanded upon in Section 4.1. network byte order. The use of this field is further expanded
upon in Section 4.1.
Version: This 8-bit field contains a message format version number. Version: This 8-bit field contains a message format version number.
The value "1" in this field indicates that exactly eight octets The value "1" in this field indicates that exactly eight octets
are present, that they appear in the order described in this are present, that they appear in the order described in this
section, and that each has the meaning defined in this section. section, and that each has the meaning defined in this section.
Further considerations about the use of this field are discussed Further considerations about the use of this field are discussed
in Section 5. in Section 5.
Flags: This 8-bit field contains bit flags that indicate the support Flags: This 8-bit field contains bit flags that indicate the support
status of optional features, such as remote invalidation. The status of optional features, such as remote invalidation. The
skipping to change at page 6, line 29 skipping to change at page 6, line 47
described in Section 5.2. described in Section 5.2.
Receive Size: This 8-bit field contains an encoded value Receive Size: This 8-bit field contains an encoded value
corresponding to the maximum number of bytes this peer is prepared corresponding to the maximum number of bytes this peer is prepared
to receive with a single RDMA Receive on this connection. The to receive with a single RDMA Receive on this connection. The
value is encoded as described in Section 5.2. value is encoded as described in Section 5.2.
4.1. Interoperability Considerations 4.1. Interoperability Considerations
The extension described in this document is designed to allow RPC- The extension described in this document is designed to allow RPC-
over-RDMA version implementations that use this extension to over-RDMA version implementations that use CM Private Data to
interoperate fully with RPC-over-RDMA version 1 implementations that interoperate fully with RPC-over-RDMA version 1 implementations that
do not exchange this information. Realizing this goal requires that do not exchange this information. Implementations that use this
implementations of this extension follow the practices described in extension must also interoperate fully with RDMA implementations that
the rest of this section. use CM Private Data for other purposes. Realizing these goals
require that implementations of this extension follow the practices
described in the rest of this section.
RPC-over-RDMA version 1 implementations that support the extension 4.1.1. Amongst RPC-over-RDMA Version 1 Implementations
described in this document are intended to interoperate fully with
RPC-over-RDMA version 1 implementations that do not recognize the
exchange of CM Private Data. When a peer does not receive a CM
Private Data message which conforms to Section 4, it needs to act as
if the remote peer supports only the default RPC-over-RDMA version 1
settings, as defined in [RFC8166]. In other words, the peer is to
behave as if a Private Data message was received in which bit 8 of
the Flags field is zero, and both Size fields contain the value zero.
The Format Identifier field is provided in order to distinguish RPC- When a peer does not receive a CM Private Data message which conforms
over-RDMA version 1 Private Data from private data inserted by layers to Section 4, it needs to act as if the remote peer supports only the
below or above RPC-over RDMA version 1. During connection default RPC-over-RDMA version 1 settings, as defined in [RFC8166].
establishment, RPC-over-RDMA version 1 implementations check for this In other words, the peer MUST behave as if a Private Data message was
protocol number before decoding subsequent fields. If this protocol received in which bit 15 of the Flags field is zero, and both Size
number is not present as the first 4 octets, an RPC-over-RDMA fields contain the value zero.
receiver needs to ignore the CM-Private Data (ie., behave as if no
RPC-over-RDMA version 1 Private Data has been provided). 4.1.2. Amongst Implementations of Other Upper-Layer Protocols
The Format Identifier field in the message format defined in this
document is provided to enable implementations to distinguish RPC-
over-RDMA version 1 Private Data from application-specific private
data inserted by applications other than RPC-over RDMA version 1.
Examples of other applications that make use of CM Private Data
include iWARP, via the MPA enhancement described in [RFC6581], and
iSCSI extensions for RDMA (iSER), as defined in [RFC7145].
During connection establishment, an implementation of the extension
described in this document checks the Format Identifier field before
decoding subsequent fields. If the RPC-over-RDMA version 1 CM
Private Data Format Identifier is not present as the first 4 octets,
an RPC-over-RDMA version 1 receiver MUST ignore the CM Private Data,
behaving as if no RPC-over-RDMA version 1 Private Data has been
provided (see above).
5. Updating the Message Format 5. Updating the Message Format
Although the message format described in this document provides the Although the message format described in this document provides the
ability for the client and server to exchange particular information ability for the client and server to exchange particular information
about the local RPC-over-RDMA implementation, it is possible that about the local RPC-over-RDMA implementation, it is possible that
there will be a future need to exchange additional properties. This there will be a future need to exchange additional properties. This
would make it necessary to extend or otherwise modify the format would make it necessary to extend or otherwise modify the format
described in this document. described in this document.
skipping to change at page 7, line 51 skipping to change at page 8, line 32
value "1", the bits in the Flags field are to be set as follows: value "1", the bits in the Flags field are to be set as follows:
Bit 15: When both connection peers have set this flag in their CM Bit 15: When both connection peers have set this flag in their CM
Private Data, the responder MAY use RDMA Send With Invalidate when Private Data, the responder MAY use RDMA Send With Invalidate when
transmitting RPC Replies. Each RDMA Send With Invalidate MUST transmitting RPC Replies. Each RDMA Send With Invalidate MUST
invalidate an STag associated only with the XID in the rdma_xid invalidate an STag associated only with the XID in the rdma_xid
field of the RPC-over-RDMA Transport Header it carries. field of the RPC-over-RDMA Transport Header it carries.
When either peer on a connection clears this flag, the responder When either peer on a connection clears this flag, the responder
MUST use only RDMA Send when transmitting RPC Replies. MUST use only RDMA Send when transmitting RPC Replies.
Bits 14 - 8: These bits are reserved and are always zero. Bits 14 - 8: These bits are reserved and are always zero when the
Version field contains 1.
5.2. Inline Threshold Values 5.2. Inline Threshold Values
Inline threshold sizes from 1KB to 256KB can be represented in the Inline threshold sizes from 1KB to 256KB can be represented in the
Send Size and Receive Size fields. A sender computes the encoded Send Size and Receive Size fields. A sender computes the encoded
value by dividing the actual value by 1024 and subtracting one from value by dividing the actual value by 1024 and subtracting one from
the result. A receiver decodes this value by performing a the result. A receiver decodes this value by performing a
complementary set of operations. complementary set of operations.
The client uses the smaller of its own send size and the server's The client uses the smaller of its own send size and the server's
reported receive size as the client-to-server inline threshold. The reported receive size as the client-to-server inline threshold. The
server uses the smaller of its own send size and the clients's server uses the smaller of its own send size and the clients's
reported receive size as the server-to-client inline threshold. reported receive size as the server-to-client inline threshold.
6. IANA Considerations 6. IANA Considerations
In accordance with [RFC8126], the author requests that IANA create a In accordance with [RFC8126], the author requests that IANA create a
new registry in the "Remote Direct Data Placement" Protocol Category new registry in the "Remote Direct Data Placement" Protocol Category
Group. The new registry is to be called the "RDMA-CM Private Data Group. The new registry is to be called the "RDMA-CM Private Data
Identifier Registry". This is a registry of 32-bit numbers that Identifier Registry". This is a registry of 32-bit numbers that
identify the Upper Layer protocol associated with data that appears identify the upper-layer protocol associated with data that appears
in the RDMA-CM Private Data area. in the application-specific RDMA-CM Private Data area. The fields in
this registry include: Format Identifier, Description, and Reference.
The information that must be provided to add an entry to this
registry will be an IESG-approved Standards Track specification
defining the semantics and interoperability requirements of the
proposed new value and the fields to be recorded in the registry.
The fields in this registry include: Field Identifier, Format
Description, and Reference.
The initial contents of this registry are a single entry: The initial contents of this registry are a single entry:
+-----------------+-------------------------------------+-----------+ +------------------+------------------------------------+-----------+
| Field | Format Description | Reference | | Format | Format Description | Reference |
| Identifier | | | | Identifier | | |
+-----------------+-------------------------------------+-----------+ +------------------+------------------------------------+-----------+
| 0xf6ab0e18 | RPC-over-RDMA version 1 CM Private | [RFC-TBD] | | 0xf6ab0e18 | RPC-over-RDMA version 1 CM Private | [RFC-TBD] |
| | Data | | | | Data | |
+-----------------+-------------------------------------+-----------+ +------------------+------------------------------------+-----------+
Table 1: RDMA-CM Private Data Identifier Registry Table 1: RDMA-CM Private Data Identifier Registry
The Expert Review policy, as defined in Section 4.5 of [RFC8126] is IANA is to assign subsequent new entries in this registry using the
to be used to handle requests to add new entries to the "File Expert Review policy as defined in Section 4.5 of [RFC8126].
Provenance Information Registry". New protocol numbers can be
assigned at random as long as they do not conflict with existing 6.1. Guidance for Designated Experts
entries in this registry.
The Designated Expert (DE), appointed by the IESG, should ascertain
the existence of suitable documentation that defines the semantics
and format of the private data, and verify that the document is
permanently and publicly available. Documentation produced outside
the IETF must not conflict with work that is active or already
published within the IETF.
The new Reference field should contain a reference to that
documentation. The DE can assign new Format Identifiers at random as
long as they do not conflict with existing entries in this registry.
The Description field should contain the name of a distinct upper-
layer RDMA consumer that will use the private data.
The DE will post the request to the nfsv4 WG mailing list (or a
successor to that list, if such a list exists), for comment and
review. The DE will approve or deny the request and publish notice
of the decision within 30 days.
7. Security Considerations 7. Security Considerations
The private data extension specified in this document inherits the The Private Data extension specified in this document inherits the
security considerations of the protocols it extends; e.g., the MPA security considerations of the protocols it extends, which in this
protocol, as specified in [RFC5044] and extended in [RFC6581]. case is defined in [RFC8166].
Additional relevant analysis of RDMA security appears in the Security
Considerations section of [RFC5042]. The integrity of CM Private Data and the authenticity of it's source
is ensured by the use of the Reliable connected (RC) Queue Pair (QP)
type, required by RPC-over-RDMA version 1. Any attempts to interfere
with or hijack an RC connection will result in the connection being
immediately terminated. Additional relevant analysis of RDMA
security appears in the Security Considerations section of [RFC5042].
Improperly setting one of the fields in the private message payload
will result in a greatly increased risk of disconnection (i.e., self-
imposed Denial of Service). There is no increased risk of exposing
upper-layer data inappropriately.
8. References 8. References
8.1. Normative References 8.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>. <https://www.rfc-editor.org/info/rfc2119>.
skipping to change at page 9, line 48 skipping to change at page 10, line 49
Memory Access Transport for Remote Procedure Call Version Memory Access Transport for Remote Procedure Call Version
1", RFC 8166, DOI 10.17487/RFC8166, June 2017, 1", RFC 8166, DOI 10.17487/RFC8166, June 2017,
<https://www.rfc-editor.org/info/rfc8166>. <https://www.rfc-editor.org/info/rfc8166>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>. May 2017, <https://www.rfc-editor.org/info/rfc8174>.
8.2. Informative References 8.2. Informative References
[IBARCH] InfiniBand Trade Association, "InfiniBand Architecture [IBA] InfiniBand Trade Association, "InfiniBand Architecture
Specification Volume 1", Release 1.3, March 2015, Specification Volume 1", Release 1.3, March 2015.
<http://www.infinibandta.org/content/
pages.php?pg=technology_download>. Available from https://www.infinibandta.org/
[RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS
Version 3 Protocol Specification", RFC 1813, Version 3 Protocol Specification", RFC 1813,
DOI 10.17487/RFC1813, June 1995, DOI 10.17487/RFC1813, June 1995,
<https://www.rfc-editor.org/info/rfc1813>. <https://www.rfc-editor.org/info/rfc1813>.
[RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J. [RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
Carrier, "Marker PDU Aligned Framing for TCP Carrier, "Marker PDU Aligned Framing for TCP
Specification", RFC 5044, DOI 10.17487/RFC5044, October Specification", RFC 5044, DOI 10.17487/RFC5044, October
2007, <https://www.rfc-editor.org/info/rfc5044>. 2007, <https://www.rfc-editor.org/info/rfc5044>.
skipping to change at page 10, line 29 skipping to change at page 11, line 31
[RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access [RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access
Transport for Remote Procedure Call", RFC 5666, Transport for Remote Procedure Call", RFC 5666,
DOI 10.17487/RFC5666, January 2010, DOI 10.17487/RFC5666, January 2010,
<https://www.rfc-editor.org/info/rfc5666>. <https://www.rfc-editor.org/info/rfc5666>.
[RFC6581] Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S. [RFC6581] Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S.
Wise, "Enhanced Remote Direct Memory Access (RDMA) Wise, "Enhanced Remote Direct Memory Access (RDMA)
Connection Establishment", RFC 6581, DOI 10.17487/RFC6581, Connection Establishment", RFC 6581, DOI 10.17487/RFC6581,
April 2012, <https://www.rfc-editor.org/info/rfc6581>. April 2012, <https://www.rfc-editor.org/info/rfc6581>.
[RFC7145] Ko, M. and A. Nezhinsky, "Internet Small Computer System
Interface (iSCSI) Extensions for the Remote Direct Memory
Access (RDMA) Specification", RFC 7145,
DOI 10.17487/RFC7145, April 2014,
<https://www.rfc-editor.org/info/rfc7145>.
[RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System [RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System
(NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530, (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530,
March 2015, <https://www.rfc-editor.org/info/rfc7530>. March 2015, <https://www.rfc-editor.org/info/rfc7530>.
[RFC7861] Adamson, A. and N. Williams, "Remote Procedure Call (RPC) [RFC7861] Adamson, A. and N. Williams, "Remote Procedure Call (RPC)
Security Version 3", RFC 7861, DOI 10.17487/RFC7861, Security Version 3", RFC 7861, DOI 10.17487/RFC7861,
November 2016, <https://www.rfc-editor.org/info/rfc7861>. November 2016, <https://www.rfc-editor.org/info/rfc7861>.
Acknowledgments Acknowledgments
 End of changes. 23 change blocks. 
75 lines changed or deleted 126 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/