draft-ietf-nfsv4-layout-types-13.txt   rfc8434.txt 
NFSv4 T. Haynes Internet Engineering Task Force (IETF) T. Haynes
Internet-Draft Hammerspace Request for Comments: 8434 Hammerspace
Updates: 5661 (if approved) April 25, 2018 Updates: 5661 August 2018
Intended status: Standards Track Category: Standards Track
Expires: October 27, 2018 ISSN: 2070-1721
Requirements for pNFS Layout Types Requirements for Parallel NFS (pNFS) Layout Types
draft-ietf-nfsv4-layout-types-13.txt
Abstract Abstract
This document defines the requirements which individual pNFS layout This document defines the requirements that individual Parallel NFS
types need to meet in order to work within the parallel NFS (pNFS) (pNFS) layout types need to meet in order to work within the pNFS
framework as defined in RFC5661. In so doing, it aims to clearly framework as defined in RFC 5661. In so doing, this document aims to
distinguish between requirements for pNFS as a whole and those clearly distinguish between requirements for pNFS as a whole and
specifically directed to the pNFS File Layout. The lack of a clear those specifically directed to the pNFS file layout. The lack of a
separation between the two set of requirements has been troublesome clear separation between the two sets of requirements has been
for those specifying and evaluating new Layout Types. In this troublesome for those specifying and evaluating new layout types. In
regard, this document updates RFC5661. this regard, this document updates RFC 5661.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This is an Internet Standards Track document.
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Further information on
Internet Standards is available in Section 2 of RFC 7841.
This Internet-Draft will expire on October 27, 2018. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8434.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction ....................................................2
2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Definitions .....................................................3
2.1. Use of the Terms "Data Server" and "Storage Device" . . . 5 2.1. Use of the Terms "Data Server" and "Storage Device" ........5
2.2. Requirements Language . . . . . . . . . . . . . . . . . . 6 2.2. Requirements Language ......................................6
3. The Control Protocol . . . . . . . . . . . . . . . . . . . . 6 3. The Control Protocol ............................................6
3.1. Control Protocol Requirements . . . . . . . . . . . . . . 8 3.1. Control Protocol Requirements ..............................8
3.2. Previously Undocumented Protocol Requirements . . . . . . 10 3.2. Previously Undocumented Protocol Requirements ..............9
3.3. Editorial Requirements . . . . . . . . . . . . . . . . . 10 3.3. Editorial Requirements ....................................10
4. Specifications of Original Layout Types . . . . . . . . . . . 11 4. Specifications of Original Layout Types ........................11
4.1. File Layout Type . . . . . . . . . . . . . . . . . . . . 11 4.1. File Layout Type ..........................................11
4.2. Block Layout Type . . . . . . . . . . . . . . . . . . . . 13 4.2. Block Layout Type .........................................12
4.3. Object Layout Type . . . . . . . . . . . . . . . . . . . 14 4.3. Object Layout Type ........................................13
5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5. Summary ........................................................14
6. Security Considerations . . . . . . . . . . . . . . . . . . . 15 6. Security Considerations ........................................15
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 7. IANA Considerations ............................................15
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 16 8. References .....................................................16
8.1. Normative References . . . . . . . . . . . . . . . . . . 16 8.1. Normative References ......................................16
8.2. Informative References . . . . . . . . . . . . . . . . . 16 8.2. Informative References ....................................16
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 16 Acknowledgments ...................................................17
Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 17 Author's Address ..................................................17
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 17
1. Introduction 1. Introduction
The concept of layout type has a central role in the definition and The concept of "layout type" has a central role in the definition and
implementation of Parallel Network File System (pNFS) (see implementation of Parallel NFS (pNFS) (see [RFC5661]). Clients and
[RFC5661]). Clients and servers implementing different layout types servers implementing different layout types behave differently in
behave differently in many ways while conforming to the overall pNFS many ways while conforming to the overall pNFS framework defined in
framework defined in [RFC5661] and this document. Layout types may [RFC5661] and this document. Layout types may differ as to:
differ as to:
o The method used to do I/O operations directed to data storage o The method used to do I/O operations directed to data storage
devices. devices.
o The requirements for communication between the metadata server o The requirements for communication between the metadata server
(MDS) and the storage devices. (MDS) and the storage devices.
o The means used to ensure that I/O requests are only processed when o The means used to ensure that I/O requests are only processed when
the client holds an appropriate layout. the client holds an appropriate layout.
o The format and interpretation of nominally opaque data fields in o The format and interpretation of nominally opaque data fields in
pNFS-related NFSv4.x data structures. pNFS-related NFSv4.x data structures.
Each layout type will define the needed details for its usage in the Each layout type will define the needed details for its usage in the
specifciation for that layout type; layout type specifications are specification for that layout type; layout type specifications are
always standards-track RFCs. Except for the files layout type, which always Standards Track RFCs. Except for the file layout type defined
was defined in Section 13 of [RFC5661], existing layout types are in Section 13 of [RFC5661], existing layout types are defined in
defined in their own standards-track documents and it is anticipated their own Standards Track documents, and it is anticipated that new
that new layout types will be defined in similar documents. layout types will be defined in similar documents.
The file layout type was defined in the Network File System (NFS) The file layout type was defined in the Network File System (NFS)
version 4.1 protocol specification [RFC5661]. The block layout type version 4.1 protocol specification [RFC5661]. The block layout type
was defined in [RFC5663] while the object layout type was defined in was defined in [RFC5663], and the object layout type was defined in
[RFC5664]. Subsequently, the SCSI layout type was defined in [RFC5664]. Subsequently, the Small Computer System Interface (SCSI)
[RFC8154]. layout type was defined in [RFC8154].
Some implementers have interpreted the text in Sections 12 ("Parallel Some implementers have interpreted the text in Sections 12 ("Parallel
NFS (pNFS)") and 13 ("NFSv4.1 as a Storage Protocol in pNFS: the File NFS (pNFS)") and 13 ("NFSv4.1 as a Storage Protocol in pNFS: the File
Layout Type") of [RFC5661] as both being applying only to the file Layout Type") of [RFC5661] as applying only to the file layout type.
layout type. Because Section 13 was not covered in a separate Because Section 13 was not covered in a separate Standards Track
standards-track document such as those for both the block and object document such as those for both the block and object layout types,
layout types, there had been some confusion as to the there was some confusion as to the responsibilities of both the
responsibilities of both the metadata server and the data servers metadata server and the data servers (DSs) that were laid out in
(DS) which were laid out in Section 12. Section 12.
As a consequence, authors of new specifications (see [FlexFiles] and As a consequence, authors of new specifications (see [RFC8435] and
[Lustre]) may struggle to meet the requirements to be a pNFS layout [Lustre]) may struggle to meet the requirements to be a pNFS layout
type. This document gathers the requirements from all of the type. This document gathers the requirements from all of the
original layout type standard documents and then specifies the original Standards Track documents regarding layout type and then
requirements placed on all layout types independent of the particular specifies the requirements placed on all layout types independent of
type chosen. the particular type chosen.
2. Definitions 2. Definitions
control communication requirement: is the specification for control communication requirement: the specification for information
information on layouts, stateids, file metadata, and file data on layouts, stateids, file metadata, and file data that must be
which must be communicated between the metadata server and the communicated between the metadata server and the storage devices.
storage devices. There is a separate set of requirements for each There is a separate set of requirements for each layout type.
layout type.
control protocol: is the particular mechanism that an implementation control protocol: the particular mechanism that an implementation of
of a layout type would use to meet the control communication a layout type would use to meet the control communication
requirement for that layout type. This need not be a protocol as requirement for that layout type. This need not be a protocol as
normally understood. In some cases the same protocol may be used normally understood. In some cases, the same protocol may be used
as a control protocol and storage protocol. as both a control protocol and storage protocol.
storage protocol: is the protocol used by clients to do I/O storage protocol: the protocol used by clients to do I/O operations
operations to the storage device. Each layout type specifies the to the storage device. Each layout type specifies the set of
set of storage protocols. storage protocols.
loose coupling: is when the control protocol is a storage protocol. loose coupling: when the control protocol is a storage protocol.
tight coupling: is an arrangement in which the control protocol is tight coupling: an arrangement in which the control protocol is one
one designed specifically for that purpose. It may be either a designed specifically for control communication. It may be either
proprietary protocol, adapted specifically to a a particular a proprietary protocol adapted specifically to a particular
metadata server, or one based on a standards-track document. metadata server or a protocol based on a Standards Track document.
(file) data: is that part of the file system object which contains (file) data: that part of the file system object that contains the
the data to be read or written. It is the contents of the object data to be read or written. It is the contents of the object
rather than the attributes of the object. rather than the attributes of the object.
data server (DS): is a pNFS server which provides the file's data data server (DS): a pNFS server that provides the file's data when
when the file system object is accessed over a file-based the file system object is accessed over a file-based protocol.
protocol. Note that this usage differs from that in [RFC5661] Note that this usage differs from that in [RFC5661], which applies
which applies the term in some cases even when other sorts of the term in some cases even when other sorts of protocols are
protocols are being used. Depending on the layout, there might be being used. Depending on the layout, there might be one or more
one or more data servers over which the data is striped. While data servers over which the data is striped. While the metadata
the metadata server is strictly accessed over the NFSv4.1 server is strictly accessed over the NFSv4.1 protocol, the data
protocol, the data server could be accessed via any file access server could be accessed via any file access protocol that meets
protocol that meets the pNFS requirements. the pNFS requirements.
See Section 2.1 for a comparison of this term and "data storage See Section 2.1 for a comparison of this term and "storage
device". device".
storage device: is the target to which clients may direct I/O storage device: the target to which clients may direct I/O requests
requests when they hold an appropriate layout. Note that each when they hold an appropriate layout. Note that each data server
data server is a storage device but that some storage device are is a storage device but that some storage device are not data
not data servers. See Section 2.1 for further discussion. servers. See Section 2.1 for further discussion.
fencing: is the process by which the metadata server prevents the fencing: the process by which the metadata server prevents the
storage devices from processing I/O from a specific client to a storage devices from processing I/O from a specific client to a
specific file. specific file.
layout: is the information a client uses to access file data on a layout: the information a client uses to access file data on a
storage device. This information will include specification of storage device. This information includes specification of the
the protocol (layout type) and the identity of the storage devices protocol (layout type) and the identity of the storage devices to
to be used. be used.
The bulk of the contents of the layout are defined in [RFC5661] The bulk of the contents of the layout are defined in [RFC5661] as
as nominally opaque, but individual layout types are responsible nominally opaque, but individual layout types are responsible for
for specifying the format of the layout data. specifying the format of the layout data.
layout iomode: is a grant of either read or read/write I/O to the layout iomode: a grant of either read-only or read/write I/O to the
client. client.
layout stateid: is a 128-bit quantity returned by a server that layout stateid: a 128-bit quantity returned by a server that
uniquely defines the layout state provided by the server for a uniquely defines the layout state provided by the server for a
specific layout that describes a layout type and file (see specific layout that describes a layout type and file (see
Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 describes Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 of
differences in handling between layout stateids and other stateid [RFC5661] describes differences in handling between layout
types. stateids and other stateid types.
layout type: is a specification of both the storage protocol used to layout type: a specification of both the storage protocol used to
access the data and the aggregation scheme used to lay out the access the data and the aggregation scheme used to lay out the
file data on the underlying storage devices. file data on the underlying storage devices.
recalling a layout: is a graceful recall, via a callback, of a recalling a layout: a graceful recall, via a callback, of a specific
specific layout by the metadata server to the client. Graceful layout by the metadata server to the client. Graceful here means
here means that the client would have the opportunity to flush any that the client would have the opportunity to flush any WRITEs,
writes, etc., before returning the layout to the metadata server. etc., before returning the layout to the metadata server.
revoking a layout: is an invalidation of a specific layout by the revoking a layout: an invalidation of a specific layout by the
metadata server. Once revocation occurs, the metadata server will metadata server. Once revocation occurs, the metadata server will
not accept as valid any reference to the revoked layout and a not accept as valid any reference to the revoked layout, and a
storage device will not accept any client access based on the storage device will not accept any client access based on the
layout. layout.
(file) metadata: is that part of the file system object that (file) metadata: the part of the file system object that contains
contains various descriptive data relevant to the file object, as various descriptive data relevant to the file object, as opposed
opposed to the file data itself. This could include the time of to the file data itself. This could include the time of last
last modification, access time, end-of-file (EOF) position, etc. modification, access time, EOF position, etc.
metadata server (MDS): is the pNFS server which provides metadata metadata server (MDS): the pNFS server that provides metadata
information for a file system object. It also is responsible for information for a file system object. It is also responsible for
generating, recalling, and revoking layouts for file system generating, recalling, and revoking layouts for file system
objects, for performing directory operations, and for performing I objects, for performing directory operations, and for performing
/O operations to regular files when the clients direct these to I/O operations to regular files when the clients direct these to
the metadata server itself. the metadata server itself.
stateid: is a 128-bit quantity returned by a server that uniquely stateid: a 128-bit quantity returned by a server that uniquely
defines the set of locking-related state provided by the server. defines the set of locking-related state provided by the server.
Stateids may designate state related to open files, to byte-range Stateids may designate state related to open files, byte-range
locks, to delegations, or to layouts. locks, delegations, or layouts.
2.1. Use of the Terms "Data Server" and "Storage Device" 2.1. Use of the Terms "Data Server" and "Storage Device"
In [RFC5661], these two terms of "Data Server" and "Storage Device" In [RFC5661], the terms "data server" and "storage device" are used
are used somewhat inconsistently: somewhat inconsistently:
o In chapter 12, where pNFS in general is discussed, the term o In Section 12, where pNFS in general is discussed, the term
"storage device" is used. "storage device" is used.
o In chapter 13, where the file layout type is discussed, the term o In Section 13, where the file layout type is discussed, the term
"data server" is used. "data server" is used.
o In other chapters, the term "data server" is used, even in o In other sections, the term "data server" is used, even in
contexts where the storage access type is not NFSv4.1 or any other contexts where the storage access type is not NFSv4.1 or any other
file access protocol. file access protocol.
As this document deals with pNFS in general, it uses the more generic As this document deals with pNFS in general, it uses the more generic
term "storage device" in preference to "data server". The term "data term "storage device" in preference to "data server". The term "data
server" is used only in contexts in which a file server is used as a server" is used only in contexts in which a file server is used as a
storage device. Note that every data server is a storage device but storage device. Note that every data server is a storage device, but
storage devices which use protocols which are not file access storage devices that use protocols that are not file access protocols
protocols (such as NFS) are not data servers. (such as NFS) are not data servers.
Since a given storage device may support multiple layout types, a Since a given storage device may support multiple layout types, a
given device can potentially act as a data server for some set of given device can potentially act as a data server for some set of
storage protocols while simultaneously acting as a storage device for storage protocols while simultaneously acting as a storage device for
others. others.
2.2. Requirements Language 2.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in
14 [RFC2119] [RFC8174] when, and only when, they appear in all BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
This document differs from most standards-track documents in that it This document differs from most Standards Track documents in that it
specifies requirements for those defining future layout types rather specifies requirements for those defining future layout types rather
than defining the requirements for implementations directly. This than defining the requirements for implementations directly. This
document makes clear whether: document makes clear whether:
(1) any particular requirement applies to implementations. (1) any particular requirement applies to implementations.
(2) any particular requirement applies to those defining layout (2) any particular requirement applies to those defining layout
types. types.
(3) the requirement is a general requirement which implementations (3) the requirement is a general requirement that implementations
need to conform to, with the specific means left to layout type need to conform to, with the specific means left to layout type
definitions type to specify. definitions type to specify.
3. The Control Protocol 3. The Control Protocol
A layout type has to meet the requirements that apply to the A layout type has to meet the requirements that apply to the
interaction between the metadata server and the storage device such interaction between the metadata server and the storage device such
that they present to the client a consistent view of stored data and that they present to the client a consistent view of stored data and
lock state (Section 12.2.6 of [RFC5661]). Particular implementations locking state (Section 12.2.6 of [RFC5661]). Particular
may satisfy these requirements in any manner they choose and the implementations may satisfy these requirements in any manner they
mechanism chosen need not be described as a protocol. Specifications choose, and the mechanism chosen need not be described as a protocol.
defining layout types need to clearly show how implementations can Specifications defining layout types need to clearly show how
meet the requirements discussed below, especially with respect to implementations can meet the requirements discussed below, especially
those that have security implications. In addition, such with respect to those that have security implications. In addition,
specifications may find it necessary to impose requirements on such specifications may find it necessary to impose requirements on
implementations of the layout type to ensure appropriate implementations of the layout type to ensure appropriate
interoperability. interoperability.
In some cases, there may be no control protocol other than the In some cases, there may be no control protocol other than the
storage protocol. This is often described as using a "loose storage protocol. This is often described as using a "loosely
coupling" model. In such cases, the assumption is that the metadata coupled" model. In such cases, the assumption is that the metadata
server, storage devices, and client may be changed independently and server, storage devices, and client may be changed independently and
that the implementation requirements in the layout type specification that the implementation requirements in the layout type specification
need to ensure this degree of interoperability. This model is used need to ensure this degree of interoperability. This model is used
in the block and object layout type specification. in the block and object layout type specification.
In other cases, it is assumed that there will be a purpose-built In other cases, it is assumed that there will be a purpose-built
control protocol which may be different for different implementations control protocol that may be different for different implementations
of the metadata server and data server. The assumption here is that of the metadata server and data server. The assumption here is that
the metadata server and data servers are designed and implemented as the metadata server and data servers are designed and implemented as
a unit and interoperability needs to be assured between clients and a unit and interoperability needs to be assured between clients and
metadata-data server pairs, developed independently. This is the metadata-data server pairs, developed independently. This is the
model used for the files layout. model used for the file layout.
Another possibility is for the definition of a control protocol to be Another possibility is for the definition of a control protocol to be
specified in a standards-track document. There are two subcases to specified in a Standards Track document. There are two subcases to
consider: consider:
o A new layout type includes a definition of a particular control o A new layout type includes a definition of a particular control
protocol whose use is obligatory for metadata servers and storage protocol whose use is obligatory for metadata servers and storage
devices implementing the layout type. In this case the devices implementing the layout type. In this case, the
interoperability model is similar to the first case above and the interoperability model is similar to the first case above, and the
defining document should assure interoperability among metadata defining document should assure interoperability among metadata
servers, storage devices, and clients developed independently. servers, storage devices, and clients developed independently.
o A control protocol is defined in a standards-track document which o A control protocol is defined in a Standards Track document that
meets the control protocol requirements for one of the existing meets the control protocol requirements for one of the existing
layout types. In this case, the new document's job is to assure layout types. In this case, the new document's job is to assure
interoperability between metadata servers and storage devices interoperability between metadata servers and storage devices
developed separately. The existing definition document for the developed separately. The existing definition document for the
selected layout type retains the function of assuring selected layout type retains the function of assuring
interoperability between clients and a given collection of interoperability between clients and a given collection of
metadata servers and storage devices. In this context, metadata servers and storage devices. In this context,
implementations that implement the new protocol are treated in the implementations that implement the new protocol are treated in the
same way as those that use an internal control protocol or a same way as those that use an internal control protocol or a
functional equivalent. functional equivalent.
An example of this last case is the SCSI layout type [RFC8154], which An example of this last case is the SCSI layout type [RFC8154], which
extends the block layout type. The block layout type had a extends the block layout type. The block layout type had a
requirement for fencing of clients, but did not present a way for the requirement for fencing of clients but did not present a way for the
control protocol (in this case the SCSI storage protocol) to fence control protocol (in this case, the SCSI storage protocol) to fence
the client. The SCSI layout type remedies that in [RFC8154] and in the client. The SCSI layout type remedies that in [RFC8154] and, in
effect has a tightly coupled model. effect, has a tightly coupled model.
3.1. Control Protocol Requirements 3.1. Control Protocol Requirements
The requirements of interactions between the metadata server and the The requirements of interactions between the metadata server and the
storage devices are: storage devices are:
(1) The metadata server MUST be able to service the client's I/O (1) The metadata server MUST be able to service the client's I/O
requests if the client decides to make such requests to the requests if the client decides to make such requests to the
metadata server instead of to the storage device. The metadata metadata server instead of to the storage device. The metadata
server must be able to retrieve the data from the constituent server must be able to retrieve the data from the constituent
skipping to change at page 8, line 34 skipping to change at page 8, line 30
requests to the metadata server. requests to the metadata server.
(2) The metadata server MUST be able to restrict access to a file on (2) The metadata server MUST be able to restrict access to a file on
the storage devices when it revokes a layout. The metadata the storage devices when it revokes a layout. The metadata
server typically would revoke a layout whenever a client fails server typically would revoke a layout whenever a client fails
to respond to a recall or a client's lease is expired due to to respond to a recall or a client's lease is expired due to
non-renewal. It might also revoke the layout as a means of non-renewal. It might also revoke the layout as a means of
enforcing a change in locking state or access permissions that enforcing a change in locking state or access permissions that
the storage device cannot directly enforce. the storage device cannot directly enforce.
Effective revocation may require client co-operation in using a Effective revocation may require client cooperation in using a
particular stateid (files layout) or principal (e,g., flexible particular stateid (file layout) or principal (e.g., flexible
files layout) when performing I/O. file layout) when performing I/O.
In contrast, there is no requirement to restrict access to a In contrast, there is no requirement to restrict access to a
file on the storage devices when a layout is recalled. It is file on the storage devices when a layout is recalled. It is
only after the metadata server determines that the client is not only after the metadata server determines that the client is not
gracefully returning the layout and starts the revocation that gracefully returning the layout and starts the revocation that
this requirement is enforced. this requirement is enforced.
(3) A pNFS implementation MUST NOT allow the violation of NFSv4.1's (3) A pNFS implementation MUST NOT allow the violation of NFSv4.1's
access controls: ACLs and file open modes. Section 12.9 of access controls: Access Control Lists (ACLs) and file open
[RFC5661] specifically lays this burden on the combination of modes. Section 12.9 of [RFC5661] specifically lays this burden
clients, storage devices, and the metadata server. However the on the combination of clients, storage devices, and the metadata
specification of the individual layout type might create server. However, the specification of the individual layout
requirements as to how this is to be done. This may include a type might create requirements as to how this is to be done.
possible requirement for the metadata server to update the This may include a possible requirement for the metadata server
storage device so that it can enforce security. to update the storage device so that it can enforce security.
The file layout requires the storage device to enforce access The file layout requires the storage device to enforce access
whereas the flex file layout requires both the storage device whereas the flexible file layout requires both the storage
and the client to enforce security. device and the client to enforce security.
(4) Interactions between locking and I/O operations MUST obey (4) Interactions between locking and I/O operations MUST obey
existing semantic restrictions. In particular, if an I/O existing semantic restrictions. In particular, if an I/O
operation would be invalid when directed at the metadata server, operation would be invalid when directed at the metadata server,
it is not to be allowed when performed on the storage device. it is not to be allowed when performed on the storage device.
For the block and SCSI layout, as the storage device is not able For the block and SCSI layouts, as the storage device is not
to reject the I/O operation, the client is responsible for able to reject the I/O operation, the client is responsible for
enforcing this requirement. enforcing this requirement.
(5) Any disagreement between the metadata server and the data server (5) Any disagreement between the metadata server and the data server
as to the value of attributes such as modify time, the change as to the value of attributes such as modify time, the change
attribute, and the EOF position MUST be of limited duration with attribute, and the EOF position MUST be of limited duration with
clear means of resolution of any discrepancies being provided. clear means of resolution of any discrepancies being provided.
Note that Note the following:
(a) Discrepancies need not be resolved unless any client has (a) Discrepancies need not be resolved unless any client has
accessed the file in question via the metadata server, accessed the file in question via the metadata server,
typically by performing a GETATTR. typically by performing a GETATTR.
(b) A particular storage device might be striped such it has no (b) A particular storage device might be striped, and as such,
information regarding the EOF position. its local view of the EOF position does not match the
global EOF position.
(c) Both clock skew and network delay can lead to the metadata (c) Both clock skew and network delay can lead to the metadata
server and the storage device having different values of server and the storage device having different values of
the time attributes. As long as those differences can be the time attributes. As long as those differences can be
accounted for in what is presented to the client in a accounted for in what is presented to the client in a
GETATTR, then no violation results. GETATTR, then no violation results.
(d) A LAYOUTCOMMIT requires that changes in attributes (d) A LAYOUTCOMMIT requires that changes in attributes
resulting from operations on the storage device need to be resulting from operations on the storage device need to be
reflected in the metadata server by the completion of the reflected in the metadata server by the completion of the
operation. operation.
These requirements may be satisfied in different ways by different These requirements may be satisfied in different ways by different
layout types. As an example, while the file layout type uses the layout types. As an example, while the file layout type uses the
stateid to fence off the client, there is no requirement that other stateid to fence off the client, there is no requirement that other
layout types use this stateid approach. layout types use this stateid approach.
Each new standards-track document for a layout types MUST address how Each new Standards Track document for a layout type MUST address how
the client, metadata server, and storage devices are to interact to the client, metadata server, and storage devices are to interact to
meet these requirements. meet these requirements.
3.2. Previously Undocumented Protocol Requirements 3.2. Previously Undocumented Protocol Requirements
While not explicitly stated as requirements in Section 12 of While not explicitly stated as requirements in Section 12 of
[RFC5661], the existing layout types do have more requirements that [RFC5661], the existing layout types do have more requirements that
they need to enforce. they need to enforce.
The client has these obligations when making I/O requests to the The client has these obligations when making I/O requests to the
storage devices: storage devices:
(1) Clients MUST NOT perform I/O to the storage device if they do (1) Clients MUST NOT perform I/O to the storage device if they do
not have layouts for the files in question. not have layouts for the files in question.
(2) Clients MUST NOT perform I/O operations outside of the specified (2) Clients MUST NOT perform I/O operations outside of the specified
ranges in the layout segment. ranges in the layout segment.
(3) Clients MUST NOT perform I/O operations which would be (3) Clients MUST NOT perform I/O operations that would be
inconsistent with the iomode specified in the layout segments it inconsistent with the iomode specified in the layout segments it
holds. holds.
Under the file layout type, the storage devices are able to reject Under the file layout type, the storage devices are able to reject
any request made not conforming to these requirements. This may not any request made not conforming to these requirements. This may not
be possible for other known layout types, which puts the burden of be possible for other known layout types, which puts the burden of
enforcing such violations solely on the client. For these layout enforcing such violations solely on the client. For these layout
types: types:
(1) The metadata server MAY use fencing operations to the storage (1) The metadata server MAY use fencing operations to the storage
devices to enforce layout revocation against the client. devices to enforce layout revocation against the client.
(2) The metadata server MUST allow the clients to perform data I/O (2) The metadata server MUST allow the clients to perform data I/O
against it, even if it has already granted the client a layout. against it, even if it has already granted the client a layout.
A layout type might discourage such I/O, but it can not forbid A layout type might discourage such I/O, but it cannot forbid
it. it.
(3) The metadata server MUST be able to do storage allocation, (3) The metadata server MUST be able to do storage allocation,
whether that is to create, delete, extend, or truncate files. whether that is to create, delete, extend, or truncate files.
The means to address these requirements will vary with the layout The means to address these requirements will vary with the layout
type. A control protocol will be used to effect these, whether a type. A control protocol will be used to effect these; the control
purpose-built one, one identical to the storage protocol, or a new protocol could be a purpose-built one, one identical to the storage
standards-track control protocol. protocol, or a new Standards Track control protocol.
3.3. Editorial Requirements 3.3. Editorial Requirements
This section discusses how the protocol requirements discussed above This section discusses how the protocol requirements discussed above
need to be addressed in documents specifying a new layout type. need to be addressed in documents specifying a new layout type.
Depending on the interoperability model for the layout type in Depending on the interoperability model for the layout type in
question, this may involve the imposition of layout-type-specific question, this may involve the imposition of layout-type-specific
requirements that ensure appropriate interoperability of pNFS requirements that ensure appropriate interoperability of pNFS
components which are developed separately. components that are developed separately.
The specification of the layout type needs to make clear how the The specification of the layout type needs to make clear how the
client, metadata server, and storage device act together to meet the client, metadata server, and storage device act together to meet the
protocol requirements discussed previously. If the document does not protocol requirements discussed previously. If the document does not
impose implementation requirements sufficient to ensure that these impose implementation requirements sufficient to ensure that these
semantic requirements are met, it is not appropriate for publication semantic requirements are met, it is not appropriate for publication
as an IETF-stream RFC. as an RFC from the IETF stream.
Some examples include: Some examples include:
o If the metadata server does not have a means to invalidate a o If the metadata server does not have a means to invalidate a
stateid issued to the storage device to keep a particular client stateid issued to the storage device to keep a particular client
from accessing a specific file, then the layout type specification from accessing a specific file, then the layout type specification
has to document how the metadata server is going to fence the has to document how the metadata server is going to fence the
client from access to the file on that storage device. client from access to the file on that storage device.
o If the metadata server implements mandatory byte-range locking o If the metadata server implements mandatory byte-range locking
skipping to change at page 11, line 34 skipping to change at page 11, line 28
specification must require that this also be done when data is specification must require that this also be done when data is
read or written using the designated storage protocol. read or written using the designated storage protocol.
4. Specifications of Original Layout Types 4. Specifications of Original Layout Types
This section discusses how the original layout types interact with This section discusses how the original layout types interact with
Section 12 of [RFC5661], which enumerates the requirements of pNFS Section 12 of [RFC5661], which enumerates the requirements of pNFS
layout type specifications. It is not normative with regards to the layout type specifications. It is not normative with regards to the
file layout type presented in Section 13 of [RFC5661], the block file layout type presented in Section 13 of [RFC5661], the block
layout type [RFC5663], and the object layout type [RFC5664]. These layout type [RFC5663], and the object layout type [RFC5664]. These
are discussed here only to illuminate the updates made in Section 3 are discussed here only to illuminate the updates Section 3 of this
of this document to Section 12 of [RFC5661]. document makes to Section 12 of [RFC5661].
4.1. File Layout Type 4.1. File Layout Type
Because the storage protocol is a subset of NFSv4.1, the semantics of Because the storage protocol is a subset of NFSv4.1, the semantics of
the file layout type comes closest to the semantics of NFSv4.1 in the the file layout type comes closest to the semantics of NFSv4.1 in the
absence of pNFS. In particular, the stateid and principal used for I absence of pNFS. In particular, the stateid and principal used for
/O MUST have the same effect and be subject to the same validation on I/O MUST have the same effect and be subject to the same validation
a data server as it would have if the I/O were being performed on the on a data server as it would have if the I/O were being performed on
metadata server itself. The same set of validations are applied the metadata server itself. The same set of validations are applied
whether pNFS is in effect or not. whether or not pNFS is in effect.
And while for most implementations the storage devices can do the While for most implementations, the storage devices can do the
following validations: following validations that are each presented as a "SHOULD" and not a
"MUST" in [RFC5661]:
(1) client holds a valid layout, (1) client holds a valid layout,
(2) client I/O matches the layout iomode, and,
(3) client does not go out of the byte ranges, (2) client I/O matches the layout iomode, and
these are each presented as a "SHOULD" and not a "MUST". Actually, (3) client does not go out of the byte ranges,
the first point is presented in [RFC5661] as both: Actually, the first point is presented in [RFC5661] as both:
"MUST": in Section 13.6 "MUST": in Section 13.6
"As described in Section 12.5.1, a client MUST NOT send an I/O to As described in Section 12.5.1, a client MUST NOT send an I/O to a
a data server for which it does not hold a valid layout; the data data server for which it does not hold a valid layout; the data
server MUST reject such an I/O." server MUST reject such an I/O.
"SHOULD": in Section 13.8 "SHOULD": in Section 13.8
"The iomode need not be checked by the data servers when clients The iomode need not be checked by the data servers when clients
perform I/O. However, the data servers SHOULD still validate that perform I/O. However, the data servers SHOULD still validate that
the client holds a valid layout and return an error if the client the client holds a valid layout and return an error if the client
does not." does not.
It should be noted that it is just these layout specific checks that It should be noted that it is just these layout-specific checks that
are optional, not the normal file access semantics. The storage are optional, not the normal file access semantics. The storage
devices MUST make all of the required access checks on each READ or devices MUST make all of the required access checks on each READ or
WRITE I/O as determined by the NFSv4.1 protocol. If the metadata WRITE I/O as determined by the NFSv4.1 protocol. If the metadata
server would deny a READ or WRITE operation on a file due to its ACL, server would deny a READ or WRITE operation on a file due to its ACL,
mode attribute, open access mode, open deny mode, mandatory byte- mode attribute, open access mode, open deny mode, mandatory byte-
range lock state, or any other attributes and state, the storage range locking state, or any other attributes and state, the storage
device MUST also deny the READ or WRITE operation. Also while the device MUST also deny the READ or WRITE operation. Also, while the
NFSv4.1 protocol does not mandate export access checks based on the NFSv4.1 protocol does not mandate export access checks based on the
client's IP address, if the metadata server implements such a policy, client's IP address, if the metadata server implements such a policy,
then that counts as such state as outlined above. then that counts as such state as outlined above.
The data filehandle provided by the PUTFH operation to the data The data filehandle provided by the PUTFH operation to the data
server provides sufficient context to enable the data server to server provides sufficient context to enable the data server to
ensure that for the subsequent READ or WRITE operation in the ensure that the client has a valid layout for the I/O being performed
compound, that the client has a valid layout for the I/O being for the subsequent READ or WRITE operation in the compound.
performed.
Finally, the data server can check the stateid presented in the READ Finally, the data server can check the stateid presented in the READ
or WRITE operation to see if that stateid has been rejected by the or WRITE operation to see if that stateid has been rejected by the
metadata server in order to cause the I/O to be fenced. Whilst it metadata server; if so, the data server will cause the I/O to be
might just be the open owner or lock owner on that client being fenced. Whilst it might just be the open owner or lock owner on that
fenced, the client should take the NFS4ERR_BAD_STATEID error code to client being fenced, the client should take the NFS4ERR_BAD_STATEID
mean it has been fenced from the file and contact the metadata error code to mean it has been fenced from the file and contact the
server. metadata server.
4.2. Block Layout Type 4.2. Block Layout Type
With the block layout type, the storage devices are generally not With the block layout type, the storage devices are generally not
able to enforce file-based security. Typically, storage area network able to enforce file-based security. Typically, storage area network
(SAN) disk arrays and SAN protocols provide coarse-grained access (SAN) disk arrays and SAN protocols provide coarse-grained access
control mechanisms (e.g., Logical Unit Number (LUN) mapping and/or control mechanisms (e.g., Logical Unit Number (LUN) mapping and/or
masking), with a target granularity of disks rather than individual masking), with a target granularity of disks rather than individual
blocks and a source granularity of individual hosts rather than of blocks and a source granularity of individual hosts rather than of
users or owners. Access to block storage is logically at a lower users or owners. Access to block storage is logically at a lower
layer of the I/O stack than NFSv4. Since NFSv4 security is not layer of the I/O stack than NFSv4. Since NFSv4 security is not
directly applicable to protocols that access such storage directly, directly applicable to protocols that access such storage directly,
Section 2.1 [RFC5663] specifies that: Section 2.1 of [RFC5663] specifies that:
"in environments where pNFS clients cannot be trusted to enforce in environments where pNFS clients cannot be trusted to enforce
such policies, pNFS block layout types SHOULD NOT be used." such policies, pNFS block layout types SHOULD NOT be used.
Due to these granularity issues, the security burden has been shifted Due to these granularity issues, the security burden has been shifted
from the storage devices to the client. Those deploying from the storage devices to the client. Those deploying
implementations of this layout type need to be sure that the client implementations of this layout type need to be sure that the client
implementation can be trusted This is not a new sort of requirement implementation can be trusted. This is not a new sort of requirement
in the context of SAN protocols. In such environments, the client is in the context of SAN protocols. In such environments, the client is
expected to provide block-based protection. expected to provide block-based protection.
This shift of the burden also extends to locks and layouts. The This shift of the burden also extends to locks and layouts. The
storage devices are not able to enforce any of these and the burden storage devices are not able to enforce any of these, and the burden
is pushed to the client to make the appropriate checks before sending is pushed to the client to make the appropriate checks before sending
I/O to the storage devices. For example, the server may use a layout I/O to the storage devices. For example, the server may use a layout
iomode only allowing reading to enforce a mandatory read-only lock, iomode only allowing reading to enforce a mandatory read-only lock.
In such cases, the client has to support that use by not sending In such cases, the client has to support that use by not sending
WRITEs to the storage devices. The fundamental issue here is that WRITEs to the storage devices. The fundamental issue here is that
the storage device is treated by this layout type in the same fashion the storage device is treated by this layout type in the same fashion
as a local disk device. Once the client has access to the storage as a local disk device. Once the client has access to the storage
device, it is able to perform both READ and WRITE I/O to the entire device, it is able to perform both READ and WRITE I/O to the entire
storage device. The byte ranges in the layout, any locks, the layout storage device. The byte ranges in the layout, any locks, the layout
iomode, etc, can only be enforced by the client. Therefore, the iomode, etc., can only be enforced by the client. Therefore, the
client is required to provide that enforcement. client is required to provide that enforcement.
In the context of fencing off of the client upon revocation of a In the context of fencing off of the client upon revocation of a
layout, these limitations come into play again, i.e., the granularity layout, these limitations come into play again, i.e., the granularity
of the fencing can only be at the host/logical-unit level. Thus, if of the fencing can only be at the level of the host and logical unit.
one of a client's layouts is revoked by the server, it will Thus, if one of a client's layouts is revoked by the server, it will
effectively revoke all of the client's layouts for files located on effectively revoke all of the client's layouts for files located on
the storage units comprising the logical volume. This may extend to the storage units comprising the logical volume. This may extend to
the client's layouts for files in other file systems. Clients need the client's layouts for files in other file systems. Clients need
to be prepared for such revocations and reacquire layouts as needed. to be prepared for such revocations and reacquire layouts as needed.
4.3. Object Layout Type 4.3. Object Layout Type
With the object layout type, security checks occur during the With the object layout type, security checks occur during the
allocation of the layout. The client will typically ask for layouts allocation of the layout. The client will typically ask for layouts
covering all of the file and may do so for either READ or READ/WRITE. covering all of the file and may do so for either READ or READ/WRITE.
skipping to change at page 14, line 22 skipping to change at page 14, line 12
server should verify permissions against the layout iomode, the file server should verify permissions against the layout iomode, the file
mode bits or ACLs, etc. As the client may be acting for multiple mode bits or ACLs, etc. As the client may be acting for multiple
local users, it MUST authenticate and authorize the user by issuing local users, it MUST authenticate and authorize the user by issuing
respective OPEN and ACCESS calls to the metadata server, similar to respective OPEN and ACCESS calls to the metadata server, similar to
having NFSv4 data delegations. having NFSv4 data delegations.
Upon successful authorization, the client receives within the layout Upon successful authorization, the client receives within the layout
a set of object capabilities allowing it I/O access to the specified a set of object capabilities allowing it I/O access to the specified
objects corresponding to the requested iomode. These capabilities objects corresponding to the requested iomode. These capabilities
are used to enforce access control and locking semantics at the are used to enforce access control and locking semantics at the
storage devices. Whenever one of the following occur on the metadata storage devices. Whenever one of the following occurs on the
server: metadata server, then the metadata server MUST change the capability
version attribute on all objects comprising the file in order to
invalidate any outstanding capabilities before committing to one of
these changes:
o the permissions on the object change, o the permissions on the object change,
o a conflicting mandatory byte-range lock is granted, or o a conflicting mandatory byte-range lock is granted, or
o a layout is revoked and reassigned to another client, o a layout is revoked and reassigned to another client.
then the metadata server MUST change the capability version attribute
on all objects comprising the file to in order to invalidate any
outstanding capabilities before committing to one of these changes.
When the metadata server wishes to fence off a client to a particular When the metadata server wishes to fence off a client to a particular
object, then it can use the above approach to invalidate the object, then it can use the above approach to invalidate the
capability attribute on the given object. The client can be informed capability attribute on the given object. The client can be informed
via the storage device that the capability has been rejected and is via the storage device that the capability has been rejected and is
allowed to fetch a refreshed set of capabilities, i.e., re-acquire allowed to fetch a refreshed set of capabilities, i.e., reacquire the
the layout. layout.
5. Summary 5. Summary
In the three original layout types, the burden of enforcing the In the three original layout types, the burden of enforcing the
security of NFSv4.1 can fall to either the storage devices (files), security of NFSv4.1 can fall to either the storage devices (files),
the client (blocks), or the metadata server (objects). Such choices the client (blocks), or the metadata server (objects). Such choices
are conditioned by the native capabilities of the storage devices - are conditioned by the native capabilities of the storage devices --
if a control protocol can be implemented, then the burden can be if a control protocol can be implemented, then the burden can be
shifted primarily to the storage devices. shifted primarily to the storage devices.
In the context of this document, we treat the control protocol as a In the context of this document, we treat the control protocol as a
set of requirements. And as new layout types are published, the set of requirements. As new layout types are published, the defining
defining documents MUST address: documents MUST address:
(1) The fencing of clients after a layout is revoked. (1) The fencing of clients after a layout is revoked.
(2) The security implications of the native capabilities of the (2) The security implications of the native capabilities of the
storage devices with respect to the requirements of the NFSv4.1 storage devices with respect to the requirements of the NFSv4.1
security model. security model.
In addition, these defining documents need to make clear how other In addition, these defining documents need to make clear how other
semantic requirements of NFSv4.1 (e.g., locking) are met in the semantic requirements of NFSv4.1 (e.g., locking) are met in the
context of the proposed layout type. context of the proposed layout type.
6. Security Considerations 6. Security Considerations
This section does not deal directly with security considerations for This section does not deal directly with security considerations for
existing or new layout types. Instead, it provides a general existing or new layout types. Instead, it provides a general
framework for understating security-related issues within the pNFS framework for understating security-related issues within the pNFS
framework. Specific security considerations will be addressed in the framework. Specific security considerations will be addressed in the
Security Considerations sections of documents specifying layout Security Considerations sections of documents specifying layout
types. For example, in Section 5 of [RFC5663], the lack of finer- types. For example, in Section 3 of [RFC5663], the lack of finer-
than-physical disk access control necessitates that the client is than-physical disk access control necessitates that the client is
delegated the responsibility to enforce the access provided to them delegated the responsibility to enforce the access provided to them
in the layout extent which they were granted by the metadata server. in the layout extent that they were granted by the metadata server.
The layout type specification must ensure that only data accesses The layout type specification must ensure that only data access
consistent with the NFSV4.1 security model are allowed. It may do consistent with the NFSV4.1 security model is allowed. It may do
this directly, by providing that appropriate checks be performed at this directly, by providing that appropriate checks be performed at
the time each access is performed. It may do it indirectly by the time each access is performed. It may do it indirectly by
allowing the client or the storage device to be responsible for allowing the client or the storage device to be responsible for
making the appropriate checks. In the latter case, I/O access rights making the appropriate checks. In the latter case, I/O access rights
are reflected in layouts and the layout type must provide a way to are reflected in layouts, and the layout type must provide a way to
prevent inappropriate access due to permissions changes between the prevent inappropriate access due to permissions changes between the
time a layout is granted and the time the access is performed. time a layout is granted and the time the access is performed.
The metadata server MUST be able to fence off a client's access to The metadata server MUST be able to fence off a client's access to
the data file on a storage device. When it revokes the layout, the the data file on a storage device. When it revokes the layout, the
client's access MUST be terminated at the storage devices. The client's access MUST be terminated at the storage devices. The
client has a subsequent opportunity to re-acquire the layout and client has a subsequent opportunity to reacquire the layout and
perform the security check in the context of the newly current access perform the security check in the context of the newly current access
permissions. permissions.
7. IANA Considerations 7. IANA Considerations
This document has no actions for IANA. This document has no IANA actions.
8. References 8. References
8.1. Normative References 8.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ Requirement Levels", BCP 14, RFC 2119,
RFC2119, March 1997, <https://www.rfc-editor.org/info/ DOI 10.17487/RFC2119, March 1997,
rfc2119>. <https://www.rfc-editor.org/info/rfc2119>.
[RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1 "Network File System (NFS) Version 4 Minor Version 1
Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010,
<https://www.rfc-editor.org/info/rfc5661>. <https://www.rfc-editor.org/info/rfc5661>.
[RFC5663] Black, D., Fridella, S., and J. Glasgow, "Parallel NFS [RFC5663] Black, D., Fridella, S., and J. Glasgow, "Parallel NFS
(pNFS) Block/Volume Layout", RFC 5663, DOI 10.17487/ (pNFS) Block/Volume Layout", RFC 5663,
RFC5663, January 2010, <https://www.rfc-editor.org/info/ DOI 10.17487/RFC5663, January 2010,
rfc5663>. <https://www.rfc-editor.org/info/rfc5663>.
[RFC5664] Halevy, B., Welch, B., and J. Zelenka, "Object-Based [RFC5664] Halevy, B., Welch, B., and J. Zelenka, "Object-Based
Parallel NFS (pNFS) Operations", RFC 5664, DOI 10.17487/ Parallel NFS (pNFS) Operations", RFC 5664,
RFC5664, January 2010, <https://www.rfc-editor.org/info/ DOI 10.17487/RFC5664, January 2010,
rfc5664>. <https://www.rfc-editor.org/info/rfc5664>.
[RFC8154] Hellwig, C., "Parallel NFS (pNFS) Small Computer System [RFC8154] Hellwig, C., "Parallel NFS (pNFS) Small Computer System
Interface (SCSI) Layout", RFC 8154, DOI 10.17487/RFC8154, Interface (SCSI) Layout", RFC 8154, DOI 10.17487/RFC8154,
May 2017, <https://www.rfc-editor.org/info/rfc8154>. May 2017, <https://www.rfc-editor.org/info/rfc8154>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>. May 2017, <https://www.rfc-editor.org/info/rfc8174>.
8.2. Informative References 8.2. Informative References
[FlexFiles] [Lustre] Faibish, S., Cote, D., and P. Tao, "Parallel NFS (pNFS)
Halevy, B. and T. Haynes, "Parallel NFS (pNFS) Flexible Lustre Layout Operations", Work in Progress,
File Layout", draft-ietf-nfsv4-flex-files-16 (Work In draft-faibish-nfsv4-pnfs-lustre-layout-07, May 2014.
Progress), January 2018.
[Lustre] Faibish, S. and P. Tao, "Parallel NFS (pNFS) Lustre Layout [RFC8435] Halevy, B. and T. Haynes, "Parallel NFS (pNFS) Flexible
Operations", draft-faibish-nfsv4-pnfs-lustre-layout-07 File Layout", RFC 8435, DOI 10.17487/RFC8435, August 2018,
(Work In Progress), April 2014. <https://www.rfc-editor.org/info/rfc8435>.
Appendix A. Acknowledgments Acknowledgments
Dave Noveck provided an early review that sharpened the clarity of Dave Noveck provided an early review that sharpened the clarity of
the definitions. He also provided a more comprehensive review of the the definitions. He also provided a more comprehensive review of the
document. document.
Both Chuck Lever and Christoph Helwig provided insightful comments Both Chuck Lever and Christoph Helwig provided insightful comments
during the WGLC. during the working group last call.
Appendix B. RFC Editor Notes
[RFC Editor: please remove this section prior to publishing this
document as an RFC]
[RFC Editor: prior to publishing this document as an RFC, please
replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the
RFC number of this document]
Author's Address Author's Address
Thomas Haynes Thomas Haynes
Hammerspace Hammerspace
4300 El Camino Real Ste 105 4300 El Camino Real Ste 105
Los Altos, CA 94022 Los Altos, CA 94022
USA United States of America
Email: loghyr@gmail.com Email: loghyr@gmail.com
 End of changes. 106 change blocks. 
278 lines changed or deleted 260 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/