draft-ietf-nfsv4-flex-files-19.txt   rfc8435.txt 
NFSv4 B. Halevy Internet Engineering Task Force (IETF) B. Halevy
Internet-Draft Request for Comments: 8435
Intended status: Standards Track T. Haynes Category: Standards Track T. Haynes
Expires: November 4, 2018 Hammerspace ISSN: 2070-1721 Hammerspace
May 03, 2018 August 2018
Parallel NFS (pNFS) Flexible File Layout Parallel NFS (pNFS) Flexible File Layout
draft-ietf-nfsv4-flex-files-19.txt
Abstract Abstract
The Parallel Network File System (pNFS) allows a separation between Parallel NFS (pNFS) allows a separation between the metadata (onto a
the metadata (onto a metadata server) and data (onto a storage metadata server) and data (onto a storage device) for a file. The
device) for a file. The flexible file layout type is defined in this flexible file layout type is defined in this document as an extension
document as an extension to pNFS which allows the use of storage to pNFS that allows the use of storage devices that require only a
devices in a fashion such that they require only a quite limited limited degree of interaction with the metadata server and use
degree of interaction with the metadata server, using already already-existing protocols. Client-side mirroring is also added to
existing protocols. Client-side mirroring is also added to provide provide replication of files.
replication of files.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This is an Internet Standards Track document.
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Further information on
Internet Standards is available in Section 2 of RFC 7841.
This Internet-Draft will expire on November 4, 2018. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8435.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction ....................................................3
1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 4 1.1. Definitions ................................................4
1.2. Requirements Language . . . . . . . . . . . . . . . . . . 6 1.2. Requirements Language ......................................6
2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 6 2. Coupling of Storage Devices .....................................6
2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 7 2.1. LAYOUTCOMMIT ...............................................7
2.2. Fencing Clients from the Storage Device . . . . . . . . . 7 2.2. Fencing Clients from the Storage Device ....................7
2.2.1. Implementation Notes for Synthetic uids/gids . . . . 8 2.2.1. Implementation Notes for Synthetic uids/gids ........8
2.2.2. Example of using Synthetic uids/gids . . . . . . . . 9 2.2.2. Example of Using Synthetic uids/gids ................9
2.3. State and Locking Models . . . . . . . . . . . . . . . . 10 2.3. State and Locking Models ..................................10
2.3.1. Loosely Coupled Locking Model . . . . . . . . . . . . 10 2.3.1. Loosely Coupled Locking Model ......................11
2.3.2. Tightly Coupled Locking Model . . . . . . . . . . . . 12 2.3.2. Tightly Coupled Locking Model ......................12
3. XDR Description of the Flexible File Layout Type . . . . . . 13 3. XDR Description of the Flexible File Layout Type ...............13
3.1. Code Components Licensing Notice . . . . . . . . . . . . 14 3.1. Code Components Licensing Notice ..........................14
4. Device Addressing and Discovery . . . . . . . . . . . . . . . 15 4. Device Addressing and Discovery ................................16
4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 16 4.1. ff_device_addr4 ...........................................16
4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 17 4.2. Storage Device Multipathing ...............................17
5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 18 5. Flexible File Layout Type ......................................18
5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 19 5.1. ff_layout4 ................................................19
5.1.1. Error Codes from LAYOUTGET . . . . . . . . . . . . . 22 5.1.1. Error Codes from LAYOUTGET .........................23
5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS . . 23 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS ...23
5.2. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 23 5.2. LAYOUTCOMMIT ..............................................24
5.3. Interactions Between Devices and Layouts . . . . . . . . 23 5.3. Interactions between Devices and Layouts ..................24
5.4. Handling Version Errors . . . . . . . . . . . . . . . . . 23 5.4. Handling Version Errors ...................................24
6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 24 6. Striping via Sparse Mapping ....................................25
7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 24 7. Recovering from Client I/O Errors ..............................25
8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 25 8. Mirroring ......................................................26
8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 26 8.1. Selecting a Mirror ........................................26
8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 26 8.2. Writing to Mirrors ........................................27
8.2.1. Single Storage Device Updates Mirrors . . . . . . . . 26 8.2.1. Single Storage Device Updates Mirrors ..............27
8.2.2. Client Updates All Mirrors . . . . . . . . . . . . . 27 8.2.2. Client Updates All Mirrors .........................27
8.2.3. Handling Write Errors . . . . . . . . . . . . . . . . 27 8.2.3. Handling Write Errors ..............................28
8.2.4. Handling Write COMMITs . . . . . . . . . . . . . . . 28 8.2.4. Handling Write COMMITs .............................28
8.3. Metadata Server Resilvering of the File . . . . . . . . . 28 8.3. Metadata Server Resilvering of the File ...................29
9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 28 9. Flexible File Layout Type Return ...............................29
9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 30 9.1. I/O Error Reporting .......................................30
9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 30 9.1.1. ff_ioerr4 ..........................................30
9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 30 9.2. Layout Usage Statistics ...................................31
9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 31 9.2.1. ff_io_latency4 .....................................31
9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 32 9.2.2. ff_layoutupdate4 ...................................32
9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 32 9.2.3. ff_iostats4 ........................................33
9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 33 9.3. ff_layoutreturn4 ..........................................34
10. Flexible File Layout Type LAYOUTERROR .........................35
10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 34 11. Flexible File Layout Type LAYOUTSTATS .........................35
11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 34 12. Flexible File Layout Type Creation Hint .......................35
12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 34 12.1. ff_layouthint4 ...........................................35
12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 35 13. Recalling a Layout ............................................36
13. Recalling a Layout . . . . . . . . . . . . . . . . . . . . . 35 13.1. CB_RECALL_ANY ............................................36
13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 36 14. Client Fencing ................................................37
14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 37 15. Security Considerations .......................................37
15. Security Considerations . . . . . . . . . . . . . . . . . . . 37 15.1. RPCSEC_GSS and Security Services .........................39
15.1. RPCSEC_GSS and Security Services . . . . . . . . . . . . 38 15.1.1. Loosely Coupled ...................................39
15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 38 15.1.2. Tightly Coupled ...................................39
15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 39 16. IANA Considerations ...........................................39
16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 39 17. References ....................................................40
17. References . . . . . . . . . . . . . . . . . . . . . . . . . 39 17.1. Normative References .....................................40
17.1. Normative References . . . . . . . . . . . . . . . . . . 39 17.2. Informative References ...................................41
17.2. Informative References . . . . . . . . . . . . . . . . . 41 Acknowledgments ...................................................42
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 41 Authors' Addresses ................................................42
Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 41
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 42
1. Introduction 1. Introduction
In the parallel Network File System (pNFS), the metadata server In Parallel NFS (pNFS), the metadata server returns layout type
returns layout type structures that describe where file data is structures that describe where file data is located. There are
located. There are different layout types for different storage different layout types for different storage systems and methods of
systems and methods of arranging data on storage devices. This arranging data on storage devices. This document defines the
document defines the flexible file layout type used with file-based flexible file layout type used with file-based data servers that are
data servers that are accessed using the Network File System (NFS) accessed using the NFS protocols: NFSv3 [RFC1813], NFSv4.0 [RFC7530],
protocols: NFSv3 [RFC1813], NFSv4.0 [RFC7530], NFSv4.1 [RFC5661], and NFSv4.1 [RFC5661], and NFSv4.2 [RFC7862].
NFSv4.2 [RFC7862].
To provide a global state model equivalent to that of the files To provide a global state model equivalent to that of the files
layout type, a back-end control protocol might be implemented between layout type, a back-end control protocol might be implemented between
the metadata server and NFSv4.1+ storage devices. This document does the metadata server and NFSv4.1+ storage devices. An implementation
not provide a standard track control protocol. An implementation can can either define its own proprietary mechanism or it could define a
either define its own mechanism or it could define a control protocol control protocol in a Standards Track document. The requirements for
in a standard's track document. The requirements for a control a control protocol are specified in [RFC5661] and clarified in
protocol are specified in [RFC5661] and clarified in [pNFSLayouts]. [RFC8434].
The control protocol described in this document is based on NFS. The The control protocol described in this document is based on NFS. It
storage devices are configured such that the metadata server has full does not provide for knowledge of stateids to be passed between the
access rights to the data file system and then the metadata server metadata server and the storage devices. Instead, the storage
uses synthetic ids to control client access to individual files. devices are configured such that the metadata server has full access
rights to the data file system and then the metadata server uses
synthetic ids to control client access to individual files.
In traditional mirroring of data, the server is responsible for In traditional mirroring of data, the server is responsible for
replicating, validating, and repairing copies of the data file. With replicating, validating, and repairing copies of the data file. With
client-side mirroring, the metadata server provides a layout which client-side mirroring, the metadata server provides a layout that
presents the available mirrors to the client. It is then the client presents the available mirrors to the client. The client then picks
which picks a mirror to read from and ensures that all writes go to a mirror to read from and ensures that all writes go to all mirrors.
all mirrors. Only if all mirrors are successfully updated, does the The client only considers the write transaction to have succeeded if
client consider the write transaction to have succeeded. In case of all mirrors are successfully updated. In case of error, the client
error, the client can use the LAYOUTERROR operation to inform the can use the LAYOUTERROR operation to inform the metadata server,
metadata server, which is then responsible for the repairing of the which is then responsible for the repairing of the mirrored copies of
mirrored copies of the file. the file.
1.1. Definitions 1.1. Definitions
control communication requirements: is the specification for control communication requirements: the specification for
information on layouts, stateids, file metadata, and file data information on layouts, stateids, file metadata, and file data
which must be communicated between the metadata server and the that must be communicated between the metadata server and the
storage devices. There is a separate set of requirements for each storage devices. There is a separate set of requirements for each
layout type. layout type.
control protocol: is the particular mechanism that an implementation control protocol: the particular mechanism that an implementation of
of a layout type would use to meet the control communication a layout type would use to meet the control communication
requirement for that layout type. This need not be a protocol as requirement for that layout type. This need not be a protocol as
normally understood. In some cases the same protocol may be used normally understood. In some cases, the same protocol may be used
as a control protocol and storage protocol. as a control protocol and storage protocol.
client-side mirroring: is a feature in which the client and not the client-side mirroring: a feature in which the client, not the
server is responsible for updating all of the mirrored copies of a server, is responsible for updating all of the mirrored copies of
layout segment. a layout segment.
(file) data: is that part of the file system object which contains (file) data: that part of the file system object that contains the
the content. data to be read or written. It is the contents of the object
rather than the attributes of the object.
data server (DS): is another term for storage device. data server (DS): a pNFS server that provides the file's data when
the file system object is accessed over a file-based protocol.
fencing: is the process by which the metadata server prevents the fencing: the process by which the metadata server prevents the
storage devices from processing I/O from a specific client to a storage devices from processing I/O from a specific client to a
specific file. specific file.
file layout type: is a layout type in which the storage devices are file layout type: a layout type in which the storage devices are
accessed via the NFS protocol (see Section 13 of [RFC5661]). accessed via the NFS protocol (see Section 13 of [RFC5661]).
gid: is the group id, a numeric value which identifies to which gid: the group id, a numeric value that identifies to which group a
group a file belongs. file belongs.
layout: is the information a client uses to access file data on a layout: the information a client uses to access file data on a
storage device. This information will include specification of storage device. This information includes specification of the
the protocol (layout type) and the identity of the storage devices protocol (layout type) and the identity of the storage devices to
to be used. be used.
layout iomode: is a grant of either read or read/write I/O to the layout iomode: a grant of either read-only or read/write I/O to the
client. client.
layout segment: is a sub-division of a layout. That sub-division layout segment: a sub-division of a layout. That sub-division might
might be by the layout iomode (see Sections 3.3.20 and 12.2.9 of be by the layout iomode (see Sections 3.3.20 and 12.2.9 of
[RFC5661]), a striping pattern (see Section 13.3 of [RFC5661]), or [RFC5661]), a striping pattern (see Section 13.3 of [RFC5661]), or
requested byte range. requested byte range.
layout stateid: is a 128-bit quantity returned by a server that layout stateid: a 128-bit quantity returned by a server that
uniquely defines the layout state provided by the server for a uniquely defines the layout state provided by the server for a
specific layout that describes a layout type and file (see specific layout that describes a layout type and file (see
Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 describes Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 of
differences in handling between layout stateids and other stateid [RFC5661] describes differences in handling between layout
types. stateids and other stateid types.
layout type: is a specification of both the storage protocol used to layout type: a specification of both the storage protocol used to
access the data and the aggregation scheme used to lay out the access the data and the aggregation scheme used to lay out the
file data on the underlying storage devices. file data on the underlying storage devices.
loose coupling: is when the control protocol is a storage protocol. loose coupling: when the control protocol is a storage protocol.
(file) metadata: is that part of the file system object which (file) metadata: the part of the file system object that contains
describes the object and not the content. E.g., it could be the various descriptive data relevant to the file object, as opposed
time since last modification, access, etc. to the file data itself. This could include the time of last
modification, access time, EOF position, etc.
metadata server (MDS): is the pNFS server which provides metadata metadata server (MDS): the pNFS server that provides metadata
information for a file system object. It also is responsible for information for a file system object. It is also responsible for
generating, recalling, and revoking layouts for file system generating, recalling, and revoking layouts for file system
objects, for performing directory operations, and for performing I objects, for performing directory operations, and for performing
/O operations to regular files when the clients direct these to I/O operations to regular files when the clients direct these to
the metadata server itself. the metadata server itself.
mirror: is a copy of a layout segment. Note that if one copy of the mirror: a copy of a layout segment. Note that if one copy of the
mirror is updated, then all copies must be updated. mirror is updated, then all copies must be updated.
recalling a layout: is when the metadata server uses a back channel recalling a layout: a graceful recall, via a callback, of a specific
to inform the client that the layout is to be returned in a layout by the metadata server to the client. Graceful here means
graceful manner. Note that the client has the opportunity to that the client would have the opportunity to flush any WRITEs,
flush any writes, etc., before replying to the metadata server. etc., before returning the layout to the metadata server.
revoking a layout: is when the metadata server invalidates the revoking a layout: an invalidation of a specific layout by the
layout such that neither the metadata server nor any storage metadata server. Once revocation occurs, the metadata server will
device will accept any access from the client with that layout. not accept as valid any reference to the revoked layout, and a
storage device will not accept any client access based on the
layout.
resilvering: is the act of rebuilding a mirrored copy of a layout resilvering: the act of rebuilding a mirrored copy of a layout
segment from a known good copy of the layout segment. Note that segment from a known good copy of the layout segment. Note that
this can also be done to create a new mirrored copy of the layout this can also be done to create a new mirrored copy of the layout
segment. segment.
rsize: is the data transfer buffer size used for reads. rsize: the data transfer buffer size used for READs.
stateid: is a 128-bit quantity returned by a server that uniquely stateid: a 128-bit quantity returned by a server that uniquely
defines the open and locking states provided by the server for a defines the set of locking-related state provided by the server.
specific open-owner or lock-owner/open-owner pair for a specific Stateids may designate state related to open files, byte-range
file and type of lock. locks, delegations, or layouts.
storage device: is the target to which clients may direct I/O storage device: the target to which clients may direct I/O requests
requests when they hold an appropriate layout. See Section 2.1 of when they hold an appropriate layout. See Section 2.1 of
[pNFSLayouts] for further discussion of the difference between a [RFC8434] for further discussion of the difference between a data
data store and a storage device. server and a storage device.
storage protocol: is the protocol used by clients to do I/O storage protocol: the protocol used by clients to do I/O operations
operations to the storage device. Each layout type specifies the to the storage device. Each layout type specifies the set of
set of storage protocols. storage protocols.
tight coupling: is an arrangement in which the control protocol is tight coupling: an arrangement in which the control protocol is one
one designed specifically for that purpose. It may be either a designed specifically for control communication. It may be either
proprietary protocol, adapted specifically to a a particular a proprietary protocol adapted specifically to a particular
metadata server, or one based on a standards-track document. metadata server or a protocol based on a Standards Track document.
uid: is the used id, a numeric value which identifies which user uid: the user id, a numeric value that identifies which user owns a
owns a file. file.
wsize: is the data transfer buffer size used for writes. wsize: the data transfer buffer size used for WRITEs.
1.2. Requirements Language 1.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in
14 [RFC2119] [RFC8174] when, and only when, they appear in all BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
2. Coupling of Storage Devices 2. Coupling of Storage Devices
A server implementation may choose either a loose or tight coupling A server implementation may choose either a loosely coupled model or
model between the metadata server and the storage devices. a tightly coupled model between the metadata server and the storage
[pNFSLayouts] describes the general problems facing pNFS devices. [RFC8434] describes the general problems facing pNFS
implementations. This document details how the new Flexible File implementations. This document details how the new flexible file
Layout Type addresses these issues. To implement the tight coupling layout type addresses these issues. To implement the tightly coupled
model, a control protocol has to be defined. As the flex file layout model, a control protocol has to be defined. As the flexible file
imposes no special requirements on the client, the control protocol layout imposes no special requirements on the client, the control
will need to provide: protocol will need to provide:
(1) for the management of both security and LAYOUTCOMMITs, and, (1) management of both security and LAYOUTCOMMITs and
(2) a global stateid model and management of these stateids. (2) a global stateid model and management of these stateids.
When implementing the loose coupling model, the only control protocol When implementing the loosely coupled model, the only control
will be a version of NFS, with no ability to provide a global stateid protocol will be a version of NFS, with no ability to provide a
model or to prevent clients from using layouts inappropriately. To global stateid model or to prevent clients from using layouts
enable client use in that environment, this document will specify how inappropriately. To enable client use in that environment, this
security, state, and locking are to be managed. document will specify how security, state, and locking are to be
managed.
2.1. LAYOUTCOMMIT 2.1. LAYOUTCOMMIT
Regardless of the coupling model, the metadata server has the Regardless of the coupling model, the metadata server has the
responsibility, upon receiving a LAYOUTCOMMIT (see Section 18.42 of responsibility, upon receiving a LAYOUTCOMMIT (see Section 18.42 of
[RFC5661]), of ensuring that the semantics of pNFS are respected (see [RFC5661]) to ensure that the semantics of pNFS are respected (see
Section 3.1 of [pNFSLayouts]). These do include a requirement that Section 3.1 of [RFC8434]). These do include a requirement that data
data written to data storage device be stable before the occurrence written to a data storage device be stable before the occurrence of
of the LAYOUTCOMMIT. the LAYOUTCOMMIT.
It is the responsibility of the client to make sure the data file is It is the responsibility of the client to make sure the data file is
stable before the metadata server begins to query the storage devices stable before the metadata server begins to query the storage devices
about the changes to the file. If any WRITE to a storage device did about the changes to the file. If any WRITE to a storage device did
not result with stable_how equal to FILE_SYNC, a LAYOUTCOMMIT to the not result with stable_how equal to FILE_SYNC, a LAYOUTCOMMIT to the
metadata server MUST be preceded by a COMMIT to the storage devices metadata server MUST be preceded by a COMMIT to the storage devices
written to. Note that if the client has not done a COMMIT to the written to. Note that if the client has not done a COMMIT to the
storage device, then the LAYOUTCOMMIT might not be synchronized to storage device, then the LAYOUTCOMMIT might not be synchronized to
the last WRITE operation to the storage device. the last WRITE operation to the storage device.
2.2. Fencing Clients from the Storage Device 2.2. Fencing Clients from the Storage Device
With loosely coupled storage devices, the metadata server uses With loosely coupled storage devices, the metadata server uses
synthetic uids (user ids) and gids (group ids) for the data file, synthetic uids (user ids) and gids (group ids) for the data file,
where the uid owner of the data file is allowed read/write access and where the uid owner of the data file is allowed read/write access and
the gid owner is allowed read only access. As part of the layout the gid owner is allowed read-only access. As part of the layout
(see ffds_user and ffds_group in Section 5.1), the client is provided (see ffds_user and ffds_group in Section 5.1), the client is provided
with the user and group to be used in the Remote Procedure Call (RPC) with the user and group to be used in the Remote Procedure Call (RPC)
[RFC5531] credentials needed to access the data file. Fencing off of [RFC5531] credentials needed to access the data file. Fencing off of
clients is achieved by the metadata server changing the synthetic uid clients is achieved by the metadata server changing the synthetic uid
and/or gid owners of the data file on the storage device to and/or gid owners of the data file on the storage device to
implicitly revoke the outstanding RPC credentials. A client implicitly revoke the outstanding RPC credentials. A client
presenting the wrong credential for the desired access will get a presenting the wrong credential for the desired access will get an
NFS4ERR_ACCESS error. NFS4ERR_ACCESS error.
With this loosely coupled model, the metadata server is not able to With this loosely coupled model, the metadata server is not able to
fence off a single client, it is forced to fence off all clients. fence off a single client; it is forced to fence off all clients.
However, as the other clients react to the fencing, returning their However, as the other clients react to the fencing, returning their
layouts and trying to get new ones, the metadata server can hand out layouts and trying to get new ones, the metadata server can hand out
a new uid and gid to allow access. a new uid and gid to allow access.
It is RECOMMENDED to implement common access control methods at the It is RECOMMENDED to implement common access control methods at the
storage device filesystem to allow only the metadata server root storage device file system to allow only the metadata server root
(super user) access to the storage device, and to set the owner of (super user) access to the storage device and to set the owner of all
all directories holding data files to the root user. This approach directories holding data files to the root user. This approach
provides a practical model to enforce access control and fence off provides a practical model to enforce access control and fence off
cooperative clients, but it can not protect against malicious cooperative clients, but it cannot protect against malicious clients;
clients; hence it provides a level of security equivalent to hence, it provides a level of security equivalent to AUTH_SYS. It is
AUTH_SYS. It is RECOMMENDED that the communication between the RECOMMENDED that the communication between the metadata server and
metadata server and storage device be secure from eavesdroppers and storage device be secure from eavesdroppers and man-in-the-middle
man-in-the-middle protocol tampering. The security measure could be protocol tampering. The security measure could be physical security
due to physical security (e.g., the servers are co-located in a (e.g., the servers are co-located in a physically secure area),
physically secure area), from encrypted communications, or some other encrypted communications, or some other technique.
technique.
With tightly coupled storage devices, the metadata server sets the With tightly coupled storage devices, the metadata server sets the
user and group owners, mode bits, and ACL of the data file to be the user and group owners, mode bits, and Access Control List (ACL) of
same as the metadata file. And the client must authenticate with the the data file to be the same as the metadata file. And the client
storage device and go through the same authorization process it would must authenticate with the storage device and go through the same
go through via the metadata server. In the case of tight coupling, authorization process it would go through via the metadata server.
fencing is the responsibility of the control protocol and is not In the case of tight coupling, fencing is the responsibility of the
described in detail here. However, implementations of the tight control protocol and is not described in detail in this document.
coupling locking model (see Section 2.3), will need a way to prevent However, implementations of the tightly coupled locking model (see
access by certain clients to specific files by invalidating the Section 2.3) will need a way to prevent access by certain clients to
corresponding stateids on the storage device. In such a scenario, specific files by invalidating the corresponding stateids on the
the client will be given an error of NFS4ERR_BAD_STATEID. storage device. In such a scenario, the client will be given an
error of NFS4ERR_BAD_STATEID.
The client need not know the model used between the metadata server The client need not know the model used between the metadata server
and the storage device. It need only react consistently to any and the storage device. It need only react consistently to any
errors in interacting with the storage device. It should both return errors in interacting with the storage device. It should both return
the layout and error to the metadata server and ask for a new layout. the layout and error to the metadata server and ask for a new layout.
At that point, the metadata server can either hand out a new layout, At that point, the metadata server can either hand out a new layout,
hand out no layout (forcing the I/O through it), or deny the client hand out no layout (forcing the I/O through it), or deny the client
further access to the file. further access to the file.
2.2.1. Implementation Notes for Synthetic uids/gids 2.2.1. Implementation Notes for Synthetic uids/gids
The selection method for the synthetic uids and gids to be used for The selection method for the synthetic uids and gids to be used for
fencing in loosely coupled storage devices is strictly an fencing in loosely coupled storage devices is strictly an
implementation issue. I.e., an administrator might restrict a range implementation issue. That is, an administrator might restrict a
of such ids available to the Lightweight Directory Access Protocol range of such ids available to the Lightweight Directory Access
(LDAP) 'uid' field [RFC4519]. She might also be able to choose an id Protocol (LDAP) 'uid' field [RFC4519]. The administrator might also
that would never be used to grant access. Then when the metadata be able to choose an id that would never be used to grant access.
server had a request to access a file, a SETATTR would be sent to the Then, when the metadata server had a request to access a file, a
storage device to set the owner and group of the data file. The user SETATTR would be sent to the storage device to set the owner and
and group might be selected in a round robin fashion from the range group of the data file. The user and group might be selected in a
of available ids. round-robin fashion from the range of available ids.
Those ids would be sent back as ffds_user and ffds_group to the Those ids would be sent back as ffds_user and ffds_group to the
client. And it would present them as the RPC credentials to the client, who would present them as the RPC credentials to the storage
storage device. When the client was done accessing the file and the device. When the client is done accessing the file and the metadata
metadata server knew that no other client was accessing the file, it server knows that no other client is accessing the file, it can reset
could reset the owner and group to restrict access to the data file. the owner and group to restrict access to the data file.
When the metadata server wanted to fence off a client, it would When the metadata server wants to fence off a client, it changes the
change the synthetic uid and/or gid to the restricted ids. Note that synthetic uid and/or gid to the restricted ids. Note that using a
using a restricted id ensures that there is a change of owner and at restricted id ensures that there is a change of owner and at least
least one id available that never gets allowed access. one id available that never gets allowed access.
Under an AUTH_SYS security model, synthetic uids and gids of 0 SHOULD Under an AUTH_SYS security model, synthetic uids and gids of 0 SHOULD
be avoided. These typically either grant super access to files on a be avoided. These typically either grant super access to files on a
storage device or are mapped to an anonymous id. In the first case, storage device or are mapped to an anonymous id. In the first case,
even if the data file is fenced, the client might still be able to even if the data file is fenced, the client might still be able to
access the file. In the second case, multiple ids might be mapped to access the file. In the second case, multiple ids might be mapped to
the anonymous ids. the anonymous ids.
2.2.2. Example of using Synthetic uids/gids 2.2.2. Example of Using Synthetic uids/gids
The user loghyr creates a file "ompha.c" on the metadata server and The user loghyr creates a file "ompha.c" on the metadata server,
it creates a corresponding data file on the storage device. which then creates a corresponding data file on the storage device.
The metadata server entry may look like: The metadata server entry may look like:
-rw-r--r-- 1 loghyr staff 1697 Dec 4 11:31 ompha.c -rw-r--r-- 1 loghyr staff 1697 Dec 4 11:31 ompha.c
On the storage device, it may be assigned some unpredictable On the storage device, the file may be assigned some unpredictable
synthetic uid/gid to deny access: synthetic uid/gid to deny access:
-rw-r----- 1 19452 28418 1697 Dec 4 11:31 data_ompha.c -rw-r----- 1 19452 28418 1697 Dec 4 11:31 data_ompha.c
When the file is opened on a client and accessed, it will try to get When the file is opened on a client and accessed, the user will try
a layout for the data file. Since the layout knows nothing about the to get a layout for the data file. Since the layout knows nothing
user (and does not care), whether the user loghyr or garbo opens the about the user (and does not care), it does not matter whether the
file does not matter. The client has to present an uid of 19452 to user loghyr or garbo opens the file. The client has to present an
get write permission. If it presents any other value for the uid, uid of 19452 to get write permission. If it presents any other value
then it must give a gid of 28418 to get read access. for the uid, then it must give a gid of 28418 to get read access.
Further, if the metadata server decides to fence the file, it should Further, if the metadata server decides to fence the file, it should
change the uid and/or gid such that these values neither match change the uid and/or gid such that these values neither match
earlier values for that file nor match a predictable change based on earlier values for that file nor match a predictable change based on
an earlier fencing. an earlier fencing.
-rw-r----- 1 19453 28419 1697 Dec 4 11:31 data_ompha.c -rw-r----- 1 19453 28419 1697 Dec 4 11:31 data_ompha.c
The set of synthetic gids on the storage device should be selected The set of synthetic gids on the storage device should be selected
such that there is no mapping in any of the name services used by the such that there is no mapping in any of the name services used by the
storage device. I.e., each group should have no members. storage device, i.e., each group should have no members.
If the layout segment has an iomode of LAYOUTIOMODE4_READ, then the If the layout segment has an iomode of LAYOUTIOMODE4_READ, then the
metadata server should return a synthetic uid that is not set on the metadata server should return a synthetic uid that is not set on the
storage device. Only the synthetic gid would be valid. storage device. Only the synthetic gid would be valid.
The client is thus solely responsible for enforcing file permissions The client is thus solely responsible for enforcing file permissions
in a loosely coupled model. To allow loghyr write access, it will in a loosely coupled model. To allow loghyr write access, it will
send an RPC to the storage device with a credential of 1066:1067. To send an RPC to the storage device with a credential of 1066:1067. To
allow garbo read access, it will send an RPC to the storage device allow garbo read access, it will send an RPC to the storage device
with a credential of 1067:1067. The value of the uid does not matter with a credential of 1067:1067. The value of the uid does not matter
as long as it is not the synthetic uid granted it when getting the as long as it is not the synthetic uid granted when getting the
layout. layout.
While pushing the enforcement of permission checking onto the client While pushing the enforcement of permission checking onto the client
may seem to weaken security, the client may already be responsible may seem to weaken security, the client may already be responsible
for enforcing permissions before modifications are sent to a server. for enforcing permissions before modifications are sent to a server.
With cached writes, the client is always responsible for tracking who With cached writes, the client is always responsible for tracking who
is modifying a file and making sure to not coalesce requests from is modifying a file and making sure to not coalesce requests from
multiple users into one request. multiple users into one request.
2.3. State and Locking Models 2.3. State and Locking Models
An implementation can always be deployed as a loosely coupled model. An implementation can always be deployed as a loosely coupled model.
There is however no way for a storage device to indicate over a NFS There is, however, no way for a storage device to indicate over an
protocol that it can definitively participate in a tightly coupled NFS protocol that it can definitively participate in a tightly
model: coupled model:
o Storage devices implementing the NFSv3 and NFSv4.0 protocols are o Storage devices implementing the NFSv3 and NFSv4.0 protocols are
always treated as loosely coupled. always treated as loosely coupled.
o NFSv4.1+ storage devices that do not return the o NFSv4.1+ storage devices that do not return the
EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID are indicating EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID are indicating
that they are to be treated as loosely coupled. From the locking that they are to be treated as loosely coupled. From the locking
viewpoint they are treated in the same way as NFSv4.0 storage viewpoint, they are treated in the same way as NFSv4.0 storage
devices. devices.
o NFSv4.1+ storage devices that do identify themselves with the o NFSv4.1+ storage devices that do identify themselves with the
EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID can potentially EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID can potentially
be tightly coupled. They would use a back-end control protocol to be tightly coupled. They would use a back-end control protocol to
implement the global stateid model as described in [RFC5661]. implement the global stateid model as described in [RFC5661].
A storage device would have to either be discovered or advertised A storage device would have to be either discovered or advertised
over the control protocol to enable a tight coupling model. over the control protocol to enable a tightly coupled model.
2.3.1. Loosely Coupled Locking Model 2.3.1. Loosely Coupled Locking Model
When locking-related operations are requested, they are primarily When locking-related operations are requested, they are primarily
dealt with by the metadata server, which generates the appropriate dealt with by the metadata server, which generates the appropriate
stateids. When an NFSv4 version is used as the data access protocol, stateids. When an NFSv4 version is used as the data access protocol,
the metadata server may make stateid-related requests of the storage the metadata server may make stateid-related requests of the storage
devices. However, it is not required to do so and the resulting devices. However, it is not required to do so, and the resulting
stateids are known only to the metadata server and the storage stateids are known only to the metadata server and the storage
device. device.
Given this basic structure, locking-related operations are handled as Given this basic structure, locking-related operations are handled as
follows: follows:
o OPENs are dealt with by the metadata server. Stateids are o OPENs are dealt with by the metadata server. Stateids are
selected by the metadata server and associated with the client id selected by the metadata server and associated with the client ID
describing the client's connection to the metadata server. The describing the client's connection to the metadata server. The
metadata server may need to interact with the storage device to metadata server may need to interact with the storage device to
locate the file to be opened, but no locking-related functionality locate the file to be opened, but no locking-related functionality
need be used on the storage device. need be used on the storage device.
OPEN_DOWNGRADE and CLOSE only require local execution on the OPEN_DOWNGRADE and CLOSE only require local execution on the
metadata server. metadata server.
o Advisory byte-range locks can be implemented locally on the o Advisory byte-range locks can be implemented locally on the
metadata server. As in the case of OPENs, the stateids associated metadata server. As in the case of OPENs, the stateids associated
with byte-range locks are assigned by the metadata server and only with byte-range locks are assigned by the metadata server and only
used on the metadata server. used on the metadata server.
o Delegations are assigned by the metadata server which initiates o Delegations are assigned by the metadata server that initiates
recalls when conflicting OPENs are processed. No storage device recalls when conflicting OPENs are processed. No storage device
involvement is required. involvement is required.
o TEST_STATEID and FREE_STATEID are processed locally on the o TEST_STATEID and FREE_STATEID are processed locally on the
metadata server, without storage device involvement. metadata server, without storage device involvement.
All I/O operations to the storage device are done using the anonymous All I/O operations to the storage device are done using the anonymous
stateid. Thus the storage device has no information about the stateid. Thus, the storage device has no information about the
openowner and lockowner responsible for issuing a particular I/O openowner and lockowner responsible for issuing a particular I/O
operation. As a result: operation. As a result:
o Mandatory byte-range locking cannot be supported because the o Mandatory byte-range locking cannot be supported because the
storage device has no way of distinguishing I/O done on behalf of storage device has no way of distinguishing I/O done on behalf of
the lock owner from those done by others. the lock owner from those done by others.
o Enforcement of share reservations is the responsibility of the o Enforcement of share reservations is the responsibility of the
client. Even though I/O is done using the anonymous stateid, the client. Even though I/O is done using the anonymous stateid, the
client must ensure that it has a valid stateid associated with the client must ensure that it has a valid stateid associated with the
openowner, that allows the I/O being done before issuing the I/O. openowner.
In the event that a stateid is revoked, the metadata server is In the event that a stateid is revoked, the metadata server is
responsible for preventing client access, since it has no way of responsible for preventing client access, since it has no way of
being sure that the client is aware that the stateid in question has being sure that the client is aware that the stateid in question has
been revoked. been revoked.
As the client never receives a stateid generated by a storage device, As the client never receives a stateid generated by a storage device,
there is no client lease on the storage device and no prospect of there is no client lease on the storage device and no prospect of
lease expiration, even when access is via NFSv4 protocols. Clients lease expiration, even when access is via NFSv4 protocols. Clients
will have leases on the metadata server. In dealing with lease will have leases on the metadata server. In dealing with lease
skipping to change at page 12, line 26 skipping to change at page 12, line 31
dealt with by the metadata server, which generates the appropriate dealt with by the metadata server, which generates the appropriate
stateids. These stateids must be made known to the storage device stateids. These stateids must be made known to the storage device
using control protocol facilities, the details of which are not using control protocol facilities, the details of which are not
discussed in this document. discussed in this document.
Given this basic structure, locking-related operations are handled as Given this basic structure, locking-related operations are handled as
follows: follows:
o OPENs are dealt with primarily on the metadata server. Stateids o OPENs are dealt with primarily on the metadata server. Stateids
are selected by the metadata server and associated with the client are selected by the metadata server and associated with the client
id describing the client's connection to the metadata server. The ID describing the client's connection to the metadata server. The
metadata server needs to interact with the storage device to metadata server needs to interact with the storage device to
locate the file to be opened, and to make the storage device aware locate the file to be opened and to make the storage device aware
of the association between the metadata-server-chosen stateid and of the association between the metadata-server-chosen stateid and
the client and openowner that it represents. the client and openowner that it represents.
OPEN_DOWNGRADE and CLOSE are executed initially on the metadata OPEN_DOWNGRADE and CLOSE are executed initially on the metadata
server but the state change made must be propagated to the storage server, but the state change made must be propagated to the
device. storage device.
o Advisory byte-range locks can be implemented locally on the o Advisory byte-range locks can be implemented locally on the
metadata server. As in the case of OPENs, the stateids associated metadata server. As in the case of OPENs, the stateids associated
with byte-range locks, are assigned by the metadata server and are with byte-range locks are assigned by the metadata server and are
available for use on the metadata server. Because I/O operations available for use on the metadata server. Because I/O operations
are allowed to present lock stateids, the metadata server needs are allowed to present lock stateids, the metadata server needs
the ability to make the storage device aware of the association the ability to make the storage device aware of the association
between the metadata-server-chosen stateid and the corresponding between the metadata-server-chosen stateid and the corresponding
open stateid it is associated with. open stateid it is associated with.
o Mandatory byte-range locks can be supported when both the metadata o Mandatory byte-range locks can be supported when both the metadata
server and the storage devices have the appropriate support. As server and the storage devices have the appropriate support. As
in the case of advisory byte-range locks, these are assigned by in the case of advisory byte-range locks, these are assigned by
the metadata server and are available for use on the metadata the metadata server and are available for use on the metadata
server. To enable mandatory lock enforcement on the storage server. To enable mandatory lock enforcement on the storage
device, the metadata server needs the ability to make the storage device, the metadata server needs the ability to make the storage
device aware of the association between the metadata-server-chosen device aware of the association between the metadata-server-chosen
stateid and the client, openowner, and lock (i.e., lockowner, stateid and the client, openowner, and lock (i.e., lockowner,
byte-range, lock-type) that it represents. Because I/O operations byte-range, and lock-type) that it represents. Because I/O
are allowed to present lock stateids, this information needs to be operations are allowed to present lock stateids, this information
propagated to all storage devices to which I/O might be directed needs to be propagated to all storage devices to which I/O might
rather than only to storage device that contain the locked region. be directed rather than only to storage device that contain the
locked region.
o Delegations are assigned by the metadata server which initiates o Delegations are assigned by the metadata server that initiates
recalls when conflicting OPENs are processed. Because I/O recalls when conflicting OPENs are processed. Because I/O
operations are allowed to present delegation stateids, the operations are allowed to present delegation stateids, the
metadata server requires the ability to make the storage device metadata server requires the ability (1) to make the storage
aware of the association between the metadata-server-chosen device aware of the association between the metadata-server-chosen
stateid and the filehandle and delegation type it represents, and stateid and the filehandle and delegation type it represents and
to break such an association. (2) to break such an association.
o TEST_STATEID is processed locally on the metadata server, without o TEST_STATEID is processed locally on the metadata server, without
storage device involvement. storage device involvement.
o FREE_STATEID is processed on the metadata server but the metadata o FREE_STATEID is processed on the metadata server, but the metadata
server requires the ability to propagate the request to the server requires the ability to propagate the request to the
corresponding storage devices. corresponding storage devices.
Because the client will possess and use stateids valid on the storage Because the client will possess and use stateids valid on the storage
device, there will be a client lease on the storage device and the device, there will be a client lease on the storage device, and the
possibility of lease expiration does exist. The best approach for possibility of lease expiration does exist. The best approach for
the storage device is to retain these locks as a courtesy. However, the storage device is to retain these locks as a courtesy. However,
if it does not do so, control protocol facilities need to provide the if it does not do so, control protocol facilities need to provide the
means to synchronize lock state between the metadata server and means to synchronize lock state between the metadata server and
storage device. storage device.
Clients will also have leases on the metadata server, which are Clients will also have leases on the metadata server that are subject
subject to expiration. In dealing with lease expiration, the to expiration. In dealing with lease expiration, the metadata server
metadata server would be expected to use control protocol facilities would be expected to use control protocol facilities enabling it to
enabling it to invalidate revoked stateids on the storage device. In invalidate revoked stateids on the storage device. In the event the
the event the client is not responsive, the metadata server may need client is not responsive, the metadata server may need to use fencing
to use fencing to prevent revoked stateids from being acted upon by to prevent revoked stateids from being acted upon by the storage
the storage device. device.
3. XDR Description of the Flexible File Layout Type 3. XDR Description of the Flexible File Layout Type
This document contains the external data representation (XDR) This document contains the External Data Representation (XDR)
[RFC4506] description of the flexible file layout type. The XDR [RFC4506] description of the flexible file layout type. The XDR
description is embedded in this document in a way that makes it description is embedded in this document in a way that makes it
simple for the reader to extract into a ready-to-compile form. The simple for the reader to extract into a ready-to-compile form. The
reader can feed this document into the following shell script to reader can feed this document into the following shell script to
produce the machine readable XDR description of the flexible file produce the machine-readable XDR description of the flexible file
layout type: layout type:
<CODE BEGINS> <CODE BEGINS>
#!/bin/sh #!/bin/sh
grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??' grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'
<CODE ENDS> <CODE ENDS>
That is, if the above script is stored in a file called "extract.sh", That is, if the above script is stored in a file called "extract.sh"
and this document is in a file called "spec.txt", then the reader can and this document is in a file called "spec.txt", then the reader can
do: do:
sh extract.sh < spec.txt > flex_files_prot.x sh extract.sh < spec.txt > flex_files_prot.x
The effect of the script is to remove leading white space from each The effect of the script is to remove leading white space from each
line, plus a sentinel sequence of "///". line, plus a sentinel sequence of "///".
The embedded XDR file header follows. Subsequent XDR descriptions, The embedded XDR file header follows. Subsequent XDR descriptions
with the sentinel sequence are embedded throughout the document. with the sentinel sequence are embedded throughout the document.
Note that the XDR code contained in this document depends on types Note that the XDR code contained in this document depends on types
from the NFSv4.1 nfs4_prot.x file [RFC5662]. This includes both nfs from the NFSv4.1 nfs4_prot.x file [RFC5662]. This includes both nfs
types that end with a 4, such as offset4, length4, etc., as well as types that end with a 4, such as offset4, length4, etc., as well as
more generic types such as uint32_t and uint64_t. more generic types such as uint32_t and uint64_t.
3.1. Code Components Licensing Notice 3.1. Code Components Licensing Notice
Both the XDR description and the scripts used for extracting the XDR Both the XDR description and the scripts used for extracting the XDR
description are Code Components as described in Section 4 of "Legal description are Code Components as described in Section 4 of "Trust
Provisions Relating to IETF Documents" [LEGAL]. These Code Legal Provisions (TLP)" [LEGAL]. These Code Components are licensed
Components are licensed according to the terms of that document. according to the terms of that document.
<CODE BEGINS> <CODE BEGINS>
/// /* /// /*
/// * Copyright (c) 2012 IETF Trust and the persons identified /// * Copyright (c) 2018 IETF Trust and the persons identified
/// * as authors of the code. All rights reserved. /// * as authors of the code. All rights reserved.
/// * /// *
/// * Redistribution and use in source and binary forms, with /// * Redistribution and use in source and binary forms, with
/// * or without modification, are permitted provided that the /// * or without modification, are permitted provided that the
/// * following conditions are met: /// * following conditions are met:
/// * /// *
/// * o Redistributions of source code must retain the above /// * - Redistributions of source code must retain the above
/// * copyright notice, this list of conditions and the /// * copyright notice, this list of conditions and the
/// * following disclaimer. /// * following disclaimer.
/// * /// *
/// * o Redistributions in binary form must reproduce the above /// * - Redistributions in binary form must reproduce the above
/// * copyright notice, this list of conditions and the /// * copyright notice, this list of conditions and the
/// * following disclaimer in the documentation and/or other /// * following disclaimer in the documentation and/or other
/// * materials provided with the distribution. /// * materials provided with the distribution.
/// * /// *
/// * o Neither the name of Internet Society, IETF or IETF /// * - Neither the name of Internet Society, IETF or IETF
/// * Trust, nor the names of specific contributors, may be /// * Trust, nor the names of specific contributors, may be
/// * used to endorse or promote products derived from this /// * used to endorse or promote products derived from this
/// * software without specific prior written permission. /// * software without specific prior written permission.
/// * /// *
/// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS /// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS
/// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED /// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
/// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE /// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
/// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS /// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
/// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO /// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
/// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE /// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
/// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, /// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
/// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT /// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
/// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR /// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
/// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS /// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
/// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF /// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
/// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, /// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
/// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING /// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
/// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF /// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
/// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. /// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
/// * /// *
/// * This code was derived from RFCTBD10. /// * This code was derived from RFC 8435.
/// * Please reproduce this note if possible. /// * Please reproduce this note if possible.
/// */ /// */
/// ///
/// /* /// /*
/// * flex_files_prot.x /// * flex_files_prot.x
/// */ /// */
/// ///
/// /* /// /*
/// * The following include statements are for example only. /// * The following include statements are for example only.
/// * The actual XDR definition files are generated separately /// * The actual XDR definition files are generated separately
skipping to change at page 16, line 8 skipping to change at page 16, line 15
4. Device Addressing and Discovery 4. Device Addressing and Discovery
Data operations to a storage device require the client to know the Data operations to a storage device require the client to know the
network address of the storage device. The NFSv4.1+ GETDEVICEINFO network address of the storage device. The NFSv4.1+ GETDEVICEINFO
operation (Section 18.40 of [RFC5661]) is used by the client to operation (Section 18.40 of [RFC5661]) is used by the client to
retrieve that information. retrieve that information.
4.1. ff_device_addr4 4.1. ff_device_addr4
The ff_device_addr4 data structure is returned by the server as the The ff_device_addr4 data structure is returned by the server as the
layout type specific opaque field da_addr_body in the device_addr4 layout-type-specific opaque field da_addr_body in the device_addr4
structure by a successful GETDEVICEINFO operation. structure by a successful GETDEVICEINFO operation.
<CODE BEGINS> <CODE BEGINS>
/// struct ff_device_versions4 { /// struct ff_device_versions4 {
/// uint32_t ffdv_version; /// uint32_t ffdv_version;
/// uint32_t ffdv_minorversion; /// uint32_t ffdv_minorversion;
/// uint32_t ffdv_rsize; /// uint32_t ffdv_rsize;
/// uint32_t ffdv_wsize; /// uint32_t ffdv_wsize;
/// bool ffdv_tightly_coupled; /// bool ffdv_tightly_coupled;
skipping to change at page 16, line 39 skipping to change at page 16, line 46
The ffda_netaddrs field is used to locate the storage device. It The ffda_netaddrs field is used to locate the storage device. It
MUST be set by the server to a list holding one or more of the device MUST be set by the server to a list holding one or more of the device
network addresses. network addresses.
The ffda_versions array allows the metadata server to present choices The ffda_versions array allows the metadata server to present choices
as to NFS version, minor version, and coupling strength to the as to NFS version, minor version, and coupling strength to the
client. The ffdv_version and ffdv_minorversion represent the NFS client. The ffdv_version and ffdv_minorversion represent the NFS
protocol to be used to access the storage device. This layout protocol to be used to access the storage device. This layout
specification defines the semantics for ffdv_versions 3 and 4. If specification defines the semantics for ffdv_versions 3 and 4. If
ffdv_version equals 3 then the server MUST set ffdv_minorversion to 0 ffdv_version equals 3, then the server MUST set ffdv_minorversion to
and ffdv_tightly_coupled to false. The client MUST then access the 0 and ffdv_tightly_coupled to false. The client MUST then access the
storage device using the NFSv3 protocol [RFC1813]. If ffdv_version storage device using the NFSv3 protocol [RFC1813]. If ffdv_version
equals 4 then the server MUST set ffdv_minorversion to one of the equals 4, then the server MUST set ffdv_minorversion to one of the
NFSv4 minor version numbers and the client MUST access the storage NFSv4 minor version numbers, and the client MUST access the storage
device using NFSv4 with the specified minor version. device using NFSv4 with the specified minor version.
Note that while the client might determine that it cannot use any of Note that while the client might determine that it cannot use any of
the configured combinations of ffdv_version, ffdv_minorversion, and the configured combinations of ffdv_version, ffdv_minorversion, and
ffdv_tightly_coupled, when it gets the device list from the metadata ffdv_tightly_coupled, when it gets the device list from the metadata
server, there is no way to indicate to the metadata server as to server, there is no way to indicate to the metadata server as to
which device it is version incompatible. If however, the client which device it is version incompatible. However, if the client
waits until it retrieves the layout from the metadata server, it can waits until it retrieves the layout from the metadata server, it can
at that time clearly identify the storage device in question (see at that time clearly identify the storage device in question (see
Section 5.4). Section 5.4).
The ffdv_rsize and ffdv_wsize are used to communicate the maximum The ffdv_rsize and ffdv_wsize are used to communicate the maximum
rsize and wsize supported by the storage device. As the storage rsize and wsize supported by the storage device. As the storage
device can have a different rsize or wsize than the metadata server, device can have a different rsize or wsize than the metadata server,
the ffdv_rsize and ffdv_wsize allow the metadata server to the ffdv_rsize and ffdv_wsize allow the metadata server to
communicate that information on behalf of the storage device. communicate that information on behalf of the storage device.
ffdv_tightly_coupled informs the client as to whether the metadata ffdv_tightly_coupled informs the client as to whether or not the
server is tightly coupled with the storage devices or not. Note that metadata server is tightly coupled with the storage devices. Note
even if the data protocol is at least NFSv4.1, it may still be the that even if the data protocol is at least NFSv4.1, it may still be
case that there is loose coupling in effect. If ffdv_tightly_coupled the case that there is loose coupling in effect. If
is not set, then the client MUST commit writes to the storage devices ffdv_tightly_coupled is not set, then the client MUST commit writes
for the file before sending a LAYOUTCOMMIT to the metadata server. to the storage devices for the file before sending a LAYOUTCOMMIT to
I.e., the writes MUST be committed by the client to stable storage the metadata server. That is, the writes MUST be committed by the
via issuing WRITEs with stable_how == FILE_SYNC or by issuing a client to stable storage via issuing WRITEs with stable_how ==
COMMIT after WRITEs with stable_how != FILE_SYNC (see Section 3.3.7 FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how !=
of [RFC1813]). FILE_SYNC (see Section 3.3.7 of [RFC1813]).
4.2. Storage Device Multipathing 4.2. Storage Device Multipathing
The flexible file layout type supports multipathing to multiple The flexible file layout type supports multipathing to multiple
storage device addresses. Storage device level multipathing is used storage device addresses. Storage-device-level multipathing is used
for bandwidth scaling via trunking and for higher availability of use for bandwidth scaling via trunking and for higher availability of use
in the event of a storage device failure. Multipathing allows the in the event of a storage device failure. Multipathing allows the
client to switch to another storage device address which may be that client to switch to another storage device address that may be that
of another storage device that is exporting the same data stripe of another storage device that is exporting the same data stripe
unit, without having to contact the metadata server for a new layout. unit, without having to contact the metadata server for a new layout.
To support storage device multipathing, ffda_netaddrs contains an To support storage device multipathing, ffda_netaddrs contains an
array of one or more storage device network addresses. This array array of one or more storage device network addresses. This array
(data type multipath_list4) represents a list of storage devices (data type multipath_list4) represents a list of storage devices
(each identified by a network address), with the possibility that (each identified by a network address), with the possibility that
some storage device will appear in the list multiple times. some storage device will appear in the list multiple times.
The client is free to use any of the network addresses as a The client is free to use any of the network addresses as a
destination to send storage device requests. If some network destination to send storage device requests. If some network
addresses are less desirable paths to the data than others, then the addresses are less desirable paths to the data than others, then the
metadata server SHOULD NOT include those network addresses in metadata server SHOULD NOT include those network addresses in
ffda_netaddrs. If less desirable network addresses exist to provide ffda_netaddrs. If less desirable network addresses exist to provide
failover, the RECOMMENDED method to offer the addresses is to provide failover, the RECOMMENDED method to offer the addresses is to provide
them in a replacement device-ID-to-device-address mapping, or a them in a replacement device-ID-to-device-address mapping or a
replacement device ID. When a client finds no response from the replacement device ID. When a client finds no response from the
storage device using all addresses available in ffda_netaddrs, it storage device using all addresses available in ffda_netaddrs, it
SHOULD send a GETDEVICEINFO to attempt to replace the existing SHOULD send a GETDEVICEINFO to attempt to replace the existing
device-ID-to-device-address mappings. If the metadata server detects device-ID-to-device-address mappings. If the metadata server detects
that all network paths represented by ffda_netaddrs are unavailable, that all network paths represented by ffda_netaddrs are unavailable,
the metadata server SHOULD send a CB_NOTIFY_DEVICEID (if the client the metadata server SHOULD send a CB_NOTIFY_DEVICEID (if the client
has indicated it wants device ID notifications for changed device has indicated it wants device ID notifications for changed device
IDs) to change the device-ID-to-device-address mappings to the IDs) to change the device-ID-to-device-address mappings to the
available addresses. If the device ID itself will be replaced, the available addresses. If the device ID itself will be replaced, the
metadata server SHOULD recall all layouts with the device ID, and metadata server SHOULD recall all layouts with the device ID and thus
thus force the client to get new layouts and device ID mappings via force the client to get new layouts and device ID mappings via
LAYOUTGET and GETDEVICEINFO. LAYOUTGET and GETDEVICEINFO.
Generally, if two network addresses appear in ffda_netaddrs, they Generally, if two network addresses appear in ffda_netaddrs, they
will designate the same storage device. When the storage device is will designate the same storage device. When the storage device is
accessed over NFSv4.1 or a higher minor version, the two storage accessed over NFSv4.1 or a higher minor version, the two storage
device addresses will support the implementation of client ID or device addresses will support the implementation of client ID or
session trunking (the latter is RECOMMENDED) as defined in [RFC5661]. session trunking (the latter is RECOMMENDED) as defined in [RFC5661].
The two storage device addresses will share the same server owner or The two storage device addresses will share the same server owner or
major ID of the server owner. It is not always necessary for the two major ID of the server owner. It is not always necessary for the two
storage device addresses to designate the same storage device with storage device addresses to designate the same storage device with
trunking being used. For example, the data could be read-only, and trunking being used. For example, the data could be read-only, and
the data consist of exact replicas. the data consist of exact replicas.
5. Flexible File Layout Type 5. Flexible File Layout Type
The layout4 type is defined in [RFC5662] as follows: The original layouttype4 introduced in [RFC5662] is modified to be:
<CODE BEGINS> <CODE BEGINS>
enum layouttype4 { enum layouttype4 {
LAYOUT4_NFSV4_1_FILES = 1, LAYOUT4_NFSV4_1_FILES = 1,
LAYOUT4_OSD2_OBJECTS = 2, LAYOUT4_OSD2_OBJECTS = 2,
LAYOUT4_BLOCK_VOLUME = 3, LAYOUT4_BLOCK_VOLUME = 3,
LAYOUT4_FLEX_FILES = 4 LAYOUT4_FLEX_FILES = 4
[[RFC Editor: please modify the LAYOUT4_FLEX_FILES
to be the layouttype assigned by IANA]]
}; };
struct layout_content4 { struct layout_content4 {
layouttype4 loc_type; layouttype4 loc_type;
opaque loc_body<>; opaque loc_body<>;
}; };
struct layout4 { struct layout4 {
offset4 lo_offset; offset4 lo_offset;
length4 lo_length; length4 lo_length;
layoutiomode4 lo_iomode; layoutiomode4 lo_iomode;
layout_content4 lo_content; layout_content4 lo_content;
}; };
<CODE ENDS> <CODE ENDS>
This document defines structures associated with the layouttype4 This document defines structures associated with the layouttype4
value LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure value LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure
as an XDR type "opaque". The opaque layout is uninterpreted by the as an XDR type "opaque". The opaque layout is uninterpreted by the
generic pNFS client layers, but is interpreted by the flexible file generic pNFS client layers but is interpreted by the flexible file
layout type implementation. This section defines the structure of layout type implementation. This section defines the structure of
this otherwise opaque value, ff_layout4. this otherwise opaque value, ff_layout4.
5.1. ff_layout4 5.1. ff_layout4
<CODE BEGINS> <CODE BEGINS>
/// const FF_FLAGS_NO_LAYOUTCOMMIT = 0x00000001; /// const FF_FLAGS_NO_LAYOUTCOMMIT = 0x00000001;
/// const FF_FLAGS_NO_IO_THRU_MDS = 0x00000002; /// const FF_FLAGS_NO_IO_THRU_MDS = 0x00000002;
/// const FF_FLAGS_NO_READ_IO = 0x00000004; /// const FF_FLAGS_NO_READ_IO = 0x00000004;
skipping to change at page 20, line 4 skipping to change at page 20, line 13
/// uint32_t ffl_stats_collect_hint; /// uint32_t ffl_stats_collect_hint;
/// }; /// };
/// ///
<CODE ENDS> <CODE ENDS>
The ff_layout4 structure specifies a layout in that portion of the The ff_layout4 structure specifies a layout in that portion of the
data file described in the current layout segment. It is either a data file described in the current layout segment. It is either a
single instance or a set of mirrored copies of that portion of the single instance or a set of mirrored copies of that portion of the
data file. When mirroring is in effect, it protects against loss of data file. When mirroring is in effect, it protects against loss of
data in layout segments. Note that while not explicitly shown in the data in layout segments.
above XDR, each layout4 element returned in the logr_layout array of
LAYOUTGET4res (see Section 18.43.1 of [RFC5661]) describes a layout
segment. Hence each ff_layout4 also describes a layout segment.
It is possible that the file is concatenated from more than one While not explicitly shown in the above XDR, each layout4 element
layout segment. Each layout segment MAY represent different striping returned in the logr_layout array of LAYOUTGET4res (see
parameters, applying respectively only to the layout segment byte Section 18.43.2 of [RFC5661]) describes a layout segment. Hence,
range. each ff_layout4 also describes a layout segment. It is possible that
the file is concatenated from more than one layout segment. Each
layout segment MAY represent different striping parameters.
The ffl_stripe_unit field is the stripe unit size in use for the The ffl_stripe_unit field is the stripe unit size in use for the
current layout segment. The number of stripes is given inside each current layout segment. The number of stripes is given inside each
mirror by the number of elements in ffm_data_servers. If the number mirror by the number of elements in ffm_data_servers. If the number
of stripes is one, then the value for ffl_stripe_unit MUST default to of stripes is one, then the value for ffl_stripe_unit MUST default to
zero. The only supported mapping scheme is sparse and is detailed in zero. The only supported mapping scheme is sparse and is detailed in
Section 6. Note that there is an assumption here that both the Section 6. Note that there is an assumption here that both the
stripe unit size and the number of stripes is the same across all stripe unit size and the number of stripes are the same across all
mirrors. mirrors.
The ffl_mirrors field is the array of mirrored storage devices which The ffl_mirrors field is the array of mirrored storage devices that
provide the storage for the current stripe, see Figure 1. provide the storage for the current stripe; see Figure 1.
The ffl_stats_collect_hint field provides a hint to the client on how The ffl_stats_collect_hint field provides a hint to the client on how
often the server wants it to report LAYOUTSTATS for a file. The time often the server wants it to report LAYOUTSTATS for a file. The time
is in seconds. is in seconds.
+-----------+ +-----------+
| | | |
| | | |
| File | | File |
| | | |
skipping to change at page 20, line 51 skipping to change at page 21, line 26
| Mirror 1 | | Mirror 2 | | Mirror 1 | | Mirror 2 |
+----+-----+ +-----+----+ +----+-----+ +-----+----+
| | | |
+-----------+ +-----------+ +-----------+ +-----------+
|+-----------+ |+-----------+ |+-----------+ |+-----------+
||+-----------+ ||+-----------+ ||+-----------+ ||+-----------+
+|| Storage | +|| Storage | +|| Storage | +|| Storage |
+| Devices | +| Devices | +| Devices | +| Devices |
+-----------+ +-----------+ +-----------+ +-----------+
Figure 1 Figure 1
The ffs_mirrors field represents an array of state information for The ffs_mirrors field represents an array of state information for
each mirrored copy of the current layout segment. Each element is each mirrored copy of the current layout segment. Each element is
described by a ff_mirror4 type. described by a ff_mirror4 type.
ffds_deviceid provides the deviceid of the storage device holding the ffds_deviceid provides the deviceid of the storage device holding the
data file. data file.
ffds_fh_vers is an array of filehandles of the data file matching to ffds_fh_vers is an array of filehandles of the data file matching the
the available NFS versions on the given storage device. There MUST available NFS versions on the given storage device. There MUST be
be exactly as many elements in ffds_fh_vers as there are in exactly as many elements in ffds_fh_vers as there are in
ffda_versions. Each element of the array corresponds to a particular ffda_versions. Each element of the array corresponds to a particular
combination of ffdv_version, ffdv_minorversion, and combination of ffdv_version, ffdv_minorversion, and
ffdv_tightly_coupled provided for the device. The array allows for ffdv_tightly_coupled provided for the device. The array allows for
server implementations which have different filehandles for different server implementations that have different filehandles for different
combinations of version, minor version, and coupling strength. See combinations of version, minor version, and coupling strength. See
Section 5.4 for how to handle versioning issues between the client Section 5.4 for how to handle versioning issues between the client
and storage devices. and storage devices.
For tight coupling, ffds_stateid provides the stateid to be used by For tight coupling, ffds_stateid provides the stateid to be used by
the client to access the file. For loose coupling and a NFSv4 the client to access the file. For loose coupling and an NFSv4
storage device, the client will have to use an anonymous stateid to storage device, the client will have to use an anonymous stateid to
perform I/O on the storage device. With no control protocol, the perform I/O on the storage device. With no control protocol, the
metadata server stateid can not be used to provide a global stateid metadata server stateid cannot be used to provide a global stateid
model. Thus the server MUST set the ffds_stateid to be the anonymous model. Thus, the server MUST set the ffds_stateid to be the
stateid. anonymous stateid.
This specification of the ffds_stateid restricts both models for This specification of the ffds_stateid restricts both models for
NFSv4.x storage protocols: NFSv4.x storage protocols:
loosely couple: the stateid has to be an anonymous stateid, loosely coupled model: the stateid has to be an anonymous stateid
tightly couple: the stateid has to be a global stateid. tightly coupled model: the stateid has to be a global stateid
A number of issues stem from a mismatch between the fact that A number of issues stem from a mismatch between the fact that
ffds_stateid is defined as a single item while ffds_fh_vers is ffds_stateid is defined as a single item while ffds_fh_vers is
defined as an array. It is possible for each open file on the defined as an array. It is possible for each open file on the
storage device to require its own open stateid. Because there are storage device to require its own open stateid. Because there are
established loosely coupled implementations of the version of the established loosely coupled implementations of the version of the
protocol described in this document, such potential issues have not protocol described in this document, such potential issues have not
been addressed here. It is possible for future layout types to be been addressed here. It is possible for future layout types to be
defined that address these issues, should it become important to defined that address these issues, should it become important to
provide multiple stateids for the same underlying file. provide multiple stateids for the same underlying file.
For loosely coupled storage devices, ffds_user and ffds_group provide For loosely coupled storage devices, ffds_user and ffds_group provide
the synthetic user and group to be used in the RPC credentials that the synthetic user and group to be used in the RPC credentials that
the client presents to the storage device to access the data files. the client presents to the storage device to access the data files.
For tightly coupled storage devices, the user and group on the For tightly coupled storage devices, the user and group on the
storage device will be the same as on the metadata server. I.e., if storage device will be the same as on the metadata server; that is,
ffdv_tightly_coupled (see Section 4.1) is set, then the client MUST if ffdv_tightly_coupled (see Section 4.1) is set, then the client
ignore both ffds_user and ffds_group. MUST ignore both ffds_user and ffds_group.
The allowed values for both ffds_user and ffds_group are specified in The allowed values for both ffds_user and ffds_group are specified as
Section 5.9 of [RFC5661]. For NFSv3 compatibility, user and group owner and owner_group, respectively, in Section 5.9 of [RFC5661].
strings that consist of decimal numeric values with no leading zeros For NFSv3 compatibility, user and group strings that consist of
can be given a special interpretation by clients and servers that decimal numeric values with no leading zeros can be given a special
choose to provide such support. The receiver may treat such a user interpretation by clients and servers that choose to provide such
or group string as representing the same user as would be represented support. The receiver may treat such a user or group string as
by an NFSv3 uid or gid having the corresponding numeric value. Note representing the same user as would be represented by an NFSv3 uid or
that if using Kerberos for security, the expectation is that these gid having the corresponding numeric value. Note that if using
values will be a name@domain string. Kerberos for security, the expectation is that these values will be a
name@domain string.
ffds_efficiency describes the metadata server's evaluation as to the ffds_efficiency describes the metadata server's evaluation as to the
effectiveness of each mirror. Note that this is per layout and not effectiveness of each mirror. Note that this is per layout and not
per device as the metric may change due to perceived load, per device as the metric may change due to perceived load,
availability to the metadata server, etc. Higher values denote availability to the metadata server, etc. Higher values denote
higher perceived utility. The way the client can select the best higher perceived utility. The way the client can select the best
mirror to access is discussed in Section 8.1. mirror to access is discussed in Section 8.1.
ffl_flags is a bitmap that allows the metadata server to inform the ffl_flags is a bitmap that allows the metadata server to inform the
client of particular conditions that may result from the more or less client of particular conditions that may result from more or less
tight coupling of the storage devices. tight coupling of the storage devices.
FF_FLAGS_NO_LAYOUTCOMMIT: can be set to indicate that the client is FF_FLAGS_NO_LAYOUTCOMMIT: can be set to indicate that the client is
not required to send LAYOUTCOMMIT to the metadata server. not required to send LAYOUTCOMMIT to the metadata server.
F_FLAGS_NO_IO_THRU_MDS: can be set to indicate that the client FF_FLAGS_NO_IO_THRU_MDS: can be set to indicate that the client
should not send I/O operations to the metadata server. I.e., even should not send I/O operations to the metadata server. That is,
if the client could determine that there was a network disconnect even if the client could determine that there was a network
to a storage device, the client should not try to proxy the I/O disconnect to a storage device, the client should not try to proxy
through the metadata server. the I/O through the metadata server.
FF_FLAGS_NO_READ_IO: can be set to indicate that the client should FF_FLAGS_NO_READ_IO: can be set to indicate that the client should
not send READ requests with the layouts of iomode not send READ requests with the layouts of iomode
LAYOUTIOMODE4_RW. Instead, it should request a layout of iomode LAYOUTIOMODE4_RW. Instead, it should request a layout of iomode
LAYOUTIOMODE4_READ from the metadata server. LAYOUTIOMODE4_READ from the metadata server.
FF_FLAGS_WRITE_ONE_MIRROR: can be set to indicate that the client FF_FLAGS_WRITE_ONE_MIRROR: can be set to indicate that the client
only needs to update one of the mirrors (see Section 8.2). only needs to update one of the mirrors (see Section 8.2).
5.1.1. Error Codes from LAYOUTGET 5.1.1. Error Codes from LAYOUTGET
[RFC5661] provides little guidance as to how the client is to proceed [RFC5661] provides little guidance as to how the client is to proceed
with a LAYOUTGET which returns an error of either with a LAYOUTGET that returns an error of either
NFS4ERR_LAYOUTTRYLATER, NFS4ERR_LAYOUTUNAVAILABLE, and NFS4ERR_DELAY. NFS4ERR_LAYOUTTRYLATER, NFS4ERR_LAYOUTUNAVAILABLE, and NFS4ERR_DELAY.
Within the context of this document: Within the context of this document:
NFS4ERR_LAYOUTUNAVAILABLE: there is no layout available and the I/O NFS4ERR_LAYOUTUNAVAILABLE: there is no layout available and the I/O
is to go to the metadata server. Note that it is possible to have is to go to the metadata server. Note that it is possible to have
had a layout before a recall and not after. had a layout before a recall and not after.
NFS4ERR_LAYOUTTRYLATER: there is some issue preventing the layout NFS4ERR_LAYOUTTRYLATER: there is some issue preventing the layout
from being granted. If the client already has an appropriate from being granted. If the client already has an appropriate
layout, it should continue with I/O to the storage devices. layout, it should continue with I/O to the storage devices.
NFS4ERR_DELAY: there is some issue preventing the layout from being NFS4ERR_DELAY: there is some issue preventing the layout from being
granted. If the client already has an appropriate layout, it granted. If the client already has an appropriate layout, it
should not continue with I/O to the storage devices. should not continue with I/O to the storage devices.
5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS
Even if the metadata server provides the FF_FLAGS_NO_IO_THRU_MDS, Even if the metadata server provides the FF_FLAGS_NO_IO_THRU_MDS
flag, the client can still perform I/O to the metadata server. The flag, the client can still perform I/O to the metadata server. The
flag functions as a hint. The flag indicates to the client that the flag functions as a hint. The flag indicates to the client that the
metadata server prefers to separate the metadata I/O from the data I/ metadata server prefers to separate the metadata I/O from the data I/
O, most likely for peformance reasons. O, most likely for performance reasons.
5.2. LAYOUTCOMMIT 5.2. LAYOUTCOMMIT
The flex file layout does not use lou_body. If lou_type is The flexible file layout does not use lou_body inside the
LAYOUT4_FLEX_FILES, the lou_body field MUST have a zero length. loca_layoutupdate argument to LAYOUTCOMMIT. If lou_type is
LAYOUT4_FLEX_FILES, the lou_body field MUST have a zero length (see
Section 18.42.1 of [RFC5661]).
5.3. Interactions Between Devices and Layouts 5.3. Interactions between Devices and Layouts
In [RFC5661], the file layout type is defined such that the In [RFC5661], the file layout type is defined such that the
relationship between multipathing and filehandles can result in relationship between multipathing and filehandles can result in
either 0, 1, or N filehandles (see Section 13.3). Some rationales either 0, 1, or N filehandles (see Section 13.3). Some rationales
for this are clustered servers which share the same filehandle or for this are clustered servers that share the same filehandle or
allowing for multiple read-only copies of the file on the same allow for multiple read-only copies of the file on the same storage
storage device. In the flexible file layout type, while there is an device. In the flexible file layout type, while there is an array of
array of filehandles, they are independent of the multipathing being filehandles, they are independent of the multipathing being used. If
used. If the metadata server wants to provide multiple read-only the metadata server wants to provide multiple read-only copies of the
copies of the same file on the same storage device, then it should same file on the same storage device, then it should provide multiple
provide multiple ff_device_addr4, each as a mirror. The client can mirrored instances, each with a different ff_device_addr4. The
then determine that since the ffds_fh_vers are different, then there client can then determine that, since the each of the ffds_fh_vers
are multiple copies of the file for the current layout segment are different, there are multiple copies of the file for the current
available. layout segment available.
5.4. Handling Version Errors 5.4. Handling Version Errors
When the metadata server provides the ffda_versions array in the When the metadata server provides the ffda_versions array in the
ff_device_addr4 (see Section 4.1), the client is able to determine if ff_device_addr4 (see Section 4.1), the client is able to determine
it can not access a storage device with any of the supplied whether or not it can access a storage device with any of the
combinations of ffdv_version, ffdv_minorversion, and supplied combinations of ffdv_version, ffdv_minorversion, and
ffdv_tightly_coupled. However, due to the limitations of reporting ffdv_tightly_coupled. However, due to the limitations of reporting
errors in GETDEVICEINFO (see Section 18.40 in [RFC5661], the client errors in GETDEVICEINFO (see Section 18.40 in [RFC5661]), the client
is not able to specify which specific device it can not communicate is not able to specify which specific device it cannot communicate
with over one of the provided ffdv_version and ffdv_minorversion with over one of the provided ffdv_version and ffdv_minorversion
combinations. Using ff_ioerr4 (see Section 9.1.1 inside either the combinations. Using ff_ioerr4 (see Section 9.1.1) inside either the
LAYOUTRETURN (see Section 18.44 of [RFC5661]) or the LAYOUTERROR (see LAYOUTRETURN (see Section 18.44 of [RFC5661]) or the LAYOUTERROR (see
Section 15.6 of [RFC7862] and Section 10 of this document), the Section 15.6 of [RFC7862] and Section 10 of this document), the
client can isolate the problematic storage device. client can isolate the problematic storage device.
The error code to return for LAYOUTRETURN and/or LAYOUTERROR is The error code to return for LAYOUTRETURN and/or LAYOUTERROR is
NFS4ERR_MINOR_VERS_MISMATCH. It does not matter whether the mismatch NFS4ERR_MINOR_VERS_MISMATCH. It does not matter whether the mismatch
is a major version (e.g., client can use NFSv3 but not NFSv4) or is a major version (e.g., client can use NFSv3 but not NFSv4) or
minor version (e.g., client can use NFSv4.1 but not NFSv4.2), the minor version (e.g., client can use NFSv4.1 but not NFSv4.2), the
error indicates that for all the supplied combinations for error indicates that for all the supplied combinations for
ffdv_version and ffdv_minorversion, the client can not communicate ffdv_version and ffdv_minorversion, the client cannot communicate
with the storage device. The client can retry the GETDEVICEINFO to with the storage device. The client can retry the GETDEVICEINFO to
see if the metadata server can provide a different combination or it see if the metadata server can provide a different combination, or it
can fall back to doing the I/O through the metadata server. can fall back to doing the I/O through the metadata server.
6. Striping via Sparse Mapping 6. Striping via Sparse Mapping
While other layout types support both dense and sparse mapping of While other layout types support both dense and sparse mapping of
logical offsets to physical offsets within a file (see for example logical offsets to physical offsets within a file (see, for example,
Section 13.4 of [RFC5661]), the flexible file layout type only Section 13.4 of [RFC5661]), the flexible file layout type only
supports a sparse mapping. supports a sparse mapping.
With sparse mappings, the logical offset within a file (L) is also With sparse mappings, the logical offset within a file (L) is also
the physical offset on the storage device. As detailed in the physical offset on the storage device. As detailed in
Section 13.4.4 of [RFC5661], this results in holes across each Section 13.4.4 of [RFC5661], this results in holes across each
storage device which does not contain the current stripe index. storage device that does not contain the current stripe index.
L: logical offset into the file L: logical offset within the file
W: stripe width W: stripe width
W = number of elements in ffm_data_servers W = number of elements in ffm_data_servers
S: number of bytes in a stripe S: number of bytes in a stripe
S = W * ffl_stripe_unit S = W * ffl_stripe_unit
N: stripe number N: stripe number
N = L / S N = L / S
skipping to change at page 25, line 24 skipping to change at page 26, line 5
the server MUST complete recovery before handing out any new layouts the server MUST complete recovery before handing out any new layouts
to the affected byte ranges. to the affected byte ranges.
Although the client implementation has the option to propagate a Although the client implementation has the option to propagate a
corresponding error to the application that initiated the I/O corresponding error to the application that initiated the I/O
operation and drop any unwritten data, the client should attempt to operation and drop any unwritten data, the client should attempt to
retry the original I/O operation by either requesting a new layout or retry the original I/O operation by either requesting a new layout or
sending the I/O via regular NFSv4.1+ READ or WRITE operations to the sending the I/O via regular NFSv4.1+ READ or WRITE operations to the
metadata server. The client SHOULD attempt to retrieve a new layout metadata server. The client SHOULD attempt to retrieve a new layout
and retry the I/O operation using the storage device first and only and retry the I/O operation using the storage device first and only
if the error persists, retry the I/O operation via the metadata retry the I/O operation via the metadata server if the error
server. persists.
8. Mirroring 8. Mirroring
The flexible file layout type has a simple model in place for the The flexible file layout type has a simple model in place for the
mirroring of the file data constrained by a layout segment. There is mirroring of the file data constrained by a layout segment. There is
no assumption that each copy of the mirror is stored identically on no assumption that each copy of the mirror is stored identically on
the storage devices. For example, one device might employ the storage devices. For example, one device might employ
compression or deduplication on the data. However, the over the wire compression or deduplication on the data. However, the over-the-wire
transfer of the file contents MUST appear identical. Note, this is a transfer of the file contents MUST appear identical. Note, this is a
constraint of the selected XDR representation in which each mirrored constraint of the selected XDR representation in which each mirrored
copy of the layout segment has the same striping pattern (see copy of the layout segment has the same striping pattern (see
Figure 1). Figure 1).
The metadata server is responsible for determining the number of The metadata server is responsible for determining the number of
mirrored copies and the location of each mirror. While the client mirrored copies and the location of each mirror. While the client
may provide a hint to how many copies it wants (see Section 12), the may provide a hint to how many copies it wants (see Section 12), the
metadata server can ignore that hint and in any event, the client has metadata server can ignore that hint; in any event, the client has no
no means to dictate either the storage device (which also means the means to dictate either the storage device (which also means the
coupling and/or protocol levels to access the layout segments) or the coupling and/or protocol levels to access the layout segments) or the
location of said storage device. location of said storage device.
The updating of mirrored layout segments is done via client-side The updating of mirrored layout segments is done via client-side
mirroring. With this approach, the client is responsible for making mirroring. With this approach, the client is responsible for making
sure modifications are made on all copies of the layout segments it sure modifications are made on all copies of the layout segments it
is informed of via the layout. If a layout segment is being is informed of via the layout. If a layout segment is being
resilvered to a storage device, that mirrored copy will not be in the resilvered to a storage device, that mirrored copy will not be in the
layout. Thus the metadata server MUST update that copy until the layout. Thus, the metadata server MUST update that copy until the
client is presented it in a layout. If the FF_FLAGS_WRITE_ONE_MIRROR client is presented it in a layout. If the FF_FLAGS_WRITE_ONE_MIRROR
is set in ffl_flags, the client need only update one of the mirrors is set in ffl_flags, the client need only update one of the mirrors
(see Section 8.2). If the client is writing to the layout segments (see Section 8.2). If the client is writing to the layout segments
via the metadata server, then the metadata server MUST update all via the metadata server, then the metadata server MUST update all
copies of the mirror. As seen in Section 8.3, during the copies of the mirror. As seen in Section 8.3, during the
resilvering, the layout is recalled, and the client has to make resilvering, the layout is recalled, and the client has to make
modifications via the metadata server. modifications via the metadata server.
8.1. Selecting a Mirror 8.1. Selecting a Mirror
skipping to change at page 26, line 27 skipping to change at page 27, line 9
arrives at the storage devices via the ffds_efficiency member. While arrives at the storage devices via the ffds_efficiency member. While
the algorithms to calculate that value are left to the metadata the algorithms to calculate that value are left to the metadata
server implementations, factors that could contribute to that server implementations, factors that could contribute to that
calculation include speed of the storage device, physical memory calculation include speed of the storage device, physical memory
available to the device, operating system version, current load, etc. available to the device, operating system version, current load, etc.
However, what should not be involved in that calculation is a However, what should not be involved in that calculation is a
perceived network distance between the client and the storage device. perceived network distance between the client and the storage device.
The client is better situated for making that determination based on The client is better situated for making that determination based on
past interaction with the storage device over the different available past interaction with the storage device over the different available
network interfaces between the two. I.e., the metadata server might network interfaces between the two; that is, the metadata server
not know about a transient outage between the client and storage might not know about a transient outage between the client and
device because it has no presence on the given subnet. storage device because it has no presence on the given subnet.
As such, it is the client which decides which mirror to access for As such, it is the client that decides which mirror to access for
reading the file. The requirements for writing to mirrored layout reading the file. The requirements for writing to mirrored layout
segments are presented below. segments are presented below.
8.2. Writing to Mirrors 8.2. Writing to Mirrors
8.2.1. Single Storage Device Updates Mirrors 8.2.1. Single Storage Device Updates Mirrors
If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is set, the client If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is set, the client
only needs to update one of the copies of the layout segment. For only needs to update one of the copies of the layout segment. For
this case, the storage device MUST ensure that all copies of the this case, the storage device MUST ensure that all copies of the
mirror are updated when any one of the mirrors is updated. If the mirror are updated when any one of the mirrors is updated. If the
storage device gets an error when updating one of the mirrors, then storage device gets an error when updating one of the mirrors, then
it MUST inform the client that the original WRITE had an error. The it MUST inform the client that the original WRITE had an error. The
client then MUST inform the metadata server (see Section 8.2.3). The client then MUST inform the metadata server (see Section 8.2.3). The
client's responsibility with respect to COMMIT is explained in client's responsibility with respect to COMMIT is explained in
Section 8.2.4. The client may choose any one of the mirrors and may Section 8.2.4. The client may choose any one of the mirrors and may
use ffds_efficiency in the same manner as for reading when making use ffds_efficiency as described in Section 8.1 when making this
this choice. choice.
8.2.2. Client Updates All Mirrors 8.2.2. Client Updates All Mirrors
If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is not set, the If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is not set, the
client is responsible for updating all mirrored copies of the layout client is responsible for updating all mirrored copies of the layout
segments that it is given in the layout. A single failed update is segments that it is given in the layout. A single failed update is
sufficient to fail the entire operation. If all but one copy is sufficient to fail the entire operation. If all but one copy is
updated successfully and the last one provides an error, then the updated successfully and the last one provides an error, then the
client needs to inform the metadata server about the error via either client needs to inform the metadata server about the error. The
LAYOUTRETURN or LAYOUTERROR that the update failed to that storage client can use either LAYOUTRETURN or LAYOUTERROR to inform the
device. If the client is updating the mirrors serially, then it metadata server that the update failed to that storage device. If
SHOULD stop at the first error encountered and report that to the the client is updating the mirrors serially, then it SHOULD stop at
metadata server. If the client is updating the mirrors in parallel, the first error encountered and report that to the metadata server.
then it SHOULD wait until all storage devices respond such that it If the client is updating the mirrors in parallel, then it SHOULD
can report all errors encountered during the update. wait until all storage devices respond so that it can report all
errors encountered during the update.
8.2.3. Handling Write Errors 8.2.3. Handling Write Errors
When the client reports a write error to the metadata server, the When the client reports a write error to the metadata server, the
metadata server is responsible for determining if it wants to remove metadata server is responsible for determining if it wants to remove
the errant mirror from the layout, if the mirror has recovered from the errant mirror from the layout, if the mirror has recovered from
some transient error, etc. When the client tries to get a new some transient error, etc. When the client tries to get a new
layout, the metadata server informs it of the decision by the layout, the metadata server informs it of the decision by the
contents of the layout. The client MUST NOT make any assumptions contents of the layout. The client MUST NOT assume that the contents
that the contents of the previous layout will match those of the new of the previous layout will match those of the new one. If it has
one. If it has updates that were not committed to all mirrors, then updates that were not committed to all mirrors, then it MUST resend
it MUST resend those updates to all mirrors. those updates to all mirrors.
There is no provision in the protocol for the metadata server to There is no provision in the protocol for the metadata server to
directly determine that the client has or has not recovered from an directly determine that the client has or has not recovered from an
error. I.e., assume that the storage device was network partitioned error. For example, if a storage device was network partitioned from
from the client and all of the copies are successfully updated after the client and the client reported the error to the metadata server,
the error was reported. There is no mechanism for the client to then the network partition would be repaired, and all of the copies
report that fact and the metadata server is forced to repair the file would be successfully updated. There is no mechanism for the client
across the mirror. to report that fact, and the metadata server is forced to repair the
file across the mirror.
If the client supports NFSv4.2, it can use LAYOUTERROR and If the client supports NFSv4.2, it can use LAYOUTERROR and
LAYOUTRETURN to provide hints to the metadata server about the LAYOUTRETURN to provide hints to the metadata server about the
recovery efforts. A LAYOUTERROR on a file is for a non-fatal error. recovery efforts. A LAYOUTERROR on a file is for a non-fatal error.
A subsequent LAYOUTRETURN without a ff_ioerr4 indicates that the A subsequent LAYOUTRETURN without a ff_ioerr4 indicates that the
client successfully replayed the I/O to all mirrors. Any client successfully replayed the I/O to all mirrors. Any
LAYOUTRETURN with a ff_ioerr4 is an error that the metadata server LAYOUTRETURN with a ff_ioerr4 is an error that the metadata server
needs to repair. The client MUST be prepared for the LAYOUTERROR to needs to repair. The client MUST be prepared for the LAYOUTERROR to
trigger a CB_LAYOUTRECALL if the metadata server determines it needs trigger a CB_LAYOUTRECALL if the metadata server determines it needs
to start repairing the file. to start repairing the file.
8.2.4. Handling Write COMMITs 8.2.4. Handling Write COMMITs
When stable writes are done to the metadata server or to a single When stable writes are done to the metadata server or to a single
replica (if allowed by the use of FF_FLAGS_WRITE_ONE_MIRROR ), it is replica (if allowed by the use of FF_FLAGS_WRITE_ONE_MIRROR), it is
the responsibility of the receiving node to propagate the written the responsibility of the receiving node to propagate the written
data stably, before replying to the client. data stably, before replying to the client.
In the corresponding cases in which unstable writes are done, the In the corresponding cases in which unstable writes are done, the
receiving node does not have any such obligation, although it may receiving node does not have any such obligation, although it may
choose to asynchronously propagate the updates. However, once a choose to asynchronously propagate the updates. However, once a
COMMIT is replied to, all replicas must reflect the writes that have COMMIT is replied to, all replicas must reflect the writes that have
been done, and this data must have been committed to stable storage been done, and this data must have been committed to stable storage
on all replicas. on all replicas.
In order to avoid situations in which stale data is read from In order to avoid situations in which stale data is read from
replicas to which writes have not been propagated: replicas to which writes have not been propagated:
o A client which has outstanding unstable writes made to single node o A client that has outstanding unstable writes made to single node
(metadata server or storage device) MUST do all reads from that (metadata server or storage device) MUST do all reads from that
same node. same node.
o When writes are flushed to the server, for example to implement, o When writes are flushed to the server (for example, to implement
close-to-open semantics, a COMMIT must be done by the client to close-to-open semantics), a COMMIT must be done by the client to
ensure that up-to-date written data will be available irrespective ensure that up-to-date written data will be available irrespective
of the particular replica read. of the particular replica read.
8.3. Metadata Server Resilvering of the File 8.3. Metadata Server Resilvering of the File
The metadata server may elect to create a new mirror of the layout The metadata server may elect to create a new mirror of the layout
segments at any time. This might be to resilver a copy on a storage segments at any time. This might be to resilver a copy on a storage
device which was down for servicing, to provide a copy of the layout device that was down for servicing, to provide a copy of the layout
segments on storage with different storage performance segments on storage with different storage performance
characteristics, etc. As the client will not be aware of the new characteristics, etc. As the client will not be aware of the new
mirror and the metadata server will not be aware of updates that the mirror and the metadata server will not be aware of updates that the
client is making to the layout segments, the metadata server MUST client is making to the layout segments, the metadata server MUST
recall the writable layout segment(s) that it is resilvering. If the recall the writable layout segment(s) that it is resilvering. If the
client issues a LAYOUTGET for a writable layout segment which is in client issues a LAYOUTGET for a writable layout segment that is in
the process of being resilvered, then the metadata server can deny the process of being resilvered, then the metadata server can deny
that request with a NFS4ERR_LAYOUTUNAVAILABLE. The client would then that request with an NFS4ERR_LAYOUTUNAVAILABLE. The client would
have to perform the I/O through the metadata server. then have to perform the I/O through the metadata server.
9. Flexible Files Layout Type Return 9. Flexible File Layout Type Return
layoutreturn_file4 is used in the LAYOUTRETURN operation to convey layoutreturn_file4 is used in the LAYOUTRETURN operation to convey
layout-type specific information to the server. It is defined in layout-type-specific information to the server. It is defined in
Section 18.44.1 of [RFC5661] as follows: Section 18.44.1 of [RFC5661] as follows:
<CODE BEGINS> <CODE BEGINS>
/* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */ /* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */
const LAYOUT4_RET_REC_FILE = 1; const LAYOUT4_RET_REC_FILE = 1;
const LAYOUT4_RET_REC_FSID = 2; const LAYOUT4_RET_REC_FSID = 2;
const LAYOUT4_RET_REC_ALL = 3; const LAYOUT4_RET_REC_ALL = 3;
enum layoutreturn_type4 { enum layoutreturn_type4 {
LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE, LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE,
LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID, LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID,
LAYOUTRETURN4_ALL = LAYOUT4_RET_REC_ALL LAYOUTRETURN4_ALL = LAYOUT4_RET_REC_ALL
}; };
skipping to change at page 29, line 33 skipping to change at page 30, line 20
union layoutreturn4 switch(layoutreturn_type4 lr_returntype) { union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
case LAYOUTRETURN4_FILE: case LAYOUTRETURN4_FILE:
layoutreturn_file4 lr_layout; layoutreturn_file4 lr_layout;
default: default:
void; void;
}; };
struct LAYOUTRETURN4args { struct LAYOUTRETURN4args {
/* CURRENT_FH: file */ /* CURRENT_FH: file */
bool lora_reclaim; bool lora_reclaim;
layoutreturn_stateid lora_recallstateid;
layouttype4 lora_layout_type; layouttype4 lora_layout_type;
layoutiomode4 lora_iomode; layoutiomode4 lora_iomode;
layoutreturn4 lora_layoutreturn; layoutreturn4 lora_layoutreturn;
}; };
<CODE ENDS> <CODE ENDS>
If the lora_layout_type layout type is LAYOUT4_FLEX_FILES and the If the lora_layout_type layout type is LAYOUT4_FLEX_FILES and the
lr_returntype is LAYOUTRETURN4_FILE, then the lrf_body opaque value lr_returntype is LAYOUTRETURN4_FILE, then the lrf_body opaque value
is defined by ff_layoutreturn4 (See Section 9.3). It allows the is defined by ff_layoutreturn4 (see Section 9.3). This allows the
client to report I/O error information or layout usage statistics client to report I/O error information or layout usage statistics
back to the metadata server as defined below. Note that while the back to the metadata server as defined below. Note that while the
data structures are built on concepts introduced in NFSv4.2, the data structures are built on concepts introduced in NFSv4.2, the
effective discriminated union (lora_layout_type combined with effective discriminated union (lora_layout_type combined with
ff_layoutreturn4) allows for a NFSv4.1 metadata server to utilize the ff_layoutreturn4) allows for an NFSv4.1 metadata server to utilize
data. the data.
9.1. I/O Error Reporting 9.1. I/O Error Reporting
9.1.1. ff_ioerr4 9.1.1. ff_ioerr4
<CODE BEGINS> <CODE BEGINS>
/// struct ff_ioerr4 { /// struct ff_ioerr4 {
/// offset4 ffie_offset; /// offset4 ffie_offset;
/// length4 ffie_length; /// length4 ffie_length;
skipping to change at page 31, line 20 skipping to change at page 32, line 4
/// uint64_t ffil_bytes_requested; /// uint64_t ffil_bytes_requested;
/// uint64_t ffil_ops_completed; /// uint64_t ffil_ops_completed;
/// uint64_t ffil_bytes_completed; /// uint64_t ffil_bytes_completed;
/// uint64_t ffil_bytes_not_delivered; /// uint64_t ffil_bytes_not_delivered;
/// nfstime4 ffil_total_busy_time; /// nfstime4 ffil_total_busy_time;
/// nfstime4 ffil_aggregate_completion_time; /// nfstime4 ffil_aggregate_completion_time;
/// }; /// };
/// ///
<CODE ENDS> <CODE ENDS>
Both operation counts and bytes transferred are kept in the Both operation counts and bytes transferred are kept in the
ff_io_latency4. As seen in ff_layoutupdate4 (See Section 9.2.2) read ff_io_latency4. As seen in ff_layoutupdate4 (see Section 9.2.2),
and write operations are aggregated separately. READ operations are READ and WRITE operations are aggregated separately. READ operations
used for the ff_io_latency4 ffl_read. Both WRITE and COMMIT are used for the ff_io_latency4 ffl_read. Both WRITE and COMMIT
operations are used for the ff_io_latency4 ffl_write. "Requested" operations are used for the ff_io_latency4 ffl_write. "Requested"
counters track what the client is attempting to do and "completed" counters track what the client is attempting to do, and "completed"
counters track what was done. There is no requirement that the counters track what was done. There is no requirement that the
client only report completed results that have matching requested client only report completed results that have matching requested
results from the reported period. results from the reported period.
ffil_bytes_not_delivered is used to track the aggregate number of ffil_bytes_not_delivered is used to track the aggregate number of
bytes requested by not fulfilled due to error conditions. bytes requested but not fulfilled due to error conditions.
ffil_total_busy_time is the aggregate time spent with outstanding RPC ffil_total_busy_time is the aggregate time spent with outstanding RPC
calls. ffil_aggregate_completion_time is the sum of all round trip calls. ffil_aggregate_completion_time is the sum of all round-trip
times for completed RPC calls. times for completed RPC calls.
In Section 3.3.1 of [RFC5661], the nfstime4 is defined as the number In Section 3.3.1 of [RFC5661], the nfstime4 is defined as the number
of seconds and nanoseconds since midnight or zero hour January 1, of seconds and nanoseconds since midnight or zero hour January 1,
1970 Coordinated Universal Time (UTC). The use of nfstime4 in 1970 Coordinated Universal Time (UTC). The use of nfstime4 in
ff_io_latency4 is to store time since the start of the first I/O from ff_io_latency4 is to store time since the start of the first I/O from
the client after receiving the layout. In other words, these are to the client after receiving the layout. In other words, these are to
be decoded as duration and not as a date and time. be decoded as duration and not as a date and time.
Note that LAYOUTSTATS are cumulative, i.e., not reset each time the Note that LAYOUTSTATS are cumulative, i.e., not reset each time the
operation is sent. If two LAYOUTSTATS ops for the same file, layout operation is sent. If two LAYOUTSTATS operations for the same file
stateid, and originating from the same NFS client are processed at and layout stateid originate from the same NFS client and are
the same time by the metadata server, then the one containing the processed at the same time by the metadata server, then the one
larger values contains the most recent time series data. containing the larger values contains the most recent time series
data.
9.2.2. ff_layoutupdate4 9.2.2. ff_layoutupdate4
<CODE BEGINS> <CODE BEGINS>
/// struct ff_layoutupdate4 { /// struct ff_layoutupdate4 {
/// netaddr4 ffl_addr; /// netaddr4 ffl_addr;
/// nfs_fh4 ffl_fhandle; /// nfs_fh4 ffl_fhandle;
/// ff_io_latency4 ffl_read; /// ff_io_latency4 ffl_read;
/// ff_io_latency4 ffl_write; /// ff_io_latency4 ffl_write;
/// nfstime4 ffl_duration; /// nfstime4 ffl_duration;
/// bool ffl_local; /// bool ffl_local;
/// }; /// };
/// ///
<CODE ENDS> <CODE ENDS>
ffl_addr differentiates which network address the client is connected
ffl_addr differentiates which network address the client connected to to on the storage device. In the case of multipathing, ffl_fhandle
on the storage device. In the case of multipathing, ffl_fhandle
indicates which read-only copy was selected. ffl_read and ffl_write indicates which read-only copy was selected. ffl_read and ffl_write
convey the latencies respectively for both read and write operations. convey the latencies for both READ and WRITE operations,
ffl_duration is used to indicate the time period over which the respectively. ffl_duration is used to indicate the time period over
statistics were collected. ffl_local if true indicates that the I/O which the statistics were collected. If true, ffl_local indicates
was serviced by the client's cache. This flag allows the client to that the I/O was serviced by the client's cache. This flag allows
inform the metadata server about "hot" access to a file it would not the client to inform the metadata server about "hot" access to a file
normally be allowed to report on. it would not normally be allowed to report on.
9.2.3. ff_iostats4 9.2.3. ff_iostats4
<CODE BEGINS> <CODE BEGINS>
/// struct ff_iostats4 { /// struct ff_iostats4 {
/// offset4 ffis_offset; /// offset4 ffis_offset;
/// length4 ffis_length; /// length4 ffis_length;
/// stateid4 ffis_stateid; /// stateid4 ffis_stateid;
/// io_info4 ffis_read; /// io_info4 ffis_read;
/// io_info4 ffis_write; /// io_info4 ffis_write;
/// deviceid4 ffis_deviceid; /// deviceid4 ffis_deviceid;
/// ff_layoutupdate4 ffis_layoutupdate; /// ff_layoutupdate4 ffis_layoutupdate;
/// }; /// };
/// ///
<CODE ENDS> <CODE ENDS>
Recall that [RFC7862] defines io_info4 as: [RFC7862] defines io_info4 as:
<CODE BEGINS> <CODE BEGINS>
struct io_info4 { struct io_info4 {
uint64_t ii_count; uint64_t ii_count;
uint64_t ii_bytes; uint64_t ii_bytes;
}; };
<CODE ENDS> <CODE ENDS>
With pNFS, data transfers are performed directly between the pNFS With pNFS, data transfers are performed directly between the pNFS
client and the storage devices. Therefore, the metadata server has client and the storage devices. Therefore, the metadata server has
no direct knowledge to the I/O operations being done and thus can not no direct knowledge of the I/O operations being done and thus cannot
create on its own statistical information about client I/O to create on its own statistical information about client I/O to
optimize data storage location. ff_iostats4 MAY be used by the optimize the data storage location. ff_iostats4 MAY be used by the
client to report I/O statistics back to the metadata server upon client to report I/O statistics back to the metadata server upon
returning the layout. returning the layout.
Since it is not feasible for the client to report every I/O that used Since it is not feasible for the client to report every I/O that used
the layout, the client MAY identify "hot" byte ranges for which to the layout, the client MAY identify "hot" byte ranges for which to
report I/O statistics. The definition and/or configuration mechanism report I/O statistics. The definition and/or configuration mechanism
of what is considered "hot" and the size of the reported byte range of what is considered "hot" and the size of the reported byte range
is out of the scope of this document. It is suggested for client are out of the scope of this document. For client implementation,
implementation to provide reasonable default values and an optional providing reasonable default values and an optional run-time
run-time management interface to control these parameters. For management interface to control these parameters is suggested. For
example, a client can define the default byte range resolution to be example, a client can define the default byte-range resolution to be
1 MB in size and the thresholds for reporting to be 1 MB/second or 10 1 MB in size and the thresholds for reporting to be 1 MB/second or 10
I/O operations per second. I/O operations per second.
For each byte range, ffis_offset and ffis_length represent the For each byte range, ffis_offset and ffis_length represent the
starting offset of the range and the range length in bytes. starting offset of the range and the range length in bytes.
ffis_read.ii_count, ffis_read.ii_bytes, ffis_write.ii_count, and ffis_read.ii_count, ffis_read.ii_bytes, ffis_write.ii_count, and
ffis_write.ii_bytes represent, respectively, the number of contiguous ffis_write.ii_bytes represent the number of contiguous READ and WRITE
read and write I/Os and the respective aggregate number of bytes I/Os and the respective aggregate number of bytes transferred within
transferred within the reported byte range. the reported byte range.
The combination of ffis_deviceid and ffl_addr uniquely identifies The combination of ffis_deviceid and ffl_addr uniquely identifies
both the storage path and the network route to it. Finally, the both the storage path and the network route to it. Finally,
ffl_fhandle allows the metadata server to differentiate between ffl_fhandle allows the metadata server to differentiate between
multiple read-only copies of the file on the same storage device. multiple read-only copies of the file on the same storage device.
9.3. ff_layoutreturn4 9.3. ff_layoutreturn4
<CODE BEGINS> <CODE BEGINS>
/// struct ff_layoutreturn4 { /// struct ff_layoutreturn4 {
/// ff_ioerr4 fflr_ioerr_report<>; /// ff_ioerr4 fflr_ioerr_report<>;
/// ff_iostats4 fflr_iostats_report<>; /// ff_iostats4 fflr_iostats_report<>;
skipping to change at page 34, line 17 skipping to change at page 35, line 5
report these errors to the metadata server as an array of elements of report these errors to the metadata server as an array of elements of
type ff_ioerr4. Each element in the array represents an error that type ff_ioerr4. Each element in the array represents an error that
occurred on the data file identified by ffie_errors.de_deviceid. If occurred on the data file identified by ffie_errors.de_deviceid. If
no errors are to be reported, the size of the fflr_ioerr_report<> no errors are to be reported, the size of the fflr_ioerr_report<>
array is set to zero. The client MAY also use fflr_iostats_report<> array is set to zero. The client MAY also use fflr_iostats_report<>
to report a list of I/O statistics as an array of elements of type to report a list of I/O statistics as an array of elements of type
ff_iostats4. Each element in the array represents statistics for a ff_iostats4. Each element in the array represents statistics for a
particular byte range. Byte ranges are not guaranteed to be disjoint particular byte range. Byte ranges are not guaranteed to be disjoint
and MAY repeat or intersect. and MAY repeat or intersect.
10. Flexible Files Layout Type LAYOUTERROR 10. Flexible File Layout Type LAYOUTERROR
If the client is using NFSv4.2 to communicate with the metadata If the client is using NFSv4.2 to communicate with the metadata
server, then instead of waiting for a LAYOUTRETURN to send error server, then instead of waiting for a LAYOUTRETURN to send error
information to the metadata server (see Section 9.1), it MAY use information to the metadata server (see Section 9.1), it MAY use
LAYOUTERROR (see Section 15.6 of [RFC7862]) to communicate that LAYOUTERROR (see Section 15.6 of [RFC7862]) to communicate that
information. For the flexible files layout type, this means that information. For the flexible file layout type, this means that
LAYOUTERROR4args is treated the same as ff_ioerr4. LAYOUTERROR4args is treated the same as ff_ioerr4.
11. Flexible Files Layout Type LAYOUTSTATS 11. Flexible File Layout Type LAYOUTSTATS
If the client is using NFSv4.2 to communicate with the metadata If the client is using NFSv4.2 to communicate with the metadata
server, then instead of waiting for a LAYOUTRETURN to send I/O server, then instead of waiting for a LAYOUTRETURN to send I/O
statistics to the metadata server (see Section 9.2), it MAY use statistics to the metadata server (see Section 9.2), it MAY use
LAYOUTSTATS (see Section 15.7 of [RFC7862]) to communicate that LAYOUTSTATS (see Section 15.7 of [RFC7862]) to communicate that
information. For the flexible files layout type, this means that information. For the flexible file layout type, this means that
LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same
contents as in ffis_layoutupdate. contents as in ffis_layoutupdate.
12. Flexible File Layout Type Creation Hint 12. Flexible File Layout Type Creation Hint
The layouthint4 type is defined in the [RFC5661] as follows: The layouthint4 type is defined in the [RFC5661] as follows:
<CODE BEGINS> <CODE BEGINS>
struct layouthint4 { struct layouthint4 {
skipping to change at page 35, line 32 skipping to change at page 36, line 17
/// ///
<CODE ENDS> <CODE ENDS>
This type conveys hints for the desired data map. All parameters are This type conveys hints for the desired data map. All parameters are
optional so the client can give values for only the parameter it optional so the client can give values for only the parameter it
cares about. cares about.
13. Recalling a Layout 13. Recalling a Layout
While Section 12.5.5 of [RFC5661] discusses layout type independent While Section 12.5.5 of [RFC5661] discusses reasons independent of
reasons for recalling a layout, the flexible file layout type layout type for recalling a layout, the flexible file layout type
metadata server should recall outstanding layouts in the following metadata server should recall outstanding layouts in the following
cases: cases:
o When the file's security policy changes, i.e., Access Control o When the file's security policy changes, i.e., ACLs or permission
Lists (ACLs) or permission mode bits are set. mode bits are set.
o When the file's layout changes, rendering outstanding layouts o When the file's layout changes, rendering outstanding layouts
invalid. invalid.
o When existing layouts are inconsistent with the need to enforce o When existing layouts are inconsistent with the need to enforce
locking constraints. locking constraints.
o When existing layouts are inconsistent with the requirements o When existing layouts are inconsistent with the requirements
regarding resilvering as described in Section 8.3. regarding resilvering as described in Section 8.3.
skipping to change at page 36, line 16 skipping to change at page 36, line 45
The metadata server can use the CB_RECALL_ANY callback operation to The metadata server can use the CB_RECALL_ANY callback operation to
notify the client to return some or all of its layouts. Section 22.3 notify the client to return some or all of its layouts. Section 22.3
of [RFC5661] defines the allowed types of the "NFSv4 Recallable of [RFC5661] defines the allowed types of the "NFSv4 Recallable
Object Types Registry". Object Types Registry".
<CODE BEGINS> <CODE BEGINS>
/// const RCA4_TYPE_MASK_FF_LAYOUT_MIN = 16; /// const RCA4_TYPE_MASK_FF_LAYOUT_MIN = 16;
/// const RCA4_TYPE_MASK_FF_LAYOUT_MAX = 17; /// const RCA4_TYPE_MASK_FF_LAYOUT_MAX = 17;
[[RFC Editor: please insert assigned constants]]
/// ///
struct CB_RECALL_ANY4args { struct CB_RECALL_ANY4args {
uint32_t craa_layouts_to_keep; uint32_t craa_layouts_to_keep;
bitmap4 craa_type_mask; bitmap4 craa_type_mask;
}; };
<CODE ENDS> <CODE ENDS>
Typically, CB_RECALL_ANY will be used to recall client state when the Typically, CB_RECALL_ANY will be used to recall client state when the
server needs to reclaim resources. The craa_type_mask bitmap server needs to reclaim resources. The craa_type_mask bitmap
specifies the type of resources that are recalled and the specifies the type of resources that are recalled, and the
craa_layouts_to_keep value specifies how many of the recalled craa_layouts_to_keep value specifies how many of the recalled
flexible file layouts the client is allowed to keep. The flexible flexible file layouts the client is allowed to keep. The mask flags
file layout type mask flags are defined as follows: for the flexible file layout type are defined as follows:
<CODE BEGINS> <CODE BEGINS>
/// enum ff_cb_recall_any_mask { /// enum ff_cb_recall_any_mask {
/// FF_RCA4_TYPE_MASK_READ = -2, /// PNFS_FF_RCA4_TYPE_MASK_READ = 16,
/// FF_RCA4_TYPE_MASK_RW = -1 /// PNFS_FF_RCA4_TYPE_MASK_RW = 17
[[RFC Editor: please insert assigned constants]]
/// }; /// };
/// ///
<CODE ENDS> <CODE ENDS>
They represent the iomode of the recalled layouts. In response, the The flags represent the iomode of the recalled layouts. In response,
client SHOULD return layouts of the recalled iomode that it needs the the client SHOULD return layouts of the recalled iomode that it needs
least, keeping at most craa_layouts_to_keep Flexible File Layouts. the least, keeping at most craa_layouts_to_keep flexible file
layouts.
The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return
layouts of iomode LAYOUTIOMODE4_READ. Similarly, the layouts of iomode LAYOUTIOMODE4_READ. Similarly, the
PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts
of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client
is notified to return layouts of either iomode. is notified to return layouts of either iomode.
14. Client Fencing 14. Client Fencing
In cases where clients are uncommunicative and their lease has In cases where clients are uncommunicative and their lease has
expired or when clients fail to return recalled layouts within a expired or when clients fail to return recalled layouts within a
lease period, the server MAY revoke client layouts and reassign these lease period, the server MAY revoke client layouts and reassign these
resources to other clients (see Section 12.5.5 in [RFC5661]). To resources to other clients (see Section 12.5.5 of [RFC5661]). To
avoid data corruption, the metadata server MUST fence off the revoked avoid data corruption, the metadata server MUST fence off the revoked
clients from the respective data files as described in Section 2.2. clients from the respective data files as described in Section 2.2.
15. Security Considerations 15. Security Considerations
The combination of components in a pNFS system is required to The combination of components in a pNFS system is required to
preserve the security properties of NFSv4.1+ with respect to an preserve the security properties of NFSv4.1+ with respect to an
entity accessing data via a client. The pNFS feature partitions the entity accessing data via a client. The pNFS feature partitions the
NFSv4.1+ file system protocol into two parts, the control protocol NFSv4.1+ file system protocol into two parts: the control protocol
and the data protocol. As the control protocol in this document is and the data protocol. As the control protocol in this document is
NFS, the security properties are equivalent to that version of NFS. NFS, the security properties are equivalent to the version of NFS
The Flexible File Layout further divides the data protocol into being used. The flexible file layout further divides the data
metadata and data paths. The security properties of the metadata protocol into metadata and data paths. The security properties of
path are equivalent to those of NFSv4.1x (see Sections 1.7.1 and the metadata path are equivalent to those of NFSv4.1x (see Sections
2.2.1 of [RFC5661]). And the security properties of the data path 1.7.1 and 2.2.1 of [RFC5661]). And the security properties of the
are equivalent to those of the version of NFS used to access the data path are equivalent to those of the version of NFS used to
storage device, with the provision that the metadata server is access the storage device, with the provision that the metadata
responsible for authenticating client access to the data file. The server is responsible for authenticating client access to the data
metadata server provides appropriate credentials to the client to file. The metadata server provides appropriate credentials to the
access data files on the storage device. It is also responsible for client to access data files on the storage device. It is also
revoking access for a client to the storage device. responsible for revoking access for a client to the storage device.
The metadata server enforces the file access-control policy at The metadata server enforces the file access control policy at
LAYOUTGET time. The client should use RPC authorization credentials LAYOUTGET time. The client should use RPC authorization credentials
for getting the layout for the requested iomode (READ or RW) and the for getting the layout for the requested iomode ((LAYOUTIOMODE4_READ
server verifies the permissions and ACL for these credentials, or LAYOUTIOMODE4_RW), and the server verifies the permissions and ACL
possibly returning NFS4ERR_ACCESS if the client is not allowed the for these credentials, possibly returning NFS4ERR_ACCESS if the
requested iomode. If the LAYOUTGET operation succeeds the client client is not allowed the requested iomode. If the LAYOUTGET
receives, as part of the layout, a set of credentials allowing it I/O operation succeeds, the client receives, as part of the layout, a set
access to the specified data files corresponding to the requested of credentials allowing it I/O access to the specified data files
iomode. When the client acts on I/O operations on behalf of its corresponding to the requested iomode. When the client acts on I/O
local users, it MUST authenticate and authorize the user by issuing operations on behalf of its local users, it MUST authenticate and
respective OPEN and ACCESS calls to the metadata server, similar to authorize the user by issuing respective OPEN and ACCESS calls to the
having NFSv4 data delegations. metadata server, similar to having NFSv4 data delegations.
The combination of file handle, synthetic uid, and gid in the layout The combination of filehandle, synthetic uid, and gid in the layout
are the way that the metadata server enforces access control to the is the way that the metadata server enforces access control to the
data server. The client only has access to file handles of file data server. The client only has access to filehandles of file
objects and not directory objects. Thus, given a file handle in a objects and not directory objects. Thus, given a filehandle in a
layout, it is not possible to guess the parent directory file handle. layout, it is not possible to guess the parent directory filehandle.
Further, as the data file permissions only allow the given synthetic Further, as the data file permissions only allow the given synthetic
uid read/write permission and the given synthetic gid read uid read/write permission and the given synthetic gid read
permission, knowing the synthetic ids of one file does not permission, knowing the synthetic ids of one file does not
necessarily allow access to any other data file on the storage necessarily allow access to any other data file on the storage
device. device.
The metadata server can also deny access at any time by fencing the The metadata server can also deny access at any time by fencing the
data file, which means changing the synthetic ids. In turn, that data file, which means changing the synthetic ids. In turn, that
forces the client to return its current layout and get a new layout forces the client to return its current layout and get a new layout
if it wants to continue IO to the data file. if it wants to continue I/O to the data file.
If access is allowed, the client uses the corresponding (READ or RW) If access is allowed, the client uses the corresponding (read-only or
credentials to perform the I/O operations at the data file's storage read/write) credentials to perform the I/O operations at the data
devices. When the metadata server receives a request to change a file's storage devices. When the metadata server receives a request
file's permissions or ACL, it SHOULD recall all layouts for that file to change a file's permissions or ACL, it SHOULD recall all layouts
and then MUST fence off any clients still holding outstanding layouts for that file and then MUST fence off any clients still holding
for the respective files by implicitly invalidating the previously outstanding layouts for the respective files by implicitly
distributed credential on all data file comprising the file in invalidating the previously distributed credential on all data file
question. It is REQUIRED that this be done before committing to the comprising the file in question. It is REQUIRED that this be done
new permissions and/or ACL. By requesting new layouts, the clients before committing to the new permissions and/or ACL. By requesting
will reauthorize access against the modified access control metadata. new layouts, the clients will reauthorize access against the modified
Recalling the layouts in this case is intended to prevent clients access control metadata. Recalling the layouts in this case is
from getting an error on I/Os done after the client was fenced off. intended to prevent clients from getting an error on I/Os done after
the client was fenced off.
15.1. RPCSEC_GSS and Security Services 15.1. RPCSEC_GSS and Security Services
Because of the special use of principals within the loose coupling Because of the special use of principals within the loosely coupled
model, the issues are different depending on the coupling model. model, the issues are different depending on the coupling model.
15.1.1. Loosely Coupled 15.1.1. Loosely Coupled
RPCSEC_GSS version 3 (RPCSEC_GSSv3) [RFC7861] contains facilities RPCSEC_GSS version 3 (RPCSEC_GSSv3) [RFC7861] contains facilities
that would allow it to be used to authorize the client to the storage that would allow it to be used to authorize the client to the storage
device on behalf of the metadata server. Doing so would require that device on behalf of the metadata server. Doing so would require that
each of the metadata server, storage device, and client would need to each of the metadata server, storage device, and client would need to
implement RPCSEC_GSSv3 using an RPC-application-defined structured implement RPCSEC_GSSv3 using an RPC-application-defined structured
privilege assertion in a manner described in Section 4.9.1 of privilege assertion in a manner described in Section 4.9.1 of
[RFC7862]. The specifics necessary to do so are not described in [RFC7862]. The specifics necessary to do so are not described in
this document. This is principally because any such specification this document. This is principally because any such specification
would require extensive implementation work on a wide range of would require extensive implementation work on a wide range of
storage devices, which would be unlikely to result in a widely usable storage devices, which would be unlikely to result in a widely usable
specification for a considerable time. specification for a considerable time.
As a result, the layout type described in this document will not As a result, the layout type described in this document will not
provide support for use of RPCSEC_GSS together with the loosely provide support for use of RPCSEC_GSS together with the loosely
coupled model. However, future layout types could be specified which coupled model. However, future layout types could be specified,
would allow such support, either through the use of RPCSEC_GSSv3, or which would allow such support, either through the use of
in other ways. RPCSEC_GSSv3 or in other ways.
15.1.2. Tightly Coupled 15.1.2. Tightly Coupled
With tight coupling, the principal used to access the metadata file With tight coupling, the principal used to access the metadata file
is exactly the same as used to access the data file. The storage is exactly the same as used to access the data file. The storage
device can use the control protocol to validate any RPC credentials. device can use the control protocol to validate any RPC credentials.
As a result there are no security issues related to using RPCSEC_GSS As a result, there are no security issues related to using RPCSEC_GSS
with a tightly coupled system. For example, if Kerberos V5 GSS-API with a tightly coupled system. For example, if Kerberos V5 Generic
[RFC4121] is used as the security mechanism, then the storage device Security Service Application Program Interface (GSS-API) [RFC4121] is
could use a control protocol to validate the RPC credentials to the used as the security mechanism, then the storage device could use a
metadata server. control protocol to validate the RPC credentials to the metadata
server.
16. IANA Considerations 16. IANA Considerations
[RFC5661] introduced a registry for "pNFS Layout Types Registry" and [RFC5661] introduced the "pNFS Layout Types Registry"; new layout
as such, new layout type numbers need to be assigned by IANA. This type numbers in this registry need to be assigned by IANA. This
document defines the protocol associated with the existing layout document defines the protocol associated with an existing layout type
type number, LAYOUT4_FLEX_FILES (see Table 1). number: LAYOUT4_FLEX_FILES. See Table 1.
+--------------------+-------+----------+-----+----------------+ +--------------------+------------+----------+-----+----------------+
| Layout Type Name | Value | RFC | How | Minor Versions | | Layout Type Name | Value | RFC | How | Minor Versions |
+--------------------+-------+----------+-----+----------------+ +--------------------+------------+----------+-----+----------------+
| LAYOUT4_FLEX_FILES | 0x4 | RFCTBD10 | L | 1 | | LAYOUT4_FLEX_FILES | 0x00000004 | RFC 8435 | L | 1 |
+--------------------+-------+----------+-----+----------------+ +--------------------+------------+----------+-----+----------------+
Table 1: Layout Type Assignments Table 1: Layout Type Assignments
[RFC5661] also introduced a registry called "NFSv4 Recallable Object [RFC5661] also introduced the "NFSv4 Recallable Object Types
Types Registry". This document defines new recallable objects for Registry". This document defines new recallable objects for
RCA4_TYPE_MASK_FF_LAYOUT_MIN and RCA4_TYPE_MASK_FF_LAYOUT_MAX (see RCA4_TYPE_MASK_FF_LAYOUT_MIN and RCA4_TYPE_MASK_FF_LAYOUT_MAX (see
Table 2). Table 2).
+------------------------------+-------+----------+-----+-----------+ +------------------------------+-------+--------+-----+-------------+
| Recallable Object Type Name | Value | RFC | How | Minor | | Recallable Object Type Name | Value | RFC | How | Minor |
| | | | | Versions | | | | | | Versions |
+------------------------------+-------+----------+-----+-----------+ +------------------------------+-------+--------+-----+-------------+
| RCA4_TYPE_MASK_FF_LAYOUT_MIN | 16 | RFCTBD10 | L | 1 | | RCA4_TYPE_MASK_FF_LAYOUT_MIN | 16 | RFC | L | 1 |
| RCA4_TYPE_MASK_FF_LAYOUT_MAX | 17 | RFCTBD10 | L | 1 | | | | 8435 | | |
+------------------------------+-------+----------+-----+-----------+ | RCA4_TYPE_MASK_FF_LAYOUT_MAX | 17 | RFC | L | 1 |
| | | 8435 | | |
+------------------------------+-------+--------+-----+-------------+
Table 2: Recallable Object Type Assignments Table 2: Recallable Object Type Assignments
17. References 17. References
17.1. Normative References 17.1. Normative References
[LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents", [LEGAL] IETF Trust, "Trust Legal Provisions (TLP)",
November 2008, <http://trustee.ietf.org/docs/ <https://trustee.ietf.org/trust-legal-provisions.html>.
IETF-Trust-License-Policy.pdf>.
[RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813, [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS
June 1995. Version 3 Protocol Specification", RFC 1813,
DOI 10.17487/RFC1813, June 1995,
<https://www.rfc-editor.org/info/rfc1813>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ Requirement Levels", BCP 14, RFC 2119,
RFC2119, March 1997, <https://www.rfc-editor.org/info/ DOI 10.17487/RFC2119, March 1997,
rfc2119>. <https://www.rfc-editor.org/info/rfc2119>.
[RFC4121] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos [RFC4121] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos
Version 5 Generic Security Service Application Program Version 5 Generic Security Service Application Program
Interface (GSS-API) Mechanism Version 2", RFC 4121, July Interface (GSS-API) Mechanism: Version 2", RFC 4121,
2005. DOI 10.17487/RFC4121, July 2005,
<https://www.rfc-editor.org/info/rfc4121>.
[RFC4506] Eisler, M., "XDR: External Data Representation Standard", [RFC4506] Eisler, M., Ed., "XDR: External Data Representation
STD 67, RFC 4506, May 2006. Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May
2006, <https://www.rfc-editor.org/info/rfc4506>.
[RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol
Specification Version 2", RFC 5531, May 2009. Specification Version 2", RFC 5531, DOI 10.17487/RFC5531,
May 2009, <https://www.rfc-editor.org/info/rfc5531>.
[RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1 "Network File System (NFS) Version 4 Minor Version 1
Protocol", RFC 5661, January 2010. Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010,
<https://www.rfc-editor.org/info/rfc5661>.
[RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., [RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1 "Network File System (NFS) Version 4 Minor Version 1
External Data Representation Standard (XDR) Description", External Data Representation Standard (XDR) Description",
RFC 5662, January 2010. RFC 5662, DOI 10.17487/RFC5662, January 2010,
<https://www.rfc-editor.org/info/rfc5662>.
[RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) [RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System
version 4 Protocol", RFC 7530, March 2015. (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530,
March 2015, <https://www.rfc-editor.org/info/rfc7530>.
[RFC7861] Adamson, W. and N. Williams, "Remote Procedure Call (RPC) [RFC7861] Adamson, A. and N. Williams, "Remote Procedure Call (RPC)
Security Version 3", November 2016. Security Version 3", RFC 7861, DOI 10.17487/RFC7861,
November 2016, <https://www.rfc-editor.org/info/rfc7861>.
[RFC7862] Haynes, T., "NFS Version 4 Minor Version 2", RFC 7862, [RFC7862] Haynes, T., "Network File System (NFS) Version 4 Minor
November 2016. Version 2 Protocol", RFC 7862, DOI 10.17487/RFC7862,
November 2016, <https://www.rfc-editor.org/info/rfc7862>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>. May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[pNFSLayouts] [RFC8434] Haynes, T., "Requirements for Parallel NFS (pNFS) Layout
Haynes, T., "Requirements for pNFS Layout Types", draft- Types", RFC 8434, DOI 10.17487/RFC8434, August 2018,
ietf-nfsv4-layout-types-07 (Work In Progress), August <https://www.rfc-editor.org/info/rfc8434>.
2017.
17.2. Informative References 17.2. Informative References
[RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol [RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol
(LDAP): Schema for User Applications", RFC 4519, DOI (LDAP): Schema for User Applications", RFC 4519,
10.17487/RFC4519, June 2006, DOI 10.17487/RFC4519, June 2006,
<http://www.rfc-editor.org/info/rfc4519>. <https://www.rfc-editor.org/info/rfc4519>.
Appendix A. Acknowledgments Acknowledgments
Those who provided miscellaneous comments to early drafts of this The following individuals provided miscellaneous comments to early
document include: Matt W. Benjamin, Adam Emerson, J. Bruce Fields, draft versions of this document: Matt W. Benjamin, Adam Emerson,
and Lev Solomonov. J. Bruce Fields, and Lev Solomonov.
Those who provided miscellaneous comments to the final drafts of this The following individuals provided miscellaneous comments to the
document include: Anand Ganesh, Robert Wipfel, Gobikrishnan final draft versions of this document: Anand Ganesh, Robert Wipfel,
Sundharraj, Trond Myklebust, Rick Macklem, and Jim Sermersheim. Gobikrishnan Sundharraj, Trond Myklebust, Rick Macklem, and Jim
Sermersheim.
Idan Kedar caught a nasty bug in the interaction of client side Idan Kedar caught a nasty bug in the interaction of client-side
mirroring and the minor versioning of devices. mirroring and the minor versioning of devices.
Dave Noveck provided comprehensive reviews of the document during the Dave Noveck provided comprehensive reviews of the document during the
working group last calls. He also rewrote Section 2.3. working group last calls. He also rewrote Section 2.3.
Olga Kornievskaia made a convincing case against the use of a Olga Kornievskaia made a convincing case against the use of a
credential versus a principal in the fencing approach. Andy Adamson credential versus a principal in the fencing approach. Andy Adamson
and Benjamin Kaduk helped to sharpen the focus. and Benjamin Kaduk helped to sharpen the focus.
Benjamin Kaduk and Olga Kornievskaia also helped provide concrete Benjamin Kaduk and Olga Kornievskaia also helped provide concrete
scenarios for loosely coupled security mechanisms. And in the end, scenarios for loosely coupled security mechanisms. In the end, Olga
Olga proved that as defined, the loosely coupled model would not work proved that as defined, the loosely coupled model would not work with
with RPCSEC_GSS. RPCSEC_GSS.
Tigran Mkrtchyan provided the use case for not allowing the client to Tigran Mkrtchyan provided the use case for not allowing the client to
proxy the I/O through the data server. proxy the I/O through the data server.
Rick Macklem provided the use case for only writing to a single Rick Macklem provided the use case for only writing to a single
mirror. mirror.
Appendix B. RFC Editor Notes
[RFC Editor: please remove this section prior to publishing this
document as an RFC]
[RFC Editor: prior to publishing this document as an RFC, please
replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the
RFC number of this document]
Authors' Addresses Authors' Addresses
Benny Halevy Benny Halevy
Email: bhalevy@gmail.com Email: bhalevy@gmail.com
Thomas Haynes Thomas Haynes
Hammerspace Hammerspace
4300 El Camino Real Ste 105 4300 El Camino Real Ste 105
Los Altos, CA 94022 Los Altos, CA 94022
USA United States of America
Email: loghyr@gmail.com Email: loghyr@gmail.com
 End of changes. 222 change blocks. 
599 lines changed or deleted 605 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/