draft-ietf-nfsv4-flex-files-00.txt   draft-ietf-nfsv4-flex-files-01.txt 
NFSv4 B. Halevy NFSv4 B. Halevy
Internet-Draft T. Haynes Internet-Draft T. Haynes
Intended status: Informational Primary Data Intended status: Informational Primary Data
Expires: February 1, 2015 July 31, 2014 Expires: March 8, 2015 September 04, 2014
Parallel NFS (pNFS) Flexible File Layout Parallel NFS (pNFS) Flexible File Layout
draft-ietf-nfsv4-flex-files-00.txt draft-ietf-nfsv4-flex-files-01.txt
Abstract Abstract
The Parallel Network File System (pNFS) allows a separation between The Parallel Network File System (pNFS) allows a separation between
the metadata and data for a file. The metadata file access is the metadata and data for a file. The metadata file access is
handled via Network File System version 4 (NFSv4) minor version 1 handled via Network File System version 4 (NFSv4) minor version 1
(NFSv4.1) and the data file access is specific to the protocol being (NFSv4.1) and the data file access is specific to the protocol being
used between the client and storage device. The client is informed used between the client and storage device. The client is informed
by the metadata server as to which protocol to use via a Layout Type. by the metadata server as to which protocol to use via a Layout Type.
The Flexible File Layout Type is defined in this document as an The Flexible File Layout Type is defined in this document as an
skipping to change at page 1, line 38 skipping to change at page 1, line 38
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on February 1, 2015. This Internet-Draft will expire on March 8, 2015.
Copyright Notice Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Difference Between a Data Server and a Storage Device . . 5 1.2. Difference Between a Data Server and a Storage Device . . 5
1.3. Requirements Language . . . . . . . . . . . . . . . . . . 5 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 5
2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 5 2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 5
2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6 2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6
2.2. Security Models . . . . . . . . . . . . . . . . . . . . . 6 2.2. Security Models . . . . . . . . . . . . . . . . . . . . . 6
2.3. State and Locking Models . . . . . . . . . . . . . . . . 6 2.3. State and Locking Models . . . . . . . . . . . . . . . . 6
3. XDR Description of the Flexible File Layout Type . . . . . . 7 3. XDR Description of the Flexible File Layout Type . . . . . . 7
3.1. Code Components Licensing Notice . . . . . . . . . . . . 7 3.1. Code Components Licensing Notice . . . . . . . . . . . . 8
4. Device Addressing and Discovery . . . . . . . . . . . . . . . 9 4. Device Addressing and Discovery . . . . . . . . . . . . . . . 9
4.1. ff_device_addr . . . . . . . . . . . . . . . . . . . . . 9 4.1. ff_device_addr . . . . . . . . . . . . . . . . . . . . . 9
4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 10 4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 10
5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 11 5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 11
5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 11 5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 12
6. Recovering from Client I/O Errors . . . . . . . . . . . . . . 14 6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 14
7. Flexible Files Layout Type Return . . . . . . . . . . . . . . 14 7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 14
7.1. ff_ioerr . . . . . . . . . . . . . . . . . . . . . . . . 15 8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 15
7.2. ff_iostats . . . . . . . . . . . . . . . . . . . . . . . 16 8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 15
7.3. ff_layoutreturn . . . . . . . . . . . . . . . . . . . . . 17 8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 16
8. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 17 8.3. Metadata Server Resilvering of the File . . . . . . . . . 16
9. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 17 9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 16
10. Flexible File Layout Type Creation Hint . . . . . . . . . . . 17 9.1. ff_ioerr . . . . . . . . . . . . . . . . . . . . . . . . 17
10.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 18 9.2. ff_iostats . . . . . . . . . . . . . . . . . . . . . . . 18
11. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 18 9.3. ff_layoutreturn . . . . . . . . . . . . . . . . . . . . . 19
11.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 18 10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 19
12. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 19 11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 19
13. Security Considerations . . . . . . . . . . . . . . . . . . . 19 12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 20
14. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 20
15. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 13. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 20
15.1. Normative References . . . . . . . . . . . . . . . . . . 20 13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 21
15.2. Informative References . . . . . . . . . . . . . . . . . 21 14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 21
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 21 15. Security Considerations . . . . . . . . . . . . . . . . . . . 22
Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 22 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 23
17.1. Normative References . . . . . . . . . . . . . . . . . . 23
17.2. Informative References . . . . . . . . . . . . . . . . . 23
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 24
Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 24
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24
1. Introduction 1. Introduction
In the parallel Network File System (pNFS), the metadata server In the parallel Network File System (pNFS), the metadata server
returns Layout Type structures that describe where file data is returns Layout Type structures that describe where file data is
located. There are different Layout Types for different storage located. There are different Layout Types for different storage
systems and methods of arranging data on storage devices. This systems and methods of arranging data on storage devices. This
document defines the Flexible File Layout Type used with file-based document defines the Flexible File Layout Type used with file-based
data servers that are accessed using the Network File System (NFS) data servers that are accessed using the Network File System (NFS)
protocols: NFSv3 [RFC1813], NFSv4 [RFC3530], NFSv4.1 [RFC5661], and protocols: NFSv3 [RFC1813], NFSv4 [RFC3530], NFSv4.1 [RFC5661], and
NFSv4.2 [NFSv42]. NFSv4.2 [NFSv42].
In contrast to the File Layout Type [RFC5661] that also uses NFSv4.1
to access the data server, the Flexible File Layout Type defines a
simple device information model suitable for aggregating standalone
NFS servers into a centrally managed pNFS cluster. In particular,
unlike the File Layout Type, the Flexible File Layout Type does not
provide striping of the data file across multiple storage devices.
To provide a global state model equivalent to that of the Files To provide a global state model equivalent to that of the Files
Layout Type, a back-end control protocol MAY be implemented between Layout Type, a back-end control protocol MAY be implemented between
the metadata server and NFSv4.1 storage devices. It is out of scope the metadata server and NFSv4.1 storage devices. It is out of scope
for this document to specify the wire protocol of such a protocol, for this document to specify the wire protocol of such a protocol,
yet the requirements for the protocol are specified in [RFC5661] and yet the requirements for the protocol are specified in [RFC5661] and
clarified in [pNFSLayouts]. clarified in [pNFSLayouts].
1.1. Definitions 1.1. Definitions
control protocol: is a set of requirements for the communication of control protocol: is a set of requirements for the communication of
skipping to change at page 4, line 32 skipping to change at page 4, line 30
metadata file: is that part of the file system object which metadata file: is that part of the file system object which
describes the object and not the payload. E.g., it could be the describes the object and not the payload. E.g., it could be the
time since last modification, access, etc. time since last modification, access, etc.
Metadata Server (MDS): is the pNFS server which provides metadata Metadata Server (MDS): is the pNFS server which provides metadata
information for a file system object. It also is responsible for information for a file system object. It also is responsible for
generating layouts for file system objects. Note that the MDS is generating layouts for file system objects. Note that the MDS is
responsible for directory-based operations. responsible for directory-based operations.
Mirror: is a copy of a file. While mirroring can be used for
backing up a file, the copies can be distrbuted such that each
remote site has a locally cached copy. Note that if one copy of
the mirror is updated, then all copies must be updated.
Object Layout Type: is a Layout Type in which the storage devices Object Layout Type: is a Layout Type in which the storage devices
are accessed via the OSD protocol [ANSI400-2004]. It is defined are accessed via the OSD protocol [ANSI400-2004]. It is defined
in [RFC5664]. in [RFC5664].
recalling a layout: is when the metadata server uses a back channel recalling a layout: is when the metadata server uses a back channel
to inform the client that the layout is to be returned in a to inform the client that the layout is to be returned in a
graceful manner. Note that the client could be able to flush any graceful manner. Note that the client could be able to flush any
writes, etc., before replying to the metadata server. writes, etc., before replying to the metadata server.
revoking a layout: is when the metadata server invalidates the revoking a layout: is when the metadata server invalidates the
layout such that neither the metadata server nor any storage layout such that neither the metadata server nor any storage
device will accept any access from the client with that layout. device will accept any access from the client with that layout.
resilvering: is the act of rebuilding a mirrored copy of a file from
a known good copy of the file. Note that this can also be done to
create a new mirrored copy of the file.
rsize: is the data transfer buffer size used for reads. rsize: is the data transfer buffer size used for reads.
stateid: is a 128-bit quantity returned by a server that uniquely stateid: is a 128-bit quantity returned by a server that uniquely
defines the open and locking states provided by the server for a defines the open and locking states provided by the server for a
specific open-owner or lock-owner/open-owner pair for a specific specific open-owner or lock-owner/open-owner pair for a specific
file and type of lock. file and type of lock.
storage device: is another term used almost interchangeably with storage device: is another term used almost interchangeably with
data server. See Section 1.2 for the nuances between the two. data server. See Section 1.2 for the nuances between the two.
skipping to change at page 6, line 20 skipping to change at page 6, line 22
a loosely coupled system, a LAYOUTCOMMIT to the metadata server MUST a loosely coupled system, a LAYOUTCOMMIT to the metadata server MUST
be proceeded with a COMMIT to the storage device. I.e., it is the be proceeded with a COMMIT to the storage device. I.e., it is the
responsibility of the client to make sure the data file is stable responsibility of the client to make sure the data file is stable
before the metadata server begins to query the storage devices about before the metadata server begins to query the storage devices about
the changes to the file. Note that if the client has not done a the changes to the file. Note that if the client has not done a
COMMIT to the storage device, then the LAYOUTCOMMIT might not be COMMIT to the storage device, then the LAYOUTCOMMIT might not be
synchronized to the last WRITE operation to the storage device. synchronized to the last WRITE operation to the storage device.
2.2. Security Models 2.2. Security Models
With NFSv3 storage devices, the metadata server uses synthetic uids With loosely coupled storage devices, the metadata server uses
and gids for the data file, where the uid owner of the data file is synthetic uids and gids for the data file, where the uid owner of the
allowed read/write access and the gid owner is allowed read only data file is allowed read/write access and the gid owner is allowed
access. As part of the layout, the client is provided with the rpc read only access. As part of the layout, the client is provided with
credentials to be used (see ffm_auth in Section 5.1) to access the the rpc credentials to be used (see ffm_auth in Section 5.1) to
data file. Fencing off clients is achieved by using SETATTR by the access the data file. Fencing off clients is achieved by using
server to change the uid and/or gid owners of the data file to SETATTR by the server to change the uid and/or gid owners of the data
implicitly revoke the outstanding rpc credentials. Note: it is file to implicitly revoke the outstanding rpc credentials. Note: it
recommended to implement common access control methods at the storage is recommended to implement common access control methods at the
device filesystem exports level to allow only the metadata server storage device filesystem exports level to allow only the metadata
root (super user) access to the storage device, and to set the owner server root (super user) access to the storage device, and to set the
of all directories holding data files to the root user. This owner of all directories holding data files to the root user. This
security method, when using weak auth flavors such as AUTH_SYS, security method, when using weak auth flavors such as AUTH_SYS,
provides a practical model to enforce access control and fence off provides a practical model to enforce access control and fence off
cooperative clients, but it can not protect against malicious cooperative clients, but it can not protect against malicious
clients; hence it provides a level of security equivalent to NFSv3. clients; hence it provides a level of security equivalent to NFSv3.
With NFSv4.x storage devices, the metadata server sets the user and With tightly coupled storage devices, the metadata server sets the
group owners, mode bits, and ACL of the data file to be the same as user and group owners, mode bits, and ACL of the data file to be the
the User File. And the client must authenticate with the storage same as the metadata file. And the client must authenticate with the
device and go through the same authorization process it would go storage device and go through the same authorization process it would
through via the metadata server. go through via the metadata server.
2.3. State and Locking Models 2.3. State and Locking Models
Metadata file OPEN, LOCK, and DELEGATION operations are always Metadata file OPEN, LOCK, and DELEGATION operations are always
executed only against the metadata server. executed only against the metadata server.
With NFSv4 storage devices, the metadata server, in response to the With NFSv4 storage devices, the metadata server, in response to the
state changing operation, executes them against the respective data state changing operation, executes them against the respective data
files on the storage devices. It then sends the storage device open files on the storage devices. It then sends the storage device open
stateid as part of the layout (see the ffm_stateid in Section 5.1) stateid as part of the layout (see the ffm_stateid in Section 5.1)
skipping to change at page 10, line 27 skipping to change at page 10, line 35
The Flexible File Layout Type supports multipathing to multiple The Flexible File Layout Type supports multipathing to multiple
storage device addresses. Storage device level multipathing is used storage device addresses. Storage device level multipathing is used
for bandwidth scaling via trunking and for higher availability of use for bandwidth scaling via trunking and for higher availability of use
in the case of a storage device failure. Multipathing allows the in the case of a storage device failure. Multipathing allows the
client to switch to another storage device address which may be that client to switch to another storage device address which may be that
of another storage device that is exporting the same data stripe of another storage device that is exporting the same data stripe
unit, without having to contact the metadata server for a new layout. unit, without having to contact the metadata server for a new layout.
To support storage device multipathing, ffda_netaddrs contains an To support storage device multipathing, ffda_netaddrs contains an
array of one more storage device network addresses. This array (data array of one or more storage device network addresses. This array
type multipath_list4) represents a list of storage device (each (data type multipath_list4) represents a list of storage device (each
identified by a network address), with the possibility that some identified by a network address), with the possibility that some
storage device will appear in the list multiple times. storage device will appear in the list multiple times.
The client is free to use any of the network addresses as a The client is free to use any of the network addresses as a
destination to send storage device requests. If some network destination to send storage device requests. If some network
addresses are less optimal paths to the data than others, then the addresses are less optimal paths to the data than others, then the
MDS SHOULD NOT include those network addresses in ffda_netaddrs. If MDS SHOULD NOT include those network addresses in ffda_netaddrs. If
less optimal network addresses exist to provide failover, the less optimal network addresses exist to provide failover, the
RECOMMENDED method to offer the addresses is to provide them in a RECOMMENDED method to offer the addresses is to provide them in a
replacement device-ID-to-device-address mapping, or a replacement replacement device-ID-to-device-address mapping, or a replacement
skipping to change at page 12, line 4 skipping to change at page 12, line 8
}; };
This document defines structure associated with the layouttype4 value This document defines structure associated with the layouttype4 value
LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an
XDR type "opaque". The opaque layout is uninterpreted by the generic XDR type "opaque". The opaque layout is uninterpreted by the generic
pNFS client layers, but obviously must be interpreted by the Flexible pNFS client layers, but obviously must be interpreted by the Flexible
File Layout Type implementation. This section defines the structure File Layout Type implementation. This section defines the structure
of this opaque value, ff_layout4. of this opaque value, ff_layout4.
5.1. ff_layout4 5.1. ff_layout4
/// struct ff_mirror4 {
/// deviceid4 ffm_deviceid; /// struct ff_data_server4 {
/// nfs_fh4 ffm_fhandle; /// deviceid4 ffds_deviceid;
/// stateid4 ffm_stateid; /// uint32_t ffds_efficiency;
/// opaque_auth ffm_auth; /// stateid4 ffds_stateid;
/// nfs_fh4 ffds_fhandle;
/// opaque_auth ffds_auth;
/// }; /// };
/// ///
/// struct ff_stripe4 { /// struct ff_mirror4 {
/// ff_mirror4 ffs_mirrors<>; /// ff_data_server4 ffm_data_servers<>;
/// }; /// };
/// ///
/// struct ff_layout4 { /// struct ff_layout4 {
/// length4 ffl_stripe_unit; /// length4 ffl_stripe_unit;
/// ff_stripe4 ffl_stripes<>; /// ff_mirror4 ffl_mirrors<>;
/// }; /// };
/// ///
The ff_layout4 structure specifies a layout over a set of mirrored The ff_layout4 structure specifies a layout over a set of mirrored
copies of the data file. This mirroring protects against loss of copies of the data file. This mirroring protects against loss of
data files. data files.
It is possible that the file is concatenated from more than one It is possible that the file is concatenated from more than one
layout segment. Each layout segment MAY represent different striping layout segment. Each layout segment MAY represent different striping
parameters, applying respectively only to the layout segment byte parameters, applying respectively only to the layout segment byte
range. range.
The ffl_stripe_unit field is the stripe unit size in use for the The ffl_stripe_unit field is the stripe unit size in use for the
current layout segment. The number of stripes is given by the number current layout segment. The number of stripes is given inside each
of elements in ffl_stripes. mirror by the number of elements in ffm_data_servers. The only
supported mapping scheme is sparse and is detailed in Section 6.
Note that there is an assumption here that both the stripe unit size
and the number of of stripes is the same across all mirrors.
The ffl_stripes field is the array of mirrored storage devices which The ffl_mirrors field is the array of mirrored storage devices which
provide the storage for the current stripe. I.e., as shown in provide the storage for the current stripe, see Figure 1.
Figure 1, each i'th storage device in ffl_stripes is part of the i'th
mirror of the layout segment.
+-----------+ +-----------+
| | | |
| | | |
| Server | | File |
| | | |
| | | |
+-----+-----+ +-----+-----+
| |
+------------+------------+ +------------+------------+
| | | |
+----+-----+ +-----+----+
| Mirror 1 | | Mirror 2 |
+----+-----+ +-----+----+
| | | |
+-----------+ +-----------+ +-----------+ +-----------+
|+-----------+ |+-----------+ |+-----------+ |+-----------+
||+-----------+ ||+-----------+ ||+-----------+ ||+-----------+
||| | ||| |
+|| Storage | +|| Storage | +|| Storage | +|| Storage |
+| Devices | +| Devices | +| Devices | +| Devices |
+-----------+ +-----------+ +-----------+ +-----------+
Figure 1 Figure 1
The ffs_mirrors field represents an array of state information for The ffs_mirrors field represents an array of state information for
each mirrored copy of the file. Each element is described by a each mirrored copy of the file. Each element is described by a
ff_mirror4 type. [[AI9: Still need to describe SPARSE vs DENSE! ff_mirror4 type.
--TH]]
ffm_deviceid provides the deviceid of the storage device holding the ffds_deviceid provides the deviceid of the storage device holding the
data file. data file.
ffm_fhandle provides the filehandle of the data file on the given ffds_fhandle provides the filehandle of the data file on the given
storage device. For tight coupling, ffm_stateid provides the stateid storage device. For tight coupling, ffds_stateid provides the
to be used by the client to access the file. For loose coupling and stateid to be used by the client to access the file. For loose
a NFSv4 storage device, the client may use an anonymous stateid to coupling and a NFSv4 storage device, the client may use an anonymous
perform I/O on the storage device as there is no use for the metadata stateid to perform I/O on the storage device as there is no use for
server stateid (no control protocol). In such a scenario, the server the metadata server stateid (no control protocol). In such a
MUST set the ffm_stateid to be zero. scenario, the server MUST set the ffds_stateid to be zero.
For NFSv3 storage devices, ffm_auth provides the RPC credentials to For NFSv3 storage devices, ffds_auth provides the RPC credentials to
be used by the client to access the data files. For NFSv4.x storage be used by the client to access the data files. For NFSv4.x storage
devices, the server SHOULD use the AUTH_NONE flavor and a zero length devices, the server SHOULD use the AUTH_NONE flavor and a zero length
opaque body to minimize the returned structure length. The client opaque body to minimize the returned structure length. The client
MUST ignore ffm_auth in this case. [[AI6: Even for tightly coupled MUST ignore ffds_auth in this case. [[AI6: Even for tightly coupled
systems, that cannot be correct! --TH]] systems, that cannot be correct! --TH]]
6. Recovering from Client I/O Errors ffds_efficiency describes the metadata server's evaluation as to the
effectiveness of each mirror. Note that this is per layout and not
per device as the metric may change due to perceived load,
availability to the metadata server, etc. Higher values denote
higher perceived utility. The way the client can select the best
mirror to access is discussed in Section 8.1.
6. Striping via Sparse Mapping
While other Layout Types support both dense and sparse mapping of
logical offsets to phyisical offsets within a file (see for example
Section 13.4 of [RFC5661]), the Flexible File Layout Type only
supports a sparse mapping.
With sparse mappings, the logical offset within a file (L) is also
the physical offset on the storage device. As detailed in
Section 13.4.4 of [RFC5661], this results in holes across each
storage device which does not contain the current stripe index.
L: logical offset into the file
W: stripe width
W = number of elements in ffm_data_servers
S: number of bytes in a stripe
S = W * ffl_stripe_unit
N: stripe number
N = L / S
7. Recovering from Client I/O Errors
The pNFS client may encounter errors when directly accessing the The pNFS client may encounter errors when directly accessing the
storage devices. However, it is the responsibility of the metadata storage devices. However, it is the responsibility of the metadata
server to recover from the I/O errors. When the LAYOUT4_FLEX_FILES server to recover from the I/O errors. When the LAYOUT4_FLEX_FILES
layout type is used, the client MUST report the I/O errors to the layout type is used, the client MUST report the I/O errors to the
server at LAYOUTRETURN time using the ff_ioerr structure (see server at LAYOUTRETURN time using the ff_ioerr structure (see
Section 7.1). Section 9.1).
The metadata server analyzes the error and determines the required The metadata server analyzes the error and determines the required
recovery operations such as recovering media failures or recovery operations such as recovering media failures or
reconstructing missing data files. reconstructing missing data files.
The metadata server SHOULD recall any outstanding layouts to allow it The metadata server SHOULD recall any outstanding layouts to allow it
exclusive write access to the stripes being recovered and to prevent exclusive write access to the stripes being recovered and to prevent
other clients from hitting the same error condition. In these cases, other clients from hitting the same error condition. In these cases,
the server MUST complete recovery before handing out any new layouts the server MUST complete recovery before handing out any new layouts
to the affected byte ranges. to the affected byte ranges.
skipping to change at page 14, line 35 skipping to change at page 15, line 13
corresponding error to the application that initiated the I/O corresponding error to the application that initiated the I/O
operation and drop any unwritten data, the client SHOULD attempt to operation and drop any unwritten data, the client SHOULD attempt to
retry the original I/O operation by requesting a new layout using retry the original I/O operation by requesting a new layout using
LAYOUTGET and retry the I/O operation(s) using the new layout, or the LAYOUTGET and retry the I/O operation(s) using the new layout, or the
client MAY just retry the I/O operation(s) using regular NFS READ or client MAY just retry the I/O operation(s) using regular NFS READ or
WRITE operations via the metadata server. The client SHOULD attempt WRITE operations via the metadata server. The client SHOULD attempt
to retrieve a new layout and retry the I/O operation using the to retrieve a new layout and retry the I/O operation using the
storage device first and only if the error persists, retry the I/O storage device first and only if the error persists, retry the I/O
operation via the metadata server. operation via the metadata server.
7. Flexible Files Layout Type Return 8. Mirroring
The Flexible File Layout Type has a simple model in place for the
mirroring of files. There is no assumption that each copy of the
mirror is stored identically on the storage devices, i.e., one device
might employ compression or deduplication on the file. However, the
over the wire transfer of the file contents MUST appear identical.
Note, this is a construct of the selected XDR representation that
each mirrored copy of the file has the same striping pattern (see
Figure 1).
The metadata server is responsible for determining the number of
mirrored copies and the location of each mirror. While the client
may provide a hint to how many copies it wants (see Section 12), the
metadata server can ignore that hint and in any event, the client has
no means to dictate neither the storage device (which also means the
coupling and/or protocol levels to access the file) nor the location
of said storage device.
8.1. Selecting a Mirror
When the metadata server grants a layout to a client, it can let the
client know how fast it expects each mirror to be once the request
arrives at the storage devices via the ffds_efficiency member. While
the algorithms to calculate that value are left to the metadata
server implementations, factors that could contribute to that
calculation include speed of the storage device, physical memory
available to the device, operating system version, current load, etc.
However, what SHOULD not be involved in that calculation is a
perceived network distance between the client and the storage device.
The client is better situated for making that determination based on
past interaction with the storage device over the different available
network intefaces bewteen the two. I.e., the metadata server might
not know about a transient outage between the client and storage
device because it has no presence on the given subnet.
As such, it is the client which decides which mirror to access for
reading the file. The requirements for writing to a mirrored file
are presented below.
8.2. Writing to Mirrors
The client is responsible for updating all mirrored copies of the
file that it is given in the layout. If all but one copy is updated
successfully and the last one provides an error, then the client
needs to return the layout to the metadata server with an error
indicating that the update failed to that storage device.
The metadata server is then responsible for determining if it wants
to remove the errant mirror from the layout, if the mirror has
recovered from some transient error, etc. When the client tries to
get a new layout, the metadata server informs it of the decision by
the contents of the layout. The client MUST not make any assumptions
that the contents of the previous layout will match those of the new
one. If it has updates that were not committed, it MUST resend those
updates to all mirrors.
8.3. Metadata Server Resilvering of the File
The metadata server may elect to create a new mirror of the file at
any time. This might be to resilver a copy on a storage device which
was down for servicing, to provide a copy of the file on storage with
different storage performance characteristics, etc. As the client
will not be aware of the new mirror and the metadata server will not
be aware of updates that the client is making to the file, the
metadata server MUST recall the writable layout segment(s) that it is
resilvering. If the client issues a LAYOUTGET for a writable layout
segment which is in the process of being resilvered, then the
metadata server MUST deny that request with a NFS4ERR_LAYOUTTRYLATER.
The client can then perform the IO through the metadata server.
9. Flexible Files Layout Type Return
layoutreturn_file4 is used in the LAYOUTRETURN operation to convey layoutreturn_file4 is used in the LAYOUTRETURN operation to convey
layout-type specific information to the server. It is defined in layout-type specific information to the server. It is defined in
[RFC5661] as follows: [RFC5661] as follows:
struct layoutreturn_file4 { struct layoutreturn_file4 {
offset4 lrf_offset; offset4 lrf_offset;
length4 lrf_length; length4 lrf_length;
stateid4 lrf_stateid; stateid4 lrf_stateid;
/* layouttype4 specific data */ /* layouttype4 specific data */
skipping to change at page 15, line 25 skipping to change at page 17, line 34
layouttype4 lora_layout_type; layouttype4 lora_layout_type;
layoutiomode4 lora_iomode; layoutiomode4 lora_iomode;
layoutreturn4 lora_layoutreturn; layoutreturn4 lora_layoutreturn;
}; };
If the lora_layout_type layout type is LAYOUT4_FLEX_FILES, then the If the lora_layout_type layout type is LAYOUT4_FLEX_FILES, then the
lrf_body opaque value is defined by the ff_layoutreturn4 type. The lrf_body opaque value is defined by the ff_layoutreturn4 type. The
new type allows the client to report I/O error information or layout new type allows the client to report I/O error information or layout
usage statistics back to the metadata server as defined below. usage statistics back to the metadata server as defined below.
7.1. ff_ioerr 9.1. ff_ioerr
/// struct ff_ioerr4 { /// struct ff_ioerr4 {
/// offset4 ffie_offset; /// offset4 ffie_offset;
/// length4 ffie_length; /// length4 ffie_length;
/// stateid4 ffie_stateid; /// stateid4 ffie_stateid;
/// device_error4 ffie_errors; /// device_error4 ffie_errors;
/// }; /// };
/// ///
Recall that [NFSv42] defines device_error4 as: Recall that [NFSv42] defines device_error4 as:
skipping to change at page 16, line 7 skipping to change at page 18, line 20
The ff_ioerr4 structure is used to return error indications for data The ff_ioerr4 structure is used to return error indications for data
files that generated errors during data transfers. These are hints files that generated errors during data transfers. These are hints
to the metadata server that there are problems with that file. For to the metadata server that there are problems with that file. For
each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length
represent the storage device and byte range within the file in which represent the storage device and byte range within the file in which
the error occurred; ffie_errors represents the operation and type of the error occurred; ffie_errors represents the operation and type of
error. The use of device_error4 is described in Section 16.6 of error. The use of device_error4 is described in Section 16.6 of
[NFSv42]. [NFSv42].
7.2. ff_iostats 9.2. ff_iostats
/// struct ff_iostats4 { /// struct ff_iostats4 {
/// offset4 ffis_offset; /// offset4 ffis_offset;
/// length4 ffis_length; /// length4 ffis_length;
/// stateid4 ffis_stateid; /// stateid4 ffis_stateid;
/// uint32_t ffis_duration; /// uint32_t ffis_duration;
/// io_info4 ffis_read; /// io_info4 ffis_read;
/// io_info4 ffis_write; /// io_info4 ffis_write;
/// layoutupdate4 ffis_layoutupdate; /// layoutupdate4 ffis_layoutupdate;
/// }; /// };
skipping to change at page 17, line 7 skipping to change at page 19, line 18
length in bytes. ffis_duration represents the number of seconds the length in bytes. ffis_duration represents the number of seconds the
reported burst of I/O lasted. ffis_read.ii_count, reported burst of I/O lasted. ffis_read.ii_count,
ffis_read.ii_bytes, ffis_write.ii_count, and ffis_write.ii_bytes ffis_read.ii_bytes, ffis_write.ii_count, and ffis_write.ii_bytes
represent, respectively, the number of contiguous read and write I/Os represent, respectively, the number of contiguous read and write I/Os
and the respective aggregate number of bytes transferred within the and the respective aggregate number of bytes transferred within the
reported byte range. [[AI7: Need to define whether we are using reported byte range. [[AI7: Need to define whether we are using
ffis_layoutupdate or not. --TH]] [[AI8: Actually, ffis_duration ffis_layoutupdate or not. --TH]] [[AI8: Actually, ffis_duration
might be what we plop down in there. In any event, ffis_duration might be what we plop down in there. In any event, ffis_duration
needs some work. --TH]] needs some work. --TH]]
7.3. ff_layoutreturn 9.3. ff_layoutreturn
/// struct ff_layoutreturn { /// struct ff_layoutreturn {
/// ff_ioerr4 fflr_ioerr_report<>; /// ff_ioerr4 fflr_ioerr_report<>;
/// ff_iostats4 fflr_iostats_report<>; /// ff_iostats4 fflr_iostats_report<>;
/// }; /// };
/// ///
When data file I/O operations fail, fflr_ioerr_report<> is used to When data file I/O operations fail, fflr_ioerr_report<> is used to
report these errors to the metadata server as an array of elements of report these errors to the metadata server as an array of elements of
type ff_ioerr4. Each element in the array represents an error that type ff_ioerr4. Each element in the array represents an error that
occurred on the data file identified by ffie_errors.de_deviceid. If occurred on the data file identified by ffie_errors.de_deviceid. If
no errors are to be reported, the size of the fflr_ioerr_report<> no errors are to be reported, the size of the fflr_ioerr_report<>
array is set to zero. The client MAY also use fflr_iostats_report<> array is set to zero. The client MAY also use fflr_iostats_report<>
to report a list of I/O statistics as an array of elements of type to report a list of I/O statistics as an array of elements of type
ff_iostats4. Each element in the array represents statistics for a ff_iostats4. Each element in the array represents statistics for a
particular byte range. Byte ranges are not guaranteed to be disjoint particular byte range. Byte ranges are not guaranteed to be disjoint
and MAY repeat or intersect. and MAY repeat or intersect.
8. Flexible Files Layout Type LAYOUTERROR 10. Flexible Files Layout Type LAYOUTERROR
If the client is using NFSv4.2 to communicate with the metadata If the client is using NFSv4.2 to communicate with the metadata
server, then instead of waiting for a LAYOUTRETURN to send error server, then instead of waiting for a LAYOUTRETURN to send error
information to the metadata server (see Section 7.1), it can use information to the metadata server (see Section 9.1), it can use
LAYOUTERROR (see Section 16.6 of [NFSv42]) to communicate that LAYOUTERROR (see Section 16.6 of [NFSv42]) to communicate that
information. information.
9. Flexible Files Layout Type LAYOUTSTATS 11. Flexible Files Layout Type LAYOUTSTATS
If the client is using NFSv4.2 to communicate with the metadata If the client is using NFSv4.2 to communicate with the metadata
server, then instead of waiting for a LAYOUTRETURN to send I/O server, then instead of waiting for a LAYOUTRETURN to send I/O
statistics to the metadata server (see Section 7.2), it can use statistics to the metadata server (see Section 9.2), it can use
LAYOUTSTATS (see Section 16.7 of [NFSv42]) to communicate that LAYOUTSTATS (see Section 16.7 of [NFSv42]) to communicate that
information. information.
10. Flexible File Layout Type Creation Hint 12. Flexible File Layout Type Creation Hint
The layouthint4 type is defined in the [RFC5661] as follows: The layouthint4 type is defined in the [RFC5661] as follows:
struct layouthint4 { struct layouthint4 {
layouttype4 loh_type; layouttype4 loh_type;
opaque loh_body<>; opaque loh_body<>;
}; };
The layouthint4 structure is used by the client to pass a hint about The layouthint4 structure is used by the client to pass a hint about
the type of layout it would like created for a particular file. If the type of layout it would like created for a particular file. If
the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body
opaque value is defined by the ff_layouthint4 type. opaque value is defined by the ff_layouthint4 type.
10.1. ff_layouthint4 12.1. ff_layouthint4
/// union ff_mirrors_hint switch (bool ffmc_valid) { /// union ff_mirrors_hint switch (bool ffmc_valid) {
/// case TRUE: /// case TRUE:
/// uint32_t ffmc_mirrors; /// uint32_t ffmc_mirrors;
/// case FALSE: /// case FALSE:
/// void; /// void;
/// }; /// };
/// ///
/// struct ff_layouthint4 { /// struct ff_layouthint4 {
/// ff_mirrors_hint fflh_mirrors_hint; /// ff_mirrors_hint fflh_mirrors_hint;
/// }; /// };
/// ///
This type conveys hints for the desired data map. All parameters are This type conveys hints for the desired data map. All parameters are
optional so the client can give values for only the parameter it optional so the client can give values for only the parameter it
cares about. cares about.
11. Recalling Layouts 13. Recalling Layouts
The Flexible File Layout Type metadata server should recall The Flexible File Layout Type metadata server should recall
outstanding layouts in the following cases: outstanding layouts in the following cases:
o When the file's security policy changes, i.e., Access Control o When the file's security policy changes, i.e., Access Control
Lists (ACLs) or permission mode bits are set. Lists (ACLs) or permission mode bits are set.
o When the file's layout changes, rendering outstanding layouts o When the file's layout changes, rendering outstanding layouts
invalid. invalid.
o When there are sharing conflicts. o When there are sharing conflicts.
11.1. CB_RECALL_ANY 13.1. CB_RECALL_ANY
The metadata server can use the CB_RECALL_ANY callback operation to The metadata server can use the CB_RECALL_ANY callback operation to
notify the client to return some or all of its layouts. The notify the client to return some or all of its layouts. The
[RFC5661] defines the following types: [RFC5661] defines the following types:
const RCA4_TYPE_MASK_FF_LAYOUT_MIN = -2; const RCA4_TYPE_MASK_FF_LAYOUT_MIN = -2;
const RCA4_TYPE_MASK_FF_LAYOUT_MAX = -1; const RCA4_TYPE_MASK_FF_LAYOUT_MAX = -1;
[[RFC Editor: please insert assigned constants]] [[RFC Editor: please insert assigned constants]]
struct CB_RECALL_ANY4args { struct CB_RECALL_ANY4args {
skipping to change at page 19, line 38 skipping to change at page 21, line 44
They represent the iomode of the recalled layouts. In response, the They represent the iomode of the recalled layouts. In response, the
client SHOULD return layouts of the recalled iomode that it needs the client SHOULD return layouts of the recalled iomode that it needs the
least, keeping at most craa_layouts_to_keep Flexible File Layouts. least, keeping at most craa_layouts_to_keep Flexible File Layouts.
The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return
layouts of iomode LAYOUTIOMODE4_READ. Similarly, the layouts of iomode LAYOUTIOMODE4_READ. Similarly, the
PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts
of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client
is notified to return layouts of either iomode. is notified to return layouts of either iomode.
12. Client Fencing 14. Client Fencing
In cases where clients are uncommunicative and their lease has In cases where clients are uncommunicative and their lease has
expired or when clients fail to return recalled layouts within a expired or when clients fail to return recalled layouts within a
lease period, at the least the server MAY revoke client layouts and/ lease period, at the least the server MAY revoke client layouts and/
or device address mappings and reassign these resources to other or device address mappings and reassign these resources to other
clients (see "Recalling a Layout" in [RFC5661]). To avoid data clients (see "Recalling a Layout" in [RFC5661]). To avoid data
corruption, the metadata server MUST fence off the revoked clients corruption, the metadata server MUST fence off the revoked clients
from the respective data files as described in Section 2.2. from the respective data files as described in Section 2.2.
13. Security Considerations 15. Security Considerations
The pNFS extension partitions the NFSv4 file system protocol into two The pNFS extension partitions the NFSv4 file system protocol into two
parts, the control path and the data path (storage protocol). The parts, the control path and the data path (storage protocol). The
control path contains all the new operations described by this control path contains all the new operations described by this
extension; all existing NFSv4 security mechanisms and features apply extension; all existing NFSv4 security mechanisms and features apply
to the control path. The combination of components in a pNFS system to the control path. The combination of components in a pNFS system
is required to preserve the security properties of NFSv4 with respect is required to preserve the security properties of NFSv4 with respect
to an entity accessing data via a client, including security to an entity accessing data via a client, including security
countermeasures to defend against threats that NFSv4 provides countermeasures to defend against threats that NFSv4 provides
defenses for in environments where these threats are considered defenses for in environments where these threats are considered
skipping to change at page 20, line 37 skipping to change at page 22, line 43
permissions or ACL, it SHOULD recall all layouts for that file and it permissions or ACL, it SHOULD recall all layouts for that file and it
MUST fence off the clients holding outstanding layouts for the MUST fence off the clients holding outstanding layouts for the
respective file by implicitly invalidating the outstanding respective file by implicitly invalidating the outstanding
credentials on all data files comprising before committing to the new credentials on all data files comprising before committing to the new
permissions and ACL. Doing this will ensure that clients re- permissions and ACL. Doing this will ensure that clients re-
authorize their layouts according to the modified permissions and ACL authorize their layouts according to the modified permissions and ACL
by requesting new layouts. Recalling the layouts in this case is by requesting new layouts. Recalling the layouts in this case is
courtesy of the server intended to prevent clients from getting an courtesy of the server intended to prevent clients from getting an
error on I/Os done after the client was fenced off. error on I/Os done after the client was fenced off.
14. IANA Considerations 16. IANA Considerations
As described in [RFC5661], new layout type numbers have been assigned As described in [RFC5661], new layout type numbers have been assigned
by IANA. This document defines the protocol associated with the by IANA. This document defines the protocol associated with the
existing layout type number, LAYOUT4_FLEX_FILES. existing layout type number, LAYOUT4_FLEX_FILES.
15. References 17. References
15.1. Normative References 17.1. Normative References
[LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents", [LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents",
November 2008, <http://trustee.ietf.org/docs/ November 2008, <http://trustee.ietf.org/docs/
IETF-Trust-License-Policy.pdf>. IETF-Trust-License-Policy.pdf>.
[NFSv42] Haynes, T., "NFS Version 4 Minor Version 2", draft-ietf- [NFSv42] Haynes, T., "NFS Version 4 Minor Version 2", draft-ietf-
nfsv4-minorversion2-22 (Work In Progress), April 2014. nfsv4-minorversion2-22 (Work In Progress), April 2014.
[RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813, [RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813,
June 1995. June 1995.
skipping to change at page 21, line 39 skipping to change at page 23, line 47
[RFC5664] Halevy, B., Ed., Welch, B., Ed., and J. Zelenka, Ed., [RFC5664] Halevy, B., Ed., Welch, B., Ed., and J. Zelenka, Ed.,
"Object-Based Parallel NFS (pNFS) Operations", RFC 5664, "Object-Based Parallel NFS (pNFS) Operations", RFC 5664,
January 2010. January 2010.
[pNFSLayouts] [pNFSLayouts]
Haynes, T., "Considerations for a New pNFS Layout Type", Haynes, T., "Considerations for a New pNFS Layout Type",
draft-haynes-nfsv4-layout-types-02 (Work In Progress), draft-haynes-nfsv4-layout-types-02 (Work In Progress),
April 2014. April 2014.
15.2. Informative References 17.2. Informative References
[ANSI400-2004] [ANSI400-2004]
Weber, R., Ed., "ANSI INCITS 400-2004, Information Weber, R., Ed., "ANSI INCITS 400-2004, Information
Technology - SCSI Object-Based Storage Device Commands Technology - SCSI Object-Based Storage Device Commands
(OSD)", December 2004. (OSD)", December 2004.
Appendix A. Acknowledgments Appendix A. Acknowledgments
Those who provided miscellaneous comments to early drafts of this Those who provided miscellaneous comments to early drafts of this
document include: Matt W. Benjamin, Adam Emerson, Tom Haynes, J. document include: Matt W. Benjamin, Adam Emerson, Tom Haynes, J.
 End of changes. 46 change blocks. 
100 lines changed or deleted 214 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/