draft-ietf-nfsv4-flex-files-03.txt   draft-ietf-nfsv4-flex-files-04.txt 
NFSv4 B. Halevy NFSv4 B. Halevy
Internet-Draft T. Haynes Internet-Draft T. Haynes
Intended status: Informational Primary Data Intended status: Informational Primary Data
Expires: June 4, 2015 December 01, 2014 Expires: June 7, 2015 December 04, 2014
Parallel NFS (pNFS) Flexible File Layout Parallel NFS (pNFS) Flexible File Layout
draft-ietf-nfsv4-flex-files-03.txt draft-ietf-nfsv4-flex-files-04.txt
Abstract Abstract
The Parallel Network File System (pNFS) allows a separation between The Parallel Network File System (pNFS) allows a separation between
the metadata and data for a file. The metadata file access is the metadata and data for a file. The metadata file access is
handled via Network File System version 4 (NFSv4) minor version 1 handled via Network File System version 4 (NFSv4) minor version 1
(NFSv4.1) and the data file access is specific to the protocol being (NFSv4.1) and the data file access is specific to the protocol being
used between the client and storage device. The client is informed used between the client and storage device. The client is informed
by the metadata server as to which protocol to use via a Layout Type. by the metadata server as to which protocol to use via a Layout Type.
The Flexible File Layout Type is defined in this document as an The Flexible File Layout Type is defined in this document as an
skipping to change at page 1, line 38 skipping to change at page 1, line 38
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on June 4, 2015. This Internet-Draft will expire on June 7, 2015.
Copyright Notice Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 23 skipping to change at page 2, line 23
1.2. Difference Between a Data Server and a Storage Device . . 5 1.2. Difference Between a Data Server and a Storage Device . . 5
1.3. Requirements Language . . . . . . . . . . . . . . . . . . 5 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 5
2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 6 2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 6
2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6 2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6
2.2. Security Models . . . . . . . . . . . . . . . . . . . . . 6 2.2. Security Models . . . . . . . . . . . . . . . . . . . . . 6
2.3. State and Locking Models . . . . . . . . . . . . . . . . 7 2.3. State and Locking Models . . . . . . . . . . . . . . . . 7
3. XDR Description of the Flexible File Layout Type . . . . . . 7 3. XDR Description of the Flexible File Layout Type . . . . . . 7
3.1. Code Components Licensing Notice . . . . . . . . . . . . 8 3.1. Code Components Licensing Notice . . . . . . . . . . . . 8
4. Device Addressing and Discovery . . . . . . . . . . . . . . . 9 4. Device Addressing and Discovery . . . . . . . . . . . . . . . 9
4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 9 4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 9
4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 10 4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 11
5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 11 5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 12
5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 12 5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 12
5.2. Interactions Between Devices and Layouts . . . . . . . . 14 5.2. Interactions Between Devices and Layouts . . . . . . . . 15
6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 14 5.3. Handling Version Errors . . . . . . . . . . . . . . . . . 15
7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 15 6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 16
8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 15 7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 16
8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 16 8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 17 8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 17
8.3. Metadata Server Resilvering of the File . . . . . . . . . 17 8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 18
9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 17 8.3. Metadata Server Resilvering of the File . . . . . . . . . 18
9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 18 9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 19
9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 18 9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 19
9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 19 9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 19
9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 19 9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 20
9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 19 9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 20
9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 20 9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 21
9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 21 9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 22
10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 21 9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 23
11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 21 10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 23
12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 21 11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 23
12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 22 12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 24
13. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 22 12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 24
13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 22 13. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 25
14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 23 13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 25
15. Security Considerations . . . . . . . . . . . . . . . . . . . 24 14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 26
15.1. Kerberized File Access . . . . . . . . . . . . . . . . . 24 15. Security Considerations . . . . . . . . . . . . . . . . . . . 26
15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 24 15.1. Kerberized File Access . . . . . . . . . . . . . . . . . 27
15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 25 15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 27
15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 27
16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 25 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28
17. References . . . . . . . . . . . . . . . . . . . . . . . . . 25 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 28
17.1. Normative References . . . . . . . . . . . . . . . . . . 25 17.1. Normative References . . . . . . . . . . . . . . . . . . 28
17.2. Informative References . . . . . . . . . . . . . . . . . 26 17.2. Informative References . . . . . . . . . . . . . . . . . 29
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 26 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 29
Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 26 Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 29
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 29
1. Introduction 1. Introduction
In the parallel Network File System (pNFS), the metadata server In the parallel Network File System (pNFS), the metadata server
returns Layout Type structures that describe where file data is returns Layout Type structures that describe where file data is
located. There are different Layout Types for different storage located. There are different Layout Types for different storage
systems and methods of arranging data on storage devices. This systems and methods of arranging data on storage devices. This
document defines the Flexible File Layout Type used with file-based document defines the Flexible File Layout Type used with file-based
data servers that are accessed using the Network File System (NFS) data servers that are accessed using the Network File System (NFS)
protocols: NFSv3 [RFC1813], NFSv4 [RFC3530], NFSv4.1 [RFC5661], and protocols: NFSv3 [RFC1813], NFSv4 [RFC3530], NFSv4.1 [RFC5661], and
skipping to change at page 3, line 38 skipping to change at page 3, line 38
yet the requirements for the protocol are specified in [RFC5661] and yet the requirements for the protocol are specified in [RFC5661] and
clarified in [pNFSLayouts]. clarified in [pNFSLayouts].
1.1. Definitions 1.1. Definitions
control protocol: is a set of requirements for the communication of control protocol: is a set of requirements for the communication of
information on layouts, stateids, file metadata, and file data information on layouts, stateids, file metadata, and file data
between the metadata server and the storage devices (see between the metadata server and the storage devices (see
[pNFSLayouts]). [pNFSLayouts]).
Client-side Mirroring: is when the client and not the server is client-side mirroring: is when the client and not the server is
responsible for updating all of the mirrored copies of a file. responsible for updating all of the mirrored copies of a file.
data file: is that part of the file system object which describes data file: is that part of the file system object which describes
the payload and not the object. E.g., it is the file contents. the payload and not the object. E.g., it is the file contents.
Data Server (DS): is one of the pNFS servers which provide the data server (DS): is one of the pNFS servers which provide the
contents of a file system object which is a regular file. contents of a file system object which is a regular file.
Depending on the layout, there might be one or more data servers Depending on the layout, there might be one or more data servers
over which the data is striped. Note that while the metadata over which the data is striped. Note that while the metadata
server is strictly accessed over the NFSv4.1 protocol, depending server is strictly accessed over the NFSv4.1 protocol, depending
on the Layout Type, the data server could be accessed via any on the Layout Type, the data server could be accessed via any
protocol that meets the pNFS requirements. protocol that meets the pNFS requirements.
fencing: is when the metadata server prevents the storage devices fencing: is when the metadata server prevents the storage devices
from processing I/O from a specific client to a specific file. from processing I/O from a specific client to a specific file.
skipping to change at page 4, line 26 skipping to change at page 4, line 26
layout iomode: describes whether the layout granted to the client is layout iomode: describes whether the layout granted to the client is
for read or read/write I/O. for read or read/write I/O.
layout stateid: is a 128-bit quantity returned by a server that layout stateid: is a 128-bit quantity returned by a server that
uniquely defines the layout state provided by the server for a uniquely defines the layout state provided by the server for a
specific layout that describes a Layout Type and file (see specific layout that describes a Layout Type and file (see
Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 describes Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 describes
the difference between a layout stateid and a normal stateid. the difference between a layout stateid and a normal stateid.
Layout Type: describes both the storage protocol used to access the layout type: describes both the storage protocol used to access the
data and the aggregation scheme used to lays out the file data on data and the aggregation scheme used to lays out the file data on
the underlying storage devices. the underlying storage devices.
loose coupling: is when the metadata server and the storage devices loose coupling: is when the metadata server and the storage devices
do not have a control protocol present. do not have a control protocol present.
metadata file: is that part of the file system object which metadata file: is that part of the file system object which
describes the object and not the payload. E.g., it could be the describes the object and not the payload. E.g., it could be the
time since last modification, access, etc. time since last modification, access, etc.
Metadata Server (MDS): is the pNFS server which provides metadata metadata server (MDS): is the pNFS server which provides metadata
information for a file system object. It also is responsible for information for a file system object. It also is responsible for
generating layouts for file system objects. Note that the MDS is generating layouts for file system objects. Note that the MDS is
responsible for directory-based operations. responsible for directory-based operations.
Mirror: is a copy of a file. While mirroring can be used for mirror: is a copy of a file. While mirroring can be used for
backing up a file, the copies can be distributed such that each backing up a file, the copies can be distributed such that each
remote site has a locally cached copy. Note that if one copy of remote site has a locally cached copy. Note that if one copy of
the mirror is updated, then all copies must be updated. the mirror is updated, then all copies must be updated.
Object Layout Type: is a Layout Type in which the storage devices Object Layout Type: is a Layout Type in which the storage devices
are accessed via the OSD protocol [ANSI400-2004]. It is defined are accessed via the OSD protocol [ANSI400-2004]. It is defined
in [RFC5664]. in [RFC5664].
recalling a layout: is when the metadata server uses a back channel recalling a layout: is when the metadata server uses a back channel
to inform the client that the layout is to be returned in a to inform the client that the layout is to be returned in a
skipping to change at page 7, line 39 skipping to change at page 7, line 39
3. XDR Description of the Flexible File Layout Type 3. XDR Description of the Flexible File Layout Type
This document contains the external data representation (XDR) This document contains the external data representation (XDR)
[RFC4506] description of the Flexible File Layout Type. The XDR [RFC4506] description of the Flexible File Layout Type. The XDR
description is embedded in this document in a way that makes it description is embedded in this document in a way that makes it
simple for the reader to extract into a ready-to-compile form. The simple for the reader to extract into a ready-to-compile form. The
reader can feed this document into the following shell script to reader can feed this document into the following shell script to
produce the machine readable XDR description of the Flexible File produce the machine readable XDR description of the Flexible File
Layout Type: Layout Type:
<CODE BEGINS>
#!/bin/sh #!/bin/sh
grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??' grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'
<CODE ENDS>
That is, if the above script is stored in a file called "extract.sh", That is, if the above script is stored in a file called "extract.sh",
and this document is in a file called "spec.txt", then the reader can and this document is in a file called "spec.txt", then the reader can
do: do:
sh extract.sh < spec.txt > flex_files_prot.x sh extract.sh < spec.txt > flex_files_prot.x
The effect of the script is to remove leading white space from each The effect of the script is to remove leading white space from each
line, plus a sentinel sequence of "///". line, plus a sentinel sequence of "///".
The embedded XDR file header follows. Subsequent XDR descriptions, The embedded XDR file header follows. Subsequent XDR descriptions,
with the sentinel sequence are embedded throughout the document. with the sentinel sequence are embedded throughout the document.
Note that the XDR code contained in this document depends on types Note that the XDR code contained in this document depends on types
from the NFSv4.1 nfs4_prot.x file [RFC5662]. This includes both nfs from the NFSv4.1 nfs4_prot.x file [RFC5662]. This includes both nfs
types that end with a 4, such as offset4, length4, etc., as well as types that end with a 4, such as offset4, length4, etc., as well as
more generic types such as uint32_t and uint64_t. more generic types such as uint32_t and uint64_t.
skipping to change at page 8, line 20 skipping to change at page 8, line 22
types that end with a 4, such as offset4, length4, etc., as well as types that end with a 4, such as offset4, length4, etc., as well as
more generic types such as uint32_t and uint64_t. more generic types such as uint32_t and uint64_t.
3.1. Code Components Licensing Notice 3.1. Code Components Licensing Notice
Both the XDR description and the scripts used for extracting the XDR Both the XDR description and the scripts used for extracting the XDR
description are Code Components as described in Section 4 of "Legal description are Code Components as described in Section 4 of "Legal
Provisions Relating to IETF Documents" [LEGAL]. These Code Provisions Relating to IETF Documents" [LEGAL]. These Code
Components are licensed according to the terms of that document. Components are licensed according to the terms of that document.
<CODE BEGINS>
/// /* /// /*
/// * Copyright (c) 2012 IETF Trust and the persons identified /// * Copyright (c) 2012 IETF Trust and the persons identified
/// * as authors of the code. All rights reserved. /// * as authors of the code. All rights reserved.
/// * /// *
/// * Redistribution and use in source and binary forms, with /// * Redistribution and use in source and binary forms, with
/// * or without modification, are permitted provided that the /// * or without modification, are permitted provided that the
/// * following conditions are met: /// * following conditions are met:
/// * /// *
/// * o Redistributions of source code must retain the above /// * o Redistributions of source code must retain the above
/// * copyright notice, this list of conditions and the /// * copyright notice, this list of conditions and the
skipping to change at page 9, line 26 skipping to change at page 9, line 31
/// ///
/// /* /// /*
/// * The following include statements are for example only. /// * The following include statements are for example only.
/// * The actual XDR definition files are generated separately /// * The actual XDR definition files are generated separately
/// * and independently and are likely to have a different name. /// * and independently and are likely to have a different name.
/// * %#include <nfsv42.x> /// * %#include <nfsv42.x>
/// * %#include <rpc_prot.x> /// * %#include <rpc_prot.x>
/// */ /// */
/// ///
<CODE ENDS>
4. Device Addressing and Discovery 4. Device Addressing and Discovery
Data operations to a storage device require the client to know the Data operations to a storage device require the client to know the
network address of the storage device. The NFSv4.1 GETDEVICEINFO network address of the storage device. The NFSv4.1 GETDEVICEINFO
operation (Section 18.40 of [RFC5661]) is used by the client to operation (Section 18.40 of [RFC5661]) is used by the client to
retrieve that information. retrieve that information.
4.1. ff_device_addr4 4.1. ff_device_addr4
The ff_device_addr4 data structure is returned by the server as the The ff_device_addr4 data structure is returned by the server as the
storage protocol specific opaque field da_addr_body in the storage protocol specific opaque field da_addr_body in the
device_addr4 structure by a successful GETDEVICEINFO operation. device_addr4 structure by a successful GETDEVICEINFO operation.
<CODE BEGINS>
/// struct ff_device_versions4 {
/// uint32_t ffdv_version;
/// uint32_t ffdv_minorversion;
/// uint32_t ffdv_rsize;
/// uint32_t ffdv_wsize;
/// bool ffdv_tightly_coupled;
/// };
///
/// struct ff_device_addr4 { /// struct ff_device_addr4 {
/// multipath_list4 ffda_netaddrs; /// multipath_list4 ffda_netaddrs;
/// uint32_t ffda_version; /// ff_device_versions4 ffda_versions<>;
/// uint32_t ffda_minorversion;
/// uint32_t ffda_rsize;
/// uint32_t ffda_wsize;
/// bool ffda_tightly_coupled;
/// }; /// };
/// ///
<CODE ENDS>
The ffda_netaddrs field is used to locate the storage device. It The ffda_netaddrs field is used to locate the storage device. It
MUST be set by the server to a list holding one or more of the device MUST be set by the server to a list holding one or more of the device
network addresses. network addresses.
The ffda_version and ffda_minorversion represent the NFS protocol to The ffda_versions array allows the metadata server to present
be used to access the storage device. This layout specification multiple NFS versions and/or minor versions to the client. The
defines the semantics for ffda_versions 3 and 4. If ffda_version ffdv_version and ffdv_minorversion represent the NFS protocol to be
equals 3 then server MUST set ffda_minorversion to 0 and the client used to access the storage device. This layout specification defines
MUST access the storage device using the NFSv3 protocol [RFC1813]. the semantics for ffdv_versions 3 and 4. If ffdv_version equals 3
If ffda_version equals 4 then the server MUST set ffda_minorversion then server MUST set ffdv_minorversion to 0 and the client MUST
to one of the NFSv4 minor version numbers and the client MUST access access the storage device using the NFSv3 protocol [RFC1813]. If
the storage device using NFSv4. ffdv_version equals 4 then the server MUST set ffdv_minorversion to
one of the NFSv4 minor version numbers and the client MUST access the
storage device using NFSv4.
The ffda_rsize and ffda_wsize are used to communicate the maximum Note that while the client might determine that it can not use any of
the configured ffdv_version or ffdv_minorversion, when it gets the
device list from the metadata server, there is no way to indicate to
the metadata server as to which device it is version incompatible.
If however the client waits until it retrieves the layout from the
metadata server, it can at that time clearly identify the storage
device in question (see Section 5.3).
The ffdv_rsize and ffdv_wsize are used to communicate the maximum
rsize and wsize supported by the storage device. As the storage rsize and wsize supported by the storage device. As the storage
device can have a different rsize or wsize than the metadata server, device can have a different rsize or wsize than the metadata server,
the ffda_rsize and ffda_wsize allow the metadata server to the ffdv_rsize and ffdv_wsize allow the metadata server to
communicate that information on behalf of the storage device. communicate that information on behalf of the storage device.
ffda_tightly_coupled informs the client as to whether the metadata ffdv_tightly_coupled informs the client as to whether the metadata
server is tightly coupled with the storage devices or not. Note that server is tightly coupled with the storage devices or not. Note that
even if the data protocol is at least NFSv4.1, it may still be the even if the data protocol is at least NFSv4.1, it may still be the
case that there is no control protocol present. If case that there is no control protocol present. If
ffda_tightly_coupled is not set, then the client MUST commit writes ffdv_tightly_coupled is not set, then the client MUST commit writes
to the storage devices for the file before sending a LAYOUTCOMMIT to to the storage devices for the file before sending a LAYOUTCOMMIT to
the metadata server. I.e., the writes MUST be committed by the the metadata server. I.e., the writes MUST be committed by the
client to stable storage via issuing WRITEs with stable_how == client to stable storage via issuing WRITEs with stable_how ==
FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how != FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how !=
FILE_SYNC (see Section 3.3.7 of [RFC1813]). FILE_SYNC (see Section 3.3.7 of [RFC1813]).
4.2. Storage Device Multipathing 4.2. Storage Device Multipathing
The Flexible File Layout Type supports multipathing to multiple The Flexible File Layout Type supports multipathing to multiple
storage device addresses. Storage device level multipathing is used storage device addresses. Storage device level multipathing is used
skipping to change at page 11, line 32 skipping to change at page 12, line 15
two storage device addresses will share the same server owner or two storage device addresses will share the same server owner or
major ID of the server owner. It is not always necessary for the two major ID of the server owner. It is not always necessary for the two
storage device addresses to designate the same storage device with storage device addresses to designate the same storage device with
trunking being used. For example, the data could be read-only, and trunking being used. For example, the data could be read-only, and
the data consist of exact replicas. the data consist of exact replicas.
5. Flexible File Layout Type 5. Flexible File Layout Type
The layout4 type is defined in [RFC5662] as follows: The layout4 type is defined in [RFC5662] as follows:
<CODE BEGINS>
enum layouttype4 { enum layouttype4 {
LAYOUT4_NFSV4_1_FILES = 1, LAYOUT4_NFSV4_1_FILES = 1,
LAYOUT4_OSD2_OBJECTS = 2, LAYOUT4_OSD2_OBJECTS = 2,
LAYOUT4_BLOCK_VOLUME = 3, LAYOUT4_BLOCK_VOLUME = 3,
LAYOUT4_FLEX_FILES = 0x80000005 LAYOUT4_FLEX_FILES = 0x80000005
[[RFC Editor: please modify the LAYOUT4_FLEX_FILES [[RFC Editor: please modify the LAYOUT4_FLEX_FILES
to be the layouttype assigned by IANA]] to be the layouttype assigned by IANA]]
}; };
struct layout_content4 { struct layout_content4 {
skipping to change at page 12, line 5 skipping to change at page 12, line 38
opaque loc_body<>; opaque loc_body<>;
}; };
struct layout4 { struct layout4 {
offset4 lo_offset; offset4 lo_offset;
length4 lo_length; length4 lo_length;
layoutiomode4 lo_iomode; layoutiomode4 lo_iomode;
layout_content4 lo_content; layout_content4 lo_content;
}; };
<CODE ENDS>
This document defines structure associated with the layouttype4 value This document defines structure associated with the layouttype4 value
LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an
XDR type "opaque". The opaque layout is uninterpreted by the generic XDR type "opaque". The opaque layout is uninterpreted by the generic
pNFS client layers, but obviously must be interpreted by the Flexible pNFS client layers, but obviously must be interpreted by the Flexible
File Layout Type implementation. This section defines the structure File Layout Type implementation. This section defines the structure
of this opaque value, ff_layout4. of this opaque value, ff_layout4.
5.1. ff_layout4 5.1. ff_layout4
<CODE BEGINS>
/// struct ff_data_server4 { /// struct ff_data_server4 {
/// deviceid4 ffds_deviceid; /// deviceid4 ffds_deviceid;
/// uint32_t ffds_efficiency; /// uint32_t ffds_efficiency;
/// stateid4 ffds_stateid; /// stateid4 ffds_stateid;
/// nfs_fh4 ffds_fhandle; /// nfs_fh4 ffds_fh_vers<>;
/// opaque_auth ffds_auth; /// opaque_auth ffds_auth;
/// }; /// };
/// ///
/// struct ff_mirror4 { /// struct ff_mirror4 {
/// ff_data_server4 ffm_data_servers<>; /// ff_data_server4 ffm_data_servers<>;
/// }; /// };
/// ///
/// struct ff_layout4 { /// struct ff_layout4 {
/// length4 ffl_stripe_unit; /// length4 ffl_stripe_unit;
/// ff_mirror4 ffl_mirrors<>; /// ff_mirror4 ffl_mirrors<>;
/// }; /// };
/// ///
<CODE ENDS>
The ff_layout4 structure specifies a layout over a set of mirrored The ff_layout4 structure specifies a layout over a set of mirrored
copies of the data file. This mirroring protects against loss of copies of the data file. This mirroring protects against loss of
data files. data files.
It is possible that the file is concatenated from more than one It is possible that the file is concatenated from more than one
layout segment. Each layout segment MAY represent different striping layout segment. Each layout segment MAY represent different striping
parameters, applying respectively only to the layout segment byte parameters, applying respectively only to the layout segment byte
range. range.
The ffl_stripe_unit field is the stripe unit size in use for the The ffl_stripe_unit field is the stripe unit size in use for the
skipping to change at page 13, line 38 skipping to change at page 14, line 35
Figure 1 Figure 1
The ffs_mirrors field represents an array of state information for The ffs_mirrors field represents an array of state information for
each mirrored copy of the file. Each element is described by a each mirrored copy of the file. Each element is described by a
ff_mirror4 type. ff_mirror4 type.
ffds_deviceid provides the deviceid of the storage device holding the ffds_deviceid provides the deviceid of the storage device holding the
data file. data file.
ffds_fhandle provides the filehandle of the data file on the given ffds_fh_vers is an array of filehandles of the data file matching to
storage device. For tight coupling, ffds_stateid provides the the available NFS versions on the given storage device. There MUST
stateid to be used by the client to access the file. For loose be exactly as many elements in ffds_fh_vers as there are in
coupling and a NFSv4 storage device, the client may use an anonymous ffda_versions. Each element of the array corresponds to each
stateid to perform I/O on the storage device as there is no use for ffdv_version and ffdv_minorversion provided for the device. The
the metadata server stateid (no control protocol). In such a array allows for server implementations which have different
scenario, the server MUST set the ffds_stateid to be zero. filehandles for different version and minor version combinations.
See Section 5.3 for how to handle versioning issues between the
client and storage devices.
For tight coupling, ffds_stateid provides the stateid to be used by
the client to access the file. For loose coupling and a NFSv4
storage device, the client may use an anonymous stateid to perform I/
O on the storage device as there is no use for the metadata server
stateid (no control protocol). In such a scenario, the server MUST
set the ffds_stateid to be zero.
For loosely coupled storage devices, ffds_auth provides the RPC For loosely coupled storage devices, ffds_auth provides the RPC
credentials to be used by the client to access the data files. For credentials to be used by the client to access the data files. For
tightly coupled storage devices, the server SHOULD use the AUTH_NONE tightly coupled storage devices, the server SHOULD use the AUTH_NONE
flavor and a zero length opaque body to minimize the returned flavor and a zero length opaque body to minimize the returned
structure length. I.e., if ffda_tightly_coupled (see Section 4.1) is structure length. I.e., if ffdv_tightly_coupled (see Section 4.1) is
set, then the client MUST ignore ffds_auth in this case. set, then the client MUST ignore ffds_auth in this case.
ffds_efficiency describes the metadata server's evaluation as to the ffds_efficiency describes the metadata server's evaluation as to the
effectiveness of each mirror. Note that this is per layout and not effectiveness of each mirror. Note that this is per layout and not
per device as the metric may change due to perceived load, per device as the metric may change due to perceived load,
availability to the metadata server, etc. Higher values denote availability to the metadata server, etc. Higher values denote
higher perceived utility. The way the client can select the best higher perceived utility. The way the client can select the best
mirror to access is discussed in Section 8.1. mirror to access is discussed in Section 8.1.
5.2. Interactions Between Devices and Layouts 5.2. Interactions Between Devices and Layouts
In [RFC5661], the File Layout Type is defined such that the In [RFC5661], the File Layout Type is defined such that the
relationship between multipathing and filehandles can result in relationship between multipathing and filehandles can result in
either 0, 1, or N filehandles (see Section 13.3). Some rationals for either 0, 1, or N filehandles (see Section 13.3). Some rationals for
this are clustered servers which share the same filehandle or this are clustered servers which share the same filehandle or
allowing for multiple read-only copies of the file on the same allowing for multiple read-only copies of the file on the same
storage device. In the Flexible File Layout Type, there is only one storage device. In the Flexible File Layout Type, while there is an
filehandle, independent of the multipathing being used. If the array of filehandles, they are independent of the multipathing being
metadata server wants to provide multiple read-only copies of the used. If the metadata server wants to provide multiple read-only
same file on the same storage device, then it should provide multiple copies of the same file on the same storage device, then it should
ff_device_addr4, each as a mirror. The client can then determine provide multiple ff_device_addr4, each as a mirror. The client can
that since the ffds_fhandle are different, then there a multiple then determine that since the ffds_fh_vers are different, then there
copies of the file available. multiple copies of the file available.
If the metadata server wants to allow access to the file with 5.3. Handling Version Errors
different versions and/or minor versions of NFS, then for each
allowed version and/or minor version, a new ff_device_addr4 must be When the metadata server provides the ffda_versions array in the
defined. The client should not assume any relationship (or lack of ff_device_addr4 (see Section 4.1), the client is able to determine if
relationship) between the filehandles presented in ffds_fhandle. it can not access a storage device with any of the supplied
I.e., even if the filehandles are binary equivalent for different ffdv_version and ffdv_minorversion combinations. However, due to the
versions, they may have varying semantics. limitations of reporting errors in GETDEVICEINFO (see Section 18.40
in [RFC5661], the client is not able to specify which specific device
it can not communicate with over one of the provided ffdv_version and
ffdv_minorversion combinations. Using ff_ioerr4 (see Section 9.1.1
inside either the LAYOUTRETURN (see Section 18.44 of [RFC5661]) or
the LAYOUTERROR (see Section 15.6 of [NFSv42] and Section 10 of this
document), the client can isolate the problematic storage device.
The error code to return for LAYOUTRETURN and/or LAYOUTERROR is
NFS4ERR_MINOR_VERS_MISMATCH. It does not matter whether the mismatch
is a major version (e.g., client can use NFSv3 but not NFSv4) or
minor version (e.g., client can use NFSv4.1 but not NFSv4.2), the
error indicates that for all the supplied combinations for
ffdv_version and ffdv_minorversion, the client can not communicate
with the storage device. The client can retry the GETDEVICEINFO to
see if the metadata server can provide a different combination or it
can fall back to doing the I/O through the metadata server.
6. Striping via Sparse Mapping 6. Striping via Sparse Mapping
While other Layout Types support both dense and sparse mapping of While other Layout Types support both dense and sparse mapping of
logical offsets to phyisical offsets within a file (see for example logical offsets to phyisical offsets within a file (see for example
Section 13.4 of [RFC5661]), the Flexible File Layout Type only Section 13.4 of [RFC5661]), the Flexible File Layout Type only
supports a sparse mapping. supports a sparse mapping.
With sparse mappings, the logical offset within a file (L) is also With sparse mappings, the logical offset within a file (L) is also
the physical offset on the storage device. As detailed in the physical offset on the storage device. As detailed in
skipping to change at page 16, line 39 skipping to change at page 18, line 8
8.1. Selecting a Mirror 8.1. Selecting a Mirror
When the metadata server grants a layout to a client, it can let the When the metadata server grants a layout to a client, it can let the
client know how fast it expects each mirror to be once the request client know how fast it expects each mirror to be once the request
arrives at the storage devices via the ffds_efficiency member. While arrives at the storage devices via the ffds_efficiency member. While
the algorithms to calculate that value are left to the metadata the algorithms to calculate that value are left to the metadata
server implementations, factors that could contribute to that server implementations, factors that could contribute to that
calculation include speed of the storage device, physical memory calculation include speed of the storage device, physical memory
available to the device, operating system version, current load, etc. available to the device, operating system version, current load, etc.
However, what SHOULD not be involved in that calculation is a However, what should not be involved in that calculation is a
perceived network distance between the client and the storage device. perceived network distance between the client and the storage device.
The client is better situated for making that determination based on The client is better situated for making that determination based on
past interaction with the storage device over the different available past interaction with the storage device over the different available
network interfaces between the two. I.e., the metadata server might network interfaces between the two. I.e., the metadata server might
not know about a transient outage between the client and storage not know about a transient outage between the client and storage
device because it has no presence on the given subnet. device because it has no presence on the given subnet.
As such, it is the client which decides which mirror to access for As such, it is the client which decides which mirror to access for
reading the file. The requirements for writing to a mirrored file reading the file. The requirements for writing to a mirrored file
are presented below. are presented below.
skipping to change at page 17, line 17 skipping to change at page 18, line 32
The client is responsible for updating all mirrored copies of the The client is responsible for updating all mirrored copies of the
file that it is given in the layout. If all but one copy is updated file that it is given in the layout. If all but one copy is updated
successfully and the last one provides an error, then the client successfully and the last one provides an error, then the client
needs to return the layout to the metadata server with an error needs to return the layout to the metadata server with an error
indicating that the update failed to that storage device. indicating that the update failed to that storage device.
The metadata server is then responsible for determining if it wants The metadata server is then responsible for determining if it wants
to remove the errant mirror from the layout, if the mirror has to remove the errant mirror from the layout, if the mirror has
recovered from some transient error, etc. When the client tries to recovered from some transient error, etc. When the client tries to
get a new layout, the metadata server informs it of the decision by get a new layout, the metadata server informs it of the decision by
the contents of the layout. The client MUST not make any assumptions the contents of the layout. The client MUST NOT make any assumptions
that the contents of the previous layout will match those of the new that the contents of the previous layout will match those of the new
one. If it has updates that were not committed, it MUST resend those one. If it has updates that were not committed, it MUST resend those
updates to all mirrors. updates to all mirrors.
8.3. Metadata Server Resilvering of the File 8.3. Metadata Server Resilvering of the File
The metadata server may elect to create a new mirror of the file at The metadata server may elect to create a new mirror of the file at
any time. This might be to resilver a copy on a storage device which any time. This might be to resilver a copy on a storage device which
was down for servicing, to provide a copy of the file on storage with was down for servicing, to provide a copy of the file on storage with
different storage performance characteristics, etc. As the client different storage performance characteristics, etc. As the client
skipping to change at page 17, line 42 skipping to change at page 19, line 11
segment which is in the process of being resilvered, then the segment which is in the process of being resilvered, then the
metadata server MUST deny that request with a NFS4ERR_LAYOUTTRYLATER. metadata server MUST deny that request with a NFS4ERR_LAYOUTTRYLATER.
The client can then perform the I/O through the metadata server. The client can then perform the I/O through the metadata server.
9. Flexible Files Layout Type Return 9. Flexible Files Layout Type Return
layoutreturn_file4 is used in the LAYOUTRETURN operation to convey layoutreturn_file4 is used in the LAYOUTRETURN operation to convey
layout-type specific information to the server. It is defined in layout-type specific information to the server. It is defined in
[RFC5661] as follows: [RFC5661] as follows:
<CODE BEGINS>
struct layoutreturn_file4 { struct layoutreturn_file4 {
offset4 lrf_offset; offset4 lrf_offset;
length4 lrf_length; length4 lrf_length;
stateid4 lrf_stateid; stateid4 lrf_stateid;
/* layouttype4 specific data */ /* layouttype4 specific data */
opaque lrf_body<>; opaque lrf_body<>;
}; };
union layoutreturn4 switch(layoutreturn_type4 lr_returntype) { union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
case LAYOUTRETURN4_FILE: case LAYOUTRETURN4_FILE:
layoutreturn_file4 lr_layout; layoutreturn_file4 lr_layout;
default: default:
void; void;
}; };
struct LAYOUTRETURN4args { struct LAYOUTRETURN4args {
/* CURRENT_FH: file */ /* CURRENT_FH: file */
bool lora_reclaim; bool lora_reclaim;
skipping to change at page 18, line 20 skipping to change at page 19, line 37
struct LAYOUTRETURN4args { struct LAYOUTRETURN4args {
/* CURRENT_FH: file */ /* CURRENT_FH: file */
bool lora_reclaim; bool lora_reclaim;
layoutreturn_stateid lora_recallstateid; layoutreturn_stateid lora_recallstateid;
layouttype4 lora_layout_type; layouttype4 lora_layout_type;
layoutiomode4 lora_iomode; layoutiomode4 lora_iomode;
layoutreturn4 lora_layoutreturn; layoutreturn4 lora_layoutreturn;
}; };
<CODE ENDS>
If the lora_layout_type layout type is LAYOUT4_FLEX_FILES, then the If the lora_layout_type layout type is LAYOUT4_FLEX_FILES, then the
lrf_body opaque value is defined by ff_layoutreturn4 (See lrf_body opaque value is defined by ff_layoutreturn4 (See
Section 9.3). It allows the client to report I/O error information Section 9.3). It allows the client to report I/O error information
or layout usage statistics back to the metadata server as defined or layout usage statistics back to the metadata server as defined
below. below.
9.1. I/O Error Reporting 9.1. I/O Error Reporting
9.1.1. ff_ioerr4 9.1.1. ff_ioerr4
<CODE BEGINS>
/// struct ff_ioerr4 { /// struct ff_ioerr4 {
/// offset4 ffie_offset; /// offset4 ffie_offset;
/// length4 ffie_length; /// length4 ffie_length;
/// stateid4 ffie_stateid; /// stateid4 ffie_stateid;
/// device_error4 ffie_errors; /// device_error4 ffie_errors;
/// }; /// };
/// ///
<CODE ENDS>
Recall that [NFSv42] defines device_error4 as: Recall that [NFSv42] defines device_error4 as:
<CODE BEGINS>
struct device_error4 { struct device_error4 {
deviceid4 de_deviceid; deviceid4 de_deviceid;
nfsstat4 de_status; nfsstat4 de_status;
nfs_opnum4 de_opnum; nfs_opnum4 de_opnum;
}; };
<CODE ENDS>
The ff_ioerr4 structure is used to return error indications for data The ff_ioerr4 structure is used to return error indications for data
files that generated errors during data transfers. These are hints files that generated errors during data transfers. These are hints
to the metadata server that there are problems with that file. For to the metadata server that there are problems with that file. For
each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length
represent the storage device and byte range within the file in which represent the storage device and byte range within the file in which
the error occurred; ffie_errors represents the operation and type of the error occurred; ffie_errors represents the operation and type of
error. The use of device_error4 is described in Section 15.6 of error. The use of device_error4 is described in Section 15.6 of
[NFSv42]. [NFSv42].
Even though the storage device might be accessed via NFSv3 and Even though the storage device might be accessed via NFSv3 and
reports back NFSv3 errors to the client, the client is responsible reports back NFSv3 errors to the client, the client is responsible
for mapping these to appropriate NFSv4 status codes as de_status. for mapping these to appropriate NFSv4 status codes as de_status.
Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4 Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4
operations. operations.
9.2. Layout Usage Statistics 9.2. Layout Usage Statistics
9.2.1. ff_io_latency4 9.2.1. ff_io_latency4
<CODE BEGINS>
/// struct ff_io_latency4 { /// struct ff_io_latency4 {
/// nfstime4 ffil_min; /// nfstime4 ffil_min;
/// nfstime4 ffil_max; /// nfstime4 ffil_max;
/// nfstime4 ffil_avg; /// nfstime4 ffil_avg;
/// uint32_t ffil_count; /// uint32_t ffil_count;
/// }; /// };
/// ///
<CODE ENDS>
When determining latencies, the client can collect the minimum via When determining latencies, the client can collect the minimum via
ffil_min, the maximum via ffil_max, and the average via ffil_avg. ffil_min, the maximum via ffil_max, and the average via ffil_avg.
Further, ffil_count relates how many data points were collected in Further, ffil_count relates how many data points were collected in
the reported period. the reported period.
9.2.2. ff_layoutupdate4 9.2.2. ff_layoutupdate4
<CODE BEGINS>
/// struct ff_layoutupdate4 { /// struct ff_layoutupdate4 {
/// netaddr4 ffl_addr; /// netaddr4 ffl_addr;
/// nfs_fh4 ffl_fhandle; /// nfs_fh4 ffl_fhandle;
/// ff_io_latency4 ffl_read; /// ff_io_latency4 ffl_read;
/// ff_io_latency4 ffl_write; /// ff_io_latency4 ffl_write;
/// uint32_t ffl_queue_depth; /// uint32_t ffl_queue_depth;
/// nfstime4 ffl_duration; /// nfstime4 ffl_duration;
/// bool ffl_local; /// bool ffl_local;
/// }; /// };
/// ///
<CODE ENDS>
ffl_addr differentiates which network address the client connected to ffl_addr differentiates which network address the client connected to
on the storage device. In the case of multipathing, ffl_fhandle on the storage device. In the case of multipathing, ffl_fhandle
indicates which read-only copy was selected. ffl_read and ffl_write indicates which read-only copy was selected. ffl_read and ffl_write
convey the latencies respectively for both read and write operations. convey the latencies respectively for both read and write operations.
ffl_queue_depth can be used to indicate how long the I/O had to wait ffl_queue_depth can be used to indicate how long the I/O had to wait
on internal queues before being serviced. ffl_duration is used to on internal queues before being serviced. ffl_duration is used to
indicate the time period over which the statistics were collected. indicate the time period over which the statistics were collected.
ffl_local if true indicates that the I/O was serviced by the client's ffl_local if true indicates that the I/O was serviced by the client's
cache. This flag allows the client to inform the metadata server cache. This flag allows the client to inform the metadata server
about "hot" access to a file it would not normally be allowed to about "hot" access to a file it would not normally be allowed to
report on. report on.
9.2.3. ff_iostats4 9.2.3. ff_iostats4
<CODE BEGINS>
/// struct ff_iostats4 { /// struct ff_iostats4 {
/// offset4 ffis_offset; /// offset4 ffis_offset;
/// length4 ffis_length; /// length4 ffis_length;
/// stateid4 ffis_stateid; /// stateid4 ffis_stateid;
/// io_info4 ffis_read; /// io_info4 ffis_read;
/// io_info4 ffis_write; /// io_info4 ffis_write;
/// deviceid4 ffis_deviceid; /// deviceid4 ffis_deviceid;
/// layoutupdate4 ffis_layoutupdate; /// layoutupdate4 ffis_layoutupdate;
/// }; /// };
/// ///
<CODE ENDS>
Recall that [NFSv42] defines io_info4 as: Recall that [NFSv42] defines io_info4 as:
<CODE BEGINS>
struct io_info4 { struct io_info4 {
uint32_t ii_count; uint32_t ii_count;
uint64_t ii_bytes; uint64_t ii_bytes;
}; };
<CODE ENDS>
With pNFS, the data transfers are performed directly between the pNFS With pNFS, the data transfers are performed directly between the pNFS
client and the storage devices. Therefore, the metadata server has client and the storage devices. Therefore, the metadata server has
no visibility to the I/O stream and cannot use any statistical no visibility to the I/O stream and cannot use any statistical
information about client I/O to optimize data storage location. information about client I/O to optimize data storage location.
ff_iostats4 MAY be used by the client to report I/O statistics back ff_iostats4 MAY be used by the client to report I/O statistics back
to the metadata server upon returning the layout. Since it is to the metadata server upon returning the layout. Since it is
infeasible for the client to report every I/O that used the layout, infeasible for the client to report every I/O that used the layout,
the client MAY identify "hot" byte ranges for which to report I/O the client MAY identify "hot" byte ranges for which to report I/O
statistics. The definition and/or configuration mechanism of what is statistics. The definition and/or configuration mechanism of what is
considered "hot" and the size of the reported byte range is out of considered "hot" and the size of the reported byte range is out of
skipping to change at page 21, line 10 skipping to change at page 23, line 16
The combination of ffis_deviceid and ffl_addr uniquely identify both The combination of ffis_deviceid and ffl_addr uniquely identify both
the storage path and the network route to it. Additionally, the the storage path and the network route to it. Additionally, the
ffis_deviceid informs the metadata server as to the version and/or ffis_deviceid informs the metadata server as to the version and/or
minor version being used for I/O to the storage device. Finally, the minor version being used for I/O to the storage device. Finally, the
ffl_fhandle allows the metadata server to differentiate between ffl_fhandle allows the metadata server to differentiate between
multiple read-only copies of the file on the same storage device. multiple read-only copies of the file on the same storage device.
9.3. ff_layoutreturn4 9.3. ff_layoutreturn4
<CODE BEGINS>
/// struct ff_layoutreturn4 { /// struct ff_layoutreturn4 {
/// ff_ioerr4 fflr_ioerr_report<>; /// ff_ioerr4 fflr_ioerr_report<>;
/// ff_iostats4 fflr_iostats_report<>; /// ff_iostats4 fflr_iostats_report<>;
/// }; /// };
/// ///
<CODE ENDS>
When data file I/O operations fail, fflr_ioerr_report<> is used to When data file I/O operations fail, fflr_ioerr_report<> is used to
report these errors to the metadata server as an array of elements of report these errors to the metadata server as an array of elements of
type ff_ioerr4. Each element in the array represents an error that type ff_ioerr4. Each element in the array represents an error that
occurred on the data file identified by ffie_errors.de_deviceid. If occurred on the data file identified by ffie_errors.de_deviceid. If
no errors are to be reported, the size of the fflr_ioerr_report<> no errors are to be reported, the size of the fflr_ioerr_report<>
array is set to zero. The client MAY also use fflr_iostats_report<> array is set to zero. The client MAY also use fflr_iostats_report<>
to report a list of I/O statistics as an array of elements of type to report a list of I/O statistics as an array of elements of type
ff_iostats4. Each element in the array represents statistics for a ff_iostats4. Each element in the array represents statistics for a
particular byte range. Byte ranges are not guaranteed to be disjoint particular byte range. Byte ranges are not guaranteed to be disjoint
and MAY repeat or intersect. and MAY repeat or intersect.
skipping to change at page 22, line 5 skipping to change at page 24, line 12
statistics to the metadata server (see Section 9.2), it can use statistics to the metadata server (see Section 9.2), it can use
LAYOUTSTATS (see Section 15.7 of [NFSv42]) to communicate that LAYOUTSTATS (see Section 15.7 of [NFSv42]) to communicate that
information. For the Flexible Files Layout Type, this means that information. For the Flexible Files Layout Type, this means that
LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same
contents as in ffis_layoutupdate. contents as in ffis_layoutupdate.
12. Flexible File Layout Type Creation Hint 12. Flexible File Layout Type Creation Hint
The layouthint4 type is defined in the [RFC5661] as follows: The layouthint4 type is defined in the [RFC5661] as follows:
<CODE BEGINS>
struct layouthint4 { struct layouthint4 {
layouttype4 loh_type; layouttype4 loh_type;
opaque loh_body<>; opaque loh_body<>;
}; };
<CODE ENDS>
The layouthint4 structure is used by the client to pass a hint about The layouthint4 structure is used by the client to pass a hint about
the type of layout it would like created for a particular file. If the type of layout it would like created for a particular file. If
the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body
opaque value is defined by the ff_layouthint4 type. opaque value is defined by the ff_layouthint4 type.
12.1. ff_layouthint4 12.1. ff_layouthint4
<CODE BEGINS>
/// union ff_mirrors_hint switch (bool ffmc_valid) { /// union ff_mirrors_hint switch (bool ffmc_valid) {
/// case TRUE: /// case TRUE:
/// uint32_t ffmc_mirrors; /// uint32_t ffmc_mirrors;
/// case FALSE: /// case FALSE:
/// void; /// void;
/// }; /// };
/// ///
/// struct ff_layouthint4 { /// struct ff_layouthint4 {
/// ff_mirrors_hint fflh_mirrors_hint; /// ff_mirrors_hint fflh_mirrors_hint;
/// }; /// };
/// ///
<CODE ENDS>
This type conveys hints for the desired data map. All parameters are This type conveys hints for the desired data map. All parameters are
optional so the client can give values for only the parameter it optional so the client can give values for only the parameter it
cares about. cares about.
13. Recalling Layouts 13. Recalling Layouts
The Flexible File Layout Type metadata server should recall The Flexible File Layout Type metadata server should recall
outstanding layouts in the following cases: outstanding layouts in the following cases:
o When the file's security policy changes, i.e., Access Control o When the file's security policy changes, i.e., Access Control
skipping to change at page 23, line 5 skipping to change at page 25, line 24
invalid. invalid.
o When there are sharing conflicts. o When there are sharing conflicts.
13.1. CB_RECALL_ANY 13.1. CB_RECALL_ANY
The metadata server can use the CB_RECALL_ANY callback operation to The metadata server can use the CB_RECALL_ANY callback operation to
notify the client to return some or all of its layouts. The notify the client to return some or all of its layouts. The
[RFC5661] defines the following types: [RFC5661] defines the following types:
<CODE BEGINS>
const RCA4_TYPE_MASK_FF_LAYOUT_MIN = -2; const RCA4_TYPE_MASK_FF_LAYOUT_MIN = -2;
const RCA4_TYPE_MASK_FF_LAYOUT_MAX = -1; const RCA4_TYPE_MASK_FF_LAYOUT_MAX = -1;
[[RFC Editor: please insert assigned constants]] [[RFC Editor: please insert assigned constants]]
struct CB_RECALL_ANY4args { struct CB_RECALL_ANY4args {
uint32_t craa_layouts_to_keep; uint32_t craa_layouts_to_keep;
bitmap4 craa_type_mask; bitmap4 craa_type_mask;
}; };
<CODE ENDS>
[[AI13: No, 5661 does not define these above values. The ask here is [[AI13: No, 5661 does not define these above values. The ask here is
to create these and _add_ them to 5661. --TH]] to create these and _add_ them to 5661. --TH]]
Typically, CB_RECALL_ANY will be used to recall client state when the Typically, CB_RECALL_ANY will be used to recall client state when the
server needs to reclaim resources. The craa_type_mask bitmap server needs to reclaim resources. The craa_type_mask bitmap
specifies the type of resources that are recalled and the specifies the type of resources that are recalled and the
craa_layouts_to_keep value specifies how many of the recalled craa_layouts_to_keep value specifies how many of the recalled
Flexible File Layouts the client is allowed to keep. The Flexible Flexible File Layouts the client is allowed to keep. The Flexible
File Layout Type mask flags are defined as follows: File Layout Type mask flags are defined as follows:
<CODE BEGINS>
/// enum ff_cb_recall_any_mask { /// enum ff_cb_recall_any_mask {
/// FF_RCA4_TYPE_MASK_READ = -2, /// FF_RCA4_TYPE_MASK_READ = -2,
/// FF_RCA4_TYPE_MASK_RW = -1 /// FF_RCA4_TYPE_MASK_RW = -1
[[RFC Editor: please insert assigned constants]] [[RFC Editor: please insert assigned constants]]
/// }; /// };
/// ///
<CODE ENDS>
They represent the iomode of the recalled layouts. In response, the They represent the iomode of the recalled layouts. In response, the
client SHOULD return layouts of the recalled iomode that it needs the client SHOULD return layouts of the recalled iomode that it needs the
least, keeping at most craa_layouts_to_keep Flexible File Layouts. least, keeping at most craa_layouts_to_keep Flexible File Layouts.
The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return
layouts of iomode LAYOUTIOMODE4_READ. Similarly, the layouts of iomode LAYOUTIOMODE4_READ. Similarly, the
PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts
of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client
is notified to return layouts of either iomode. is notified to return layouts of either iomode.
skipping to change at page 25, line 21 skipping to change at page 27, line 48
In all, while either an elaborate schema could be used to In all, while either an elaborate schema could be used to
automatically authenticate principals or RPCSEC_GSSv3 aware clients, automatically authenticate principals or RPCSEC_GSSv3 aware clients,
metadata server, and storage devices could be deployed, if more metadata server, and storage devices could be deployed, if more
secure authentication is desired, tight coupling should be considered secure authentication is desired, tight coupling should be considered
as described in the next section. as described in the next section.
15.1.2. Tightly Coupled 15.1.2. Tightly Coupled
With tight coupling, the principal used to access the metadata file With tight coupling, the principal used to access the metadata file
is exactly the same as used to access the data file. Thus there are is exactly the same as used to access the data file. Thus there are
no security issues related to kerberization of a tightly coupled no security issues related to using Kerberos with a tightly coupled
system. system.
16. IANA Considerations 16. IANA Considerations
As described in [RFC5661], new layout type numbers have been assigned As described in [RFC5661], new layout type numbers have been assigned
by IANA. This document defines the protocol associated with the by IANA. This document defines the protocol associated with the
existing layout type number, LAYOUT4_FLEX_FILES. existing layout type number, LAYOUT4_FLEX_FILES.
17. References 17. References
skipping to change at page 26, line 40 skipping to change at page 29, line 22
[rpcsec_gssv3] [rpcsec_gssv3]
Adamson, W. and N. Williams, "Remote Procedure Call (RPC) Adamson, W. and N. Williams, "Remote Procedure Call (RPC)
Security Version 3", November 2014. Security Version 3", November 2014.
Appendix A. Acknowledgments Appendix A. Acknowledgments
Those who provided miscellaneous comments to early drafts of this Those who provided miscellaneous comments to early drafts of this
document include: Matt W. Benjamin, Adam Emerson, Tom Haynes, J. document include: Matt W. Benjamin, Adam Emerson, Tom Haynes, J.
Bruce Fields, and Lev Solomonov. Bruce Fields, and Lev Solomonov.
Idan Kedar caught a nasty bug in the interaction of client side
mirroring and the minor versioning of devices.
Appendix B. RFC Editor Notes Appendix B. RFC Editor Notes
[RFC Editor: please remove this section prior to publishing this [RFC Editor: please remove this section prior to publishing this
document as an RFC] document as an RFC]
[RFC Editor: prior to publishing this document as an RFC, please [RFC Editor: prior to publishing this document as an RFC, please
replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the
RFC number of this document] RFC number of this document]
Authors' Addresses Authors' Addresses
 End of changes. 61 change blocks. 
89 lines changed or deleted 195 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/