draft-ietf-nfsv4-pnfs-obj-02.txt   draft-ietf-nfsv4-pnfs-obj-03.txt 
NFSv4 B. Halevy NFSv4 B. Halevy
Internet-Draft B. Welch Internet-Draft B. Welch
Expires: March 2, 2007 J. Zelenka Intended status: Standards Track J. Zelenka
Panasas Expires: September 6, 2007 Panasas
T. Pisek March 5, 2007
Sun
August 29, 2006
Object-based pNFS Operations Object-based pNFS Operations
draft-ietf-nfsv4-pnfs-obj-02.txt draft-ietf-nfsv4-pnfs-obj-03.txt
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 37 skipping to change at page 1, line 35
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on March 2, 2007. This Internet-Draft will expire on September 6, 2007.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2006). Copyright (C) The IETF Trust (2007).
Abstract Abstract
This Internet-Draft provides a description of the object-based pNFS This Internet-Draft provides a description of the object-based pNFS
extension for NFSv4. This is a companion to the main pnfs extension for NFSv4. This is a companion to the main pnfs
specification in the NFSv4 Minor Version 1 Internet Draft, which is specification in the NFSv4 Minor Version 1 Internet Draft, which is
currently draft-ietf-nfsv4-minorversion1-03.txt. currently draft-ietf-nfsv4-minorversion1-10.txt.
Requirements Language Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [1]. document are to be interpreted as described in RFC 2119 [1].
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Object Storage Device Addressing and Discovery . . . . . . . . 3 2. Object Storage Device Addressing and Discovery . . . . . . . . 4
2.1 pnfs_osd_addr_type4 . . . . . . . . . . . . . . . . . . . 4 2.1. pnfs_osd_addr_type4 . . . . . . . . . . . . . . . . . . . 5
2.2 pnfs_osd_deviceaddr4 . . . . . . . . . . . . . . . . . . . 4 2.2. pnfs_osd_deviceaddr4 . . . . . . . . . . . . . . . . . . . 5
3. Object-Based Layout . . . . . . . . . . . . . . . . . . . . . 4 3. Object-Based Layout . . . . . . . . . . . . . . . . . . . . . 5
3.1 pnfs_osd_layout4 . . . . . . . . . . . . . . . . . . . . . 5 3.1. pnfs_osd_layout4 . . . . . . . . . . . . . . . . . . . . . 6
3.1.1 pnfs_osd_objid4 . . . . . . . . . . . . . . . . . . . 6 3.1.1. pnfs_osd_objid4 . . . . . . . . . . . . . . . . . . . 7
3.1.3 pnfs_osd_object_cred4 . . . . . . . . . . . . . . . . 7 3.1.3. pnfs_osd_object_cred4 . . . . . . . . . . . . . . . . 8
3.1.4 pnfs_osd_raid_algorithm4 . . . . . . . . . . . . . . . 7 3.1.4. pnfs_osd_raid_algorithm4 . . . . . . . . . . . . . . . 8
3.1.5 pnfs_osd_data_map4 . . . . . . . . . . . . . . . . . . 7 3.1.5. pnfs_osd_data_map4 . . . . . . . . . . . . . . . . . . 8
3.2 Data Mapping Schemes . . . . . . . . . . . . . . . . . . . 8 3.2. Data Mapping Schemes . . . . . . . . . . . . . . . . . . . 9
3.2.1 Simple Striping . . . . . . . . . . . . . . . . . . . 8 3.2.1. Simple Striping . . . . . . . . . . . . . . . . . . . 9
3.2.2 Nested Striping . . . . . . . . . . . . . . . . . . . 9 3.2.2. Nested Striping . . . . . . . . . . . . . . . . . . . 10
3.2.3 Mirroring . . . . . . . . . . . . . . . . . . . . . . 11 3.2.3. Mirroring . . . . . . . . . . . . . . . . . . . . . . 12
3.3 RAID Algorithms . . . . . . . . . . . . . . . . . . . . . 11 3.3. RAID Algorithms . . . . . . . . . . . . . . . . . . . . . 13
3.3.1 PNFS_OSD_RAID_0 . . . . . . . . . . . . . . . . . . . 11 3.3.1. PNFS_OSD_RAID_0 . . . . . . . . . . . . . . . . . . . 13
3.3.2 PNFS_OSD_RAID_4 . . . . . . . . . . . . . . . . . . . 11 3.3.2. PNFS_OSD_RAID_4 . . . . . . . . . . . . . . . . . . . 13
3.3.3 PNFS_OSD_RAID_5 . . . . . . . . . . . . . . . . . . . 12 3.3.3. PNFS_OSD_RAID_5 . . . . . . . . . . . . . . . . . . . 13
3.3.4 PNFS_OSD_RAID_PQ . . . . . . . . . . . . . . . . . . . 12 3.3.4. PNFS_OSD_RAID_PQ . . . . . . . . . . . . . . . . . . . 14
3.3.5 RAID Usage and implementation notes . . . . . . . . . 13 3.3.5. RAID Usage and implementation notes . . . . . . . . . 14
4. Object-Based Layout Update . . . . . . . . . . . . . . . . . . 13 4. Object-Based Layout Update . . . . . . . . . . . . . . . . . . 15
4.1 pnfs_osd_layoutupdate4 . . . . . . . . . . . . . . . . . . 13 4.1. pnfs_osd_layoutupdate4 . . . . . . . . . . . . . . . . . . 15
4.1.1 pnfs_osd_deltaspaceused4 . . . . . . . . . . . . . . . 14 4.1.1. pnfs_osd_deltaspaceused4 . . . . . . . . . . . . . . . 15
4.1.2 pnfs_osd_errno4 . . . . . . . . . . . . . . . . . . . 14 4.1.2. pnfs_osd_errno4 . . . . . . . . . . . . . . . . . . . 16
4.1.3 pnfs_osd_ioerr4 . . . . . . . . . . . . . . . . . . . 15 4.1.3. pnfs_osd_ioerr4 . . . . . . . . . . . . . . . . . . . 17
5. Object-Based Creation Layout Hint . . . . . . . . . . . . . . 15 5. Object-Based Creation Layout Hint . . . . . . . . . . . . . . 17
5.1 pnfs_osd_layouthint4 . . . . . . . . . . . . . . . . . . . 16 5.1. pnfs_osd_layouthint4 . . . . . . . . . . . . . . . . . . . 17
6. Layout Segments . . . . . . . . . . . . . . . . . . . . . . . 17 6. Layout Segments . . . . . . . . . . . . . . . . . . . . . . . 19
6.1 CB_LAYOUTRECALL and LAYOUTRETURN . . . . . . . . . . . . . 17 6.1. CB_LAYOUTRECALL and LAYOUTRETURN . . . . . . . . . . . . . 19
6.2 LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . . 18 6.2. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . . 19
7. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 18 7. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 20
8. Security Considerations . . . . . . . . . . . . . . . . . . . 18 8. Security Considerations . . . . . . . . . . . . . . . . . . . 20
8.1 OSD Security Data Types . . . . . . . . . . . . . . . . . 19 8.1. OSD Security Data Types . . . . . . . . . . . . . . . . . 20
8.2 The OSD Security Protocol . . . . . . . . . . . . . . . . 19 8.2. The OSD Security Protocol . . . . . . . . . . . . . . . . 21
8.3 Revoking capabilities . . . . . . . . . . . . . . . . . . 21 8.3. Revoking capabilities . . . . . . . . . . . . . . . . . . 22
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23
9.1 Normative References . . . . . . . . . . . . . . . . . . . 21 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23
9.2 Informative References . . . . . . . . . . . . . . . . . . 22 10.1. Normative References . . . . . . . . . . . . . . . . . . . 23
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 22 10.2. Informative References . . . . . . . . . . . . . . . . . . 24
Intellectual Property and Copyright Statements . . . . . . . . 24 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . . 24
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 24
Intellectual Property and Copyright Statements . . . . . . . . . . 26
1. Introduction 1. Introduction
In pNFS, the file server returns typed layout structures that In pNFS, the file server returns typed layout structures that
describe where file data is located. There are different layouts for describe where file data is located. There are different layouts for
different storage systems and methods of arranging data on storage different storage systems and methods of arranging data on storage
devices. This document describes the layouts used with object-based devices. This document describes the layouts used with object-based
storage devices (OSD) that are accessed according to the iSCSI/OSD storage devices (OSD) that are accessed according to the iSCSI/OSD
storage protocol standard (SNIA T10/1355-D [2]). storage protocol standard (SNIA T10/1355-D [2]).
skipping to change at page 4, line 9 skipping to change at page 5, line 10
of this is iSCSI targets that are not known to the client until a of this is iSCSI targets that are not known to the client until a
layout has been requested. Eventually iSCSI will adopt ANSI T10 layout has been requested. Eventually iSCSI will adopt ANSI T10
SAM-3, at which time the World Wide Name (WWN aka, EUI-64/EUI-128) SAM-3, at which time the World Wide Name (WWN aka, EUI-64/EUI-128)
naming conventions can be specified. In addition, Fibre Channel (FC) naming conventions can be specified. In addition, Fibre Channel (FC)
SCSI targets have a unique WWN. Although these FC targets have SCSI targets have a unique WWN. Although these FC targets have
already been discovered, some implementations may want to specify the already been discovered, some implementations may want to specify the
WWN in addition to the label. This information appears as the WWN in addition to the label. This information appears as the
"target" and "lun" fields in the pnfs_osd_deviceaddr4 type described "target" and "lun" fields in the pnfs_osd_deviceaddr4 type described
below. below.
2.1 pnfs_osd_addr_type4 2.1. pnfs_osd_addr_type4
The following enum specifies the manner in which a scsi target can be The following enum specifies the manner in which a scsi target can be
specified. The target can be specified as a network address, as an specified. The target can be specified as a network address, as an
Internet Qualified Name (IQN), or by the World-Wide Name (WWN) of the Internet Qualified Name (IQN), or by the World-Wide Name (WWN) of the
target. target.
enum pnfs_obj_addr_type4 { enum pnfs_obj_addr_type4 {
OBJ_TARGET_NETADDR = 1, OBJ_TARGET_NETADDR = 1,
OBJ_TARGET_IQN = 2, OBJ_TARGET_IQN = 2,
OBJ_TARGET_WWN = 3 OBJ_TARGET_WWN = 3
}; };
2.2 pnfs_osd_deviceaddr4 2.2. pnfs_osd_deviceaddr4
The specification for an object device address is as follows: The specification for an object device address is as follows:
struct pnfs_osd_deviceaddr4 { struct pnfs_osd_deviceaddr4 {
union target switch (pnfs_osd_addr_type4 type) { union target switch (pnfs_osd_addr_type4 type) {
case OBJ_TARGET_NETADDR: case OBJ_TARGET_NETADDR:
pnfs_netaddr4 netaddr; pnfs_netaddr4 netaddr;
case OBJ_TARGET_IQN: case OBJ_TARGET_IQN:
string iqn<>; string iqn<>;
skipping to change at page 4, line 46 skipping to change at page 5, line 47
default: default:
void; void;
}; };
uint64_t lun; uint64_t lun;
opaque root_id<>; opaque root_id<>;
}; };
3. Object-Based Layout 3. Object-Based Layout
The pnfs_layout4 type is defined in the NFSv4.1 draft [5] as follows: The layout4 type is defined in the NFSv4.1 draft [5] as follows:
enum pnfs_layouttype4 { enum layouttype4 {
LAYOUT_NFSV4_FILES = 1, LAYOUT4_NFSV4_1_FILES = 1,
LAYOUT_OSD2_OBJECTS = 2, LAYOUT4_OSD2_OBJECTS = 2,
LAYOUT_BLOCK_VOLUME = 3 LAYOUT4_BLOCK_VOLUME = 3
}; };
struct pnfs_layout4 { struct layout_content4 {
offset4 offset; layouttype4 loc_type;
length4 length; opaque loc_body<>;
pnfs_layoutiomode4 iomode;
pnfs_layouttype4 type;
opaque layout<>;
}; };
This draft defines structure associated with the pnfs_layouttype4 struct layout4 {
value, LAYOUT_OSD2_OBJECTS. The NFSv4.1 draft specifies the offset4 lo_offset;
length4 lo_length;
layoutiomode4 lo_iomode;
layout_content4 lo_content;
};
This draft defines structure associated with the layouttype4 value,
LAYOUT4_OSD2_OBJECTS. The NFSv4.1 draft specifies the loc_body
structure as an XDR type "opaque". The opaque layout is structure as an XDR type "opaque". The opaque layout is
uninterpreted by the generic pNFS client layers, but obviously must uninterpreted by the generic pNFS client layers, but obviously must
be interpreted by the object-storage layout driver. This document be interpreted by the object-storage layout driver. This document
defines the structure of this opaque value, pnfs_osd_layout4. defines the structure of this opaque value, pnfs_osd_layout4.
3.1 pnfs_osd_layout4 3.1. pnfs_osd_layout4
struct pnfs_osd_layout4 { struct pnfs_osd_layout4 {
pnfs_osd_data_map4 map; pnfs_osd_data_map4 map;
pnfs_osd_object_cred4 components<>; pnfs_osd_object_cred4 components<>;
}; };
The pnfs_osd_layout4 structure specifies a layout over a set of The pnfs_osd_layout4 structure specifies a layout over a set of
component objects. The components field is an array of object component objects. The components field is an array of object
identifiers and security credentials that grant access to each identifiers and security credentials that grant access to each
object. The organization of the data is defined by the object. The organization of the data is defined by the
pnfs_osd_data_map4 type that specifies how the file's data is mapped pnfs_osd_data_map4 type that specifies how the file's data is mapped
onto the component objects (i.e., the striping pattern). The data onto the component objects (i.e., the striping pattern). The data
placement algorithm that maps file data onto component objects assume placement algorithm that maps file data onto component objects assume
that each component object occurs exactly once in the array of that each component object occurs exactly once in the array of
components. Therefore, component objects MUST appear in the components. Therefore, component objects MUST appear in the
component array only once. component array only once.
Note that the layout depends on the file size, which the client Note that the layout depends on the file size, which the client
learns from the generic return parameters of LAYOUTGET, by doing learns from the generic return parameters of LAYOUTGET, by doing
GETATTR commands to the metadata server, and by getting GETATTR commands to the metadata server. The client uses the file
CB_SIZE_CHANGED callbacks from the metadata server. The client uses size to decide if it should fill holes with zeros, or return a short
the file size to decide if it should fill holes with zeros, or return read. Striping patterns can cause cases where component objects are
a short read. Striping patterns can cause cases where component shorter than other components because a hole happens to correspond to
objects are shorter than other components because a hole happens to the last part of the component object.
correspond to the last part of the component object.
3.1.1 pnfs_osd_objid4 3.1.1. pnfs_osd_objid4
An object is identified by a number, somewhat like an inode number. An object is identified by a number, somewhat like an inode number.
The object storage model has a two level scheme, where the objects The object storage model has a two level scheme, where the objects
within an object storage device are grouped into partitions. within an object storage device are grouped into partitions.
struct pnfs_osd_objid4 { struct pnfs_osd_objid4 {
pnfs_deviceid4 device_id; deviceid4 device_id;
uint64_t partition_id; uint64_t partition_id;
uint64_t object_id; uint64_t object_id;
}; };
The pnfs_osd_objid4 type is used to identify an object within a The pnfs_osd_objid4 type is used to identify an object within a
partition on a specified object storage device. "device_id" selects partition on a specified object storage device. "device_id" selects
the object storage device from the set of available storage devices. the object storage device from the set of available storage devices.
The device is identified with the pnfs_deviceid4 type, which is an The device is identified with the deviceid4 type, which is an index
index into addressing information about that device returned by the into addressing information about that device returned by the
GETDEVICEINFO pnfs operation. Within an OSD, a partition is GETDEVICELIST and GETDEVICEINFO pnfs operations. Within an OSD, a
identified with a 64-bit number, "partition_id". Within a partition, partition is identified with a 64-bit number, "partition_id". Within
an object is identified with a 64-bit number, "object_id". Creation a partition, an object is identified with a 64-bit number,
and management of partitions is outside the scope of this standard, "object_id". Creation and management of partitions is outside the
and is a facility provided by the object storage file system. scope of this standard, and is a facility provided by the object
storage file system.
3.1.2 pnfs_osd_version4 3.1.2. pnfs_osd_version4
enum pnfs_osd_version4 { enum pnfs_osd_version4 {
PNFS_OSD_MISSING = 0, PNFS_OSD_MISSING = 0,
PNFS_OSD_VERSION_1 = 1, PNFS_OSD_VERSION_1 = 1,
PNFS_OSD_VERSION_2 = 2 PNFS_OSD_VERSION_2 = 2
}; };
The osd_version is used to indicate the OSD protocol version or The osd_version is used to indicate the OSD protocol version or
whether an object is missing (i.e., unavailable). Some layout whether an object is missing (i.e., unavailable). Some layout
schemes encode redundant information and can compensate for missing schemes encode redundant information and can compensate for missing
components, but the data placement algorithm needs to know what parts components, but the data placement algorithm needs to know what parts
are missing. are missing.
At this time the OSD standard is at version 1.0, and we anticipate a At this time the OSD standard is at version 1.0, and we anticipate a
version 2.0 of the standard. The second generation OSD protocol has version 2.0 of the standard ((SNIA T10/1729-D [6])). The second
additional proposed features to support more robust error recovery, generation OSD protocol has additional proposed features to support
snapshots, and byte-range capabilities. Therefore, the OSD version more robust error recovery, snapshots, and byte-range capabilities.
is explicitly called out in the information returned in the layout. Therefore, the OSD version is explicitly called out in the
(This information can also be deduced by looking inside the information returned in the layout. (This information can also be
capability type at the format field, which is the first byte. The deduced by looking inside the capability type at the format field,
format value is 0x1 for an OSD v1 capability. However, it seems most which is the first byte. The format value is 0x1 for an OSD v1
robust to call out the version explicitly.) capability. However, it seems most robust to call out the version
explicitly.)
3.1.3 pnfs_osd_object_cred4 3.1.3. pnfs_osd_object_cred4
struct pnfs_osd_object_cred4 { struct pnfs_osd_object_cred4 {
pnfs_osd_objid4 object_id; pnfs_osd_objid4 object_id;
pnfs_osd_version4 osd_version; pnfs_osd_version4 osd_version;
opaque credential<>; opaque credential<>;
}; };
The pnfs_osd_object_cred4 structure is used to identify each The pnfs_osd_object_cred4 structure is used to identify each
component comprising the file. The object_id identifies the component comprising the file. The object_id identifies the
component object, the osd_version represents the osd protocol component object, the osd_version represents the osd protocol
version, or whether that component is unavailable, and the credential version, or whether that component is unavailable, and the credential
provides the OSD security credentials needed to access that object provides the OSD security credentials needed to access that object
(see Section 8.1 for more details). (see Section 8.1 for more details).
3.1.4 pnfs_osd_raid_algorithm4 3.1.4. pnfs_osd_raid_algorithm4
enum pnfs_osd_raid_algorithm4 { enum pnfs_osd_raid_algorithm4 {
PNFS_OSD_RAID_0 = 1, PNFS_OSD_RAID_0 = 1,
PNFS_OSD_RAID_4 = 2, PNFS_OSD_RAID_4 = 2,
PNFS_OSD_RAID_5 = 3, PNFS_OSD_RAID_5 = 3,
PNFS_OSD_RAID_PQ = 4 /* Reed-Solomon P+Q */ PNFS_OSD_RAID_PQ = 4 /* Reed-Solomon P+Q */
}; };
pnfs_osd_raid_algorithm4 represents the data redundancy algorithm pnfs_osd_raid_algorithm4 represents the data redundancy algorithm
used to protect the file's contents. See Section 3.3 for more used to protect the file's contents. See Section 3.3 for more
details. details.
3.1.5 pnfs_osd_data_map4 3.1.5. pnfs_osd_data_map4
struct pnfs_osd_data_map4 { struct pnfs_osd_data_map4 {
length4 stripe_unit; length4 stripe_unit;
uint16_t group_width; uint32_t group_width;
uint16_t group_depth; uint32_t group_depth;
uint16_t mirror_cnt; uint32_t mirror_cnt;
pnfs_osd_raid_algorithm4 raid_algorithm; pnfs_osd_raid_algorithm4 raid_algorithm;
}; };
The pnfs_osd_data_map4 structure parameterizes the algorithm that The pnfs_osd_data_map4 structure parameterizes the algorithm that
maps a file's contents over the component objects. Instead of maps a file's contents over the component objects. Instead of
limiting the system to simple striping scheme where loss of a single limiting the system to simple striping scheme where loss of a single
component object results in data loss, the map parameters support component object results in data loss, the map parameters support
mirroring and more complicated schemes that protect against loss of a mirroring and more complicated schemes that protect against loss of a
component object. component object.
skipping to change at page 8, line 4 skipping to change at page 9, line 9
The pnfs_osd_data_map4 structure parameterizes the algorithm that The pnfs_osd_data_map4 structure parameterizes the algorithm that
maps a file's contents over the component objects. Instead of maps a file's contents over the component objects. Instead of
limiting the system to simple striping scheme where loss of a single limiting the system to simple striping scheme where loss of a single
component object results in data loss, the map parameters support component object results in data loss, the map parameters support
mirroring and more complicated schemes that protect against loss of a mirroring and more complicated schemes that protect against loss of a
component object. component object.
The stripe_unit is the number of bytes placed on one component before The stripe_unit is the number of bytes placed on one component before
advancing to the next one in the list of components. The number of advancing to the next one in the list of components. The number of
bytes in a full stripe is stripe_unit times the number of components. bytes in a full stripe is stripe_unit times the number of components.
In some raid schemes, a stripe includes redundant information (i.e., In some raid schemes, a stripe includes redundant information (i.e.,
parity) that lets the system recover from loss or damage to a parity) that lets the system recover from loss or damage to a
component object. component object.
The group_width and group_depth parameters allow a nested striping The group_width and group_depth parameters allow a nested striping
pattern. If there is no nesting, then group_width and group_depth pattern. If there is no nesting, then group_width and group_depth
MUST be zero. Otherwise, the group_width defines the width of a data MUST be zero. Otherwise, the group_width defines the width of a data
stripe, and the group_depth defines how many stripes are written stripe, and the group_depth defines how many stripes are accessed
before advancing to the next group of components in the list of before advancing to the next group of components in the list of
component objects for the file. The size of the components array component objects for the file. The size of the components array
MUST be a multiple of group_width. MUST be a multiple of group_width.
The mirror_cnt is used to replicate a file by replicating its The mirror_cnt is used to replicate a file by replicating its
component objects. If there is no mirroring, then mirror_cnt MUST be component objects. If there is no mirroring, then mirror_cnt MUST be
0. If mirror_cnt is greater than zero, then the size of the 0. If mirror_cnt is greater than zero, then the size of the
component array MUST be a multiple of (mirror_cnt+1). component array MUST be a multiple of (mirror_cnt+1).
See Section 3.2 for more details. See Section 3.2 for more details.
3.2 Data Mapping Schemes 3.2. Data Mapping Schemes
This section describes the different data mapping schemes in detail. This section describes the different data mapping schemes in detail.
3.2.1 Simple Striping 3.2.1. Simple Striping
The object layout always uses a "dense" layout as described in the The object layout always uses a "dense" layout as described in the
pNFS document. This means that the second stripe unit of the file pNFS document. This means that the second stripe unit of the file
starts at offset 0 of the second component, rather than at offset starts at offset 0 of the second component, rather than at offset
stripe_unit bytes. After a full stripe has been written, the next stripe_unit bytes. After a full stripe has been written, the next
stripe unit is appended to the first component object in the list stripe unit is appended to the first component object in the list
without any holes in the component objects. The mapping from the without any holes in the component objects. The mapping from the
logical offset within a file (L) to do the component object C and logical offset within a file (L) to do the component object C and
object-specific offset O is defined by the following equations: object-specific offset O is defined by the following equations:
skipping to change at page 9, line 29 skipping to change at page 10, line 33
Offset 9000: Offset 9000:
N = 9000 / 16384 = 0 N = 9000 / 16384 = 0
C = (9000-(0*16384)) / 4096 = 2 (D2) C = (9000-(0*16384)) / 4096 = 2 (D2)
O = (0*4096)+(9000%4096) = 808 O = (0*4096)+(9000%4096) = 808
Offset 132000: Offset 132000:
N = 132000 / 16384 = 8 N = 132000 / 16384 = 8
C = (132000-(8*16384)) / 4096 = 0 C = (132000-(8*16384)) / 4096 = 0
O = (8*4096) + (132000%4096) = 33696 O = (8*4096) + (132000%4096) = 33696
3.2.2 Nested Striping 3.2.2. Nested Striping
The group_width and group_depth parameters allow a nested striping The group_width and group_depth parameters allow a nested striping
pattern. If there is no nesting, then group_width and group_depth pattern. If there is no nesting, then group_width and group_depth
MUST be zero. Otherwise, the group_width defines the width of a data MUST be zero. Otherwise, the group_width defines the width of a data
stripe, and the group_depth defines how many stripes are written stripe, and the group_depth defines how many stripes are written
before advancing to the next group of components in the list of before advancing to the next group of components in the list of
component objects for the file. The size of the components array component objects for the file. The size of the components array
MUST be a multiple of group_width. The math used to map from a file MUST be a multiple of group_width. The math used to map from a file
offset to a component object and offset within that object is shown offset to a component object and offset within that object is shown
below. The computations map from the logical offset L to the below. The computations map from the logical offset L to the
skipping to change at page 11, line 5 skipping to change at page 12, line 33
O = 27 MB % 1 MB + 2 * 1 MB + 0 * 50 * 1 MB = 2 MB O = 27 MB % 1 MB + 2 * 1 MB + 0 * 50 * 1 MB = 2 MB
Offset 7232 MB: Offset 7232 MB:
M = 7232 MB / 5000 MB = 1 M = 7232 MB / 5000 MB = 1
G = (7232 MB - (1 * 5000 MB)) / 500 MB = 4 G = (7232 MB - (1 * 5000 MB)) / 500 MB = 4
H = (7232 MB - (1 * 5000 MB)) % 500 MB = 232 MB H = (7232 MB - (1 * 5000 MB)) % 500 MB = 232 MB
N = 232 MB / 10 MB = 23 N = 232 MB / 10 MB = 23
C = (232 MB - (23 * 10 MB)) / 1 MB + 4 * 10 = 42 C = (232 MB - (23 * 10 MB)) / 1 MB + 4 * 10 = 42
O = 7232 MB % 1 MB + 23 * 1 MB + 1 * 50 * 1 MB = 73 MB O = 7232 MB % 1 MB + 23 * 1 MB + 1 * 50 * 1 MB = 73 MB
3.2.3 Mirroring 3.2.3. Mirroring
The mirror_cnt is used to replicate a file by replicating its The mirror_cnt is used to replicate a file by replicating its
component objects. If there is no mirroring, then mirror_cnt MUST be component objects. If there is no mirroring, then mirror_cnt MUST be
0. If mirror_cnt is greater than zero, then the size of the 0. If mirror_cnt is greater than zero, then the size of the
component array MUST be a multiple of (mirror_cnt+1). Thus, for a component array MUST be a multiple of (mirror_cnt+1). Thus, for a
classic mirror on two objects, mirror_cnt is one. If group_width is classic mirror on two objects, mirror_cnt is one. If group_width is
also non-zero, then the size MUST be a multiple of group_width * also non-zero, then the size MUST be a multiple of group_width *
(mirror_cnt+1). Replicas are adjacent in the components array, and (mirror_cnt+1). Replicas are adjacent in the components array, and
the value C produced by the above equations is not a direct index the value C produced by the above equations is not a direct index
into the components array. Instead, the following equations into the components array. Instead, the following equations
determine the replica component index RCi, where i ranges from 0 to determine the replica component index RCi, where i ranges from 0 to
mirror_cnt. mirror_cnt.
C = component index for striping or two-level striping C = component index for striping or two-level striping
i ranges from 0 to mirror_cnt, inclusive i ranges from 0 to mirror_cnt, inclusive
RCi = C * (mirror_cnt+1) + i RCi = C * (mirror_cnt+1) + i
3.3 RAID Algorithms 3.3. RAID Algorithms
pnfs_osd_raid_algorithm4 determines the algorithm and placement of pnfs_osd_raid_algorithm4 determines the algorithm and placement of
redundant data. This section defines the different RAID algorithms. redundant data. This section defines the different RAID algorithms.
3.3.1 PNFS_OSD_RAID_0 3.3.1. PNFS_OSD_RAID_0
PNFS_OSD_RAID_0 means there is no parity data, so all bytes in the PNFS_OSD_RAID_0 means there is no parity data, so all bytes in the
component objects are data bytes located by the above equations for C component objects are data bytes located by the above equations for C
and O. If a component object is unavailable, the pNFS client can and O. If a component object is unavailable, the pNFS client can
choose to return NULLs for the missing data, or it can retry the READ choose to return NULLs for the missing data, or it can retry the READ
against the pNFS server, or it can return an EIO error. against the pNFS server, or it can return an EIO error.
3.3.2 PNFS_OSD_RAID_4 3.3.2. PNFS_OSD_RAID_4
PNFS_OSD_RAID_4 means that the last component object, or the last in PNFS_OSD_RAID_4 means that the last component object, or the last in
each group if group_width is > zero, contains parity information each group if group_width is > zero, contains parity information
computed over the rest of the stripe with an XOR operation. If a computed over the rest of the stripe with an XOR operation. If a
component object is unavailable, the client can read the rest of the component object is unavailable, the client can read the rest of the
stripe units in the damaged stripe and recompute the missing stripe stripe units in the damaged stripe and recompute the missing stripe
unit by XORing the other stripe units in the stripe. Or the client unit by XORing the other stripe units in the stripe. Or the client
can replay the READ against the pNFS server which will presumably can replay the READ against the pNFS server which will presumably
perform the reconstructed read on the client's behalf. perform the reconstructed read on the client's behalf.
skipping to change at page 12, line 12 skipping to change at page 13, line 41
for embedded parity, L'. First compute L', and then use L' in the for embedded parity, L'. First compute L', and then use L' in the
above equations for C and O. above equations for C and O.
L = file offset, not accounting for parity L = file offset, not accounting for parity
P = number of parity devices in each stripe P = number of parity devices in each stripe
W = group_width, if not zero, else size of component array W = group_width, if not zero, else size of component array
N = L / (W-P * stripe_unit) N = L / (W-P * stripe_unit)
L' = N * (W * stripe_unit) + L' = N * (W * stripe_unit) +
(L % (W-P * stripe_unit)) (L % (W-P * stripe_unit))
3.3.3 PNFS_OSD_RAID_5 3.3.3. PNFS_OSD_RAID_5
PNFS_OSD_RAID_5 means that the position of the parity data is rotated PNFS_OSD_RAID_5 means that the position of the parity data is rotated
on each stripe. In the first stripe, the last component holds the on each stripe. In the first stripe, the last component holds the
parity. In the second stripe, the next-to-last component holds the parity. In the second stripe, the next-to-last component holds the
parity, and so on. In this scheme, all stripe units are rotated so parity, and so on. In this scheme, all stripe units are rotated so
that I/O is evenly spread across objects as the file is read that I/O is evenly spread across objects as the file is read
sequentially. The rotated parity layout is illustrated here, with sequentially. The rotated parity layout is illustrated here, with
numbers indicating the stripe unit. numbers indicating the stripe unit.
0 1 2 P 0 1 2 P
skipping to change at page 12, line 45 skipping to change at page 14, line 28
W = group_width, if not zero, else size of component array W = group_width, if not zero, else size of component array
N = L / (W-1 * stripe_unit) N = L / (W-1 * stripe_unit)
(Compute L' as describe above) (Compute L' as describe above)
(Compute C based on L' as described above) (Compute C based on L' as described above)
C' = (C - (N%W)) % W C' = (C - (N%W)) % W
I = W - (N%W) - 1 I = W - (N%W) - 1
if (C' <= I) { if (C' <= I) {
C'++ C'++
} }
3.3.4 PNFS_OSD_RAID_PQ 3.3.4. PNFS_OSD_RAID_PQ
PNFS_OSD_RAID_PQ is a double-parity scheme that uses the Reed-Solomon PNFS_OSD_RAID_PQ is a double-parity scheme that uses the Reed-Solomon
P+Q encoding scheme. In this layout, the last two component objects P+Q encoding scheme. In this layout, the last two component objects
hold the P and Q data, respectively. P is parity computed with XOR, hold the P and Q data, respectively. P is parity computed with XOR,
and Q is a more complex equation that is not described here. The and Q is a more complex equation that is not described here. The
equations given above for embedded parity can be used to map a file equations given above for embedded parity can be used to map a file
offset to the correct component object by setting the number of offset to the correct component object by setting the number of
parity components to 2 instead of 1 for RAID4 or RAID5. Clients may parity components to 2 instead of 1 for RAID4 or RAID5. Clients may
simply choose to read data through the metadata server if two simply choose to read data through the metadata server if two
components are missing or damaged. components are missing or damaged.
Issue: This scheme also has a RAID_4 like layout where the ECC blocks Issue: This scheme also has a RAID_4 like layout where the ECC blocks
are stored on the same components on every stripe and a rotated, are stored on the same components on every stripe and a rotated,
RAID-5 like layout where the stripe units are rotated. Should we RAID-5 like layout where the stripe units are rotated. Should we
make the following properties orthogonal: RAID_4 or RAID_5 (i.e., make the following properties orthogonal: RAID_4 or RAID_5 (i.e.,
non-rotated or rotated), and then have the number of parity non-rotated or rotated), and then have the number of parity
components and the associated algorithm be the orthogonal parameter? components and the associated algorithm be the orthogonal parameter?
3.3.5 RAID Usage and implementation notes 3.3.5. RAID Usage and implementation notes
RAID layouts with redundant data in their stripes require additional RAID layouts with redundant data in their stripes require additional
serialization of updates to ensure correct operation. Otherwise, if serialization of updates to ensure correct operation. Otherwise, if
two clients simultaneously write to the same logical range of an two clients simultaneously write to the same logical range of an
object, the result could include different data in the same ranges of object, the result could include different data in the same ranges of
mirrored tuples, or corrupt parity information. It is the mirrored tuples, or corrupt parity information. It is the
responsibility of the metadata server to enforce serialization responsibility of the metadata server to enforce serialization
requirements such as this. For example, the metadata server may do requirements such as this. For example, the metadata server may do
so by not granting overlapping write layouts within mirrored objects. so by not granting overlapping write layouts within mirrored objects.
4. Object-Based Layout Update 4. Object-Based Layout Update
pnfs_layoutupdate4 is used in the LAYOUTCOMMIT operation to convey layoutupdate4 is used in the LAYOUTCOMMIT operation to convey updates
updates to the layout and additional information to the metadata to the layout and additional information to the metadata server. It
server. It is defined in the NFSv4.1 draft [5] as follows: is defined in the NFSv4.1 draft [5] as follows:
struct pnfs_layoutupdate4 { struct layoutupdate4 {
pnfs_layouttype4 type; layouttype4 lou_type;
opaque layoutupdate_data<>; opaque lou_body<>;
}; };
The pnfs_layoutupdate4 type is an opaque value at the generic pNFS The layoutupdate4 type is an opaque value at the generic pNFS client
client level. If the type is LAYOUT_OSD2_OBJECTS, then the opaque level. If the lou_type layout type is LAYOUT4_OSD2_OBJECTS, then the
value is described by the pnfs_osd_layoutupdate4 type. lou_body opaque value is defined by the pnfs_osd_layoutupdate4 type.
4.1 pnfs_osd_layoutupdate4 4.1. pnfs_osd_layoutupdate4
struct pnfs_osd_layoutupdate4 { struct pnfs_osd_layoutupdate4 {
pnfs_osd_deltaspaceused4 delta_space_used; pnfs_osd_deltaspaceused4 delta_space_used;
pnfs_osd_ioerr4 ioerr<>; pnfs_osd_ioerr4 ioerr<>;
}; };
Object-Based pNFS clients are not allowed to modify the layout. Object-Based pNFS clients are not allowed to modify the layout.
"delta_space_used" is used to convey capacity usage information back "delta_space_used" is used to convey capacity usage information back
to the metadata server and, in case OSD I/O operations failed, to the metadata server and, in case OSD I/O operations failed,
"ioerr" is used to report these errors to the metadata server. "ioerr" is used to report these errors to the metadata server.
4.1.1 pnfs_osd_deltaspaceused4 4.1.1. pnfs_osd_deltaspaceused4
union pnfs_osd_deltaspaceused4 switch (bool valid) { union pnfs_osd_deltaspaceused4 switch (bool valid) {
case TRUE: case TRUE:
length4 delta; /* Bytes consumed by write activity */ int64_t delta; /* Bytes consumed by write activity */
case FALSE: case FALSE:
void; void;
}; };
pnfs_osd_deltaspaceused4 is used to convey space utilization pnfs_osd_deltaspaceused4 is used to convey space utilization
information at the time of LAYOUTCOMMIT. For the file system to information at the time of LAYOUTCOMMIT. For the file system to
properly maintain capacity used information, it needs to track how properly maintain capacity used information, it needs to track how
much capacity was consumed by WRITE operations performed by the much capacity was consumed by WRITE operations performed by the
client. In this protocol, the OSD returns the capacity consumed by a client. In this protocol, the OSD returns the capacity consumed by a
write, which can be different because of internal overhead like write, which can be different because of internal overhead like
block-based allocation and indirect blocks, and the client reflects block-based allocation and indirect blocks, and the client reflects
this back to the pNFS server so it can accurately track quota. The this back to the pNFS server so it can accurately track quota. The
pNFS server can choose to trust this information coming from the pNFS server can choose to trust this information coming from the
clients and therefore avoid querying the OSDs at the time of clients and therefore avoid querying the OSDs at the time of
LAYOUTCOMMIT. If the client is unable to obtain this information LAYOUTCOMMIT. If the client is unable to obtain this information
from the OSD, it simply returns invalid deltaspaceused4. from the OSD, it simply returns invalid delta_space_used.
4.1.2 pnfs_osd_errno4 4.1.2. pnfs_osd_errno4
enum pnfs_osd_errno4 { enum pnfs_osd_errno4 {
PNFS_OSD_NOT_FOUND = 1, PNFS_OSD_NOT_FOUND = 1,
PNFS_OSD_NO_SPACE = 2, PNFS_OSD_NO_SPACE = 2,
PNFS_OSD_EIO = 3, PNFS_OSD_EIO = 3,
PNFS_OSD_BAD_CRED = 4, PNFS_OSD_BAD_CRED = 4,
PNFS_OSD_NO_ACCESS = 5, PNFS_OSD_NO_ACCESS = 5,
PNFS_OSD_UNREACHABLE = 6 PNFS_OSD_UNREACHABLE = 6
}; };
pnfs_osd_errno4 is used to represent error types when read/write pnfs_osd_errno4 is used to represent error types when read/write
errors are reported to the metadata server. errors are reported to the metadata server.
o PNFS_OSD_NOT_FOUND indicates the object ID specifics an object o PNFS_OSD_NOT_FOUND indicates the object ID specifics an object that
that does not exist on the Object Storage Device. does not exist on the Object Storage Device.
o PNFS_OSD_NO_SPACE indicates the operation failed because the o PNFS_OSD_NO_SPACE indicates the operation failed because the Object
Object Storage Device ran out of free capacity during the operation. Storage Device ran out of free capacity during the operation.
o PNFS_OSD_EIO indicates the operation failed because the Object o PNFS_OSD_EIO indicates the operation failed because the Object
Storage Device experienced a failure trying to access the object. Storage Device experienced a failure trying to access the object.
The most common source of these errors is media errors, but other The most common source of these errors is media errors, but other
internal errors might cause this. In this case, the metadata server internal errors might cause this. In this case, the metadata server
should go examine the broken object more closely. should go examine the broken object more closely.
o PNFS_OSD_BAD_CRED indicates the security parameters are not valid. o PNFS_OSD_BAD_CRED indicates the security parameters are not valid.
The primary cause of this is that the capability has expired, or the The primary cause of this is that the capability has expired, or the
security policy tag (i.e., capability version number) has been security policy tag (i.e., capability version number) has been
changed to revoke capabilities. The client will need to return the changed to revoke capabilities. The client will need to return the
layout and get a new one with fresh capabilities. layout and get a new one with fresh capabilities.
o PNFS_OSD_NO_ACCESS indicates the capability does not allow the o PNFS_OSD_NO_ACCESS indicates the capability does not allow the
requested operation. This should not occur in normal operation requested operation. This should not occur in normal operation
because the metadata server should give out correct capabilities, or because the metadata server should give out correct capabilities, or
none at all. none at all.
o PNFS_OSD_UNREACHABLE indicates the client was unable to contact o PNFS_OSD_UNREACHABLE indicates the client was unable to contact the
the Object Storage Device due to a communication failure. Object Storage Device due to a communication failure.
4.1.3 pnfs_osd_ioerr4 4.1.3. pnfs_osd_ioerr4
struct pnfs_osd_ioerr4 { struct pnfs_osd_ioerr4 {
pnfs_osd_objid4 component; pnfs_osd_objid4 component;
length4 offset; length4 offset;
length4 length; length4 length;
bool iswrite; bool iswrite;
pnfs_osd_errno4 errno; pnfs_osd_errno4 errno;
}; };
The pnfs_osd_ioerr4 structure is used to return error indications for The pnfs_osd_ioerr4 structure is used to return error indications for
objects that generated errors during data transfers. These are hints objects that generated errors during data transfers. These are hints
to the metadata server that there are problems with that object. For to the metadata server that there are problems with that object. For
each error, "component", "offset", and "length" represent the object each error, "component", "offset", and "length" represent the object
and byte range in which the error occurred. "iswrite" is set to and byte range within the component object in which the error
"true" if the failed OSD operation was data modifying, and "errno" occurred. "iswrite" is set to "true" if the failed OSD operation was
represents the type of error. data modifying, and "errno" represents the type of error.
5. Object-Based Creation Layout Hint 5. Object-Based Creation Layout Hint
The pnfs_layouthint4 type is defined in the NFSv4.1 draft [5] as The layouthint4 type is defined in the NFSv4.1 draft [5] as follows:
follows:
struct pnfs_layouthint4 { struct layouthint4 {
pnfs_layouttype4 type; layouttype4 loh_type;
opaque layouthint_data<>; opaque loh_body<>;
}; };
The pnfs_layouthint4 type is an opaque value at the generic pNFS The layouthint4 structure is used by the client to pass in a hint
client level. If the layout type is LAYOUT_OSD2_OBJECTS, then the about the type of layout it would like created for a particular file.
opaque value is described by the pnfs_osd_layouthint4 type. If the loh_type layout type is LAYOUT4_OSD2_OBJECTS, then the
loh_body opaque value is defined by the pnfs_osd_layouthint4 type.
5.1 pnfs_osd_layouthint4 5.1. pnfs_osd_layouthint4
union num_comps_hint4 switch (bool valid) { union num_comps_hint4 switch (bool valid) {
case TRUE: case TRUE:
uint32_t num_comps; uint32_t num_comps;
case FALSE: case FALSE:
void; void;
}; };
union stripe_unit_hint4 switch (bool valid) { union stripe_unit_hint4 switch (bool valid) {
case TRUE: case TRUE:
skipping to change at page 16, line 20 skipping to change at page 18, line 4
case FALSE: case FALSE:
void; void;
}; };
union stripe_unit_hint4 switch (bool valid) { union stripe_unit_hint4 switch (bool valid) {
case TRUE: case TRUE:
length4 stripe_unit; length4 stripe_unit;
case FALSE: case FALSE:
void; void;
}; };
union group_width_hint4 switch (bool valid) { union group_width_hint4 switch (bool valid) {
case TRUE: case TRUE:
uint16_t group_width; uint32_t group_width;
case FALSE: case FALSE:
void; void;
}; };
union group_depth_hint4 switch (bool valid) { union group_depth_hint4 switch (bool valid) {
case TRUE: case TRUE:
uint16_t group_depth; uint32_t group_depth;
case FALSE: case FALSE:
void; void;
}; };
union mirror_cnt_hint4 switch (bool valid) { union mirror_cnt_hint4 switch (bool valid) {
case TRUE: case TRUE:
uint16_t mirror_cnt; uint32_t mirror_cnt;
case FALSE: case FALSE:
void; void;
}; };
union raid_algorithm_hint4 switch (bool valid) { union raid_algorithm_hint4 switch (bool valid) {
case TRUE: case TRUE:
pnfs_osd_raid_algorithm4 raid_algorithm; pnfs_osd_raid_algorithm4 raid_algorithm;
case FALSE: case FALSE:
void; void;
}; };
skipping to change at page 17, line 32 skipping to change at page 19, line 16
The pnfs layout operations operate on logical byte ranges. There is The pnfs layout operations operate on logical byte ranges. There is
no requirement in the protocol for any relationship between byte no requirement in the protocol for any relationship between byte
ranges used in LAYOUTGET to acquire layouts and byte ranges used in ranges used in LAYOUTGET to acquire layouts and byte ranges used in
CB_LAYOUTRECALL, LAYOUTCOMMIT, or LAYOUTRETURN. However, using OSD CB_LAYOUTRECALL, LAYOUTCOMMIT, or LAYOUTRETURN. However, using OSD
capabilities poses limitations on these operations since the capabilities poses limitations on these operations since the
capabilities associated with layout segments cannot be merged or capabilities associated with layout segments cannot be merged or
split. The following guidelines should be followed for proper split. The following guidelines should be followed for proper
operation of object-based layouts. operation of object-based layouts.
6.1 CB_LAYOUTRECALL and LAYOUTRETURN 6.1. CB_LAYOUTRECALL and LAYOUTRETURN
In general, the object-based layout driver should keep track of each In general, the object-based layout driver should keep track of each
layout segment it got, keeping record of the segment's iomode, layout segment it got, keeping record of the segment's iomode,
offset, and length. The server should allow the client to get offset, and length. The server should allow the client to get
multiple overlapping layout segments but is free to recall the layout multiple overlapping layout segments but is free to recall the layout
to prevent overlap. to prevent overlap.
In response to CB_LAYOUTRECALL, the client should return all layout In response to CB_LAYOUTRECALL, the client should return all layout
segments matching the given iomode and overlapping with the recalled segments matching the given iomode and overlapping with the recalled
range. When returning the layouts for this byte range with range. When returning the layouts for this byte range with
skipping to change at page 18, line 10 skipping to change at page 19, line 42
the clientid, iomode, and byte range given in LAYOUTRETURN. If no the clientid, iomode, and byte range given in LAYOUTRETURN. If no
exact match is found then the server should release all layout exact match is found then the server should release all layout
segments matching the clientid and iomode and that are fully segments matching the clientid and iomode and that are fully
contained in the returned byte range. If none are found and the byte contained in the returned byte range. If none are found and the byte
range is a subset of an outstanding layout segment with for the same range is a subset of an outstanding layout segment with for the same
clientid and iomode, then the client can be considered malfunctioning clientid and iomode, then the client can be considered malfunctioning
and the server SHOULD recall all layouts from this client to reset and the server SHOULD recall all layouts from this client to reset
its state. If this behavior repeats the server SHOULD deny all its state. If this behavior repeats the server SHOULD deny all
LAYOUTGETs from this client. LAYOUTGETs from this client.
6.2 LAYOUTCOMMIT 6.2. LAYOUTCOMMIT
LAYOUTCOMMIT is only used by object-based pNFS to convey modified LAYOUTCOMMIT is only used by object-based pNFS to convey modified
attributes hints and/or to report I/O errors to the MDS. Therefore, attributes hints and/or to report I/O errors to the MDS. Therefore,
the offset and length in LAYOUTCOMMIT4args are reserved for future the offset and length in LAYOUTCOMMIT4args are reserved for future
use and should be set to 0. However, component byte ranges in the use and should be set to 0. However, component byte ranges in the
optional pnfs_osd_ioerr4 structure are used for recovering the object optional pnfs_osd_ioerr4 structure are used for recovering the object
and MUST be set by the client to cover all failed I/O operations to and MUST be set by the client to cover all failed I/O operations to
the component. the component.
7. Recalling Layouts 7. Recalling Layouts
skipping to change at page 19, line 13 skipping to change at page 20, line 45
significant. significant.
The object storage protocol MUST implement the security aspects The object storage protocol MUST implement the security aspects
described in version 1 of the T10 OSD protocol definition [2]. The described in version 1 of the T10 OSD protocol definition [2]. The
remainder of this section gives an overview of the security mechanism remainder of this section gives an overview of the security mechanism
described in that standard. The goal is to give the reader a basic described in that standard. The goal is to give the reader a basic
understanding of the object security model. Any discrepancies understanding of the object security model. Any discrepancies
between this text and the actual standard are obviously to be between this text and the actual standard are obviously to be
resolved in favor of the OSD standard. resolved in favor of the OSD standard.
8.1 OSD Security Data Types 8.1. OSD Security Data Types
There are three main data types associated with object security: a There are three main data types associated with object security: a
capability, a credential, and security parameters. The capability is capability, a credential, and security parameters. The capability is
a set of fields that specifies an object and what operations can be a set of fields that specifies an object and what operations can be
performed on it. A credential is a signed capability. Only a performed on it. A credential is a signed capability. Only a
security manager that knows the secret device keys can correctly sign security manager that knows the secret device keys can correctly sign
a capability to form a valid credential. In pNFS, the file server a capability to form a valid credential. In pNFS, the file server
acts as the security manager and returns signed capabilities (i.e., acts as the security manager and returns signed capabilities (i.e.,
credentials) to the pNFS client. The security parameters are values credentials) to the pNFS client. The security parameters are values
computed by the issuer of OSD commands (i.e., the client) that prove computed by the issuer of OSD commands (i.e., the client) that prove
skipping to change at page 19, line 36 skipping to change at page 21, line 20
resulting signatures into the security_parameters field of the OSD resulting signatures into the security_parameters field of the OSD
command. The object storage device uses the secret keys it shares command. The object storage device uses the secret keys it shares
with the security manager to validate the signature values in the with the security manager to validate the signature values in the
security parameters. security parameters.
The security types are opaque to the generic layers of the pNFS The security types are opaque to the generic layers of the pNFS
client. The credential is defined as opaque within the client. The credential is defined as opaque within the
pnfs_osd_and_cred type. Instead of repeating the definitions here, pnfs_osd_and_cred type. Instead of repeating the definitions here,
the reader is referred to section 4.9.2.2 of the OSD standard. the reader is referred to section 4.9.2.2 of the OSD standard.
8.2 The OSD Security Protocol 8.2. The OSD Security Protocol
The object storage protocol relies on a cryptographically secure The object storage protocol relies on a cryptographically secure
capability to control accesses at the object storage devices. capability to control accesses at the object storage devices.
Capabilities are generated by the metadata server, returned to the Capabilities are generated by the metadata server, returned to the
client, and used by the client as described below to authenticate client, and used by the client as described below to authenticate
their requests to the Object Storage Device (OSD). Capabilities their requests to the Object Storage Device (OSD). Capabilities
therefore achieve the required access and open mode checking. They therefore achieve the required access and open mode checking. They
allow the file server to define and check a policy (e.g., open mode) allow the file server to define and check a policy (e.g., open mode)
and the OSD to enforce that policy without knowing the details (e.g., and the OSD to enforce that policy without knowing the details (e.g.,
user IDs and ACLs). user IDs and ACLs).
skipping to change at page 21, line 5 skipping to change at page 22, line 39
generate valid OSD requests (within the CapArgs access restriction). generate valid OSD requests (within the CapArgs access restriction).
To provide the required privacy requirements for the capabilities To provide the required privacy requirements for the capabilities
returned by LAYOUTGET, the GSS-API can be used, e.g. by using a returned by LAYOUTGET, the GSS-API can be used, e.g. by using a
session key known to the file server and to the client to encrypt the session key known to the file server and to the client to encrypt the
whole layout or parts of it. Two general ways to provide privacy in whole layout or parts of it. Two general ways to provide privacy in
the absence of GSS-API that are independent of NFSv4 are either an the absence of GSS-API that are independent of NFSv4 are either an
isolated network such as a VLAN or a secure channel provided by isolated network such as a VLAN or a secure channel provided by
IPsec. IPsec.
8.3 Revoking capabilities 8.3. Revoking capabilities
At any time, the metadata server may invalidate all outstanding At any time, the metadata server may invalidate all outstanding
capabilities on an object by changing its capability version capabilities on an object by changing its capability version
attribute. There is also a "fence bit" attribute that the metadata attribute. There is also a "fence bit" attribute that the metadata
server can toggle to temporarily block access without permanently server can toggle to temporarily block access without permanently
revoking capabilities. The value of the fence bit and the capability revoking capabilities. The value of the fence bit and the capability
version are part of a capability, and they must match the state of version are part of a capability, and they must match the state of
the attributes. If they do not match, the OSD rejects accesses to the attributes. If they do not match, the OSD rejects accesses to
the object. When a client attempts to use a capability and discovers the object. When a client attempts to use a capability and discovers
a capability version mismatch, it should issue a LAYOUTRETURN for the a capability version mismatch, it should issue a LAYOUTRETURN for the
skipping to change at page 21, line 40 skipping to change at page 23, line 25
either READ or READ/WRITE. It is the pNFS client's responsibility to either READ or READ/WRITE. It is the pNFS client's responsibility to
enforce access control among multiple users accessing the same file. enforce access control among multiple users accessing the same file.
It is neither required nor expected that the pNFS client will obtain It is neither required nor expected that the pNFS client will obtain
a separate layout for each user accessing a shared object. The a separate layout for each user accessing a shared object. The
client SHOULD use ACCESS calls to check user permissions when client SHOULD use ACCESS calls to check user permissions when
performing I/O so that the server's access control policies are performing I/O so that the server's access control policies are
correctly enforced. The result of the ACCESS operation may be cached correctly enforced. The result of the ACCESS operation may be cached
indefinitely, as the server is expected to recall layouts when the indefinitely, as the server is expected to recall layouts when the
file's access permissions or ACL change. file's access permissions or ACL change.
9. References 9. IANA Considerations
9.1 Normative References As described in the NFSv4.1 draft [5], new layout type numbers will
be requested from IANA. This document defines the protocol
associated with the existing layout type number,
LAYOUT4_OSD2_OBJECTS, and it requires no further actions for IANA.
10. References
10.1. Normative References
[1] Bradner, S., "Key words for use in RFCs to Indicate Requirement [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", RFC 2119, March 1997. Levels", RFC 2119, March 1997.
[2] Weber, R., "SCSI Object-Based Storage Device Commands", [2] Weber, R., "SCSI Object-Based Storage Device Commands",
July 2004, <http://www.t10.org/ftp/t10/drafts/osd/osd-r10.pdf>. July 2004, <http://www.t10.org/ftp/t10/drafts/osd/osd-r10.pdf>.
[3] Eisler, M., "XDR: External Data Representation Standard", [3] Eisler, M., "XDR: External Data Representation Standard",
RFC 4506, May 2006. STD 67, RFC 4506, May 2006.
[4] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, [4] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame,
C., Eisler, M., and D. Noveck, "Network File System (NFS) C., Eisler, M., and D. Noveck, "Network File System (NFS)
version 4 Protocol", RFC 3530, April 2003. version 4 Protocol", RFC 3530, April 2003.
9.2 Informative References 10.2. Informative References
[5] Shepler, S., "NFSv4 Minor Version 1", June 2006, <http:// [5] Shepler, S., Eisler, M., and D. Noveck, "NFSv4 Minor Version 1",
www.ietf.org/internet-drafts/ March 2007, <http://www.ietf.org/internet-drafts/
draft-ietf-nfsv4-minorversion1-03.txt>. draft-ietf-nfsv4-minorversion1-10.txt>.
[6] Weber, R., "SCSI Object-Based Storage Device Commands -2 [6] Weber, R., "SCSI Object-Based Storage Device Commands -2
(OSD-2)", October 2004, (OSD-2)", January 2007,
<http://www.t10.org/ftp/t10/drafts/osd2/osd2r00.pdf>. <http://www.t10.org/ftp/t10/drafts/osd2/osd2r01.pdf>.
Appendix A. Acknowledgments
Todd Pisek was a co-editor of the initial drafts for this document.
Authors' Addresses Authors' Addresses
Benny Halevy Benny Halevy
Panasas, Inc. Panasas, Inc.
1501 Reedsdale St. Suite 400 1501 Reedsdale St. Suite 400
Pittsburgh, PA 15233 Pittsburgh, PA 15233
USA USA
Phone: +1-412-323-3500 Phone: +1-412-323-3500
skipping to change at page 23, line 14 skipping to change at page 26, line 5
Jim Zelenka Jim Zelenka
Panasas, Inc. Panasas, Inc.
1501 Reedsdale St. Suite 400 1501 Reedsdale St. Suite 400
Pittsburgh, PA 15233 Pittsburgh, PA 15233
USA USA
Phone: +1-412-323-3500 Phone: +1-412-323-3500
Email: jimz@panasas.com Email: jimz@panasas.com
URI: http://www.panasas.com/ URI: http://www.panasas.com/
Todd Pisek Full Copyright Statement
Sun Microsystems, Inc.
1270 Eagan Industrial Rd. - Suite 160
Eagant, MN 55121-1231
USA
Phone: +1-651-552-6415 Copyright (C) The IETF Trust (2007).
Email: trp@sun.com
URI: http://www.sun.com/
Intellectual Property Statement This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79. found in BCP 78 and BCP 79.
skipping to change at page 24, line 29 skipping to change at page 26, line 45
such proprietary rights by implementers or users of this such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr. http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at this standard. Please address the information to the IETF at
ietf-ipr@ietf.org. ietf-ipr@ietf.org.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2006). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Acknowledgment Acknowledgment
Funding for the RFC Editor function is currently provided by the Funding for the RFC Editor function is provided by the IETF
Internet Society. Administrative Support Activity (IASA).
 End of changes. 72 change blocks. 
183 lines changed or deleted 187 lines changed or added

This html diff was produced by rfcdiff 1.33. The latest version is available from http://tools.ietf.org/tools/rfcdiff/