draft-ietf-nfsv4-flex-files-17.txt   draft-ietf-nfsv4-flex-files-18.txt 
NFSv4 B. Halevy NFSv4 B. Halevy
Internet-Draft Internet-Draft
Intended status: Standards Track T. Haynes Intended status: Standards Track T. Haynes
Expires: August 31, 2018 Primary Data Expires: October 27, 2018 Hammerspace
February 27, 2018 April 25, 2018
Parallel NFS (pNFS) Flexible File Layout Parallel NFS (pNFS) Flexible File Layout
draft-ietf-nfsv4-flex-files-17.txt draft-ietf-nfsv4-flex-files-18.txt
Abstract Abstract
The Parallel Network File System (pNFS) allows a separation between The Parallel Network File System (pNFS) allows a separation between
the metadata (onto a metadata server) and data (onto a storage the metadata (onto a metadata server) and data (onto a storage
device) for a file. The flexible file layout type is defined in this device) for a file. The flexible file layout type is defined in this
document as an extension to pNFS which allows the use of storage document as an extension to pNFS which allows the use of storage
devices in a fashion such that they require only a quite limited devices in a fashion such that they require only a quite limited
degree of interaction with the metadata server, using already degree of interaction with the metadata server, using already
existing protocols. Client-side mirroring is also added to provide existing protocols. Client-side mirroring is also added to provide
skipping to change at page 1, line 38 skipping to change at page 1, line 38
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 31, 2018. This Internet-Draft will expire on October 27, 2018.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 25 skipping to change at page 2, line 25
2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 7 2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 7
2.2. Fencing Clients from the Storage Device . . . . . . . . . 7 2.2. Fencing Clients from the Storage Device . . . . . . . . . 7
2.2.1. Implementation Notes for Synthetic uids/gids . . . . 8 2.2.1. Implementation Notes for Synthetic uids/gids . . . . 8
2.2.2. Example of using Synthetic uids/gids . . . . . . . . 9 2.2.2. Example of using Synthetic uids/gids . . . . . . . . 9
2.3. State and Locking Models . . . . . . . . . . . . . . . . 10 2.3. State and Locking Models . . . . . . . . . . . . . . . . 10
2.3.1. Loosely Coupled Locking Model . . . . . . . . . . . . 10 2.3.1. Loosely Coupled Locking Model . . . . . . . . . . . . 10
2.3.2. Tightly Coupled Locking Model . . . . . . . . . . . . 12 2.3.2. Tightly Coupled Locking Model . . . . . . . . . . . . 12
3. XDR Description of the Flexible File Layout Type . . . . . . 13 3. XDR Description of the Flexible File Layout Type . . . . . . 13
3.1. Code Components Licensing Notice . . . . . . . . . . . . 14 3.1. Code Components Licensing Notice . . . . . . . . . . . . 14
4. Device Addressing and Discovery . . . . . . . . . . . . . . . 15 4. Device Addressing and Discovery . . . . . . . . . . . . . . . 15
4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 15 4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 16
4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 17 4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 17
5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 18 5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 18
5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 19 5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 19
5.1.1. Error Codes from LAYOUTGET . . . . . . . . . . . . . 22 5.1.1. Error Codes from LAYOUTGET . . . . . . . . . . . . . 22
5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS . . 23 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS . . 23
5.2. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 23 5.2. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 23
5.3. Interactions Between Devices and Layouts . . . . . . . . 23 5.3. Interactions Between Devices and Layouts . . . . . . . . 23
5.4. Handling Version Errors . . . . . . . . . . . . . . . . . 23 5.4. Handling Version Errors . . . . . . . . . . . . . . . . . 23
6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 24 6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 24
7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 24 7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 24
8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 25 8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 25
8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 26 8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 26
8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 26 8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 26
8.2.1. Single Storage Device Updates Mirrors . . . . . . . . 26 8.2.1. Single Storage Device Updates Mirrors . . . . . . . . 26
8.2.2. Client Updates All Mirrors . . . . . . . . . . . . . 26 8.2.2. Client Updates All Mirrors . . . . . . . . . . . . . 27
8.2.3. Handling Write Errors . . . . . . . . . . . . . . . . 27 8.2.3. Handling Write Errors . . . . . . . . . . . . . . . . 27
8.2.4. Handling Write COMMITs . . . . . . . . . . . . . . . 27 8.2.4. Handling Write COMMITs . . . . . . . . . . . . . . . 28
8.3. Metadata Server Resilvering of the File . . . . . . . . . 28 8.3. Metadata Server Resilvering of the File . . . . . . . . . 28
9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 28 9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 28
9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 29 9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 30
9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 29 9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 30
9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 30 9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 30
9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 30 9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 31
9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 31 9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 32
9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 32 9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 32
9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 33 9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 33
10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 34 10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 34
11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 34 11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 34
12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 34 12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 34
12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 35 12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 35
13. Recalling a Layout . . . . . . . . . . . . . . . . . . . . . 35 13. Recalling a Layout . . . . . . . . . . . . . . . . . . . . . 35
13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 35 13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 36
14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 36 14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 37
15. Security Considerations . . . . . . . . . . . . . . . . . . . 37 15. Security Considerations . . . . . . . . . . . . . . . . . . . 37
15.1. RPCSEC_GSS and Security Services . . . . . . . . . . . . 38 15.1. RPCSEC_GSS and Security Services . . . . . . . . . . . . 38
15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 38 15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 38
15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 39 15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 39
16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 39 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 39
17. References . . . . . . . . . . . . . . . . . . . . . . . . . 40 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 39
17.1. Normative References . . . . . . . . . . . . . . . . . . 40 17.1. Normative References . . . . . . . . . . . . . . . . . . 39
17.2. Informative References . . . . . . . . . . . . . . . . . 41 17.2. Informative References . . . . . . . . . . . . . . . . . 41
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 41 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 41
Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 42 Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 41
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 42 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 42
1. Introduction 1. Introduction
In the parallel Network File System (pNFS), the metadata server In the parallel Network File System (pNFS), the metadata server
returns layout type structures that describe where file data is returns layout type structures that describe where file data is
located. There are different layout types for different storage located. There are different layout types for different storage
systems and methods of arranging data on storage devices. This systems and methods of arranging data on storage devices. This
document defines the flexible file layout type used with file-based document defines the flexible file layout type used with file-based
data servers that are accessed using the Network File System (NFS) data servers that are accessed using the Network File System (NFS)
skipping to change at page 4, line 12 skipping to change at page 4, line 12
presents the available mirrors to the client. It is then the client presents the available mirrors to the client. It is then the client
which picks a mirror to read from and ensures that all writes go to which picks a mirror to read from and ensures that all writes go to
all mirrors. Only if all mirrors are successfully updated, does the all mirrors. Only if all mirrors are successfully updated, does the
client consider the write transaction to have succeeded. In case of client consider the write transaction to have succeeded. In case of
error, the client can use the LAYOUTERROR operation to inform the error, the client can use the LAYOUTERROR operation to inform the
metadata server, which is then responsible for the repairing of the metadata server, which is then responsible for the repairing of the
mirrored copies of the file. mirrored copies of the file.
1.1. Definitions 1.1. Definitions
control communication requirements: are for a layout type the control communication requirements: is the specification for
details regarding information on layouts, stateids, file metadata, information on layouts, stateids, file metadata, and file data
and file data which must be communicated between the metadata which must be communicated between the metadata server and the
server and the storage devices. storage devices. There is a separate set of requirements for each
layout type.
control protocol: is the particular mechanism that an implementation control protocol: is the particular mechanism that an implementation
of a layout type would use to meet the control communication of a layout type would use to meet the control communication
requirement for that layout type. This need not be a protocol as requirement for that layout type. This need not be a protocol as
normally understood. In some cases the same protocol may be used normally understood. In some cases the same protocol may be used
as a control protocol and storage protocol. as a control protocol and storage protocol.
client-side mirroring: is a feature in which the client and not the client-side mirroring: is a feature in which the client and not the
server is responsible for updating all of the mirrored copies of a server is responsible for updating all of the mirrored copies of a
layout segment. layout segment.
skipping to change at page 6, line 29 skipping to change at page 6, line 32
metadata server, or one based on a standards-track document. metadata server, or one based on a standards-track document.
uid: is the used id, a numeric value which identifies which user uid: is the used id, a numeric value which identifies which user
owns a file. owns a file.
wsize: is the data transfer buffer size used for writes. wsize: is the data transfer buffer size used for writes.
1.2. Requirements Language 1.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
document are to be interpreted as described in [RFC2119]. "OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
2. Coupling of Storage Devices 2. Coupling of Storage Devices
A server implementation may choose either a loose or tight coupling A server implementation may choose either a loose or tight coupling
model between the metadata server and the storage devices. model between the metadata server and the storage devices.
[pNFSLayouts] describes the general problems facing pNFS [pNFSLayouts] describes the general problems facing pNFS
implementations. This document details how the new Flexible File implementations. This document details how the new Flexible File
Layout Type addresses these issues. To implement the tight coupling Layout Type addresses these issues. To implement the tight coupling
model, a control protocol has to be defined. As the flex file layout model, a control protocol has to be defined. As the flex file layout
imposes no special requirements on the client, the control protocol imposes no special requirements on the client, the control protocol
skipping to change at page 37, line 41 skipping to change at page 37, line 48
requested iomode. If the LAYOUTGET operation succeeds the client requested iomode. If the LAYOUTGET operation succeeds the client
receives, as part of the layout, a set of credentials allowing it I/O receives, as part of the layout, a set of credentials allowing it I/O
access to the specified data files corresponding to the requested access to the specified data files corresponding to the requested
iomode. When the client acts on I/O operations on behalf of its iomode. When the client acts on I/O operations on behalf of its
local users, it MUST authenticate and authorize the user by issuing local users, it MUST authenticate and authorize the user by issuing
respective OPEN and ACCESS calls to the metadata server, similar to respective OPEN and ACCESS calls to the metadata server, similar to
having NFSv4 data delegations. having NFSv4 data delegations.
The combination of file handle, synthetic uid, and gid in the layout The combination of file handle, synthetic uid, and gid in the layout
are the way that the metadata server enforces access control to the are the way that the metadata server enforces access control to the
data server. The directory namespace on the storage device SHOULD data server. The client only has access to file handles of file
only be accessible to the metadata server and not the clients. In objects and not directory objects. Thus, given a file handle in a
that case, the client only has access to file handles of file objects layout, it is not possible to guess the parent directory file handle.
and not directory objects. Thus, given a file handle in a layout, it Further, as the data file permissions only allow the given synthetic
is not possible to guess the parent directory file handle. Further, uid read/write permission and the given synthetic gid read
as the data file permissions only allow the given synthetic uid read/ permission, knowing the synthetic ids of one file does not
write permission and the given synthetic gid read permission, knowing necessarily allow access to any other data file on the storage
the synthetic ids of one file does not necessarily allow access to device.
any other data file on the storage device.
The metadata server can also deny access at any time by fencing the The metadata server can also deny access at any time by fencing the
data file, which means changing the synthetic ids. In turn, that data file, which means changing the synthetic ids. In turn, that
forces the client to return its current layout and get a new layout forces the client to return its current layout and get a new layout
if it wants to continue IO to the data file. if it wants to continue IO to the data file.
If the configuration of the storage device is such that clients can
access the directory namespace, then the access control degrades to
that of a typical NFS server with exports with a security flavor of
AUTH_SYS. Any client which is allowed access can forge credentials
to access any data file. The caveat is that the rogue client might
have no knowledge of the data file's type or position in the metadata
directory namespace.
If access is allowed, the client uses the corresponding (READ or RW) If access is allowed, the client uses the corresponding (READ or RW)
credentials to perform the I/O operations at the data file's storage credentials to perform the I/O operations at the data file's storage
devices. When the metadata server receives a request to change a devices. When the metadata server receives a request to change a
file's permissions or ACL, it SHOULD recall all layouts for that file file's permissions or ACL, it SHOULD recall all layouts for that file
and then MUST fence off any clients still holding outstanding layouts and then MUST fence off any clients still holding outstanding layouts
for the respective files by implicitly invalidating the previously for the respective files by implicitly invalidating the previously
distributed credential on all data file comprising the file in distributed credential on all data file comprising the file in
question. It is REQUIRED that this be done before committing to the question. It is REQUIRED that this be done before committing to the
new permissions and/or ACL. By requesting new layouts, the clients new permissions and/or ACL. By requesting new layouts, the clients
will reauthorize access against the modified access control metadata. will reauthorize access against the modified access control metadata.
skipping to change at page 39, line 48 skipping to change at page 39, line 46
+------------------------------+-------+----------+-----+-----------+ +------------------------------+-------+----------+-----+-----------+
| Recallable Object Type Name | Value | RFC | How | Minor | | Recallable Object Type Name | Value | RFC | How | Minor |
| | | | | Versions | | | | | | Versions |
+------------------------------+-------+----------+-----+-----------+ +------------------------------+-------+----------+-----+-----------+
| RCA4_TYPE_MASK_FF_LAYOUT_MIN | 16 | RFCTBD10 | L | 1 | | RCA4_TYPE_MASK_FF_LAYOUT_MIN | 16 | RFCTBD10 | L | 1 |
| RCA4_TYPE_MASK_FF_LAYOUT_MAX | 17 | RFCTBD10 | L | 1 | | RCA4_TYPE_MASK_FF_LAYOUT_MAX | 17 | RFCTBD10 | L | 1 |
+------------------------------+-------+----------+-----+-----------+ +------------------------------+-------+----------+-----+-----------+
Table 2: Recallable Object Type Assignments Table 2: Recallable Object Type Assignments
Note, [RFC5661] should have also defined (see Table 3):
+-------------------------------+------+-----------+-----+----------+
| Recallable Object Type Name | Valu | RFC | How | Minor |
| | e | | | Versions |
+-------------------------------+------+-----------+-----+----------+
| RCA4_TYPE_MASK_OTHER_LAYOUT_M | 12 | [RFC5661] | L | 1 |
| IN | | | | |
| RCA4_TYPE_MASK_OTHER_LAYOUT_M | 15 | [RFC5661] | L | 1 |
| AX | | | | |
+-------------------------------+------+-----------+-----+----------+
Table 3: Recallable Object Type Assignments
17. References 17. References
17.1. Normative References 17.1. Normative References
[LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents", [LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents",
November 2008, <http://trustee.ietf.org/docs/ November 2008, <http://trustee.ietf.org/docs/
IETF-Trust-License-Policy.pdf>. IETF-Trust-License-Policy.pdf>.
[RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813, [RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813,
June 1995. June 1995.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/
RFC2119, March 1997, <https://www.rfc-editor.org/info/
rfc2119>.
[RFC4121] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos [RFC4121] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos
Version 5 Generic Security Service Application Program Version 5 Generic Security Service Application Program
Interface (GSS-API) Mechanism Version 2", RFC 4121, July Interface (GSS-API) Mechanism Version 2", RFC 4121, July
2005. 2005.
[RFC4506] Eisler, M., "XDR: External Data Representation Standard", [RFC4506] Eisler, M., "XDR: External Data Representation Standard",
STD 67, RFC 4506, May 2006. STD 67, RFC 4506, May 2006.
[RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol
skipping to change at page 41, line 11 skipping to change at page 40, line 42
[RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS)
version 4 Protocol", RFC 7530, March 2015. version 4 Protocol", RFC 7530, March 2015.
[RFC7861] Adamson, W. and N. Williams, "Remote Procedure Call (RPC) [RFC7861] Adamson, W. and N. Williams, "Remote Procedure Call (RPC)
Security Version 3", November 2016. Security Version 3", November 2016.
[RFC7862] Haynes, T., "NFS Version 4 Minor Version 2", RFC 7862, [RFC7862] Haynes, T., "NFS Version 4 Minor Version 2", RFC 7862,
November 2016. November 2016.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[pNFSLayouts] [pNFSLayouts]
Haynes, T., "Requirements for pNFS Layout Types", draft- Haynes, T., "Requirements for pNFS Layout Types", draft-
ietf-nfsv4-layout-types-07 (Work In Progress), August ietf-nfsv4-layout-types-07 (Work In Progress), August
2017. 2017.
17.2. Informative References 17.2. Informative References
[RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol [RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol
(LDAP): Schema for User Applications", RFC 4519, DOI (LDAP): Schema for User Applications", RFC 4519, DOI
10.17487/RFC4519, June 2006, 10.17487/RFC4519, June 2006,
skipping to change at page 42, line 21 skipping to change at page 42, line 12
replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the
RFC number of this document] RFC number of this document]
Authors' Addresses Authors' Addresses
Benny Halevy Benny Halevy
Email: bhalevy@gmail.com Email: bhalevy@gmail.com
Thomas Haynes Thomas Haynes
Primary Data, Inc. Hammerspace
4300 El Camino Real Ste 100 4300 El Camino Real Ste 105
Los Altos, CA 94022 Los Altos, CA 94022
USA USA
Email: loghyr@gmail.com Email: loghyr@gmail.com
 End of changes. 19 change blocks. 
56 lines changed or deleted 42 lines changed or added

This html diff was produced by rfcdiff 1.46. The latest version is available from http://tools.ietf.org/tools/rfcdiff/