draft-ietf-nfsv4-nfs-rdma-problem-statement-07.txt   draft-ietf-nfsv4-nfs-rdma-problem-statement-08.txt 
NFSv4 Working Group Tom Talpey NFSv4 Working Group Tom Talpey
Internet-Draft Network Appliance, Inc. Internet-Draft NetApp
Intended status: Informational Chet Juszczak Intended status: Informational Chet Juszczak
Expires: January 1, 2008 July 1, 2007 Expires: August 23, 2008 February 21, 2008
NFS RDMA Problem Statement NFS RDMA Problem Statement
draft-ietf-nfsv4-nfs-rdma-problem-statement-07 draft-ietf-nfsv4-nfs-rdma-problem-statement-08
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 35 skipping to change at page 1, line 35
documents at any time. It is inappropriate to use Internet-Drafts documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in as reference material or to cite them other than as "work in
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on January 1, 2008. This Internet-Draft will expire on August 23, 2008.
Copyright Notice Copyright Notice
Copyright (C) The IETF Trust (2007). Copyright (C) The IETF Trust (2008).
Abstract Abstract
This draft addresses applying Remote Direct Memory Access to the This draft addresses enabling the use of Remote Direct Memory
NFS protocols. NFS implementations historically incur significant Access (RDMA) by the Network File System (NFS) protocols. NFS
overhead due to data copies on end-host systems, as well as other implementations historically incur significant overhead due to data
processing overhead. The potential benefits of RDMA to these copies on end-host systems, as well as other processing overhead.
implementations are explored, and the reasons why RDMA is The potential benefits of RDMA to these implementations are
especially well-suited to NFS and network file protocols in general explored, and the reasons why RDMA is especially well-suited to NFS
are evaluated. and network file protocols in general are evaluated.
Table Of Contents Table Of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2
2. Problem Statement . . . . . . . . . . . . . . . . . . . . 5 2. Problem Statement . . . . . . . . . . . . . . . . . . . . 5
3. File Protocol Architecture . . . . . . . . . . . . . . . . 6 3. File Protocol Architecture . . . . . . . . . . . . . . . . 6
4. Sources of Overhead . . . . . . . . . . . . . . . . . . . 8 4. Sources of Overhead . . . . . . . . . . . . . . . . . . . 8
4.1. Savings from TOE . . . . . . . . . . . . . . . . . . . . 9 4.1. Savings from TOE . . . . . . . . . . . . . . . . . . . . 9
4.2. Savings from RDMA . . . . . . . . . . . . . . . . . . . 10 4.2. Savings from RDMA . . . . . . . . . . . . . . . . . . . 10
5. Application of RDMA to NFS . . . . . . . . . . . . . . . . 10 5. Application of RDMA to NFS . . . . . . . . . . . . . . . . 10
6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 11 6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 11
Security Considerations . . . . . . . . . . . . . . . . . 12 Security Considerations . . . . . . . . . . . . . . . . . 12
IANA Considerations . . . . . . . . . . . . . . . . . . . 12 IANA Considerations . . . . . . . . . . . . . . . . . . . 13
Acknowledgements . . . . . . . . . . . . . . . . . . . . . 12 Acknowledgements . . . . . . . . . . . . . . . . . . . . . 13
Normative References . . . . . . . . . . . . . . . . . . . 12 Normative References . . . . . . . . . . . . . . . . . . . 13
Informative References . . . . . . . . . . . . . . . . . . 12 Informative References . . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . 15 Authors' Addresses . . . . . . . . . . . . . . . . . . . . 16
Intellectual Property and Copyright Statements . . . . . . 15 Intellectual Property and Copyright Statements . . . . . . 16
Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . 16 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . 17
1. Introduction 1. Introduction
The Network File System (NFS) protocol (as described in [RFC1094], The Network File System (NFS) protocol (as described in [RFC1094],
[RFC1813], and [RFC3530]) is one of several remote file access [RFC1813], and [RFC3530]) is one of several remote file access
protocols used in the class of processing architecture sometimes protocols used in the class of processing architecture sometimes
called Network Attached Storage (NAS). called Network Attached Storage (NAS).
Historically, remote file access has proven to be a convenient, Historically, remote file access has proven to be a convenient,
cost-effective way to share information over a network, a concept cost-effective way to share information over a network, a concept
skipping to change at page 10, line 13 skipping to change at page 10, line 13
enough to drive the full network bandwidth without RDMA. It can enough to drive the full network bandwidth without RDMA. It can
also be shown that the speedup may be higher if network bandwidth also be shown that the speedup may be higher if network bandwidth
grows faster than Moore's Law, although the higher benefits will grows faster than Moore's Law, although the higher benefits will
apply to a narrow range of applications. apply to a narrow range of applications.
4.2. Savings from RDMA 4.2. Savings from RDMA
Performance measurements directly comparing an NFS over RDMA Performance measurements directly comparing an NFS over RDMA
prototype with conventional network-based NFS processing are prototype with conventional network-based NFS processing are
described in [CAL+03]. Comparisons of Read throughput and CPU described in [CAL+03]. Comparisons of Read throughput and CPU
overhead were performed on two Gigabit Ethernet adapters, one overhead were performed on two types of Gigabit Ethernet adapters,
conventional and one with RDMA capability. The prototype RDMA one type being a conventional adapter, and another type with RDMA
protocol performed all transfers via RDMA Read. capability. The prototype RDMA protocol performed all transfers
via RDMA Read. The NFS layer in the study was measured while
performing read transfers, varying the transfer size and readahead
depth across ranges used by typical NFS deployments.
In these results, conventional network-based throughput was In these results, conventional network-based throughput was
severely limited by the client's CPU being saturated at 100% for severely limited by the client's CPU being saturated at 100% for
all transfers. Read throughput reached no more than 60MBytes/s. all transfers. Read throughput reached no more than 60MBytes/s.
I/O Type Size Read Throughput CPU Utilization I/O Type Size Read Throughput CPU Utilization
Conventional 2KB 20MB/s 100% Conventional 2KB 20MB/s 100%
Conventional 16KB 40MB/s 100% Conventional 16KB 40MB/s 100%
Conventional 256KB 60MB/s 100% Conventional 256KB 60MB/s 100%
skipping to change at page 12, line 18 skipping to change at page 12, line 23
7. Security Considerations 7. Security Considerations
The NFS protocol, in conjunction with its layering on RPC, provides The NFS protocol, in conjunction with its layering on RPC, provides
a rich and widely interoperable security model to applications and a rich and widely interoperable security model to applications and
systems. Any layering of NFS over RDMA transports must address the systems. Any layering of NFS over RDMA transports must address the
NFS security requirements, and additionally must ensure that no new NFS security requirements, and additionally must ensure that no new
vulnerabilities are introduced. For RDMA, the integrity, and any vulnerabilities are introduced. For RDMA, the integrity, and any
privacy, of the data stream are of particular importance. privacy, of the data stream are of particular importance.
Security Considerations must be addressed by any relevant RDMA The core goals of an NFS-to-RDMA binding are to reduce overhead and
transport layering. The protocol described in [RPCRDMA] provides to enable high performance. To support these goals while
one such approach. maintaining required NFS security protection presents a special
challenge. Historically, the provision of integrity and privacy
have been implemented within the RPC layer, and their operation
requires local processing of messages exchanged with the RPC peer.
This procesing imposes memory and processing overhead on a per-
message basis, exactly the overhead that RDMA is designed to avoid.
Therefore, it is a requirement that the RDMA transport binding
provide a means to delegate the integrity and privacy processing to
the RDMA hardware, in order to maintain the high level of
performance desired from the approach, while simultaneously
providing the existing highest levels of security required by the
NFS protocol. This in turn requires a means by which the RPC layer
may invoke these services from the RDMA provider, and for the NFS
layer to negotiate their use end-to-end.
The "Channel Binding" concept [RFC5056] provides a means by which
the RPC and NFS layers may delegate their session protection to the
lower RDMA layers. An extension to the RPCSEC_GSS protocol
[RPCSECGSSV2] may then be specified to negotiate the use of these
bindings, and to establish the shared secrets necessary to protect
the sessions.
The protocol described in [RPCRDMA] specifies the use of these
mechanisms, and they are required to implement the protocol.
An additional consideration is protection of the integrity and
privacy of local memory by the RDMA transport itself. The use of
RDMA by NFS must not introduce any vulnerabilities to system memory
contents, or to memory owned by user processes. These protections
are provided by the RDMA layer specifications, and specifically
their security models. It is required that any RDMA provider used
for NFS transport be conformant to the requirements of [RFC5042] in
order to satisfy these protections.
8. IANA Considerations 8. IANA Considerations
This document has no IANA considerations. This document has no IANA considerations.
9. Acknowledgements 9. Acknowledgements
The authors wish to thank Jeff Chase who provided many useful The authors wish to thank Jeff Chase who provided many useful
suggestions. suggestions.
skipping to change at page 13, line 5 skipping to change at page 13, line 38
Specification Version 2", Standards Track RFC Specification Version 2", Standards Track RFC
[RFC4506] [RFC4506]
M. Eisler, Ed. "XDR: External Data Representation Standard", M. Eisler, Ed. "XDR: External Data Representation Standard",
Standards Track RFC Standards Track RFC
[RFC1813] [RFC1813]
B. Callaghan, B. Pawlowski, P. Staubach, "NFS Version 3 B. Callaghan, B. Pawlowski, P. Staubach, "NFS Version 3
Protocol Specification", Informational RFC Protocol Specification", Informational RFC
[RPCSECGSSV2]
M. Eisler, "RPCSEC_GSS Version 2", Internet Draft Work In
Progress, draft-ietf-nfsv4-rpcsec-gss-v2
[RFC5056]
N. Williams, "On the Use of Channel Bindings to Secure
Channels", Standards Track RFC
[RFC5042]
J. Pinkerton, E. Deleganes, "Direct Data Placement Protocol
(DDP) / Remote Direct Memory Access Protocol (RDMAP) Security"
Standards Track RFC
11. Informative References 11. Informative References
[BRU99] [BRU99]
J. Brustoloni, "Interoperation of copy avoidance in network J. Brustoloni, "Interoperation of copy avoidance in network
and file I/O", in Proc. INFOCOM '99, pages 534-542, New York, and file I/O", in Proc. INFOCOM '99, pages 534-542, New York,
NY, Mar. 1999., IEEE. Also available from NY, Mar. 1999., IEEE. Also available from
http://www.cs.pitt.edu/~jcb/publs.html http://www.cs.pitt.edu/~jcb/publs.html
[CAL+03] [CAL+03]
B. Callaghan, T. Lingutla-Raj, A. Chiu, P. Staubach, O. Asad, B. Callaghan, T. Lingutla-Raj, A. Chiu, P. Staubach, O. Asad,
skipping to change at page 15, line 4 skipping to change at page 16, line 4
[RFC1094] [RFC1094]
Sun Microsystems, "NFS: Network File System Protocol Sun Microsystems, "NFS: Network File System Protocol
Specification" Specification"
[RPCRDMA] [RPCRDMA]
T. Talpey, B. Callaghan, "RDMA Transport for ONC RPC", T. Talpey, B. Callaghan, "RDMA Transport for ONC RPC",
Internet Draft Work in Progress, draft-ietf-nfsv4-rpcrdma Internet Draft Work in Progress, draft-ietf-nfsv4-rpcrdma
[SHI+03] [SHI+03]
P. Shivam, J. Chase, "On the Elusive Benefits of Protocol P. Shivam, J. Chase, "On the Elusive Benefits of Protocol
Offload", to be published in Proceedings of ACM SIGCOMM Summer Offload", Proceedings of ACM SIGCOMM Summer 2003 NICELI
2003 NICELI Workshop, also available from Workshop, also available from
http://issg.cs.duke.edu/publications/niceli03.pdf http://issg.cs.duke.edu/publications/niceli03.pdf
[SKE+01] [SKE+01]
K.-A. Skevik, T. Plagemann, V. Goebel, P. Halvorsen, K.-A. Skevik, T. Plagemann, V. Goebel, P. Halvorsen,
"Evaluation of a Zero-Copy Protocol Implementation", in "Evaluation of a Zero-Copy Protocol Implementation", in
Proceedings of the 27th Euromicro Conference - Multimedia and Proceedings of the 27th Euromicro Conference - Multimedia and
Telecommunications Track (MTT'2001), Warsaw, Poland, September Telecommunications Track (MTT'2001), Warsaw, Poland, September
2001. 2001.
Authors' Addresses Authors' Addresses
Tom Talpey Tom Talpey
Network Appliance, Inc. Network Appliance, Inc.
375 Totten Pond Road 1601 Trapelo Road, #16
Waltham, MA 02451 USA Waltham, MA 02451 USA
Phone: +1 781 768 5329 Phone: +1 781 768 5329
Email: thomas.talpey@netapp.com Email: thomas.talpey@netapp.com
Chet Juszczak Chet Juszczak
Chet's Boathouse Co. Chet's Boathouse Co.
P.O. Box 1467 P.O. Box 1467
Merrimack, NH 03054 Merrimack, NH 03054
Email: chetnh@earthlink.net Email: chetnh@earthlink.net
Intellectual Property and Copyright Statements Intellectual Property and Copyright Statements
Full Copyright Statement Full Copyright Statement
Copyright (C) The IETF Trust (2007). Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided on This document and the information contained herein are provided on
an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE
IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL
WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY
WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE
 End of changes. 13 change blocks. 
29 lines changed or deleted 78 lines changed or added

This html diff was produced by rfcdiff 1.34. The latest version is available from http://tools.ietf.org/tools/rfcdiff/