draft-ietf-nfsv4-nfs-rdma-problem-statement-03.txt   draft-ietf-nfsv4-nfs-rdma-problem-statement-04.txt 
INTERNET-DRAFT Tom Talpey INTERNET-DRAFT Tom Talpey
Expires: April 2006 Chet Juszczak Expires: December 2006 Chet Juszczak
October, 2005 June, 2006
NFS RDMA Problem Statement NFS RDMA Problem Statement
draft-ietf-nfsv4-nfs-rdma-problem-statement-03 draft-ietf-nfsv4-nfs-rdma-problem-statement-04
Status of this Memo Status of this Memo
By submitting this Internet-Draft, I certify that any applicable By submitting this Internet-Draft, each author represents that any
patent or other IPR claims of which I am aware have been disclosed, applicable patent or other IPR claims of which he or she is aware
or will be disclosed, and any of which I become aware will be have been or will be disclosed, and any of which he or she becomes
disclosed, in accordance with RFC 3668. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in as reference material or to cite them other than as "work in
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt The list of http://www.ietf.org/ietf/1id-abstracts.txt The list of
Internet-Draft Shadow Directories can be accessed at Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (2005). All Rights Reserved.
Abstract Abstract
This draft addresses applying Remote Direct Memory Access to the This draft addresses applying Remote Direct Memory Access to the
NFS protocols. NFS implementations historically incur significant NFS protocols. NFS implementations historically incur significant
overhead due to data copies on end-host systems, as well as other overhead due to data copies on end-host systems, as well as other
sources. The potential benefits of RDMA to these implementations sources. The potential benefits of RDMA to these implementations
are explored, and the reasons why RDMA is especially well-suited to are explored, and the reasons why RDMA is especially well-suited to
NFS and network file protocols in general are evaluated. NFS and network file protocols in general are evaluated.
Table Of Contents Table Of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2
2. Problem Statement . . . . . . . . . . . . . . . . . . . . 4 2. Problem Statement . . . . . . . . . . . . . . . . . . . . 4
3. File Protocol Architecture . . . . . . . . . . . . . . . . 5 3. File Protocol Architecture . . . . . . . . . . . . . . . . 5
4. Sources of Overhead . . . . . . . . . . . . . . . . . . . 7 4. Sources of Overhead . . . . . . . . . . . . . . . . . . . 7
4.1. Savings from TOE . . . . . . . . . . . . . . . . . . . . 8 4.1. Savings from TOE . . . . . . . . . . . . . . . . . . . . 8
4.2. Savings from RDMA . . . . . . . . . . . . . . . . . . . 9 4.2. Savings from RDMA . . . . . . . . . . . . . . . . . . . 9
5. Application of RDMA to NFS . . . . . . . . . . . . . . . . 10 5. Application of RDMA to NFS . . . . . . . . . . . . . . . . 10
6. Improved Semantics . . . . . . . . . . . . . . . . . . . . 10 6. Improved Semantics . . . . . . . . . . . . . . . . . . . . 10
7. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 11 7. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 11
Acknowledgements . . . . . . . . . . . . . . . . . . . . . 11 Security Considerations . . . . . . . . . . . . . . . . . 12
IANA Considerations . . . . . . . . . . . . . . . . . . . 12
Acknowledgements . . . . . . . . . . . . . . . . . . . . . 12
Normative References . . . . . . . . . . . . . . . . . . . 12 Normative References . . . . . . . . . . . . . . . . . . . 12
Informative References . . . . . . . . . . . . . . . . . . 12 Informative References . . . . . . . . . . . . . . . . . . 12
Authors' Addresses . . . . . . . . . . . . . . . . . . . . 14 Authors' Addresses . . . . . . . . . . . . . . . . . . . . 14
Full Copyright Statement . . . . . . . . . . . . . . . . . 14 Intellectual Property and Copyright Statements . . . . . . 15
1. Introduction 1. Introduction
The Network File System (NFS) protocol (as described in [RFC1094], The Network File System (NFS) protocol (as described in [RFC1094],
[RFC1813], and [RFC3530]) is one of several remote file access [RFC1813], and [RFC3530]) is one of several remote file access
protocols used in the class of processing architecture sometimes protocols used in the class of processing architecture sometimes
called Network Attached Storage (NAS). called Network Attached Storage (NAS).
Historically, remote file access has proved to be a convenient, Historically, remote file access has proved to be a convenient,
cost-effective way to share information over a network, a concept cost-effective way to share information over a network, a concept
skipping to change at page 3, line 13 skipping to change at page 3, line 7
implementations. implementations.
Replication of local file access performance on NAS using Replication of local file access performance on NAS using
traditional network protocol stacks has proven difficult, not traditional network protocol stacks has proven difficult, not
because of protocol processing overheads, but because of data copy because of protocol processing overheads, but because of data copy
costs in the network endpoints. This is especially true since host costs in the network endpoints. This is especially true since host
buses are now often the main bottleneck in NAS architectures buses are now often the main bottleneck in NAS architectures
[MOG03] [CHA+01]. [MOG03] [CHA+01].
The External Data Representation [RFC1832] employed beneath NFS and The External Data Representation [RFC1832] employed beneath NFS and
RPC [RPC1831] can add more data copies, exacerbating the problem. RPC [RFC1831] can add more data copies, exacerbating the problem.
Data copy-avoidance designs have not been widely adopted for a Data copy-avoidance designs have not been widely adopted for a
variety of reasons. [BRU99] points out that "many copy avoidance variety of reasons. [BRU99] points out that "many copy avoidance
techniques for network I/O are not applicable or may even backfire techniques for network I/O are not applicable or may even backfire
if applied to file I/O." Other designs that eliminate unnecessary if applied to file I/O." Other designs that eliminate unnecessary
copies, such as [PAI+00], are incompatible with existing APIs and copies, such as [PAI+00], are incompatible with existing APIs and
therefore force application changes. therefore force application changes.
Over the past year, an effort to standardize a set of protocols for In recent years, an effort to standardize a set of protocols for
Remote Direct Memory Access, RDMA, over the standard Internet Remote Direct Memory Access, RDMA, over the standard Internet
Protocol Suite has been chartered [RDDP]. Several drafts have been Protocol Suite has been chartered [RDDP]. Several drafts have been
proposed and are under discussion. proposed and are being considered for Standards Track.
RDMA is a general solution to the problem of CPU overhead incurred RDMA is a general solution to the problem of CPU overhead incurred
due to data copies, primarily at the receiver. Substantial due to data copies, primarily at the receiver. Substantial
research has addressed this and has borne out the efficacy of the research has addressed this and has borne out the efficacy of the
approach. An overview of this is the RDDP Problem Statement approach. An overview of this is the RDDP Problem Statement
document, [RDDPPS]. document, [RFC4297].
In addition to the per-byte savings of off-loading data copies, In addition to the per-byte savings of off-loading data copies,
RDMA-enabled NICs (RNICS) offload the underlying protocol layers as RDMA-enabled NICs (RNICS) offload the underlying protocol layers as
well, e.g. TCP, further reducing CPU overhead due to NAS well, e.g. TCP, further reducing CPU overhead due to NAS
processing. processing.
1.1. Background 1.1. Background
The RDDP Problem Statement [RDDPPS] asserts: The RDDP Problem Statement [RFC4297] asserts:
"High costs associated with copying are an issue primarily for "High costs associated with copying are an issue primarily for
large scale systems ... with high bandwidth feeds, usually large scale systems ... with high bandwidth feeds, usually
multiprocessors and clusters, that are adversely affected by multiprocessors and clusters, that are adversely affected by
copying overhead. Examples of such machines include all copying overhead. Examples of such machines include all
varieties of servers: database servers, storage servers, varieties of servers: database servers, storage servers,
application servers for transaction processing, for e- application servers for transaction processing, for e-
commerce, and web serving, content distribution, video commerce, and web serving, content distribution, video
distribution, backups, data mining and decision support, and distribution, backups, data mining and decision support, and
scientific computing. Note that such servers almost scientific computing.
exclusively service many concurrent sessions (transport
connections), which, in aggregate, are responsible for > 1
Gbits/s of communication. Nonetheless, the cost of copying
overhead for a particular load is the same whether from few or
many sessions."
Note that such servers almost exclusively service many
concurrent sessions (transport connections), which, in
aggregate, are responsible for > 1 Gbits/s of communication.
Nonetheless, the cost of copying overhead for a particular
load is the same whether from few or many sessions."
Note that each of the servers listed above could be accessing their Note that each of the servers listed above could be accessing their
file data as an NFS client, or NFS serving the data to such file data as an NFS client, or NFS serving the data to such
clients, or acting as both. clients, or acting as both.
The CPU overhead of the NFS and TCP/IP protocol stacks (including The CPU overhead of the NFS and TCP/IP protocol stacks (including
data copies or reduced copy workarounds) becomes a significant data copies or reduced copy workarounds) becomes a significant
matter in these clients and servers. File access using locally matter in these clients and servers. File access using locally
attached disks imposes relatively low overhead due to the highly attached disks imposes relatively low overhead due to the highly
optimized I/O path and direct memory access afforded to the storage optimized I/O path and direct memory access afforded to the storage
controller. This is not the case with NFS, which must pass data controller. This is not the case with NFS, which must pass data
skipping to change at page 4, line 34 skipping to change at page 4, line 30
buffer caches, in XDR marshalling and unmarshalling, and within buffer caches, in XDR marshalling and unmarshalling, and within
network stacks and network drivers. Other overheads such as network stacks and network drivers. Other overheads such as
serialization among multiple threads of execution sharing a single serialization among multiple threads of execution sharing a single
NFS mount point and transport connection are additionally NFS mount point and transport connection are additionally
encountered. encountered.
Numerous upper layer protocols achieve extremely high bandwidth and Numerous upper layer protocols achieve extremely high bandwidth and
low overhead through the use of RDMA. [MAF+02] show that the RDMA- low overhead through the use of RDMA. [MAF+02] show that the RDMA-
based Direct Access File System (with a user-level implementation based Direct Access File System (with a user-level implementation
of the file system client) can outperform even a zero-copy of the file system client) can outperform even a zero-copy
implementation of NFS [CHA+01] [CHA+99] [GAL+99]. Also, file data implementation of NFS [CHA+01] [CHA+99] [GAL+99] [KM02]. Also,
access implies the use of large ULP messages. These large messages file data access implies the use of large ULP messages. These
tend to amortize any increase in per-message costs due to the large messages tend to amortize any increase in per-message costs
offload of protocol processing incurred when using RNICs while due to the offload of protocol processing incurred when using RNICs
gaining the benefits of reduced per-byte costs. Finally, the while gaining the benefits of reduced per-byte costs. Finally, the
direct memory addressing afforded by RDMA avoids many sources of direct memory addressing afforded by RDMA avoids many sources of
contention on network resources. contention on network resources.
2. Problem Statement 2. Problem Statement
The principal performance problem encountered by NFS The principal performance problem encountered by NFS
implementations is the CPU overhead required to implement the implementations is the CPU overhead required to implement the
protocol. Primary among the sources of this overhead is the protocol. Primary among the sources of this overhead is the
movement of data from NFS protocol messages to its eventual movement of data from NFS protocol messages to its eventual
destination in user buffers or aligned kernel buffers. Due to the destination in user buffers or aligned kernel buffers. Due to the
nature of the RPC and XDR protocols, the NFS data payload arrives nature of the RPC and XDR protocols, the NFS data payload arrives
at arbitrary alignment and the NFS requests are completed in an at arbitrary alignment and the NFS requests are completed in an
arbitrary sequence. arbitrary sequence.
The data copies consume system bus bandwidth and CPU time, reducing The data copies consume system bus bandwidth and CPU time, reducing
the available system capacity for applications [RDDPPS]. Achieving the available system capacity for applications [RFC4297].
zero-copy with NFS has, to date, required sophisticated, version- Achieving zero-copy with NFS has, to date, required sophisticated,
specific "header cracking" hardware and/or extensive platform- version-specific "header cracking" hardware and/or extensive
specific virtual memory mapping tricks. Such approaches become platform-specific virtual memory mapping tricks. Such approaches
even more difficult for NFS version 4 due to the existence of the become even more difficult for NFS version 4 due to the existence
COMPOUND operation, which further reduces alignment and greatly of the COMPOUND operation, which further reduces alignment and
complicates ULP offload. greatly complicates ULP offload.
Furthermore, NFS will soon be challenged by emerging high-speed Furthermore, NFS will soon be challenged by emerging high-speed
network fabrics such as 10 Gbits/s Ethernet. Performing even raw network fabrics such as 10 Gbits/s Ethernet. Performing even raw
network I/O such as TCP is an issue at such speeds with today's network I/O such as TCP is an issue at such speeds with today's
hardware. The problem is fundamental in nature and has led the hardware. The problem is fundamental in nature and has led the
IETF to explore RDMA [RDDPPS]. IETF to explore RDMA [RFC4297].
Zero-copy techniques benefit file protocols extensively, as they Zero-copy techniques benefit file protocols extensively, as they
enable direct user I/O, reduce the overhead of protocol stacks, enable direct user I/O, reduce the overhead of protocol stacks,
provide perfect alignment into caches, etc. Many studies have provide perfect alignment into caches, etc. Many studies have
already shown the performance benefits of such techniques [SKE+01, already shown the performance benefits of such techniques [SKE+01]
DCK+03, FJNFS, FJDAFS, MAF+02]. [DCK+03] [FJNFS] [FJDAFS] [KM02] [MAF+02].
RDMA implementations generally have other interesting properties, RDMA implementations generally have other interesting properties,
such as hardware assisted protocol access, and support for user such as hardware assisted protocol access, and support for user
space access to I/O. RDMA is compelling here for another reason; space access to I/O. RDMA is compelling here for another reason;
hardware offloaded networking support in itself does not avoid data hardware offloaded networking support in itself does not avoid data
copies, without resorting to implementing part of the NFS protocol copies, without resorting to implementing part of the NFS protocol
in the NIC. Support of RDMA by NFS enables the highest performance in the NIC. Support of RDMA by NFS enables the highest performance
at the architecture level rather than by implementation; this at the architecture level rather than by implementation; this
enables ubiquitous and interoperable solutions. enables ubiquitous and interoperable solutions.
skipping to change at page 8, line 20 skipping to change at page 8, line 17
With copies crossing the bus twice per copy, network processing With copies crossing the bus twice per copy, network processing
overhead is high whenever network bandwidth is large in comparison overhead is high whenever network bandwidth is large in comparison
to CPU and memory bandwidths. Generally with today's end-systems, to CPU and memory bandwidths. Generally with today's end-systems,
the effects are observable at network speeds at or above 1 Gbits/s. the effects are observable at network speeds at or above 1 Gbits/s.
A common question is whether increase in CPU processing power A common question is whether increase in CPU processing power
alleviates the problem of high processing costs of network I/O. alleviates the problem of high processing costs of network I/O.
The answer is no, it is the memory bandwidth that is the issue. The answer is no, it is the memory bandwidth that is the issue.
Faster CPUs do not help if the CPU spends most of its time waiting Faster CPUs do not help if the CPU spends most of its time waiting
for memory [RDDPPS]. for memory [RFC4297].
TCP offload engine (TOE) technology aims to offload the CPU by TCP offload engine (TOE) technology aims to offload the CPU by
moving TCP/IP protocol processing to the NIC. However, TOE moving TCP/IP protocol processing to the NIC. However, TOE
technology by itself does nothing to avoid necessary data copies technology by itself does nothing to avoid necessary data copies
within upper layer protocols. [MOG03] provides a description of within upper layer protocols. [MOG03] provides a description of
the role TOE can play in reducing per-packet and per-message costs. the role TOE can play in reducing per-packet and per-message costs.
Beyond the offloads commonly provided by today's network interface Beyond the offloads commonly provided by today's network interface
hardware, TOE alone (w/o RDMA) helps in protocol header processing, hardware, TOE alone (w/o RDMA) helps in protocol header processing,
but this has been shown to be a minority component of the total but this has been shown to be a minority component of the total
protocol processing overhead. [CHA+01] protocol processing overhead. [CHA+01]
skipping to change at page 10, line 43 skipping to change at page 10, line 43
part of the problem: data copies. RDMA also addresses other part of the problem: data copies. RDMA also addresses other
overheads, e.g. underlying protocol offload, and offers separation overheads, e.g. underlying protocol offload, and offers separation
of control information from data. of control information from data.
The current NFS message layout provides the performance enhancing The current NFS message layout provides the performance enhancing
opportunity for an NFS over RDMA protocol that separates the opportunity for an NFS over RDMA protocol that separates the
control information from data chunks while meeting the alignment control information from data chunks while meeting the alignment
needs of both. The data chunks can be copied "directly" between needs of both. The data chunks can be copied "directly" between
the client and server memory addresses above (with a single the client and server memory addresses above (with a single
occurrence on each memory bus) while the control information can be occurrence on each memory bus) while the control information can be
passed "inline". [ONCRDMA] describes such a protocol. passed "inline". [RPCRDMA] describes such a protocol.
6. Improved Semantics 6. Improved Semantics
Network file protocols need to export the application programming Network file protocols need to export the application programming
interfaces and semantics that applications, especially mission interfaces and semantics that applications, especially mission
critical ones like database and clusters, have been developed to critical ones like database and clusters, have been developed to
expect. These APIs and semantics are historical in nature and expect. These APIs and semantics are historical in nature and
successful deprecation is doubtful. NFS has not delivered all of successful deprecation is doubtful. NFS has not delivered all of
the semantics (for example, reliable filesystem transactions) for the semantics (for example, reliable filesystem transactions) for
the sake of acceptable performance. the sake of acceptable performance.
The advanced properties of RDMA-capable transports allow improved The advanced properties of RDMA-capable transports allow improved
semantics. [DAFS] is an example of a protocol which exports semantics. DAFS [DCK+03] is an example of a protocol which exports
semantics which are similar to those of NFSv4, but improved in semantics which are similar to those of NFSv4, but improved in
specific areas. Improved NFS semantics can also be delivered. As specific areas. Improved NFS semantics can also be delivered. As
an example, [RPCRDMA] describes an implementation of RPC for RDMA an example, [RPCRDMA] describes an implementation of RPC for RDMA
transport that is evolutionary in nature yet enables the provision transport that is evolutionary in nature yet enables the provision
of reliable and idempotent filesystem operation. This proposal of reliable and idempotent filesystem operation. This proposal
shows that it is possible to deliver extended semantics with an shows that it is possible to deliver extended semantics with an
RPC/XDR layer implementation with no changes required above the NFS RPC/XDR layer implementation with no changes required above the NFS
layer, and few within. layer, and few within.
7. Conclusions 7. Conclusions
NFS version 4 [RFC3530] has recently been granted "Proposed NFS version 4 [RFC3530] has been granted "Proposed Standard"
Standard" status. The NFSv4 protocol was developed along several status. The NFSv4 protocol was developed along several design
design points, important among them: effective operation over wide- points, important among them: effective operation over wide- area
area networks, including the Internet itself; strong security networks, including the Internet itself; strong security
integrated into the protocol; extensive cross-platform integrated into the protocol; extensive cross-platform
interoperability including integrated locking semantics compatible interoperability including integrated locking semantics compatible
with multiple operating systems; and (this is key), protocol with multiple operating systems; and (this is key), protocol
extension. extension.
NFS version 4 is an excellent base on which to add the needed NFS version 4 is an excellent base on which to add the needed
performance enhancements and improved semantics described above. performance enhancements and improved semantics described above.
The minor versioning support defined in NFS version 4 was designed The minor versioning support defined in NFS version 4 was designed
to support protocol improvements without disruption to the to support protocol improvements without disruption to the
installed base. Evolutionary improvement of the protocol via minor installed base. Evolutionary improvement of the protocol via minor
skipping to change at page 12, line 5 skipping to change at page 12, line 5
It is vital that the NFS protocol continue to provide these It is vital that the NFS protocol continue to provide these
benefits to a wide range of applications, without its usefulness benefits to a wide range of applications, without its usefulness
being compromised by concerns about performance and semantic being compromised by concerns about performance and semantic
inadequacies. This can reasonably be addressed in the existing NFS inadequacies. This can reasonably be addressed in the existing NFS
protocol framework. A cautious evolutionary improvement of protocol framework. A cautious evolutionary improvement of
performance and semantics allows building on the value already performance and semantics allows building on the value already
present in the NFS protocol, while addressing new requirements that present in the NFS protocol, while addressing new requirements that
have arisen from the application of networking technology. have arisen from the application of networking technology.
8. Acknowledgements 8. Security Considerations
Security Considerations are not covered by this document. Please
refer to the appropriate protocol documents for any security
issues.
9. IANA Considerations
IANA Considerations are not covered by this document. Please refer
to the appropriate protocol documents for any IANA issues.
10. Acknowledgements
The authors wish to thank Jeff Chase who provided many useful The authors wish to thank Jeff Chase who provided many useful
suggestions. suggestions.
9. Normative References 11. Normative References
[RFC3530] [RFC3530]
S. Shepler, et. al., "NFS Version 4 Protocol", Standards Track S. Shepler, et. al., "NFS Version 4 Protocol", Standards Track
RFC RFC
[RFC1831] [RFC1831]
R. Srinivasan, "RPC: Remote Procedure Call Protocol R. Srinivasan, "RPC: Remote Procedure Call Protocol
Specification Version 2", Standards Track RFC Specification Version 2", Standards Track RFC
[RFC1832] [RFC1832]
R. Srinivasan, "XDR: External Data Representation Standard", R. Srinivasan, "XDR: External Data Representation Standard",
Standards Track RFC Standards Track RFC
[RFC1813] [RFC1813]
B. Callaghan, B. Pawlowski, P. Staubach, "NFS Version 3 B. Callaghan, B. Pawlowski, P. Staubach, "NFS Version 3
Protocol Specification", Informational RFC Protocol Specification", Informational RFC
10. Informative References 12. Informative References
[BRU99] [BRU99]
J. Brustoloni, "Interoperation of copy avoidance in network J. Brustoloni, "Interoperation of copy avoidance in network
and file I/O", in Proc. INFOCOM '99, pages 534-542, New York, and file I/O", in Proc. INFOCOM '99, pages 534-542, New York,
NY, Mar. 1999., IEEE. Also available from NY, Mar. 1999., IEEE. Also available from
http://www.cs.pitt.edu/~jcb/publs.html http://www.cs.pitt.edu/~jcb/publs.html
[CAL+03] [CAL+03]
B. Callaghan, T. Lingutla-Raj, A. Chiu, P. Staubach, O. Asad, B. Callaghan, T. Lingutla-Raj, A. Chiu, P. Staubach, O. Asad,
"NFS over RDMA", in Proceedings of ACM SIGCOMM Summer 2003 "NFS over RDMA", in Proceedings of ACM SIGCOMM Summer 2003
skipping to change at page 13, line 6 skipping to change at page 13, line 19
[CHA+99] [CHA+99]
J. S. Chase, D. C. Anderson, A. J. Gallatin, A. R. Lebeck, K. J. S. Chase, D. C. Anderson, A. J. Gallatin, A. R. Lebeck, K.
G. Yocum, "Network I/O with Trapeze", in 1999 Hot G. Yocum, "Network I/O with Trapeze", in 1999 Hot
Interconnects Symposium, August 1999. Interconnects Symposium, August 1999.
[CHU96] [CHU96]
H.K. Chu, "Zero-copy TCP in Solaris", Proc. of the USENIX 1996 H.K. Chu, "Zero-copy TCP in Solaris", Proc. of the USENIX 1996
Annual Technical Conference, San Diego, CA, January 1996 Annual Technical Conference, San Diego, CA, January 1996
[DAFS]
Direct Access File System Specification version 1.0, available
from http://www.dafscollaborative.org, September 2001
[DCK+03] [DCK+03]
M. DeBergalis, P. Corbett, S. Kleiman, A. Lent, D. Noveck, T. M. DeBergalis, P. Corbett, S. Kleiman, A. Lent, D. Noveck, T.
Talpey, M. Wittle, "The Direct Access File System", in Talpey, M. Wittle, "The Direct Access File System", in
Proceedings of 2nd USENIX Conference on File and Storage Proceedings of 2nd USENIX Conference on File and Storage
Technologies (FAST '03), San Francisco, CA, March 31 - April Technologies (FAST '03), San Francisco, CA, March 31 - April
2, 2003 2, 2003
[FJDAFS] [FJDAFS]
Fujitsu Prime Software Technologies, "Meet the DAFS Fujitsu Prime Software Technologies, "Meet the DAFS
Performance with DAFS/VI Kernel Implementation using cLAN", Performance with DAFS/VI Kernel Implementation using cLAN",
skipping to change at page 14, line 5 skipping to change at page 14, line 11
Gallatin, R. Kisley, R. Wickremesinghe, E. Gabber, "Structure Gallatin, R. Kisley, R. Wickremesinghe, E. Gabber, "Structure
and Performance of the Direct Access File System (DAFS)", in and Performance of the Direct Access File System (DAFS)", in
Proceedings of 2002 USENIX Annual Technical Conference, Proceedings of 2002 USENIX Annual Technical Conference,
Monterey, CA, June 9-14, 2002. Monterey, CA, June 9-14, 2002.
[MOG03] [MOG03]
J. Mogul, "TCP offload is a dumb idea whose time has come", J. Mogul, "TCP offload is a dumb idea whose time has come",
9th Workshop on Hot Topics in Operating Systems (HotOS IX), 9th Workshop on Hot Topics in Operating Systems (HotOS IX),
Lihue, HI, May 2003. USENIX. Lihue, HI, May 2003. USENIX.
[NFSv4.1]
S. Shepler, ed., "NFSv4 Minor Version 1" Internet Draft work-
in-progress, draft-ietf-nfsv4-minorversion1
[PAI+00] [PAI+00]
V. S. Pai, P. Druschel, W. Zwaenepoel, "IO-Lite: a unified I/O V. S. Pai, P. Druschel, W. Zwaenepoel, "IO-Lite: a unified I/O
buffering and caching system", ACM Trans. Computer Systems, buffering and caching system", ACM Trans. Computer Systems,
18(1):37-66, Feb. 2000. 18(1):37-66, Feb. 2000.
[RDDPPS] [RDDP]
RDDP Working Group charter,
http://www.ietf.org/html.charters/rddp-charter.html
[RFC4297]
Remote Direct Data Placement Working Group Problem Statement, Remote Direct Data Placement Working Group Problem Statement,
A. Romanow, J. Mogul, T. Talpey, S. Bailey, Internet Draft A. Romanow, J. Mogul, T. Talpey, S. Bailey, Informational RFC
Work in Progress, draft-ietf-rddp-problem-statement
[RFC1094]
Sun Microsystems, "NFS: Network File System Protocol
Specification"
[RPCRDMA] [RPCRDMA]
B. Callaghan, T. Talpey, "RDMA Transport for ONC RPC", T. Talpey, B. Callaghan, "RDMA Transport for ONC RPC",
Internet Draft Work in Progress, draft-ietf-nfsv4-rpcrdma Internet Draft Work in Progress, draft-ietf-nfsv4-rpcrdma
[SHI+03] [SHI+03]
P. Shivam, J. Chase, "On the Elusive Benefits of Protocol P. Shivam, J. Chase, "On the Elusive Benefits of Protocol
Offload", to be published in Proceedings of ACM SIGCOMM Summer Offload", to be published in Proceedings of ACM SIGCOMM Summer
2003 NICELI Workshop, also available from 2003 NICELI Workshop, also available from
http://issg.cs.duke.edu/publications/niceli03.pdf http://issg.cs.duke.edu/publications/niceli03.pdf
[SKE+01] [SKE+01]
K.-A. Skevik, T. Plagemann, V. Goebel, P. Halvorsen, K.-A. Skevik, T. Plagemann, V. Goebel, P. Halvorsen,
skipping to change at page 15, line 5 skipping to change at page 15, line 22
Phone: +1 781 768 5329 Phone: +1 781 768 5329
Email: thomas.talpey@netapp.com Email: thomas.talpey@netapp.com
Chet Juszczak Chet Juszczak
Chet's Boathouse Co. Chet's Boathouse Co.
P.O. Box 1467 P.O. Box 1467
Merrimack, NH 03054 Merrimack, NH 03054
Email: chetnh@earthlink.net Email: chetnh@earthlink.net
Full Copyright Statement Intellectual Property and Copyright Statements
Copyright (C) The Internet Society (2005). This document is
subject to the rights, licenses and restrictions contained in BCP
78 and except as set forth therein, the authors retain all their
rights.
This document and the information contained herein are provided on
an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT
THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE.
Intellectual Property Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed Intellectual Property Rights or other rights that might be claimed
to pertain to the implementation or use of the technology described to pertain to the implementation or use of the technology described
in this document or the extent to which any license under such in this document or the extent to which any license under such
rights might or might not be available; nor does it represent that rights might or might not be available; nor does it represent that
it has made any independent effort to identify any such rights. it has made any independent effort to identify any such rights.
Information on the procedures with respect to rights in RFC Information on the procedures with respect to rights in RFC
documents can be found in BCP 78 and BCP 79. documents can be found in BCP 78 and BCP 79.
skipping to change at page 15, line 45 skipping to change at page 15, line 48
of such proprietary rights by implementers or users of this of such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository specification can be obtained from the IETF on-line IPR repository
at http://www.ietf.org/ipr. at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf- this standard. Please address the information to the IETF at ietf-
ipr@ietf.org. ipr@ietf.org.
Acknowledgement Disclaimer of Validity
This document and the information contained herein are provided on
an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT
THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
Acknowledgement
Funding for the RFC Editor function is currently provided by the Funding for the RFC Editor function is currently provided by the
Internet Society. Internet Society.
 End of changes. 34 change blocks. 
74 lines changed or deleted 94 lines changed or added

This html diff was produced by rfcdiff 1.32. The latest version is available from http://www.levkowetz.com/ietf/tools/rfcdiff/