draft-ietf-nfsv4-migration-issues-00.txt   draft-ietf-nfsv4-migration-issues-01.txt 
NFSv4 D. Noveck, Ed. NFSv4 D. Noveck, Ed.
Internet-Draft EMC Internet-Draft EMC
Intended status: Informational P. Shivam Intended status: Informational P. Shivam
Expires: October 13, 2012 C. Lever Expires: January 8, 2013 C. Lever
B. Baker B. Baker
ORACLE ORACLE
April 11, 2012 July 7, 2012
NFSv4 migration: Implementation experience and spec issues to resolve NFSv4 migration: Implementation experience and spec issues to resolve
draft-ietf-nfsv4-migration-issues-00 draft-ietf-nfsv4-migration-issues-01
Abstract Abstract
The migration feature of NFSv4 provides for moving responsibility for The migration feature of NFSv4 provides for moving responsibility for
a single filesystem from one server to another, without disruption to a single filesystem from one server to another, without disruption to
clients. Recent implementation experience has shown problems in the clients. Recent implementation experience has shown problems in the
existing specification for this feature. This document discusses the existing specification for this feature. This document discusses the
issues which have arisen and explores the options available for issues which have arisen and explores the options available for
curing the issues via clarification and correction of the NFSv4.0 and curing the issues via clarification and correction of the NFSv4.0 and
NFSv4.1 specifications. NFSv4.1 specifications.
skipping to change at page 1, line 39 skipping to change at page 1, line 39
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on October 13, 2012. This Internet-Draft will expire on January 8, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 27 skipping to change at page 2, line 27
3.1.2. Server reboots resulting in a confused lease 3.1.2. Server reboots resulting in a confused lease
situation . . . . . . . . . . . . . . . . . . . . . . 6 situation . . . . . . . . . . . . . . . . . . . . . . 6
3.1.3. Client complexity issues . . . . . . . . . . . . . . . 7 3.1.3. Client complexity issues . . . . . . . . . . . . . . . 7
3.2. Sources of Protocol difficulties . . . . . . . . . . . . . 9 3.2. Sources of Protocol difficulties . . . . . . . . . . . . . 9
3.2.1. Issues with nfs_client_id4 generation and use . . . . 9 3.2.1. Issues with nfs_client_id4 generation and use . . . . 9
3.2.2. Issues with lease proliferation . . . . . . . . . . . 11 3.2.2. Issues with lease proliferation . . . . . . . . . . . 11
4. Issues to be resolved in NFSv4.0 . . . . . . . . . . . . . . . 11 4. Issues to be resolved in NFSv4.0 . . . . . . . . . . . . . . . 11
4.1. Possible changes to nfs_client_id4 client-string . . . . . 11 4.1. Possible changes to nfs_client_id4 client-string . . . . . 11
4.2. Possible changes to handle differing nfs_client_id4 4.2. Possible changes to handle differing nfs_client_id4
string values . . . . . . . . . . . . . . . . . . . . . . 12 string values . . . . . . . . . . . . . . . . . . . . . . 12
4.3. Other issues within migration-state sections . . . . . . . 12 4.3. Other issues within migration-state sections . . . . . . . 13
4.4. Issues within other sections . . . . . . . . . . . . . . . 13 4.4. Issues within other sections . . . . . . . . . . . . . . . 13
5. Proposed resolution of NFSv4.0 protocol difficulties . . . . . 13 5. Proposed resolution of NFSv4.0 protocol difficulties . . . . . 14
5.1. Proposed changes: nfs_client_id4 client-string . . . . . . 13 5.1. Proposed changes: nfs_client_id4 client-string . . . . . . 14
5.2. Client-string Models (AS PROPOSED) . . . . . . . . . . . . 14 5.2. Client-string Approaches (AS PROPOSED) . . . . . . . . . . 14
5.2.1. Non-Uniform Client-string Model . . . . . . . . . . . 15 5.2.1. Non-Uniform Client-string Approach . . . . . . . . . . 16
5.2.2. Uniform Client-string Model . . . . . . . . . . . . . 16 5.2.2. Uniform Client-string Approach . . . . . . . . . . . . 16
5.2.3. Trunking Determination in the Uniform 5.2.3. Mixing Client-string Approaches . . . . . . . . . . . 18
Client-string Model . . . . . . . . . . . . . . . . . 17 5.2.4. Trunking Determination using Uniform Client-strings . 19
5.3. Proposed changes: merged (vs. synchronized) leases . . . . 20 5.3. Proposed changes: merged (vs. synchronized) leases . . . . 24
5.4. Other proposed changes to migration-state sections . . . . 21 5.4. Other proposed changes to migration-state sections . . . . 25
5.4.1. Proposed changes: Client ID migration . . . . . . . . 22 5.4.1. Proposed changes: Client ID migration . . . . . . . . 25
5.4.2. Proposed changes: Callback re-establishment . . . . . 23 5.4.2. Proposed changes: Callback re-establishment . . . . . 26
5.4.3. Proposed changes: NFS4ERR_LEASE_MOVED rework . . . . . 23 5.4.3. Proposed changes: NFS4ERR_LEASE_MOVED rework . . . . . 26
5.5. Proposed changes to other sections . . . . . . . . . . . . 23 5.5. Proposed changes to other sections . . . . . . . . . . . . 27
5.5.1. Proposed changes: callback update . . . . . . . . . . 23 5.5.1. Proposed changes: callback update . . . . . . . . . . 27
5.5.2. Proposed changes: clientid4 handling . . . . . . . . . 24 5.5.2. Proposed changes: clientid4 handling . . . . . . . . . 27
5.6. Migration, Replication and State (AS PROPOSED) . . . . . . 25 5.5.3. Proposed changes: NFS4ERR_CLID_INUSE . . . . . . . . . 29
5.6.1. Migration and State . . . . . . . . . . . . . . . . . 26 5.6. Migration, Replication and State (AS PROPOSED) . . . . . . 29
5.6.2. Replication and State . . . . . . . . . . . . . . . . 28 5.6.1. Migration and State . . . . . . . . . . . . . . . . . 30
5.6.3. Notification of Migrated Lease . . . . . . . . . . . . 28 5.6.2. Replication and State . . . . . . . . . . . . . . . . 32
5.6.4. Migration and the Lease_time Attribute . . . . . . . . 30 5.6.3. Notification of Migrated Lease . . . . . . . . . . . . 32
6. Results of proposed changes for NFSv4.0 . . . . . . . . . . . 31 5.6.4. Migration and the Lease_time Attribute . . . . . . . . 35
6. Results of proposed changes for NFSv4.0 . . . . . . . . . . . 35
6.1. Results: Failure to free migrated state on client 6.1. Results: Failure to free migrated state on client
reboot . . . . . . . . . . . . . . . . . . . . . . . . . . 31 reboot . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.2. Results: Server reboots resulting in confused lease 6.2. Results: Server reboots resulting in confused lease
situation . . . . . . . . . . . . . . . . . . . . . . . . 32 situation . . . . . . . . . . . . . . . . . . . . . . . . 36
6.3. Results: Client complexity issues . . . . . . . . . . . . 33 6.3. Results: Client complexity issues . . . . . . . . . . . . 38
6.4. Result summary . . . . . . . . . . . . . . . . . . . . . . 34 6.4. Result summary . . . . . . . . . . . . . . . . . . . . . . 39
7. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . . 34 7. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . . 39
8. Security Considerations . . . . . . . . . . . . . . . . . . . 35 7.1. Addressing state merger in NFSv4.1 . . . . . . . . . . . . 39
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 7.2. Addressing pNFS relationship with migration . . . . . . . 40
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 35 7.3. Addressing server owner changes in NFSv4.1 . . . . . . . . 40
11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 35 8. Lock State and File System Transitions (AS PROPOSED) . . . . . 41
11.1. Normative References . . . . . . . . . . . . . . . . . . . 35 8.1. File System Transitions with Matching Server Scopes . . . 42
11.2. Informative References . . . . . . . . . . . . . . . . . . 36 8.2. File System Transitions with Non-Matching Server Scopes . 43
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36 8.3. FS Transitions Involving Reobtaining Locking State . . . . 44
9. Security Considerations . . . . . . . . . . . . . . . . . . . 45
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 45
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 45
12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 46
12.1. Normative References . . . . . . . . . . . . . . . . . . . 46
12.2. Informative References . . . . . . . . . . . . . . . . . . 46
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 46
1. Introduction 1. Introduction
This document is in the informational category, and while the facts This document is in the informational category, and while the facts
it reports may have normative implications, any such normative it reports may have normative implications, any such normative
significance reflects the readers' preferences. For example, we may significance reflects the readers' preferences. For example, we may
report that the reboot of a client with migrated state results in report that the reboot of a client with migrated state results in
state not being promptly cleared and that this will prevent granting state not being promptly cleared and that this will prevent granting
of conflicting lock requests at least for the lease time, which is a of conflicting lock requests at least for the lease time, which is a
fact. While it is to be expected that client and server implementers fact. While it is to be expected that client and server implementers
skipping to change at page 5, line 26 skipping to change at page 5, line 26
each examined in turn. each examined in turn.
3. NFSv4.0 Implementation Experience 3. NFSv4.0 Implementation Experience
3.1. Implementation issues 3.1. Implementation issues
Note that the examples below reflect current experience which arises Note that the examples below reflect current experience which arises
from clients implementing the recommendation to use different from clients implementing the recommendation to use different
nfs_client_id4 id strings for different server addresses, i.e. using nfs_client_id4 id strings for different server addresses, i.e. using
what is later referred to herein as the "non-uniform client-string what is later referred to herein as the "non-uniform client-string
model" approach"
This is simply because that is the experience implementers have had. This is simply because that is the experience implementers have had.
The reader should not assume that in all cases, this practice is the The reader should not assume that in all cases, this practice is the
source of the difficulty. It may be so in some cases but clearly it source of the difficulty. It may be so in some cases but clearly it
is not in all cases. is not in all cases.
3.1.1. Failure to free migrated state on client reboot 3.1.1. Failure to free migrated state on client reboot
The following sort of situation has proved troublesome: The following sort of situation has proved troublesome:
o A client C establishes a clientid4 C1 with server ABC specifying o A client C establishes a clientid4 C1 with server ABC specifying
an nfs_client_id4 with "id" value "C-ABC" and verifier 0x111. an nfs_client_id4 with id string value "C-ABC" and boot verifier
0x111.
o The client begins to access files in filesystem F on server ABC, o The client begins to access files in filesystem F on server ABC,
resulting in generating stateids S1, S2, etc. under the lease for resulting in generating stateids S1, S2, etc. under the lease for
clientid C1. It may also access files on other filesystems on the clientid C1. It may also access files on other filesystems on the
same server. same server.
o The filesystem is migrated from ABC to server XYZ. When o The filesystem is migrated from ABC to server XYZ. When
transparent state migration is in effect, stateids S1 and S2 and transparent state migration is in effect, stateids S1 and S2 and
clientid4 C1 are now available for use by client C at server XYZ. clientid4 C1 are now available for use by client C at server XYZ.
So far, so good. So far, so good.
o Client C reboots and attempts to access data on server XYZ, o Client C reboots and attempts to access data on server XYZ,
whether in filesystem F or another. It does a SETCLIENTID with an whether in filesystem F or another. It does a SETCLIENTID with an
nfs_client_id4 with "id" value "C-XYZ" and verifier 0x112. There nfs_client_id4 with id string value "C-XYZ" and boot verifier
is thus no occasion to free stateids S1 and S2 since they are 0x112. There is thus no occasion to free stateids S1 and S2 since
associated with a different client name and so lease expiration is they are associated with a different client name and so lease
the only way that they can be gotten rid of. expiration is the only way that they can be gotten rid of.
Note here that while it seems clear to us in this example that C-XYZ Note here that while it seems clear to us in this example that C-XYZ
and C-ABC are from the same client, the server has no way to and C-ABC are from the same client, the server has no way to
determine the structure of the "opaque" id. In the protocol, it determine the structure of the "opaque" id string. In the protocol,
really is opaque. Only the client knows which nfs_client_id4 values it really is treated as opaque. Only the client knows which
designate the same client on a different server. nfs_client_id4 values designate the same client on a different
server.
3.1.2. Server reboots resulting in a confused lease situation 3.1.2. Server reboots resulting in a confused lease situation
Further problems arise from scenarios like the following. Further problems arise from scenarios like the following.
o Client C talks to server ABC using an nfs_client_id4 id like o Client C talks to server ABC using an nfs_client_id4 id string
"C-ABC" and verifier v1. As a result a lease with clientid4 c.i such as "C-ABC" and a boot verifier v1. As a result, a lease with
is established: {v1, "C-ABC", c.i}. clientid4 c.i is established: {v1, "C-ABC", c.i}.
o fs_a1 migrates from server ABC to server XYZ along with its state. o fs_a1 migrates from server ABC to server XYZ along with its state.
Now server XYZ also has a lease: {v1, "C-ABC", c.i}. Now server XYZ also has a lease: {v1, "C-ABC", c.i}.
o Server ABC reboots. o Server ABC reboots.
o Client C talks to server ABC using an nfs_client_id4 id like o Client C talks to server ABC using an nfs_client_id4 id string
"C-ABC" and verifier v1. As a result a lease with clientid4 c.j such as "C-ABC" and a boot verifier v1. As a result, a lease with
is established: {v1, "C-ABC", c.j}. clientid4 c.j is established: {v1, "C-ABC", c.j}.
o fs_a2 migrates from server ABC to server XYZ. Now server XYZ also o fs_a2 migrates from server ABC to server XYZ. Now server XYZ also
has a lease: {v1, "C-ABC", c.j}. has a lease: {v1, "C-ABC", c.j}.
o Now server XYZ has two leases that match {v1, "C-ABC", *}, when o Now server XYZ has two leases that match {v1, "C-ABC", *}, when
the protocol clearly assumes there can be only one. the protocol clearly assumes there can be only one.
Note that if the client used "C" (rather than "C-ABC") as the Note that if the client used "C" (rather than "C-ABC") as the
nfs_client_id4 id string, the exact same situation would arise. nfs_client_id4 id string, the exact same situation would arise.
One of the first cases in which this sort of situation has resulted One of the first cases in which this sort of situation has resulted
in difficulties is in connection with doing a SETCLIENTID for in difficulties is in connection with doing a SETCLIENTID for
callback update. callback update.
The SETCLIENTID for callback update only includes the nfs_client_id4, The SETCLIENTID for callback update only includes the nfs_client_id4,
assuming there can only be one such with a given nfs_client_id4 assuming there can only be one such with a given nfs_client_id4
value. If there are multiple, confirmed client records with value. If there were multiple, confirmed client records with
identical nfs_client_id4 values, there is no way to map the callback identical nfs_client_id4 id string values, there would be no way to
update request to the correct client record. map the callback update request to the correct client record. Apart
from the migration handling specified in [RFC3530], such a situation
cannot arise.
One possible accommodation for this particular issue that has been One possible accommodation for this particular issue that has been
used is to add a RENEW operation along with SETCLIENTID (on a used is to add a RENEW operation along with SETCLIENTID (on a
callback update) to disambiguate the client. callback update) to disambiguate the client.
When the client updates the callback info to the destination, the When the client updates the callback info to the destination, the
client would, by convention, send a compound like this: client would, by convention, send a compound like this:
{ RENEW clientid4, SETCLIENTID nfs_client_id4,verf,cb } { RENEW clientid4, SETCLIENTID nfs_client_id4,verf,cb }
skipping to change at page 10, line 12 skipping to change at page 10, line 12
client, so that there is no occasion for the difference in client, so that there is no occasion for the difference in
interpretation to matter. interpretation to matter.
o When transparent migration of client state occurs between two o When transparent migration of client state occurs between two
servers, it becomes important to determine when state on two servers, it becomes important to determine when state on two
different servers is for the same client or not, and this different servers is for the same client or not, and this
distinction becomes very important. distinction becomes very important.
Given the need for the server to be aware of client identity with Given the need for the server to be aware of client identity with
regard to migrated state, either client-string construction rules regard to migrated state, either client-string construction rules
will have to change or there will be need to get around current will have to change or there will be a need to get around current
issues, or perhaps a combination of these two will be required. issues, or perhaps a combination of these two will be required.
Later sections will examine the options and propose a solution. Later sections will examine the options and propose a solution.
One consideration that may indicate that this cannot remain exactly One consideration that may indicate that this cannot remain exactly
as it is today has to do with the fact that the current explanation as it is today has to do with the fact that the current explanation
for this behavior is not correct. The current definitive definition for this behavior is not correct. The current definitive definition
of the NFSv4.0 protocol [RFC3530], and the current pending draft of of the NFSv4.0 protocol [RFC3530], and the current pending draft of
RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Client RFC3530bis [cur-v4.0-bis] both agree. The section entitled "Client
ID" says: ID" says:
skipping to change at page 10, line 35 skipping to change at page 10, line 35
the client issues SETCLIENTID with the same id string to each the client issues SETCLIENTID with the same id string to each
network address of such a server, the server will think it is the network address of such a server, the server will think it is the
same client, and each successive SETCLIENTID will cause the server same client, and each successive SETCLIENTID will cause the server
to begin the process of removing the client's previous leased to begin the process of removing the client's previous leased
state. state.
In point of fact, a "SETCLIENTID with the same id string" sent to In point of fact, a "SETCLIENTID with the same id string" sent to
multiple network addresses will be treated as all from the same multiple network addresses will be treated as all from the same
client but will not "cause the server to begin the process of client but will not "cause the server to begin the process of
removing the client's previous leased state" unless the server removing the client's previous leased state" unless the server
believes it is a newer instance of the same client, i.e. if the id is believes it is a different instance of the same client, i.e. if the
the same and there is a different verifier. If the client does not id string is the same and there is a different boot verifier. If the
reboot, the verifier should not change. If it does reboot, the client does not reboot, the verifier should not change. If it does
verifier will change, and the server should "begin the process of reboot, the verifier will change, and the server should "begin the
removing the client's previous leased state. process of removing the client's previous leased state.
The situation of multiple SETCLIENTID requests received by a server The situation of multiple SETCLIENTID requests received by a server
on multiple network addresses is exactly the same, from the protocol on multiple network addresses is exactly the same, from the protocol
design point of view, as when multiple (i.e. duplicate) SETCLIENTID design point of view, as when multiple (i.e. duplicate) SETCLIENTID
requests are received by the server on a single network address. The requests are received by the server on a single network address. The
same protocol mechanisms that prevent erroneous state deletion in the same protocol mechanisms that prevent erroneous state deletion in the
latter case prevent it in the former case. There is no reason for latter case prevent it in the former case. There is no reason for
special handling of the multiple-network-appearance case, in this special handling of the multiple-network-appearance case, in this
regard. regard.
skipping to change at page 11, line 29 skipping to change at page 11, line 29
This could be enough only if we are prepared to do away with the This could be enough only if we are prepared to do away with the
"should" recommending non-uniform client-strings and replace it with "should" recommending non-uniform client-strings and replace it with
a "should not" or even a "SHOULD NOT". Current client implementation a "should not" or even a "SHOULD NOT". Current client implementation
patterns make this an unpalatable choice for use as a general patterns make this an unpalatable choice for use as a general
solution, but it is reasonable to "RECOMMEND" this choice for a well- solution, but it is reasonable to "RECOMMEND" this choice for a well-
defined subset of clients. One alternative would be to create a way defined subset of clients. One alternative would be to create a way
for the server to infer from client behavior which leases are held by for the server to infer from client behavior which leases are held by
the same client and use this information to do appropriate lease the same client and use this information to do appropriate lease
mergers. Prototyping and detailed specification work has shown that mergers. Prototyping and detailed specification work has shown that
this could be done but the resulting complexity is such that a better this could be done but the resulting complexity is such that a better
choice is to "RECOMMEND" use of the uniform model for clients choice is to "RECOMMEND" use of the uniform approach for clients
supporting the migration feature. supporting the migration feature.
Because of the discussion of client-string construction in [RFC3530],
most existing clients implement the non-uniform client-string
approach. As a result, existing servers may not have been tested
with clients implementing uniform client-strings. As a consequence,
care must be taken to preserve interoperability between UCS-capable
clients and servers that don't tolerate uniform client strings for
one reason or another. See Section 5.2.3 for details.
4. Issues to be resolved in NFSv4.0 4. Issues to be resolved in NFSv4.0
4.1. Possible changes to nfs_client_id4 client-string 4.1. Possible changes to nfs_client_id4 client-string
The fact that the reason given in client-string-BP3 is not valid The fact that the reason given in client-string-BP3 is not valid
makes the existing "should" insupportable. We can't either makes the existing "should" insupportable. We can't either
o Keep a reason we know is invalid. o Keep a reason we know is invalid.
o Keep saying "should" without giving a reason. o Keep saying "should" without giving a reason.
What are often presented as reasons that motivate use of the non- What are often presented as reasons that motivate use of the non-
uniform model always turn out to be cases in which, if the uniform uniform approach always turn out to be cases in which, if the uniform
model were used, the server will treat a client which accesses that approach were used, the server will treat a client which accesses
server via two different IP addresses as part of a single client, as that server via two different IP addresses as part of a single
it in fact is. This may be disconcerting to a client unaware that client, as it in fact is. This may be disconcerting to a client
the two IP addresses connect to the same server. This is thus not a unaware that the two IP addresses connect to the same server. This
reason to use the non-uniform model but rather an illustration of the is thus not a reason to use the non-uniform approach but rather an
fact that those using the uniform model must use server behavior to illustration of the fact that those using the uniform approach must
determine whether any trunking of IP addresses exists, as is use server behavior to determine whether any trunking of IP addresses
described in Section 5.2.2. exists, as is described in Section 5.2.2.
It is always possible that a valid new reason will be found, but so It is always possible that a valid new reason will be found, but so
far none has been proposed. Given the history, the burden of proof far none has been proposed. Given the history, the burden of proof
should be on those asserting the validity of a proposed new reason. should be on those asserting the validity of a proposed new reason.
So we will assume for now that the "should" will have to go. The So we will assume for now that the "should" will have to go. The
question is what to replace it with. question is what to replace it with.
o We can't say "MUST NOT", despite the problems this raises for o We can't say "MUST NOT", despite the problems this raises for
migration since this is pretty late in the day for such a change. migration since this is pretty late in the day for such a change.
skipping to change at page 13, line 15 skipping to change at page 13, line 24
o Lack of clarity in the discussion of moving clientids (as well as o Lack of clarity in the discussion of moving clientids (as well as
stateids) as part of moving state for migration. stateids) as part of moving state for migration.
o The discussion of synchronized leases is wrong in that there is no o The discussion of synchronized leases is wrong in that there is no
way to determine (in the current spec) when leases are for the way to determine (in the current spec) when leases are for the
same client and also wrong in suggesting a benefit from leases same client and also wrong in suggesting a benefit from leases
synchronized at the point of transfer. What is needed is merger synchronized at the point of transfer. What is needed is merger
of leases, which is necessary to keep client complexity of leases, which is necessary to keep client complexity
requirements from getting out of hand. requirements from getting out of hand.
o Lack of clarity in the discussion of LEASE_MOVED handling. o Lack of clarity in the discussion of LEASE_MOVED handling,
including failure to fully address situations in which transparent
state migration did not occur.
4.4. Issues within other sections 4.4. Issues within other sections
There are a number of cases in which certain sections, not There are a number of cases in which certain sections, not
specifically related to migration require additional clarification. specifically related to migration, require additional clarification.
This is generally because text that is clear in a context in which This is generally because text that is clear in a context in which
leases and clientids are created in one place and live there forever leases and clientids are created in one place and live there forever
may need further refinement in the more dynamic environment that may need further refinement in the more dynamic environment that
arises as part of migration. arises as part of migration.
Some examples: Some examples:
o Some people are under the impression that updating callback o Some people are under the impression that updating callback
endpoint information for an existing client, which is part of the endpoint information for an existing client, as used during
client's handling of migration, may cause the destination server migration, may cause the destination server to free existing
to free existing state. There needs to be additions to clarify state. There need to be additions to clarify the situation.
the situation.
o The handling of the sets of clientid4's maintained by each server o The handling of the sets of clientid4's maintained by each server
needs to be clarified. In particular, the issue of how the client needs to be clarified. In particular, the issue of how the client
adapts to the presumably independent and uncoordinated clientid4 adapts to the presumably independent and uncoordinated clientid4
sets needs to be clearly addressed sets needs to be clearly addressed
o Statements regarding handling of invalid clientid4's need to be o Statements regarding handling of invalid clientid4's need to be
clarified and/or refined in light of the possibilities that arise clarified and/or refined in light of the possibilities that arise
due to lease motion and merger. due to lease motion and merger.
o Confusion and lack of clarity about NFS4ERR_CLID_INUSE.
5. Proposed resolution of NFSv4.0 protocol difficulties 5. Proposed resolution of NFSv4.0 protocol difficulties
5.1. Proposed changes: nfs_client_id4 client-string 5.1. Proposed changes: nfs_client_id4 client-string
We propose replacing client-string-BP3 with the following text and We propose replacing client-string-BP3 with the following text and
adding the following proposed Section 5.2 to provide implementation adding the following proposed Section 5.2 to provide implementation
guidance. guidance.
o The string MAY be different for each server network address that o The string MAY be different for each server network address that
the client accesses, rather than common to all server network the client accesses, rather than common to all server network
addresses. The considerations that might influence a client to addresses.
use different strings for each are explained in Section 5.2.
o The considerations that might influence a client to use different
strings for different network server addresses are explained in
Section 5.2.
o Despite the use of the word "string" for this identifier, and the o Despite the use of the word "string" for this identifier, and the
fact that using strings will often be convenient, it should be fact that using strings will often be convenient, it should be
understood that the protocol defines this as opaque data. In understood that the protocol defines this as opaque data. In
particular, those receiving such an id should not assume that it particular, those receiving such an id should not assume that it
will be in UTF-8 format nor should they reject it if it is not. will be in UTF-8 format. Servers MUST NOT reject an
nfs_client_id4 simply because the id string is not in UTF-8
format.
5.2. Client-string Models (AS PROPOSED) 5.2. Client-string Approaches (AS PROPOSED)
One particular aspect of the construction of the nfs4_client_id4 One particular aspect of the construction of the nfs4_client_id4
string has proved recurrently troublesome. The client has a choice string has proved recurrently troublesome. The client has a choice
of: of:
o Presenting the same id string to each server address accessed. o Presenting the same id string to multiple server addresses. This
This is referred to as the "uniform client-string model" and is is referred to as the "uniform client-string approach" and is
discussed in Section 5.2.2. discussed in Section 5.2.2.
o Presenting a different id string to each server address accessed. o Presenting different id strings to multiple server addresses.
This is referred to as the "non-uniform client-string model" and This is referred to as the "non-uniform client-string approach"
is discussed in Section 5.2.1. and is discussed in Section 5.2.1.
Note that implementation considerations, including compatibility with
existing servers, may make it desirable for a client to use both
approaches, based on configuration information, such as mount
options. This issue will be discussed in Section 5.2.3.
Construction of the client-string has been a troublesome issue Construction of the client-string has been a troublesome issue
because of the way in which the NFS protocols have evolved. because of the way in which the NFS protocols have evolved.
o NFSv3 as a stateless protocol had no need to identify the state o NFSv3 as a stateless protocol had no need to identify the state
shared by a particular client-server pair. Thus there was no shared by a particular client-server pair. Thus there was no
occasion to consider the question of whether a set of requests occasion to consider the question of whether a set of requests
come from the same client, or whether two server IP addresses are come from the same client, or whether two server IP addresses are
connected to the same server. As the environment was one in which connected to the same server. As the environment was one in which
the user supplied the target server IP address as part of the user supplied the target server IP address as part of
skipping to change at page 15, line 4 skipping to change at page 15, line 25
client identity information. client identity information.
o NFSv4.1 is a stateful protocol with full support for client and o NFSv4.1 is a stateful protocol with full support for client and
server identity determination. This enables the server to be server identity determination. This enables the server to be
aware when two requests come from the same client (they are on aware when two requests come from the same client (they are on
sessions sharing a clientid4) and the client to be aware when two sessions sharing a clientid4) and the client to be aware when two
server IP addresses are connected to the same server (they return server IP addresses are connected to the same server (they return
the same server name in responding to an EXCHANGE_ID). the same server name in responding to an EXCHANGE_ID).
NFSv4.0 is unfortunately halfway between these two. The two client- NFSv4.0 is unfortunately halfway between these two. The two client-
string models have arisen in attempts to deal with the changing string approaches have arisen in attempts to deal with the changing
requirements of the protocol as implementation has proceeded and requirements of the protocol as implementation has proceeded and
features that were not very substantial in [RFC3530], got more features that were not very substantial in [RFC3530], got more
substantial. substantial.
o In the absence of any implementation of the fs_locations-related o In the absence of any implementation of the fs_locations-related
features (replication, referral, and migration), the situation is features (replication, referral, and migration), the situation is
very similar to that of NFSv3, with the addition of state but with very similar to that of NFSv3, with the addition of state but with
no concern to provide accurate client and server identity no concern to provide accurate client and server identity
determination. This is the situation that gave rise to the non- determination. This is the situation that gave rise to the non-
uniform client-string model. uniform client-string approach.
o In the presence of replication and referrals, the client may have o In the presence of replication and referrals, the client may have
occasion to take advantage of knowledge of server trunking occasion to take advantage of knowledge of server trunking
information. Even more important, migration, by transferring information. Even more important, migration, by transferring
state among servers, causes difficulties for the non-uniform state among servers, causes difficulties for the non-uniform
client-string model, in that the two different client-strings sent client-string approach, in that the two different client-strings
to different IP addresses may wind up on the same IP address, sent to different IP addresses may wind up on the same IP address,
adding confusion. adding confusion.
Both models have to deal with the asymmetry in client and server o A further consideration is that client implementations typically
provide NFSv4.1 by augmenting their existing NFSv4.0
implementation, not by providing two separate implementations.
Thus the more NFSv4.0 and NFSv4.1 can work alike, the less complex
are clients. This is a key reason why those implementing NFSv4.0
clients might prefer using the uniform client string model, even
if they have chosen not to provide fs_locations-related features
in their NFSv4.0 client.
Both approaches have to deal with the asymmetry in client and server
identity information between client and server. Each seeks to make identity information between client and server. Each seeks to make
the client's and the server's views match. In the process, each the client's and the server's views match. In the process, each
encounters some combination of inelegant protocol features and/or encounters some combination of inelegant protocol features and/or
implementation difficulties. The choice of which to use is up to the implementation difficulties. The choice of which to use is up to the
client implementer and the sections below try to give some useful client implementer and the sections below try to give some useful
guidance. guidance.
5.2.1. Non-Uniform Client-string Model 5.2.1. Non-Uniform Client-string Approach
The non-uniform client-string model is an attempt to handle these The non-uniform client-string approach is an attempt to handle these
matters in NFSv4.0 client implementations in as NFSv3-like a way as matters in NFSv4.0 client implementations in as NFSv3-like a way as
possible. possible.
For a client using the non-uniform model, all internal recording of For a client using the non-uniform approach, all internal recording
clientid4 values is to include, whether explicitly or implicitly, the of clientid4 values is to include, whether explicitly or implicitly,
server IP address so that one always has an (IP-address, clientid4) the server IP address so that one always has an (IP-address,
pair. Two such pairs from different servers are always distinct even clientid4) pair. Two such pairs from different servers are always
when the clientid4 values are the same, as they may occasionally be. distinct even when the clientid4 values are the same, as they may
In this model, such equality is always treated as simple occasionally be. In this approach, such equality is always treated
happenstance. as simple happenstance.
Making the client-string different on different servers means that a Making the client-string different on different servers means that a
server has no way of tying together information from the same client server has no way of tying together information from the same client
and so will treat a single client as multiple clients with multiple and so will treat a single client as multiple clients with multiple
leases for each server network address. Since there is no way in the leases for each server network address. Since there is no way in the
protocol for the client to determine if two network addresses are protocol for the client to determine if two network addresses are
connected to the same server, the resulting lack of knowledge is connected to the same server, the resulting lack of knowledge is
symmetrical and can result in simpler client implementations in which symmetrical and can result in simpler client implementations in which
there is a single clientid/lease per server network addresses. there is a single clientid/lease per server network addresses.
Support for migration, particularly with transparent state migration, Support for migration, particularly with transparent state migration,
is more complex in the case of non-uniform client-strings. For is more complex in the case of non-uniform client-strings. For
example, migration of a lease can result in multiple leases for the example, migration of a lease can result in multiple leases for the
same client accessing the same server addresses, vitiating many of same client accessing the same server addresses, vitiating many of
the advantages of this approach. Therefore, client implementations the advantages of this approach. Therefore, client implementations
that support migration with transparent state migration SHOULD NOT that support migration with transparent state migration SHOULD NOT
use the non-uniform client-string model. use the non-uniform client-string approach, except where it is
necessary for compatibility with existing server implementations (For
details of arranging use of multiple client-string approaches, see
Section 5.2.3).
5.2.2. Uniform Client-string Model 5.2.2. Uniform Client-string Approach
When the client-string is kept uniform, the server has the basis to When the client-string is kept uniform, the server has the basis to
have a single clientid4/lease for each distinct client. The problem have a single clientid4/lease for each distinct client. The problem
that has to be addressed is the lack of explicit server identity that has to be addressed is the lack of explicit server identity
information, which is made available in NFSv4.1. information, which is made available in NFSv4.1.
When the same client-string is given to multiple IP addresses, the When the same client-string is given to multiple IP addresses, the
client can determine whether two IP addresses correspond to a single client can determine whether two IP addresses correspond to a single
server, based on the server's behavior. This is the inverse of the server, based on the server's behavior. This is the inverse of the
strategy adopted for the non-uniform model in which different server strategy adopted for the non-uniform approach in which different
IP addresses are told about different clients, simply to prevent a server IP addresses are told about different clients, simply to
server from manifesting behavior that is inconsistent with there prevent a server from manifesting behavior that is inconsistent with
being a single server for each IP address, in line with the there being a single server for each IP address, in line with the
traditions of NFS. So, to compare: traditions of NFS. So, to compare:
o In the non-uniform model, servers are told about different clients o In the non-uniform approach, servers are told about different
because, if the server were to use accurate information as to clients because, if the server were to use accurate information as
client identity, two IP addresses on the same server would behave to client identity, two IP addresses on the same server would
as if they were talking to the same client, which might prove behave as if they were talking to the same client, which might
disconcerting to a client not expecting such behavior. prove disconcerting to a client not expecting such behavior.
o In the uniform model, the servers are told about there being a o In the uniform approach, the servers are told about there being a
single client, which is, after all, the truth. Then, when the single client, which is, after all, the truth. Then, when the
server uses this information, two IP addresses on the same server server uses this information, two IP addresses on the same server
will behave as if they are talking to the same client, and this will behave as if they are talking to the same client, and this
difference in behavior allows the client to infer the server IP difference in behavior allows the client to infer the server IP
address trunking configuration, even though NFSv4.0 does not address trunking configuration, even though NFSv4.0 does not
explicitly provide this information. explicitly provide this information.
The approach given in the section below shows one example of how The approach given in the section below shows one example of how
this might be done. this might be done.
The uniform client-string approach makes it necessary to exercise
more care in the definition of the nfs_client_id4 boot verifier:
o In [RFC3530], the client is told to change the boot verifier when
reboot occurs, but there is no explicit statement as to the
converse, so that any requirement to keep the verifier constant
unless rebooting is only present by implication.
o Many existing clients change the boot verifier every time they
destroy and recreate the data structure that tracks an <IP-
address, clientid4> pair. This might happen if the last mount of
a particular server is removed, and then a fresh mount is created.
And, note that this might result in each <IP-address, clientid4>
pair having its own boot verifier that is independent of the
others.
o Within the uniform client-string approach, an nfs_client_id4
designates a globally known client instance, so that the boot
verifier should change if and only if a new client instance is
created, typically as a result of a reboot.
The following are advantages for the implementation of using the The following are advantages for the implementation of using the
uniform client-string model: uniform client-string approach:
o Clients can take advantage of server trunking (and clustering with o Clients can take advantage of server trunking (and clustering with
single-server-equivalent semantics) to increase bandwidth or single-server-equivalent semantics) to increase bandwidth or
reliability. reliability.
o There are advantages in state management so that, for example, we o There are advantages in state management so that, for example, we
never have a delegation under one clientid revoked because of a never have a delegation under one clientid revoked because of a
reference to the same file from the same client under a different reference to the same file from the same client under a different
clientid. clientid.
o The uniform client-string model allows the server to do any o The uniform client-string approach allows the server to do any
necessary automatic lease merger in connection with migration, necessary automatic lease merger in connection with migration,
without requiring any client involvement. This consideration is without requiring any client involvement. This consideration is
of sufficient weight to cause us RECOMMEND use of the uniform of sufficient weight to cause us to RECOMMEND use of the uniform
client-string model for clients supporting transparent state client-string approach for clients supporting transparent state
migration. migration.
The following implementation considerations might cause issues for The following implementation considerations might cause issues for
client implementations. client implementations.
o This model is considerably different from the non-uniform model, o This approach is considerably different from the non-uniform
which most client implementations have been following. Until approach, which most client implementations have been following.
substantial implementation experience is obtained with this model, Until substantial implementation experience is obtained with this
reluctance to embrace something so new is to be expected. approach, reluctance to embrace something so new is to be
expected.
o Mapping between server network addresses and leases is more o Mapping between server network addresses and leases is more
complicated in that it is no longer a one-to-one mapping. complicated in that it is no longer a one-to-one mapping.
How to balance these considerations depends on implementation goals. How to balance these considerations depends on implementation goals.
5.2.3. Trunking Determination in the Uniform Client-string Model 5.2.3. Mixing Client-string Approaches
As noted above, a client which needs to use the uniform client-string
approach (e.g. to support migration), may also need to support
existing servers with implementations that do not work properly in
this case.
Some examples of such server issues include:
o Some existing NFSv4 server implementations of IP-address failover
depend on clients' use of a non-uniform client-string approach.
In particular, when a server supports both its own IP address and
one failed over from a partner server, it may have separate sets
of state applicable to the two IP addresses, owned by different
servers but residing on a single one.
In this situation, some servers have relied on clients' use of the
non-uniform client-string approach, as suggested but not mandated
by [RFC3530], to keep these sets of state separate, and will have
problems in handling clients using the uniform client-string
approach, in that such clients will see changes in trunking
relationships whenever server failover and giveback occur.
o Some existing servers incorrectly return NFS4ERR_CLID_INUSE in a
way which interferes with clients using the uniform client-string
approach. See Section 5.5.3 for details.
In order to support such servers, the client can use different
approaches for different mounts, as long as:
o The uniform client-string approach is used when accessing servers
that may return NFS4ERR_MOVED.
o The non-uniform client-string approach is used when accessing
servers whose implementations make them incompatible with the
uniform client-string approach
One effective way for clients to handle this is to support the
uniform client-string approach as the default, but allow a mount
option to specify use of the non-uniform client-string approach for
particular mount points, as long as such mount points are not used
when migration is to be supported.
In the case in which the same server has multiple mounts, and both
approaches are specified for the same server, the client could have
multiple clientids corresponding to the same server, one for each
approach and would then have to keep these separate.
5.2.4. Trunking Determination using Uniform Client-strings
This section provides an example of how trunking determination could This section provides an example of how trunking determination could
be done by a client following the uniform client-string model. be done by a client following the uniform client-string approach
Clients need not follow this procedure but implementers should make (whether this is used for all mounts or not). Clients need not
sure that the issues dealt with by this procedure are all properly follow this procedure but implementers should make sure that the
addressed. issues dealt with by this procedure are all properly addressed.
For a client using the uniform model, clientid4 values are treated as We need to clarify the various possible purposes of trunking
important information in determining server trunking patterns. For determination and the corresponding requirements as to server
two different IP addresses to return the same clientid4 value is a behavior. The following points should be noted:
necessary, though not a sufficient condition for them to be
o The primary purpose of the trunking determination algorithm is to
make sure that, if the server treats client requests on two IP
addresses as part of the same client, the client will not be
blind-sided and encounter disconcerting server behavior, as
mentioned in Section 5.2.2. Such behavior could occur if the
client were unaware that all of its client requests for the two IP
addresses were being handled as part of a single client talking to
a single server.
o A second purpose to be able to use knowledge of trunking
relationships for better performance, etc
o If a server were to give out distinct clientid's in response to
receiving the same nfs_client_id4 on different network addresses,
and acted as if these were separate clients, the primary purpose
of trunking determination would be met, as long as the server did
not treat them as part of the same client. In this case, the
server would be acting, with regard to that client, as if it were
two distinct servers. This would interfere with the secondary
purpose of trunking determination but there is nothing the client
can do about that.
o Suppose a server were to give such a client two different
clientid's but act as if they were one. That it is the only way
that the server could behave in a way that would defeat the
primary purpose of the trunking determination algorithm.
Servers MUST NOT do that.
For a client using the uniform approach, clientid4 values are treated
as important information in determining server trunking patterns.
For two different IP addresses to return the same clientid4 value is
a necessary, though not a sufficient condition for them to be
considered as connected to the same server. As a result, when two considered as connected to the same server. As a result, when two
different IP addresses return the same clientid4, the client needs to different IP addresses return the same clientid4, the client needs to
determine, using the procedure given below or otherwise, whether the determine, using the procedure given below or otherwise, whether the
IP addresses are connected to the same server. For such clients, all IP addresses are connected to the same server. For such clients, all
internal recording of clientid4 values needs to include, whether internal recording of clientid4 values needs to include, whether
explicitly or implicitly, identification of the server from which the explicitly or implicitly, identification of the server from which the
clientid4 was received so that one always has a (server clientid4) clientid4 was received so that one always has a (server, clientid4)
pair. Two such pairs from different servers are always considered pair. Two such pairs from different servers are always considered
distinct even when the clientid4 values are the same, as they may distinct even when the clientid4 values are the same, as they may
occasionally be. occasionally be.
In order to make this approach work, the client must have accessible, In order to make this approach work, the client must have accessible,
for each nfs4_client_id4 used (only one in the uniform model) a list for each nfs4_client_id4 used by the uniform approach (only one in
of all server IP addresses, together with the associated clientid4 general) a list of all server IP addresses, together with the
values. As a part of the associated data structures, there should be associated clientid4 values and authentication flavors. As a part of
the ability to mark a server IP structure as having the same server the associated data structures, there should be the ability to mark a
as another and to mark an IP-address as currently unresolved. One server IP structure as having the same server as another and to mark
way to do this is to a allow each such entry to point to another with an IP-address as currently unresolved. One way to do this is to a
the pointer value being one of: allow each such entry to point to another with the pointer value
being one of:
o A pointer to another entry for an IP address associated with the o A pointer to another entry for an IP address associated with the
same server, where that IP address is the first one referenced to same server, where that IP address is the first one referenced to
access that server. access that server.
o A pointer to the current entry if there is no earlier IP address o A pointer to the current entry if there is no earlier IP address
associated with the same server, i.e. where the current IP address associated with the same server, i.e. where the current IP address
is the first one referenced to access that server. We'll refer to is the first one referenced to access that server. We'll refer to
such an IP address as the lead IP address for a given server. such an IP address as the lead IP address for a given server.
o The value NULL if the address's server identity is currently o The value NULL if the address's server identity is currently
unresolved. unresolved.
When a SETCLIENTID is done and a clientid4 returned, the data In order to keep the above information current, in the interests of
structure is searched for a matching clientid4 and processing depends the most effective trunking determination, RENEWs should be
on what is found. We will refer to the IP address on which this periodically done on each server. However, even if this is not done,
SETCLIENTID is done as X. The SETCLIENTID will use the common the primary purpose of the trunking determination algorithm, to
nfs_client_id4 and specify X as part of the callback parameters. We prevent confusion due to trunking hidden from the client, will be
call the clientid4 and verifier returned by this operation XC and XV. achieved.
Note that at this point no SETCLIENTID_CONFIRM has yet been done. Given this apparatus, when a SETCLIENTID is done and a clientid4
This is because we have either established a new clientid4 on a returned, the data structure can be searched for a matching clientid4
previously unknown server or changed the callback parameters on a and if such is found, further processing can be done to determine
clientid4 associated with some already known server. We don't want whether the clientid4 match is accidental, or the result of trunking.
to confirm something that we are not sure we want to happen.
In this algorithm, when SETCLIENTID is done it will use the common
nfs_client_id4 and specify the current target IP address as part of
the callback parameters. We call the clientid4 and SETCLIENTID
verifier returned by this operation XC and XV.
Note that when the client has done previous SETCLIENTID's, to any IP
addresses, with more than one authentication flavor, we have the
possibility of receiving NFS4ERR_CLID_INUSE, since we do not yet know
which of our connections with existing IP addresses might be trunked
with our current one. In the event that the SETCLIENID fails with
NFS4ERR_CLID_INUSE, one must try all other authentication flavors
currently in use and eventually one will be correct and not return
NFS4ERR_CLID_INUSE.
Note that at this point, no SETCLIENTID_CONFIRM has yet been done.
This is because our SETCLIENTID has either established a new
clientid4 on a previously unknown server or changed the callback
parameters on a clientid4 associated with some already known server.
Given that we don't want to confirm something that we are not sure we
want to happen, what is to be done next depends on information about
existing clientid4's.
o If no matching clientid4 is found, the IP address X and clientid4 o If no matching clientid4 is found, the IP address X and clientid4
XC are added to the list and considered as having no existing XC are added to the list and considered as having no existing
known IP addresses trunked with it. The IP address is marked as a known IP addresses trunked with it. The IP address is marked as a
lead IP address for a new server. A SETCLIENTID_CONFIRM is done lead IP address for a new server. A SETCLIENTID_CONFIRM is done
using XC and XV. using XC and XV.
o If a matching clientid4 is found which is marked unresolved, o If a matching clientid4 is found which is marked unresolved,
processing on the new IP address is suspended. In order to processing on the new IP address is suspended. In order to
simplify processing, there can only be one unresolved IP address simplify processing, there can only be one unresolved IP address
skipping to change at page 19, line 18 skipping to change at page 22, line 29
addresses with a matching clientid4, the address will have been addresses with a matching clientid4, the address will have been
resolved: either it will be part of the same server as a new IP resolved: either it will be part of the same server as a new IP
address to be added to an existing set of IP addresses for a address to be added to an existing set of IP addresses for a
server, or it will be recognized as a new server. At the point at server, or it will be recognized as a new server. At the point at
which this determination is made, the unresolved indication is which this determination is made, the unresolved indication is
cleared and any suspended SETCLIENTID processing is restarted cleared and any suspended SETCLIENTID processing is restarted
So for each lead IP address IPn with a clientid4 matching XC, the So for each lead IP address IPn with a clientid4 matching XC, the
following steps are done. following steps are done.
o If the authentication flavor for IPn does not match that for X,
the IP address is skipped, since it is impossible or IPn and X to
be trunked in these circumstances. This avoids any possibility
that NFS4ERR_CLID_INUSE will be returned for the SETCLIENTID and
SETCLIENID_CONFIRM to be done below, as long as the server(s) at
IP addresses IPn and X are correctly implemented.
o A SETCLIENTID is done to update the callback parameters to reflect o A SETCLIENTID is done to update the callback parameters to reflect
the possibility that X will be marked as associated with the the possibility that X will be marked as associated with the
server whose lead IP address is IPn. So assume that we do that server whose lead IP address is IPn. The specific callback
SETCLIENTID on IP address IPn and get back a setclientid_confirm parameters chosen, in terms of cb_client4 and callback_ident, are
value (in the form of a verifier4) SCn. up to the client and should reflect its preferences as to callback
handling for the common clientid, in the event that X and IPn are
trunked together. So assume that we do that SETCLIENTID on IP
address IPn and get back a setclientid_confirm value (in the form
of a verifier4) SCn.
Note that the v4.0 spec requires the server to make sure that such Note that the v4.0 spec requires the server to make sure that such
value are very unlikely to be regenerated. Given that it is value are very unlikely to be regenerated. Given that it is
already highly unlikely that the clientid XC is duplicated by already highly unlikely that the clientid XC is duplicated by
distinct servers, the probability that Sc is duplicated as well distinct servers, the probability that Sc is duplicated as well
has to be considered vanishingly small. Note also that the has to be considered vanishingly small. Note also that the
callback update procedure can be repeated multiple times to reduce callback update procedure can be repeated multiple times to reduce
the probability of spurious matches further. the probability of spurious matches further.
o Note that we don't want this to happen if address X is not o Note that we don't want this to happen if address X is not
skipping to change at page 19, line 45 skipping to change at page 23, line 18
o If the setclientid_confirm value generated on X is accepted on o If the setclientid_confirm value generated on X is accepted on
IPn, then X and IPn are recognized as connected to the same server IPn, then X and IPn are recognized as connected to the same server
and the entry for X is marked as associated with IPn. The entry and the entry for X is marked as associated with IPn. The entry
is now resolved and processing can be restarted for IP addresses is now resolved and processing can be restarted for IP addresses
whose clientid4 matched XC but whose resolution had been deferred. whose clientid4 matched XC but whose resolution had been deferred.
o If the confirm value generated on IPn is not accepted on X, then X o If the confirm value generated on IPn is not accepted on X, then X
and IPn are distinct and the callback update will not be and IPn are distinct and the callback update will not be
confirmed. So we go on to the next IPn, until we run out of them. confirmed. So we go on to the next IPn, until we run out of them.
If it happens that we run out of potential matches, then we can
treat X as connected to a distinct server and then update and
confirm its callback parameters on that basis.
Note here that we may set a number of possible values for the
callback parameters to be used for XC, one for the possibility that X
is untrunked, and others for each potential match with an existing
IPn. Although there are multiple such updates at most one will be
confirmed and, if X is untrunked, its original callback parameters
will be put in effect by its SETCLIENID_CONFIRM.
The procedure above has made no explicit mention of the possibility The procedure above has made no explicit mention of the possibility
that server reboot can occur at any time. To address this that server reboot can occur at any time. To address this
possibility the client should periodically use the clientid4 XC in possibility the client should periodically use the clientid4 XC in
RENEW operations, directed to both the IP address X and the current RENEW operations, directed to both the IP address X and the current
lead IP address that is currently being tested for identity. lead IP address that is currently being tested for identity.
o When XC becomes invalid on X, the resolution process should be o When XC becomes invalid on X, the resolution process should be
terminated, subject to being redone later. Before redoing the terminated, subject to being redone later. Before redoing the
resolution, XC should be checked on all the lead IP addresses on resolution, XC should be checked on all the lead IP addresses on
skipping to change at page 21, line 23 skipping to change at page 24, line 51
lease time at the point of migration would not maintain the lease time at the point of migration would not maintain the
desired synchronization property. The leases would be desired synchronization property. The leases would be
synchronized until one of them was renewed, after which they would synchronized until one of them was renewed, after which they would
be unsynchronized again. be unsynchronized again.
To avoid client complexity, we need to have no more than one lease To avoid client complexity, we need to have no more than one lease
between a single client and a single server. This requires merger of between a single client and a single server. This requires merger of
leases since there is no real help from synchronizing them at a leases since there is no real help from synchronizing them at a
single instant. single instant.
For the uniform model, the destination server would simply merge For the uniform approach, the destination server would simply merge
leases as part of state transfer, since two leases with the same leases as part of state transfer, since two leases with the same
nfs_client_id4 values must be for the same client. nfs_client_id4 values must be for the same client.
We have made the following decisions as far as proposed normative We have made the following decisions as far as proposed normative
statements regarding for state merger. They reflect the facts that statements regarding for state merger. They reflect the facts that
we want to support fully migration support in the simplest way we want to support fully migration support in the simplest way
possible and that we can't say MUST since we have older clients and possible and that we can't say MUST since we have older clients and
servers to deal with. servers to deal with.
o Clients SHOULD use the uniform client-string model in order to get o Clients SHOULD use the uniform client-string approach in order to
good migration support. get good migration support.
o Servers SHOULD provide automatic lease merger during state o Servers SHOULD provide automatic lease merger during state
migration so that clients using the uniform id model get the migration so that clients using the uniform id approach get the
support automatically. support automatically.
If the clients and the servers obey the SHOULD's, having more than a If the clients and the servers obey the SHOULD's, having more than a
single lease for a given client-server pair will be a transient single lease for a given client-server pair will be a transient
situation, cleaned up as part of adapting to use of migrated state. situation, cleaned up as part of adapting to use of migrated state.
Since clients and servers will be a mixture of old and new and Since clients and servers will be a mixture of old and new and
because nothing is a MUST we have to ensure that no combination will because nothing is a MUST we have to ensure that no combination will
show worse behavior than is exhibited by current (i.e. old) clients show worse behavior than is exhibited by current (i.e. old) clients
and servers. and servers.
skipping to change at page 22, line 41 skipping to change at page 26, line 20
o If the text means to suggest that the same clientid4 must be used, o If the text means to suggest that the same clientid4 must be used,
the logic is not clear since the issue is not the same as for the logic is not clear since the issue is not the same as for
stateids of which there might be many. Adapting to the change of stateids of which there might be many. Adapting to the change of
a single clientid, as might happen as a part of lease migration, a single clientid, as might happen as a part of lease migration,
is relatively easy for the client. is relatively easy for the client.
We have decided to address this issue as follows, with the relevant We have decided to address this issue as follows, with the relevant
changes all reflected in Section 5.6. changes all reflected in Section 5.6.
o Make it clear that both clientid4 and nfs_client_id4 are to be o Make it clear that both clientid4 and nfs_client_id4 (including
transferred. both id string and boot verifier) are to be transferred.
o Indicate that the initial transfer will result in the same o Indicate that the initial transfer will result in the same
clientid4 after transfer but this is not guaranteed since there clientid4 after transfer but this is not guaranteed since there
may conflict with an existing clientid4 on the destination server may conflict with an existing clientid4 on the destination server
and because lease merger can result in a change of the clientid4. and because lease merger can result in a change of the clientid4.
5.4.2. Proposed changes: Callback re-establishment 5.4.2. Proposed changes: Callback re-establishment
The current definitive definition of the NFSv4.0 protocol [RFC3530], The current definitive definition of the NFSv4.0 protocol [RFC3530],
and the current pending draft of RFC3530bis [cur-v4.0-bis] both and the current pending draft of RFC3530bis [cur-v4.0-bis] both
skipping to change at page 24, line 40 skipping to change at page 28, line 14
distinct IP addresses, denoting separate entities. When trunking distinct IP addresses, denoting separate entities. When trunking
of server IP addresses is not a consideration, a client should of server IP addresses is not a consideration, a client should
keep track of (IP-address, clientid4) pairs, so that each pair is keep track of (IP-address, clientid4) pairs, so that each pair is
distinct. For a discussion of how to address the issue in the distinct. For a discussion of how to address the issue in the
face of possible trunking of server IP addresses, see Section 5.2. face of possible trunking of server IP addresses, see Section 5.2.
When a clientid4 is presented to a server and that clientid4 is When a clientid4 is presented to a server and that clientid4 is
not recognized, the server will reject the request with the error not recognized, the server will reject the request with the error
NFS4ERR_STALE_CLIENTID. This can occur for a number of reasons: NFS4ERR_STALE_CLIENTID. This can occur for a number of reasons:
* A server reboot causing loss of the server's knowledge of * A server reboot causing loss of the server's knowledge of the
client client
* Client error sending an incorrect clientid4 or valid clientid4 * Client error sending an incorrect clientid4 or valid clientid4
to the wrong server. to the wrong server.
* Loss of lease state due to lease expiration. * Loss of lease state due to lease expiration.
* Client or server error causing the server to believe that the * Client or server error causing the server to believe that the
client has rebooted (i.e. receiving a SETCLIENTID with an client has rebooted (i.e. receiving a SETCLIENTID with an
nfs_client_id4 which has a matching id and a non-matching nfs_client_id4 which has a matching id string and a non-
verifier. matching boot verifier).
* Migration of all state under the associated lease causes its * Migration of all state under the associated lease causes its
non-existence to be recognized on the source server. non-existence to be recognized on the source server.
* Merger of state under the associated lease with another lease * Merger of state under the associated lease with another lease
under a different clientid causes the clientid4 serving as the under a different clientid causes the clientid4 serving as the
source of the merge to cease being recognized on its server. source of the merge to cease being recognized on its server.
In the event of a server reboot, or loss of lease state due to In the event of a server reboot, or loss of lease state due to
lease expiration, the client must obtain a new clientid4 by use of lease expiration, the client must obtain a new clientid4 by use of
the SETCLIENTID operation and then proceed to any other necessary the SETCLIENTID operation and then proceed to any other necessary
recovery for the server reboot case (See the section entitled recovery for the server reboot case (See the section entitled
"Server Failure and Recovery"). In cases of server or client "Server Failure and Recovery"). In cases of server or client
error resulting in this error, use of SETCLIENTID to establish a error resulting in this error, use of SETCLIENTID to establish a
new lease is desirable as well. new lease is desirable as well.
In the last two cases, different recovery procedures are required. In the last two cases, different recovery procedures are required.
See Section 5.6 for details. Note that in cases in which there is See Section 5.6 for details. Note that in cases in which there is
any uncertainty about which sort of handling is applicable, the any uncertainty about which sort of handling is applicable, the
distinguishing characteristic is that in reboot-like cases, the distinguishing characteristic is that in reboot-like cases, the
clientid4 and all associated stateid cease to exist while in clientid4 and all associated stateids cease to exist while in
migration-related cases, the clientid4 ceases to exist while the migration-related cases, the clientid4 ceases to exist while the
stateids are still valid. stateids are still valid.
The client must also employ the SETCLIENTID operation when it The client must also employ the SETCLIENTID operation when it
receives a NFS4ERR_STALE_STATEID error using a stateid derived receives a NFS4ERR_STALE_STATEID error using a stateid derived
from its current clientid4, since this indicates a situation, such from its current clientid4, since this indicates a situation, such
as server reboot which has invalidated the existing clientid4 and as server reboot which has invalidated the existing clientid4 and
associated stateids (see the section entitled "lock-owner" for associated stateids (see the section entitled "lock-owner" for
details). details).
See the detailed descriptions of SETCLIENTID and See the detailed descriptions of SETCLIENTID and
SETCLIENTID_CONFIRM for a complete specification of the SETCLIENTID_CONFIRM for a complete specification of the
operations. operations.
5.5.3. Proposed changes: NFS4ERR_CLID_INUSE
It appears to be the intention that only a single authentication
flavor be used for client establishment between any client-server
pair. However:
o There is no explicit statement to this effect.
o The error that indicates an authentication flavor conflict has a
name which does not clarify this issue: NFS4ERR_CLID_INUSE.
o The definition of the error is also not very helpful: "The
SETCLIENTID operation has found that a client id is already in use
by another client".
As a result, servers exist which reject a SETCLIENTID simply because
there already exists a clientid for the same client, established
using a different IP address. Although this is generally understood
to be erroneous, such servers still exist and the spec should make
the correct behavior clear.
Although the error name cannot be changed, the following changes
should be made to avoid confusion:
o The definition of the error should be changed to read, "The
SETCLIENTID operation has found that the specified nfs_client_id4
was previously presented with a different authentication flavor
and that client instance currently holds an active lease."
o In the description of SETCLIENTID, the phrase "then the server
returns a NFS4ERR_CLID_INUSE error" should be expanded to read
"then the server returns a NFS4ERR_CLID_INUSE error, since use of
a single client with multiple principals is not allowed."
5.6. Migration, Replication and State (AS PROPOSED) 5.6. Migration, Replication and State (AS PROPOSED)
When responsibility for handling a given filesystem is transferred to When responsibility for handling a given filesystem is transferred to
a new server (migration) or the client chooses to use an alternate a new server (migration) or the client chooses to use an alternate
server (e.g., in response to server unresponsiveness) in the context server (e.g., in response to server unresponsiveness) in the context
of filesystem replication, the appropriate handling of state shared of filesystem replication, the appropriate handling of state shared
between the client and server (i.e., locks, leases, stateids, and between the client and server (i.e., locks, leases, stateids, and
client IDs) is as described below. The handling differs between client IDs) is as described below. The handling differs between
migration and replication. migration and replication.
skipping to change at page 26, line 31 skipping to change at page 30, line 40
If transferring stateids from server to server would result in a If transferring stateids from server to server would result in a
conflict for an existing stateid for the destination server with the conflict for an existing stateid for the destination server with the
existing client, transparent state migration MUST NOT happen for that existing client, transparent state migration MUST NOT happen for that
client. Servers participating in using transparent state migration client. Servers participating in using transparent state migration
should co-ordinate their stateid assignment policies to make this should co-ordinate their stateid assignment policies to make this
situation unlikely or impossible. The means by which this might be situation unlikely or impossible. The means by which this might be
done, like all of the inter-server interactions for migration, are done, like all of the inter-server interactions for migration, are
not specified by the NFS version 4.0 protocol. not specified by the NFS version 4.0 protocol.
Handling of clientid values is similar but not identical. The Handling of clientid values is similar but not identical. The
clientid4 and nfs_client_id4 information (id and verifier) will be clientid4 and nfs_client_id4 information (id string and boot
transferred with the rest of the state information and the verifier) will be transferred with the rest of the state information
destination server should use that information to determine and the destination server should use that information to determine
appropriate clientid4 handling. Although the destination server may appropriate clientid4 handling. Although the destination server may
make state stored under an existing lease available under the make state stored under an existing lease available under the
clientid4 used on the source server, the client should not assume clientid4 used on the source server, the client should not assume
that this is always so. In particular, that this is always so. In particular,
o If there is an existing lease with an nfs_client_id4 that matches o If there is an existing lease with an nfs_client_id4 that matches
a migrated lease (same id and verifier), the server SHOULD merge a migrated lease (same id string and boot verifier), the server
the two, making the union of the sets of stateids available under SHOULD merge the two, making the union of the sets of stateids
the clientid4 for the existing lease. As part of the lease available under the clientid4 for the existing lease. As part of
merger, the expiration time of the lease will reflect renewal done the lease merger, the expiration time of the lease will reflect
within either of the ancestor leases (and so will reflect the renewal done within either of the ancestor leases (and so will
latest of the renewals). reflect the latest of the renewals).
o If there is an existing lease with an nfs_client_id4 that o If there is an existing lease with an nfs_client_id4 that
partially matches a migrated lease (same id and a different partially matches a migrated lease (same id string and a different
verifier), the server MUST eliminate one of the two, possibly boot verifier), the server MUST eliminate one of the two, possibly
invalidating one of the ancestor clientid4's. Since verifiers are invalidating one of the ancestor clientid4's. Since boot
not ordered, the later lease renewal time will prevail. verifiers are not ordered, the later lease renewal time will
prevail.
When leases are not merged, the transfer of state should result in When leases are not merged, the transfer of state should result in
creation of a confirmed client record with empty callback information creation of a confirmed client record with empty callback information
but matching the {v, x, c} for the transferred client information. but matching the {v, x, c} for the transferred client information.
This should enable establishment of new callback information using This should enable establishment of new callback information using
SETCLIENTID and SETCLIENTID_CONFIRM. SETCLIENTID and SETCLIENTID_CONFIRM.
A client may determine the disposition of migrated state by using a A client may determine the disposition of migrated state by using a
stateid associated with the migrated state and in an operation on the stateid associated with the migrated state and in an operation on the
new server and using the associated clientid4 in a RENEW on the new new server and using the associated clientid4 in a RENEW on the new
server. server.
o If the stateid is not valid and an error NFS4ERR_BAD_STATEID is o If the stateid is not valid and an error NFS4ERR_BAD_STATEID is
received, either transparent state migration has not occurred or received, either transparent state migration has not occurred or
the state was purged due to verifier mismatch. the state was purged due to boot verifier mismatch.
o If the stateid is valid and an error NFS4ERR_STALE_CLIENTID is o If the stateid is valid and an error NFS4ERR_STALE_CLIENTID is
received on the RENEW, transparent state migration has occurred received on the RENEW, transparent state migration has occurred
and the lease has been merged with an existing lease on the and the lease has been merged with an existing lease on the
destination server. destination server.
o If the stateid is valid and the clientid4 is valid, the lease has o If the stateid is valid and the clientid4 is valid, the lease has
been transferred intact. been transferred intact.
Since responsibility for an entire filesystem is transferred with a Since responsibility for an entire filesystem is transferred with a
skipping to change at page 28, line 5 skipping to change at page 32, line 14
in the event of server restart. in the event of server restart.
When a lease is transferred to a new server (as opposed to being When a lease is transferred to a new server (as opposed to being
merged with a lease already on the new server), a client SHOULD re- merged with a lease already on the new server), a client SHOULD re-
establish new callback information with the new server as soon as establish new callback information with the new server as soon as
possible, according to sequences described in sections "Operation 35: possible, according to sequences described in sections "Operation 35:
SETCLIENTID - Negotiate Client ID" and "Operation 36: SETCLIENTID - Negotiate Client ID" and "Operation 36:
SETCLIENTID_CONFIRM - Confirm Client ID". This ensures that server SETCLIENTID_CONFIRM - Confirm Client ID". This ensures that server
operations are not blocked by the inability to recall delegations. operations are not blocked by the inability to recall delegations.
In those situation in which state has not been transferred, as shown
by a return of NFS4ERR_BAD_STATEID, the client may attempt to reclaim
the locks in order to take advantage of cases in which destination
server has set up a file-system-specific grace period in support of
the migration.
5.6.2. Replication and State 5.6.2. Replication and State
Since client switch-over in the case of replication is not under Since client switch-over in the case of replication is not under
server control, the handling of state is different. In this case, server control, the handling of state is different. In this case,
leases, stateids and client IDs do not have validity across a leases, stateids and client IDs do not have validity across a
transition from one server to another. The client must re-establish transition from one server to another. The client must re-establish
its locks on the new server. This can be compared to the re- its locks on the new server. This can be compared to the re-
establishment of locks by means of reclaim-type requests after a establishment of locks by means of reclaim-type requests after a
server reboot. The difference is that the server has no provision to server reboot. The difference is that the server has no provision to
distinguish requests reclaiming locks from those obtaining new locks distinguish requests reclaiming locks from those obtaining new locks
skipping to change at page 28, line 33 skipping to change at page 32, line 48
5.6.3. Notification of Migrated Lease 5.6.3. Notification of Migrated Lease
In the case of lease renewal, the client may not be submitting In the case of lease renewal, the client may not be submitting
requests for a filesystem that has been migrated to another server. requests for a filesystem that has been migrated to another server.
This can occur because of the implicit lease renewal mechanism. The This can occur because of the implicit lease renewal mechanism. The
client renews a lease containing state of multiple filesystems when client renews a lease containing state of multiple filesystems when
submitting a request to any one filesystem at the server. submitting a request to any one filesystem at the server.
In order for the client to schedule renewal of leases that may have In order for the client to schedule renewal of leases that may have
been relocated to the new server, the client must find out about been relocated to the new server, the client must find out about
lease relocation before those leases expire. To accomplish this, all lease relocation before those leases expire. Similarly, when
operations which implicitly renew leases for a client (such as OPEN, migration occurs but there has not been transparent state migration,
CLOSE, READ, WRITE, RENEW, LOCK, and others), will return the error the client needs to find out about the change soon enough to be able
NFS4ERR_LEASE_MOVED if responsibility for any of the leases to be to reclaim the lock within the destination server's grace period. To
renewed has been transferred to a new server. Note that when the accomplish this, all operations which implicitly renew leases for a
transfer of responsibility leaves remaining state for that lease on client (such as OPEN, CLOSE, READ, WRITE, RENEW, LOCK, and others),
the source server, the lease is renewed just as it would have been in will return the error NFS4ERR_LEASE_MOVED if responsibility for any
the NFS4ERR_OK case, despite returning the error. The transfer of of the leases to be renewed has been transferred to a new server.
responsibility happens when the server receives a Note that when the transfer of responsibility leaves remaining state
for that lease on the source server, the lease is renewed just as it
would have been in the NFS4ERR_OK case, despite returning the error.
The transfer of responsibility happens when the server receives a
GETATTR(fs_locations) from the client for each filesystem for which a GETATTR(fs_locations) from the client for each filesystem for which a
lease has been moved to a new server. Normally it does this after lease has been moved to a new server. Normally it does this after
receiving an NFS4ERR_MOVED for an access to the filesystem but the receiving an NFS4ERR_MOVED for an access to the filesystem but the
server is not required to verify that this happens in order to server is not required to verify that this happens in order to
terminate the return of NFS4ERR_LEASE_MOVED. By convention, the terminate the return of NFS4ERR_LEASE_MOVED. By convention, the
compounds containing GETATTR(fs_locations) SHOULD include an appended compounds containing GETATTR(fs_locations) SHOULD include an appended
RENEW operation to permit the server to identify the client getting RENEW operation to permit the server to identify the client getting
the information. the information.
Note that the NFS4ERR_LEASE_MOVED error is only required when Note that the NFS4ERR_LEASE_MOVED error is only required when
responsibility for at least one stateid has been transferred. In the responsibility for at least one stateid has been affected. In the
case of a null lease, where the only associated state is a clientid, case of a null lease, where the only associated state is a clientid,
no NFS4ERR_LEASE_MOVED error need be generated. no NFS4ERR_LEASE_MOVED error need be generated.
Upon receiving the NFS4ERR_LEASE_MOVED error, a client that supports Upon receiving the NFS4ERR_LEASE_MOVED error, a client that supports
filesystem migration MUST perform the necessary GETATTR operation for filesystem migration MUST perform the necessary GETATTR operation for
each of the filesystems containing state that have been migrated and each of the filesystems containing state that have been migrated and
so give the server evidence that it is aware of the migration of the so give the server evidence that it is aware of the migration of the
filesystem. Once the client has done this for all migrated filesystem. Once the client has done this for all migrated
filesystems on which the client holds state, the server MUST resume filesystems on which the client holds state, the server MUST resume
normal handling of stateful requests from that client. normal handling of stateful requests from that client.
skipping to change at page 29, line 36 skipping to change at page 34, line 7
fs's interrogated have been migrated while termination with fs's interrogated have been migrated while termination with
NFS4ERR_MOVED indicates that the filesystem getting the error has NFS4ERR_MOVED indicates that the filesystem getting the error has
migrated while those interrogated before it in the same COMPOUND have migrated while those interrogated before it in the same COMPOUND have
not. Those whose interrogation follows the error remain in an not. Those whose interrogation follows the error remain in an
uncertain state and can be interrogated by restarting the requests uncertain state and can be interrogated by restarting the requests
from after the point at which NFS4ERR_MOVED was returned or by from after the point at which NFS4ERR_MOVED was returned or by
issuing a new set of COMPOUND requests for the filesystems which issuing a new set of COMPOUND requests for the filesystems which
remain in an uncertain state. remain in an uncertain state.
Once the migrated filesystems have been found, all that is needed is Once the migrated filesystems have been found, all that is needed is
for client to give evidence to the server that it is aware of the for the client to give evidence to the server that it is aware of the
migrated status of filesystems found by this process, by migrated status of filesystems found by this process, by
interrogating the fs_locations attribute for an fh each of the interrogating the fs_locations attribute for an fh within each of the
migrated filesystems. The client can do this building and issuing migrated filesystems. The client can do this by building and issuing
one or more COMPOUND requests, each of which consists of a set of one or more COMPOUND requests, each of which consists of a set of
PUTFH operations, each followed by a GETATTR of the fs_locations PUTFH operations, each followed by a GETATTR of the fs_locations
attribute. A RENEW follows to help tie the operations to the lease attribute. A RENEW follows to help tie the operations to the lease
returning NFS4ERR_LEASE_MOVED. Once the client has done this for all returning NFS4ERR_LEASE_MOVED. Once the client has done this for all
migrated filesystems on which the client holds state, the server will migrated filesystems on which the client holds state, the server will
resume normal handling of stateful requests from that client. resume normal handling of stateful requests from that client.
In order to support legacy clients that do not handle the In order to support legacy clients that do not handle the
NFS4ERR_LEASE_MOVED error correctly, the server SHOULD time out after NFS4ERR_LEASE_MOVED error correctly, the server SHOULD time out after
a wait of at least two lease periods, at which time it will resume a wait of at least two lease periods, at which time it will resume
skipping to change at page 31, line 14 skipping to change at page 35, line 33
previously fetched (on the source server). previously fetched (on the source server).
In the case in which lease merger occurs as part of state transfer, In the case in which lease merger occurs as part of state transfer,
the lease_time attribute of the destination lease remains in effect. the lease_time attribute of the destination lease remains in effect.
The client can simply renew that lease with its existing lease_time The client can simply renew that lease with its existing lease_time
attribute. State in the source lease is renewed at the time of attribute. State in the source lease is renewed at the time of
transfer so that it cannot expire, as long as the destination lease transfer so that it cannot expire, as long as the destination lease
is appropriately renewed. is appropriately renewed.
If state has not been transferred transparently (i.e., the client If state has not been transferred transparently (i.e., the client
sees a real or simulated server reboot), the client should fetch the need to reclaim or re-obtain its locks), the client should fetch the
value of lease_time on the new (i.e., destination) server, and use it value of lease_time on the new (i.e., destination) server, and use it
for subsequent locking requests. However the server must respect a for subsequent locking requests. However the server must respect a
grace period at least as long as the lease_time on the source server, grace period at least as long as the lease_time on the source server,
in order to ensure that clients have ample time to reclaim their in order to ensure that clients have ample time to reclaim their
locks before potentially conflicting non-reclaimed locks are granted. locks before potentially conflicting non-reclaimed locks are granted.
The means by which the new server obtains the value of lease_time on The means by which the new server obtains the value of lease_time on
the old server is left to the server implementations. It is not the old server is left to the server implementations. It is not
specified by the NFS version 4.0 protocol. specified by the NFS version 4.0 protocol.
6. Results of proposed changes for NFSv4.0 6. Results of proposed changes for NFSv4.0
The purpose of this section is to examine the troubling results The purpose of this section is to examine the troubling results
reported in Section 3.1. We will look at the scenarios as they would reported in Section 3.1. We will look at the scenarios as they would
be handled within the proposal. be handled within the proposal.
Because the choice of uniform vs. non-uniform nfs_client_id4 id Because the choice of uniform vs. non-uniform nfs_client_id4 id
strings is a "SHOULD" in these cases, we will designate clients that strings is a "SHOULD" in these cases, we will designate clients that
follow this recommendation by SHOULD-UF-CID. follow this recommendation by SHOULD-UF-CID.
We will also have to take account of the various merger-related We will also have to take account of any merger-related "SHOULD"
"SHOULD" clauses to better understand how they have addressed the clauses to better understand how they have addressed the issues seen.
issues seen, we abbreviate these (collectively known as "SHOULD- We abbreviate as follows:
merges") as follows:
o SHOULD-SVR-AM refers to the server obeying the SHOULD which o SHOULD-SVR-AM refers to the server obeying the SHOULD which
RECOMMENDS that they merge leases with identical nfs_client_id4 id RECOMMENDS that they merge leases with identical nfs_client_id4 id
strings and verifiers. strings and boot verifiers.
6.1. Results: Failure to free migrated state on client reboot 6.1. Results: Failure to free migrated state on client reboot
Let's look at the troublesome situation cited in Section 3.1.1. We Let's look at the troublesome situation cited in Section 3.1.1. We
have already seen what happens when SHOULD-UF-CID does not hold. Now have already seen what happens when SHOULD-UF-CID does not hold. Now
let's look at the situation in which SHOULD-UF-CID holds, whether let's look at the situation in which SHOULD-UF-CID holds, whether
SHOULD-SVR-AM is in effect or not. SHOULD-SVR-AM is in effect or not.
o A client C establishes a clientid4 C1 with server ABC specifying o A client C establishes a clientid4 C1 with server ABC specifying
an nfs_client_id4 with "id" value "C" and verifier 0x111. an nfs_client_id4 with id string value "C" and boot verifier
0x111.
o The client begins to access files in filesystem F on server ABC, o The client begins to access files in filesystem F on server ABC,
resulting in generating stateids S1, S2, etc. under the lease for resulting in generating stateids S1, S2, etc. under the lease for
clientid C1. It may also access files on other filesystems on the clientid C1. It may also access files on other filesystems on the
same server. same server.
o The filesystem is migrated from ABC to server XYZ. When o The filesystem is migrated from ABC to server XYZ. When
transparent state migration is in effect, stateids S1 and S2 and transparent state migration is in effect, stateids S1 and S2 and
lease {0x111, "C", C1} are now available for use by client C at lease {0x111, "C", C1} are now available for use by client C at
server XYZ. So far, so good. server XYZ. So far, so good.
o Client C reboots and attempts to access data on server XYZ, o Client C reboots and attempts to access data on server XYZ,
whether in filesystem F or another. It does a SETCLIENID with an whether in filesystem F or another. It does a SETCLIENID with an
nfs_client_id4 with "id" value "C" and verifier 0x112. The state nfs_client_id4 with id string value "C" and boot verifier 0x112.
associated with lease {0x111, "C", C1} is deleted as part of The state associated with lease {0x111, "C", C1} is deleted as
creating {0x112, "C", C2}. No problem. part of creating {0x112, "C", C2}. No problem.
The correctness signature for this issue is The correctness signature for this issue is
SHOULD-UF-CID SHOULD-UF-CID
so if you have clients and servers that obey the SHOULD clauses, the so if you have clients and servers that obey the SHOULD clauses, the
problem is gone regardless of the choice on the MAY. problem is gone regardless of the choice on the MAY.
6.2. Results: Server reboots resulting in confused lease situation 6.2. Results: Server reboots resulting in confused lease situation
Now let's consider the scenario given in Section 3.1.2. We have Now let's consider the scenario given in Section 3.1.2. We have
already seen what happens when SHOULD-UF-CID does not hold . Now already seen what happens when SHOULD-UF-CID does not hold . Now
let's look at the situation in which SHOULD-UF-CID holds and SHOULD- let's look at the situation in which SHOULD-UF-CID holds and SHOULD-
SVR-AM holds as well. SVR-AM holds as well.
o Client C talks to server ABC using an nfs_client_id4 id like o Client C talks to server ABC using an nfs_client_id4 id string
"C-ABC" and verifier v1. As a result a lease with clientid4 c.i such as "C-ABC" and boot verifier v1. As a result a lease with
established: {v1, "C-ABC", c.i}. clientid4 c.i established: {v1, "C-ABC", c.i}.
o fs_a1 migrates from server ABC to server XYZ along with its state. o fs_a1 migrates from server ABC to server XYZ along with its state.
Now server XYZ also has a lease: {v1, "C-ABC", c.i} Now server XYZ also has a lease: {v1, "C-ABC", c.i}
o Server ABC reboots. o Server ABC reboots.
o Client C talks to server ABC using an nfs_client_id4 id like o Client C talks to server ABC using an nfs_client_id4 id string
"C-ABC" and verifier v1. As a result a lease with clientid4 c.j such as "C-ABC" and boot verifier v1. As a result a lease with
established: {v1, "C-ABC", c.j}. clientid4 c.j established: {v1, "C-ABC", c.j}.
o fs_a2 migrates from server ABC to server XYZ. As part of o fs_a2 migrates from server ABC to server XYZ. As part of
migration the incoming lease is seen to denote same Nfs_client_id4 migration the incoming lease is seen to denote same Nfs_client_id4
and so is merged with {v1, "C-ABC, c.i}. and so is merged with {v1, "C-ABC, c.i}.
o Now server XYZ has only one lease that matches {v1, "C_ABC", *}, o Now server XYZ has only one lease that matches {v1, "C_ABC", *},
so the problem is solved so the problem is solved
Now let's consider the same scenario in the situation in which Now let's consider the same scenario in the situation in which
SHOULD-UF-CID holds and SHOULD-SVR-AM holds as well. SHOULD-UF-CID holds and SHOULD-SVR-AM holds as well.
o Client C talks to server ABC using an nfs_client_id4 id like "C" o Client C talks to server ABC using an nfs_client_id4 id string "C"
and verifier v1. As a result a lease with clientid4 c.i is and boot verifier v1. As a result a lease with clientid4 c.i is
established: {v1, "C", c.i}. established: {v1, "C", c.i}.
o fs_a1 migrates from server ABC to server XYZ along with its state. o fs_a1 migrates from server ABC to server XYZ along with its state.
Now XYZ also has a lease: {v1, "C", c.i} Now XYZ also has a lease: {v1, "C", c.i}
o Server ABC reboots. o Server ABC reboots.
o Client C talks to server ABC using an nfs_client_id4 id like "C" o Client C talks to server ABC using an nfs_client_id4 id string "C"
and verifier v1. As a result a lease with clientid4 c.j is and boot verifier v1. As a result a lease with clientid4 c.j is
established: {v1, "C", c.j}. established: {v1, "C", c.j}.
o fs_a2 migrates from server ABC to server XYZ. As part of o fs_a2 migrates from server ABC to server XYZ. As part of
migration the incoming lease is seen to denote the same migration the incoming lease is seen to denote the same
nfs_client_id4 and so is merged with {v1, "C", c.i}. nfs_client_id4 and so is merged with {v1, "C", c.i}.
o Now server XYZ has only one lease that matches {v1, "C", *}, so o Now server XYZ has only one lease that matches {v1, "C", *}, so
the problem is solved the problem is solved
The correctness signature for this issue is The correctness signature for this issue is
skipping to change at page 34, line 13 skipping to change at page 38, line 29
have up to m clientid's, which we will call Cxy for server Sy. have up to m clientid's, which we will call Cxy for server Sy.
o Now assume that for load-balancing or other operational reasons, o Now assume that for load-balancing or other operational reasons,
numbers of filesystems are migrated among the servers. As a numbers of filesystems are migrated among the servers. As a
result, depending on how this handled, the number of clientids may result, depending on how this handled, the number of clientids may
explode. See below. explode. See below.
Now look what will happen under various scenarios: Now look what will happen under various scenarios:
o We have previously (in Section 3.1.3) looked at this in case of o We have previously (in Section 3.1.3) looked at this in case of
client following the non-uniform client-string model. In that client following the non-uniform client-string approach. In that
case, each client-server pair could have up to m clientid's and case, each client-server pair could have up to m clientid's and
each client will have up to m**2 clientids. If we add the each client will have up to m**2 clientids. If we add the
possibility of server reboot, the only bound on a client's possibility of server reboot, the only bound on a client's
clientid count is L. clientid count is L.
o If we look at this in the SHOULD-UF-CID case in which the SHOULD- o If we look at this in the SHOULD-UF-CID case in which the SHOULD-
SVR_AM condition holds, the situation is no different. Although SVR_AM condition holds, the situation is no different. Although
the server has the client identity information that could enable the server has the client identity information that could enable
same-client-same-server leases to be combined, it does not do so. same-client-same-server leases to be combined, it does not do so.
We still have up to L clientid's per client. We still have up to L clientid's per client.
skipping to change at page 34, line 43 skipping to change at page 39, line 12
so if you have clients and servers that obey the SHOULD clauses, the so if you have clients and servers that obey the SHOULD clauses, the
problem is gone regardless of the choice on the MAY. problem is gone regardless of the choice on the MAY.
6.4. Result summary 6.4. Result summary
We have seen that (SHOULD-SVR-AM & SHOULD-UF-CID) are sufficient to We have seen that (SHOULD-SVR-AM & SHOULD-UF-CID) are sufficient to
solve the problems people have experienced. solve the problems people have experienced.
7. Issues for NFSv4.1 7. Issues for NFSv4.1
Because NFSv4.1 includes the uniform client-string model, addressing Because NFSv4.1 embraces the uniform client-string approach,
migration issues is simpler. In the terms of Section 6, we already addressing migration issues is simpler. In the terms of Section 6,
have SHOULD-UF-CID, for NFSv4.1, as advised by section 2.4 of we already have SHOULD-UF-CID, for NFSv4.1, as advised by section 2.4
[RFC5661], simplifying the work to be done. of [RFC5661], simplifying the work to be done.
Nevertheless, there are some issues that will have to be addressed. Nevertheless, there are some issues that will have to be addressed.
Some examples:
For example, the other necessary part of addressing migration issues, o The other necessary part of addressing migration issues, which we
which we call above SHOULD-SVR-AM, is not currently addressed by call above SHOULD-SVR-AM, is not currently addressed by NFSv4.1
NFSv4.1 and it needs to be. and changes need to be made to make it clear that state needs to
be appropriately merged as part of migration, to avoid multiple
clientids between a client-server pair.
8. Security Considerations o There needs to be some clarification of how migration, and
particularly transparent state migration, should interact with
pNFS layouts.
o The current discussion (in [RFC5661]), of the possibility of
server_owner changes is incomplete and confusing.
Discussion of how to resolve these issues will appear in the sections
below.
7.1. Addressing state merger in NFSv4.1
The existing treatment of state transfer in [RFC5661], has similar
problems to that in [RFC3530] in that it assumes that the state for
multiple fs's on different servers will not be merged to so that it
appears under a single common clientid. We've already seen the
reasons that this is a problem, with regard to NFSv4.0.
Although we don't have the problems stemming from the non-uniform
client-string approach, there are a number of complexities in the
existing treatment of state management in the section entitled "Lock
State and File System Transitions" in [RFC5661] that make this non-
trivial to address:
o Migration is currently treated together with other sorts of file
system transitions including transitioning between replicas
without any NFS4ERR_MOVED errors.
o There is separate handling and discussion of the cases of matching
and non-matching server scopes.
o In the case of matching server scopes, the text calls for an
impossible degree of transparency.
o In the case of non-matching server scopes, the text does not
mention transparent state migration at all, resulting in a
functional regression from NFSV4.0
7.2. Addressing pNFS relationship with migration
This is made difficult because, within the PNFS framework, migration
might mean any of several things:
o Transfer of the MDS, leaving DS's alone.
This would be minimally disruptive to those using layouts but
would a require the pNFS control protocol to support the DS being
directed to a new MDS.
o Transfer of a DS, leaving everything else in place.
Such a transfer can be handled without using migration at all.
The server can recall/revoke layouts, as appropriate.
o Transfer of the file system to a new file system with both MDS and
DS's moving.
In such a transfer, an entirely different set of DS's will be at
the target location. There may even be no pNFS support on the
destination FS at all.
Migration needs to support both the first and last of these models.
7.3. Addressing server owner changes in NFSv4.1
Section 2.10.5 of [RFC5661] states the following.
The client should be prepared for the possibility that
eir_server_owner values may be different on subsequent EXCHANGE_ID
requests made to the same network address, as a result of various
sorts of reconfiguration events. When this happens and the
changes result in the invalidation of previously valid forms of
trunking, the client should cease to use those forms, either by
dropping connections or by adding sessions. For a discussion of
lock reclaim as it relates to such reconfiguration events, see
Section 8.4.2.1.
While this paragraph is literally true in that such reconfiguration
events can happen and clients have to deal with them, it is confusing
in that it can be read as suggesting that clients have to deal with
them without disruption, which in general is impossible.
A clearer alternative would be:
It is always possible that, as a result of various sorts of
reconfiguration events, eir_server_scope and eir_server_owner
values may be different on subsequent EXCHANGE_ID requests made to
the same network address.
In most cases such reconfiguration events will be disruptive and
indicate that an IP address formerly connected to one server is
now connected to an entirely different one.
Some guidelines on client handling of such situations follow:
* When eir_server_scope changes, the client has no assurance that
any id's it obtained previously (e.g. file handles) can be
validly used on the new server, and, even if the new server
accepts them, there is no assurance that this is not due to
accident. Thus it is best to treat all such state as lost/
stale although a client may assume that the probability of
inadvertent acceptance is low and treat this situation as
within the next case.
* When eir_server_scope remains the same and
eir_server_owner.so_major_id changes, the client can use
filehandles it has and attempt reclaims. It may find that
these are now stale but if NFS4ERR_STALE is not received, he
can proceed to reclaim his opens.
* When eir_server_scope and eir_server_owner.so_major_id remain
the same, the client has to use the now-current values of
eir_server-owner.so_minor_id in deciding on appropriate forms
of trunking.
8. Lock State and File System Transitions (AS PROPOSED)
In dealing with file system transitions, the client needs to handle
cases in which the two servers have cooperated in state management
and cases in which they have not.
The primary means by which a client finds out about state management
co-operation is by comparing eir_server_scope values returned by each
server. If the scope values do not match, then any co-operation of
the servers in state management, is limited to transferring state in
event of migration and making arrangements for the safe reclamation
of locking state. If the scope values match, then this indicates the
servers have cooperated in assigning client IDs and stateids to the
point that the same id will not refer to different things on
different servers. Servers may reject client IDs that refer to state
they do not know about. See the section entitled "Server Scope" for
more information about the use of server scope.
How the client needs to deal with locking state with regard to these
situations will depend upon:
o The type of file system transition occurring.
o The type of state involved (e.g. layout state may sometimes be
handled differently).
o The specific level of state handling co-ordination between the two
servers for the specific transition.
We will divide the basic description of these possibilities into
three sections
o In Section 8.1, we will discuss handling specific to the case of
matching server scopes.
o In Section 8.2, we will discuss handling specific to the case of
non-matching server scopes.
o In Section 8.3, we will discuss issues relating to handling common
to both cases.
8.1. File System Transitions with Matching Server Scopes
In the case of migration, the servers involved in the migration of a
file system SHOULD transfer all server state relevant to the
migrating file system from the original to the new server. When this
is done, it needs to be done in a way that is maximally transparent
to the client in that all stateids used by the client to access state
on the filesystem in question can be used on the new server, albeit
possibly under different client IDs.
When layouts are active for a migrated file system, layout state
SHOULD be included as part of the state transferred. Even if it is
the case that there are circumstances preventing the layout from
being supported on the new server, this should be dealt with by
recalling layouts either before or after the transition. Where this
cannot be done, layout revocation is possible but any such revocation
should appear to the client just as any other layout revocation
would.
With replication, such a degree of common state is typically not the
case. Clients, however, should use the information provided by the
eir_server_scope returned by EXCHANGE_ID (as modified by the
validation procedures described in the section entitled "Server
Scope") to determine whether such sharing may be in effect in non-
migration cases, rather than making assumptions based solely on the
reason for the transition.
This state transfer will reduce disruption to the client when a file
system transition occurs. If the servers are successful in
transferring all state, the client can access existing stateids,
using either existing or new sessions between the client and the new
server instance. If the server accepts such a transferred stateid as
valid, then the client may use that stateid to access the same state
that it represented on the old server.
When the two servers belong to the same server scope, it does not
mean that when dealing with the transition, the client will not have
to reclaim or otherwise reobtain state. However, it does mean that
the client may proceed using its current stateids when communicating
with the new server, and the new server will either recognize the
stateids as valid or reject them, in which case locking state must be
reobtained by the client.
File systems cooperating in state management may actually share state
or simply divide the identifier space so as to recognize (and reject
as stale) each other's stateids and client IDs. Servers that do
share state may not do so under all conditions or at all times. If
the server cannot be sure when accepting a stateid that it reflects
the locks the client was given, the server must treat the state as
stale and report it as such to the client.
8.2. File System Transitions with Non-Matching Server Scopes
When the two file system instances are on servers that do not share a
server scope value, the client must establish a new client ID on the
destination, if it does not have one already, to obtain access to its
locks. Depending on the type of file system transition and
facilities provided by the server, it may re-establish its connection
to locking and layout state in a number of ways.
In the case of migration, the servers may have transferred stateids,
making it possible for the client to access his state on the new
server, simply by using the existing stateid. The server may
transfer all state or a subset and the client can use TEST_STATEID to
determine what state has been transferred and what needs to be
reclaimed or otherwise reobtained as described in Section 8.3.
Lock reclaim may be used by the client for any sort of file system
transition, but the server is not required to support it in any
particular case.
Note that in this case, lock reclaim may be attempted even when the
servers involved in the transfer have different server scope values
(see Section 8.4.2.1 for the contrary case of reclaim after server
reboot). Servers with different server scope values may cooperate to
allow reclaim for locks associated with the transfer of a file system
even if they do not cooperate sufficiently to share a server scope.
8.3. FS Transitions Involving Reobtaining Locking State
In either case, when actual locks are not known to be maintained, the
destination server may establish a grace period specific to the given
file system, with non-reclaim locks being rejected for that file
system, even though normal locks are being granted for other file
systems. Clients should not infer the absence of a grace period for
file systems being transitioned to a server from responses to
requests for other file systems.
In the case of lock reclamation for a given file system after a file
system transition, edge conditions can arise similar to those for
reclaim after server restart (although in the case of the planned
state transfer associated with migration, these can be avoided by
securely recording lock state as part of state migration). Unless
the destination server can guarantee that locks will not be
incorrectly granted, the destination server should not allow lock
reclaims and should avoid establishing a grace period.
Once all locks have been reclaimed, or there were no locks to
reclaim, the client indicates that there are no more reclaims to be
done for the file system in question by sending a RECLAIM_COMPLETE
operation with the rca_one_fs parameter set to true. Once this has
been done, non-reclaim locking operations may be done, and any
subsequent request to do a reclaim will be rejected with the error
NFS4ERR_NO_GRACE.
Information about client identity may be propagated between servers
in the form of a client_owner4 and associated verifiers, under the
assumption that the client presents the same values to all the
servers with which it deals.
Servers are encouraged to provide facilities to allow locks to be
reclaimed on the new server after a file system transition. Often,
however, in cases in which the two servers do not share a server
scope value, such facilities may not be available and the client
should be prepared to re-obtain locks, even though it is possible
that the client may have its LOCK or OPEN request denied due to a
conflicting lock.
Layouts may be reobtained when necessary even without special
facilities for lock reclamation. However, the client MUST NOT depend
on being able to obtain such layout since pNFS or the desired mapping
type might not be supported on the new server.
The consequences of having no facilities available to reclaim locks
on the new server will depend on the type of environment. In some
environments, such as the transition between read-only file systems,
such denial of locks should not pose large difficulties in practice.
When an attempt to re-establish a lock on a new server is denied, the
client should treat the situation as if its original lock had been
revoked. Note that when the lock is granted, the client cannot
assume that no conflicting lock could have been granted in the
interim. Where change attribute continuity is present, the client
may check the change attribute to check for unwanted file
modifications. Where even this is not available, and the file system
is not read-only, a client may reasonably treat all pending locks as
having been revoked.
9. Security Considerations
The current definitive definition of the NFSv4.0 protocol [RFC3530], The current definitive definition of the NFSv4.0 protocol [RFC3530],
and the current pending draft of RFC3530bis [cur-v4.0-bis] both and the current pending draft of RFC3530bis [cur-v4.0-bis] both
agree. The section entitled "Security Considerations" encourages agree. The section entitled "Security Considerations" encourages
that clients protect the integrity of the SECINFO operation, any that clients protect the integrity of the SECINFO operation, any
GETATTR operation for the fs_locations attribute, and the operations GETATTR operation for the fs_locations attribute, and the operations
SETCLIENTID/SETCLIENTID_CONFIRM. A migration recovery event can use SETCLIENTID/SETCLIENTID_CONFIRM. A migration recovery event can use
any or all of these operations. We do not recommend any change here. any or all of these operations. We do not recommend any change here.
9. IANA Considerations 10. IANA Considerations
This document does not require actions by IANA. This document does not require actions by IANA.
10. Acknowledgements 11. Acknowledgements
The editor and authors of this document gratefully acknowledge the The editor and authors of this document gratefully acknowledge the
contributions of Trond Myklebust of NetApp and Robert Thurlow of contributions of Trond Myklebust of NetApp and Robert Thurlow of
Oracle. We also thank Tom Haynes of NetApp and Spencer Shepler of Oracle. We also thank Tom Haynes of NetApp and Spencer Shepler of
Microsoft for their guidance and suggestions. Microsoft for their guidance and suggestions.
Special thanks go to members of the Oracle Solaris NFS team, Special thanks go to members of the Oracle Solaris NFS team,
especially Rick Mesta and James Wahlig, for their work implementing especially Rick Mesta and James Wahlig, for their work implementing
an NFSv4.0 migration prototype and identifying many of the issues an NFSv4.0 migration prototype and identifying many of the issues
documented here. documented here.
11. References 12. References
11.1. Normative References 12.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., [RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Beame, C., Eisler, M., and D. Noveck, "Network File System Beame, C., Eisler, M., and D. Noveck, "Network File System
(NFS) version 4 Protocol", RFC 3530, April 2003. (NFS) version 4 Protocol", RFC 3530, April 2003.
[RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File
System (NFS) Version 4 Minor Version 1 Protocol", System (NFS) Version 4 Minor Version 1 Protocol",
RFC 5661, January 2010. RFC 5661, January 2010.
11.2. Informative References 12.2. Informative References
[cur-v4.0-bis] [cur-v4.0-bis]
Haynes, T., Ed. and D. Noveck, Ed., "Network File System Haynes, T., Ed. and D. Noveck, Ed., "Network File System
(NFS) Version 4 Protocol", 2011, <http://www.ietf.org/id/ (NFS) Version 4 Protocol", 2011, <http://www.ietf.org/id/
draft-ietf-nfsv4-rfc3530bis-17.txt>. draft-ietf-nfsv4-rfc3530bis-18.txt>.
Work in progress. Work in progress.
Authors' Addresses Authors' Addresses
David Noveck (editor) David Noveck (editor)
EMC Corporation EMC Corporation
228 South Street 228 South Street
Hopkinton, MA 01748 Hopkinton, MA 01748
US US
 End of changes. 93 change blocks. 
224 lines changed or deleted 755 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/