draft-ietf-nfsv4-migration-issues-12.txt   draft-ietf-nfsv4-migration-issues-13.txt 
NFSv4 D. Noveck, Ed. NFSv4 D. Noveck, Ed.
Internet-Draft NetApp Internet-Draft NetApp
Intended status: Informational P. Shivam Intended status: Informational P. Shivam
Expires: September 30, 2017 C. Lever Expires: November 17, 2017 C. Lever
B. Baker B. Baker
ORACLE ORACLE
March 29, 2017 May 16, 2017
NFSv4 migration: Implementation Experience and Specification Issues NFSv4 Migration and Trunking: Implementation and Specification Issues
draft-ietf-nfsv4-migration-issues-12 draft-ietf-nfsv4-migration-issues-13
Abstract Abstract
The migration feature of NFSv4 provides for moving responsibility for This document discusses a range of implementation and specification
a single filesystem from one server to another, without disruption to issues concerning features related to the use of location-related
clients. A number of problems in the specification of this feature attributes in NFSv4. These include migration, which transfers
in NFSv4.0 were resolved by the publication of RFC 7931. In responsibility for a file system from one server to another, and
addition, there are specification issues to be resolved with regard trunking which deals with the discovery and control of the set of
to the NFSv4.1 version of this feature which are discussed in this network addresses to use to access a file system. The focus of the
document. discussion, which relates to multiple minor versions, is on providing
appropriate clarification and correction of existing specifications.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 30, 2017. This Internet-Draft will expire on November 17, 2017.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Language . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. NFSv4.0 Issues and Their Resolution . . . . . . . . . . . . . 3 2.1. Requirements Language . . . . . . . . . . . . . . . . . . 4
3.1. NFSv4.0 Issues . . . . . . . . . . . . . . . . . . . . . 3 2.2. Use of Normative Terms . . . . . . . . . . . . . . . . . 4
3.2. Resolution of NFSv4.0 Protocol Difficulties . . . . . . . 4 2.3. Terminology Used in this Document . . . . . . . . . . . . 5
4. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . 5 3. Multi-version Issues and Their Resolution . . . . . . . . . . 6
4.1. Issues to Address for NFSv4.1 . . . . . . . . . . . . . . 5 3.1. Multi-Version Issues . . . . . . . . . . . . . . . . . . 6
4.1.1. Addressing state merger in NFSv4.1 . . . . . . . . . 6 3.2. Resolution of Multi-Version Issues . . . . . . . . . . . 7
4.1.2. Addressing pNFS relationship with migration . . . . . 7 3.2.1. Providing Trunking Discovery . . . . . . . . . . . . 7
4.1.3. Addressing server_owner changes in NFSv4.1 . . . . . 7 3.2.2. Interaction of Trunking and Migration . . . . . . . . 9
4.1.4. Addressing Confirmation Status of Migrated 4. NFSv4.0 Issues . . . . . . . . . . . . . . . . . . . . . . . 11
Client IDs in NFSv4.1 . . . . . . . . . . . . . . . . 8 4.1. Core NFSv4.0 Migration Issues . . . . . . . . . . . . . . 11
4.1.5. Addressing Session Migration in NFSv4.1 . . . . . . . 9 4.2. Resolution of Core Migration Protocol Difficulties in
4.2. Possible Resolutions for NFSv4.1 Issues . . . . . . . . . 9 NFSv4.0 . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.2.1. Server Responsibilities in Effecting Transparent 4.3. Additional NFSv4.0 Issues . . . . . . . . . . . . . . . . 12
State Migration . . . . . . . . . . . . . . . . . . . 10 4.4. Resolution of Additional NFSv4.0 Issues . . . . . . . . . 12
4.2.2. Determining Initial Migration Status in NFSv4.1 . . . 11 5. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . 14
4.2.3. Client Response to Migration in NFSv4.1 . . . . . . . 13 5.1. Issues to Address for NFSv4.1 . . . . . . . . . . . . . . 14
4.2.4. Dealing with Multiple Location Entries . . . . . . . 13 5.1.1. Addressing State Merger in NFSv4.1 . . . . . . . . . 15
4.2.5. Client Recovery from Migration Events . . . . . . . . 15 5.1.2. Addressing pNFS Relationship with Migration . . . . . 15
4.2.6. The Migration Discovery Process . . . . . . . . . . . 18 5.1.3. Addressing Server_owner Changes in NFSv4.1 . . . . . 16
4.2.7. Synchronzing Session Transfer . . . . . . . . . . . . 19 5.1.4. Addressing Confirmation Status of Migrated
4.2.8. Migration and pNFS . . . . . . . . . . . . . . . . . 22 Client IDs in NFSv4.1 . . . . . . . . . . . . . . . . 17
5. Security Considerations . . . . . . . . . . . . . . . . . . . 23 5.1.5. Addressing Changes in Trunking Configuration . . . . 18
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 5.1.6. Addressing Session Migration in NFSv4.1 . . . . . . . 18
7. Normative References . . . . . . . . . . . . . . . . . . . . 23 5.2. Possible Resolutions for NFSv4.1 Protocol Issues . . . . 19
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 23 5.2.1. Client ID Confirmation Issues . . . . . . . . . . . . 19
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 5.2.2. Dealing with Multiple Location Entries . . . . . . . 20
5.2.3. Migration and pNFS . . . . . . . . . . . . . . . . . 22
5.3. Defining Server Responsibilities for NFSv4.1 . . . . . . 23
5.3.1. Server Responsibilities in Effecting Transparent
State Migration . . . . . . . . . . . . . . . . . . . 24
5.3.2. Synchronizing Session Transfer . . . . . . . . . . . 25
5.4. Defining Client Responsibilities for NFSv4.1 . . . . . . 27
5.4.1. Client Recovery from Migration Events . . . . . . . . 28
5.4.2. The Migration Discovery Process . . . . . . . . . . . 30
5.4.3. Overview of Client Response to NFS4ERR_MOVED . . . . 31
5.4.4. Obtaining Access to Sessions and State after
Migration . . . . . . . . . . . . . . . . . . . . . . 33
5.4.5. Obtaining Access to Sessions and State after Network
Address Transfer . . . . . . . . . . . . . . . . . . 35
5.5. Resolution of NFSv4.1 Issues . . . . . . . . . . . . . . 35
6. Security Considerations . . . . . . . . . . . . . . . . . . . 38
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 38
8. Normative References . . . . . . . . . . . . . . . . . . . . 38
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 38
Appendix B. History of this Document . . . . . . . . . . . . . . 39
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 40
1. Introduction 1. Introduction
This document. which deals with existing issues/problems in This is an informational document that discusses a number of related
issues in multiple versions of NFSv4.
Many of these relate to the migration feature of NFSv4, which
provides for moving responsibility for a single filesystem from one
server to another, without disruption to clients. A number of
problems in the specification of this feature in NFSv4.0 were
resolved by the publication of [RFC7931], which added trunking
detection to NFSV4.0. However, NFSv4.0 remains without an
appropriate discussion of trunking discovery, which has many
important connections with migration. As a result, NFSv4.0 requires
clarification of how the client is to respond to changes in the
trunking arrangements to use, both when migration occurs and when it
does not.
In addition, there are specification issues to be resolved with
regard to the NFSv4.1 version of these features which are discussed
in this document.
All of the issues discussed relate to the handling and interpretation
of the location-related attributes fs_locations and fs_locations_info
and to the proper client and server handling of changes in the values
of these attributes
These issues are all related to the protocol features for effecting
file system migration, or to trunking discovery but it is not
possible to treat each of these features in isolation. These
features are inherently linked because migration needs to deal with
the possibility of multiple server addresses in location attributes
and because location attributes, which provide trunking-related
information, may change, which might or might not involve migration.
2. Language
2.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
2.2. Use of Normative Terms
This document, which deals with existing issues/problems in
standards-track documents, is in the informational category, and standards-track documents, is in the informational category, and
while the facts it reports may have normative implications, any such while the facts it reports may have normative implications, any such
normative significance reflects the readers' preferences. For normative significance reflects the readers' preferences. For
example, we may report that the existing definition of migration for example, we may report that the existing definition of migration for
NFSv4.1 does not properly describe how migrating state is to be NFSv4.1 does not properly describe how migrating state is to be
merged with existing state for the destination server. While it is merged with existing state for the destination server. While it is
to be expected that client and server implementers will judge this to to be expected that client and server implementers will judge this to
be a situation that is best avoided, the judgment as to how pressing be a situation that it would be appropriate to resolve, the judgment
this issue should be considered is a judgment for the reader, and as to how pressing this issue should be considered is a judgment for
eventually the nfsv4 working group to make. the reader, and eventually the nfsv4 working group to make.
We do explore possible ways in which such issues can be avoided, with We do explore possible ways in which such issues can be dealt with,
minimal negative effects, given that the working group has decided to with minimal negative effects, given that the working group has
address these issues, but the choice of exactly how to address these decided to address these issues, but the choice of exactly how to
is best given effect in one or more standards-track documents and/or address these is best given effect in one or more standards-track
errata. documents and/or errata.
This document focuses on NFSv4.1, since the analogous issues for In the context of this informational document, these normative
NFSv4.0 have already been addressed by the publication of [RFC7931]. keywords will generally occur in the context of a quotation, most
Nevertheless, the history of these issues in NFSv4.0 is presented, often direct but sometimes indirect. The context will make it clear
since understanding the similarities and differences between these whether the quotation is from:
protocols may be helpful in deciding how best to address remaining
issues.
2. Conventions o The base definition of the NFSv4.0 protocol [RFC7530].
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", o The document updating the handling of migration in NFSv4.0
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this [RFC7931].
document are to be interpreted as described in [RFC2119].
In the context of this informational document, these normative o The current definition of the NFSv4.1 protocol [RFC5661].
keywords will always occur in the context of a quotation, most often
direct but sometimes indirect. The context will make it clear
whether the quotation is from:
o The previously current definitive definition of the NFSv4.0 An additional possibility is that these terms may appear in a
protocol [RFC7530]. proposed or possible text to serve as a replacement for a current
protocol specification. Sometimes, a number of possible alternative
texts may be listed and benefits and detriments of each examined in
turn.
o The current definitive definition of the NFSv4.1 protocol 2.3. Terminology Used in this Document
[RFC5661].
o A proposed or possible text to serve as a replacement for the In this document the phrase "client ID" always refers to the 64-bit
current or previous definitive document text. Sometimes, a number shorthand identifier assigned by the server (a clientid4) and never
of possible alternative texts may be listed and benefits and to the structure which the client uses to identify itself to the
detriments of each examined in turn. server (called an nfs_client_id4 or client_owner in NFSv4.0 and
NFSv4.1 respectively). The opaque identifier within those structures
is referred to as a client id string".
3. NFSv4.0 Issues and Their Resolution Regarding trunking of network addresses, we use the following
terminology:
3.1. NFSv4.0 Issues o Trunking detection refers to ways of deciding whether two specific
network addresses are connected to the same NFSv4 server. The
means available to make this determination depends on the protocol
version, and, in some cases, on the client implementation.
o Two network addresses connected to the same server are said to be
server-trunkable.
o Two network addresses connected to the same server such that those
addresses can be used to support a single common session are
referred to as session-trunkable. Note that two addresses may be
server-trunkable without being session-trunkable.
o Trunking discovery is a process by which a client using one
network address can obtain other addresses that are trunkable
(either server-trunkable or session-trunkable) with it.
Regarding terminology relating to attributes used in trunking
discovery and other multi-server namespace features:
o Location attributes include the fs_locations and fs_locations_info
attributes.
o Location entries are the individual file system locations in the
location attributes.
o Location elements are derived from location entries. If a
location entry specifies an IP address there is only a single
corresponding location element. Location entries that contain a
host name, are resolved using DNS, and may result in one or more
location elements. All location elements consist of a location
address which is the IP address of an interface to a server and an
fs name which is the location of the file system within the
server's pseudo-fs. The fs name is empty if the server has no
pseudo-fs and only a single exported file system at the root
filehandle.
o Two location elements are said to be server-trunkable if they
specify the same fs name and the location addresses are such that
the location addresses are server-trunkable.
o Two location elements are said to be session-trunkable if they
specify the same fs name and the location addresses are such that
the location addresses are session-trunkable.
Each set of server-trunkable location elements defines the available
access paths to a particular file system. When there are multiple
such file systems, each of these, which contains the same data, is a
replica of the others. Logically, such replication is symmetric,
since the fs currently in use and an alternate fs are replicas of
each other. Often, in other documents, the term "replica" is not
applied to the fs currently in use, despite the fact that the
replication relation is inherently symmetric.
3. Multi-version Issues and Their Resolution
3.1. Multi-Version Issues
The source of these issues rises from a lack of clarity regarding the
meaning of and proper handling for location attributes that specify
more than a single server address. Such situations can arise as a
result of multiple entries in the same attribute or because a single
entry has a server name which, when processed by DNS, is mapped to
multiple server addresses.
Both [RFC7530] and [RFC5661] indicate that multiple addresses may be
present and that these addresses may be different paths to the same
server as well as different copies of the same data. However, the
following issues have, for both protocols, interfered with the
recognition of the existing location attributes as a way of providing
a trunking discovery function:
o There is no discussion of the use of these attributes when a file
system is first accessed, giving the impression that they are only
to be used as a way of overcoming access difficulties.
o The treatment of migration (and in the case of NFSv4.1 of file
system transitions in general) is written as if only a single
server address will be accessed.
o Although location attributes can contains the addresses of
migration targets and of additional replicas as well, the issues
that arise when both of these are specified are not clearly
discussed.
In addition, there are factors that relates to specific protocol
versions and documents:
o In NFSv4.0, as described solely by [RFC7530], trunking is treated
as a problem to be avoided, making the whole matter moot.
o In NFSv4.0, as described by [RFC7530] together with [RFC7931], the
situation is different. There is a means of trunking detection
suggested in [RFC7931] but it is a suggestion only valid when the
client chooses to use the uniform client id string model.
o For NFSv4.1, as described by [RFC5661], there is a standard method
of trunking detection, which can be relied upon.
The issues that need to be resolved for both versions are:
o Provision of a trunking discovery facility to allow a client to
find out about other addresses that may be used to access the
current server.
o Better integration of migration with trunking changes, including
situations in which the set of addresses to access the same server
changes (without migration) and those in which there is a shift to
a different server, but trunking of addresses on either the source
or destination is involved
3.2. Resolution of Multi-Version Issues
Although the specifics of addressing these issues will be different
for different versions, there are some common aspects discussed in
the subsections below:
o The trunking discovery function, except for the additional
location attribute fs_locations_info, is to be addressed in
substantially the same way in both versions, as explained in
Section 3.2.1.
o The interaction of trunking and migration is discussed in general
terms in Section 3.2.2. However, the specifics of the NFSv4.1
client's response to NFS4ERR_MOVED are discussed in Sections
5.4.3, 5.4.4, and 5.4.5.
3.2.1. Providing Trunking Discovery
A client can discover a set network addresses to use to access a file
system using an NFSv4 server in a number of ways:
o If the client is accessing a server using its name, that name can
be mapped to a set of IP addresses using DNS and if multiple
addresses are available, those addresses can generally be used
together to access the server.
o A client connected to a server without knowledge of its name can
obtain the value of a location attribute (e.g. fs_locations).
Where an entry within that attribute specifies a server name, DNS
can be used to obtain one or more network addresses corresponding
to that name. In cases in which one of those is the address being
used, the others that corresponding to that name can also be used
to access the server.
o A client can obtain the value of a location attribute (e.g.
fs_locations) and use location entries that specify network
addresses. When there is a means of trunking detection available
(see below), all of addresses that are determined to correspond to
the same server can be used to access the server.
Note that the last two of these are usable in situations in which
NFS4ERR_MOVED was returned. Note that this does not necessarily mean
that migration has occurred since there may be a shift in the set of
network addresses to be used without changing to a different server.
See Section 3.2.2 for further discussion.
Which of the above means of providing trunking information is
appropriate to use in a given environment will depend on security
considerations, the possible need for the server to direct different
clients to different sets of addresses, and the availability of
trunking detection facilities on the clients.
With regard to security, the possibility that requests to determine
the set of network addresses corresponding to a given server might be
interfered with or have their responses corrupted needs to be taken
account of. As a result, when use of DNSSEC is not available, it
might not be advisable to present server names in location attributes
and present the network addresses directly, eliminating the need to
use DNS to effect this translation. Fetching of location attributes
should be done with integrity protection.
In some situations, the server will want to direct clients to use
specific sets of network addresses, to effect load balancing, or to
meet quality-of-service goals. In such environments, presentation of
network addresses directly in the location attribute can help give
the server the necessary control over the paths to be used when
accessing particular file systems. When such techniques are used,
servers typically present their own network addresses in the location
attribute while adding the names of other servers, such as those used
to access replicas.
Trunking detection allows the client to determine whether two network
addresses can be used to access the same server. The availability of
trunking detection depends on the protocol version, and, in some
case, on client implementation choices:
o For NFSv4.0, a means by which it can be determined if two network
addresses correspond to the same server is suggested in [RFC7931].
However is it is optional and only available to clients using the
uniform client id string approach.
o For NFSv4.1, the client can compare the server_owner returned in
the response to EXCHANGE_ID to determine if two network addresses
correspond to the same server.
As a result, direct presentation of network addresses in location
entries may be problematic for NFSv4.0, since some clients might not
have the trunking detection facilities that allow them to take
advantage of this information. For further discussion of issues
related to NFSv4.0, see Section 4.4.
3.2.2. Interaction of Trunking and Migration
When the set of network addresses designated by a location attribute
changes, NFS4ERR_MOVED may or may not result, and in some of the
cases in which it is returned migration will occur, while in others
there a shift in the network addresses used to access a particular
file system with no migration.
o When the list of networks addresses is a superset of that
previously in effect, there is no need for migration or any other
sort of client adjustment. Nevertheless, the client is free to
use an additional address if it provides another path to the same
server. If, on the other hand, it does not do so, the client may
treat it as it does a replica, to be used if the current server
addresses become unavailable.
o When the list of networks addresses is a subset of that previously
in effect, immediate action is not needed if the address was not
being used. The client should avoid using it in the future,
whether the address is for a replica or a potential additional
path to the server being used.
o When an address being removed is one of a number of paths to the
current server, the client can cease to use it but it can continue
to use it until NFS4ERR_MOVED is received. This is not considered
a migration event, unless it is the last available path to the
server that has become unusable.
When migration does occur, multiple addresses may be in use on the
server previous to migration and multiple addresses may be available
for use on the destination server.
With regard to the server in use, it may be that return of
NFS4ERR_MOVED indicates that a particular network address is no
longer to be used, without implying that migration of the file system
to a different server is needed In light of this possibility, clients
are best off not concluding that migration has occurred until
concluding that all the network addresses known to be associated with
the server are not usable.
It should be noted that the need to defer this determination is not
absolute. If a client is not aware of all network addresses for any
reason, if may conclude that migration has occurred when it has not
and treat a switch to a different server address as if it were a
migration event. This is generally harmless since the use of the
same server via a new address will appear as a successful Transparent
State Migration.
While significant harm will not arise from this misapprehension, it
can give rise to disconcerting situations. For example, if a lock
has been revoked during the address shift, it will appear to the
client as if the lock has been lost during migration, normally
calling for it to be recoverable via an fs-specific grace period
associated with the migration event.
With regard to the destination server, it is desirable for the client
to be aware of all the valid network addresses that can be used to
access the destination server. However, there is no need for this to
be done immediately. Implementations can process the additional
location elements in parallel with normal use of the first valid
location entry found to access the destination.
Because a location attribute may include entries relating to the
current server, the migration destination and possible replicas to
use, scanning for available network addresses could potentially be a
long process. The following list of helpful practices, here
presented as suggestions, could become RECOMMENDATIONs or
REQUIREMENTs in future standards-track documents
o Servers are well advised to place location entries that represent
addresses usable with the current server or a migration target
before those associated with replicas.
o A client can cease scanning for trunkable location entries once it
encounters one whose fs_name differs from the current fs name.
o A client can cease scanning for trunkable location entries once it
encounters a location element whose address in not server-
trunkable with the one it is using.
4. NFSv4.0 Issues
4.1. Core NFSv4.0 Migration Issues
Many of the problems seen with Transparent State Migration derived Many of the problems seen with Transparent State Migration derived
from the inability of servers to determine whether two client IDs, from the inability of NFSv4.0servers to determine whether two client
issued on different servers, corresponded to the same client. This IDs, issued on different servers, corresponded to the same client.
difficulty derived in turn from the common practice, recommended by This difficulty derived in turn from the common practice, recommended
[RFC7530], in which each client presented different client by [RFC7530], in which each client presented different client
identification strings to different servers, rather than presenting identification strings to different servers, rather than presenting
the same identification string to all servers. the same identification string to all servers.
This practice, later referred to as the "non-uniform" client string This practice, later referred to as the "non-uniform" client id
approach, derived from concern that, since NFSv4.0 provided no means string approach, derived from concern that, since NFSv4.0 provided no
to determine whether two IP addresses correspond to the server, a means to determine whether two IP addresses correspond to the server,
single client connected to both might be confused by the fact that a single client connected to both might be confused by the fact that
state changes made via one IP address might unexpectedly affect the state changes made via one IP address might unexpectedly affect the
state maintained with respect to the second IP address, thought of as state maintained with respect to the second IP address, thought of as
a separate server a separate server
To avoid this unexpected behavior, clients used the non-uniform To avoid this unexpected behavior, clients used the non-uniform
client id string approach. By doing so, a client connected to two client id string approach. By doing so, a client connected to two
different servers (or to two IP addresses connected to the same different servers (or to two IP addresses connected to the same
server) appeared to be two different servers. Since the server is server) appeared to be two different servers. Since the server is
under the impression that two different clients are involved, state under the impression that two different clients are involved, state
changes made on each distinct IP address cannot be reflected on changes made on each distinct IP address cannot be reflected on
another. another.
However, by doing things this way, state migrated from server to However, by doing things in that way, state migrated from server to
server cannot be referred to the actual client which generated it, server cannot be referred to the actual client which generated it,
leading to confusion. leading to confusion.
In addition to this core problem, the following issues with regard to In addition to this core problem, the following issues with regard to
Transparent State Migration needed to be addressed: Transparent State Migration needed to be addressed:
o Clarification regarding the ability to merge state from different o Clarification regarding the ability to merge state from different
leases even though their expiration times might not be precisely leases even though their expiration times might not be precisely
synchronized. synchronized.
o Clarifying the treatment of client IDs since it is not always o Clarifying the treatment of client IDs since it is not always
clear when clientid4 and when nfs_client_id4 was intended. clear when clientid4 and when nfs_client_id4 was intended.
o Clarifying the logic of returning NFS4ERR_LEASE_MOVED. o Clarifying the logic of returning NFS4ERR_LEASE_MOVED.
o Clarifying the handling NFS4ERR_CLID_INUSE. o Clarifying the handling NFS4ERR_CLID_INUSE.
3.2. Resolution of NFSv4.0 Protocol Difficulties 4.2. Resolution of Core Migration Protocol Difficulties in NFSv4.0
The client string identification issue was addressed in [RFC7931] as The client string identification issue was addressed in [RFC7931] as
follows: follows:
o Defining both the uniform and non-uniform client id string o Defining both the uniform and non-uniform client id string
approaches as valid choices but indicating that the latter posed approaches as valid choices but indicating that the latter posed
difficulties for Transparent Stare Migration. difficulties for Transparent Stare Migration.
o Providing a way that clients could use to determine whether two IP o Providing a way that clients using the uniform approach could use
addresses are connected to the same server. to determine whether two IP addresses are connected to the same
server.
o Allowing clients using the uniform approach to avoid negative o Allowing clients using the uniform approach to avoid negative
consequences due to otherwise unexpected behavior since behavior consequences due to otherwise unexpected behavior since behavior
that is a consequence of known trunking relationships is not that is a consequence of known trunking relationships is not
unexpected. unexpected.
o As a result, servers migrating state are aware of the fact that o As a result, servers migrating state can be aware of the fact that
the same client is associated with two different items of state the same client is associated with two different items of state
even when that state was originally created on two different even when that state was originally created on two different
servers. servers.
Since all of the other issues noted in Section 3.1 were also Since all of the other issues noted in Section 4.1 were also
addressed, publication of [RFC7931] updating [RFC7530] addressed all addressed by [RFC7931], publication of that document updating
known issues with Transparent State Migration in NFSv4.0. [RFC7530] addressed all issues with Transparent State Migration in
NFSv4.0 known at that time.
4. Issues for NFSv4.1 4.3. Additional NFSv4.0 Issues
4.1. Issues to Address for NFSv4.1 In light of the fact that a large set of migration-specific issues
were addressed by the publication of [RFC7931], the remaining issues
derive from those mentioned in Section 3.1. These include:
o Introducing facilities for trunking discovery.
o Clarifying the relationship between migration and trunking.
4.4. Resolution of Additional NFSv4.0 Issues
One possible approach to addressing these issues would entail
publication of an additional standards-track document updating
[RFC7530].
Fortunately, it appears that all of the material to be updated
appears in Section 8 of that document, whether it concerns the
provision of trunking discovery or the interaction of trunking and
migration. It also appears that none of the material to be updated
is in sections updated by [RFC7931].
A review of the existing Section 8 of [RFC7530], shows the following
sections as requiring significant attention:
o The existing Section 8.1 requires a considerable expansion to
explain the various uses of the fs_locations and the possible
interactions among them.
o The existing Section 8.4 may require substantial re-organization
to reflect the facts that fs_locations has multiple functions and
may be referenced on multiple occasions.
o The existing Section 8.5 follows the previous approach for NFSv4.0
in assuming that trunking simply cannot and should not happen.
For example, the last paragraph says:
If a single location entry designates multiple server IP
addresses, the client should choose a single one to use. When
two server addresses are designated by a single location entry
and they correspond to different servers, this normally
indicates some sort of misconfiguration, and so the client
should avoid using such location entries when alternatives are
available. When they are not, clients should pick one of the
IP addresses and use it, without using others that are not
directed to the same server.
As written, this seems to foreclose any use of trunking in
connection with migration. In retrospect, it appears that this
section should have been revised as part of [RFC7931], but since
that was not done then, the issue needs to be addressed now.
Overall, it appears that, in addition to the revision of Section 8.1,
Sections 8.4 and 8.5 need to be reorganized. One likely approach is
to divide the material into three main sections based on the
circumstances in which the attribute is accessed:
A section dealing with initial use and how it is trunking
discovery and preparation for replication.
A section dealing with subsequent attribute change and how it
affects trunking discovery and preparation for replication.
A section dealing with how the attribute is used to deal with
communication problems including receiving NFS4ERR_MOVED but also
including other communication difficulties driving selection of a
new replica.
5. Issues for NFSv4.1
5.1. Issues to Address for NFSv4.1
Because NFSv4.1 embraces the uniform client-string approach, as Because NFSv4.1 embraces the uniform client-string approach, as
advised by section 2.4 of [RFC5661], addressing migration issues is advised by section 2.4 of [RFC5661], addressing migration issues is
simpler, in that a shift in client id string models is not required. simpler, in that a shift in client id string models is not required.
Instead, NFSv4 returns information in the EXCHANGE_ID response to Instead, NFSv4 returns information in the EXCHANGE_ID response to
enable trunking relationships to be determined by the client. enable trunking relationships to be determined by the client.
The other necessary part of addressing migration issues, providing Despite this simplification, there are substantial issues that need
for the server's merger of leases that relate to the same client, is to be dealt with:
not currently addressed by [RFC5661] and changes need to be made to
make it clear that state needs to be appropriately merged as part of o The other necessary part of addressing migration issues, providing
migration, to avoid multiple client IDs between a client-server pair. for the server's merger of leases that relate to the same client,
is not currently addressed by [RFC5661] and changes need to be
made to make it clear that state needs to be appropriately merged
as part of migration, to avoid multiple client IDs between a
client-server pair.
o The current discussion (in [RFC5661]), of the possibility of
server_owner changes is incomplete and confusing.
o As with NFSV4.0, the interaction of trunking with migration and
other aspects of multi-server namespace needs to be clarified.
Addressing migration in NFSv41 will also require adaptation of the
approaches used in [RFC7931] to the NFSv4.1 environment including:
o The use of EXCHANGE_ID needs to be accommodated including issues
associated with the expected confirmation status of client IDs
transferred by Transparent State Migration.
o The use of sessions needs to be addressed including discussion of
the proper use of the status bits returned by the SEQUENCE
operation.
In addition, there are a number of new features within NFSv4.1 whose In addition, there are a number of new features within NFSv4.1 whose
relationship with migration needs to be clarified. Some examples: relationship with migration needs to be clarified. Some examples:
o The interaction of trunking with migration and other aspects of
multi-server namespace needs to be clarified.
o There needs to be some clarification of how migration, and o There needs to be some clarification of how migration, and
particularly Transparent State Migration, should interact with particularly Transparent State Migration, should interact with
pNFS layouts. pNFS layouts.
o The current discussion (in [RFC5661]), of the possibility of
server_owner changes is incomplete and confusing.
o The expected confirmation status of client IDs transferred by
Transparent State Migration needs to be clarified.
o There are a number of issues related to the migration of sessions o There are a number of issues related to the migration of sessions
that need to be addressed that need to be addressed.
Discussion of how to resolve these issues will appear in the sections Discussion of how to resolve these issues will appear in the sections
below. below.
4.1.1. Addressing state merger in NFSv4.1 5.1.1. Addressing State Merger in NFSv4.1
The existing treatment of state transfer in [RFC5661], has similar The existing treatment of state transfer in [RFC5661], has similar
problems to that in [RFC7530] in that it assumes that the state for problems to that in [RFC7530] in that it assumes that the state for
multiple filesystems formerly on different servers will not be merged multiple filesystems formerly on different servers will not be merged
so that it appears under a single common client ID. We've already so that it appears under a single common client ID. We've already
seen the reasons that this is a problem with regard to NFSv4.0. seen the reasons that this is a problem with regard to NFSv4.0.
Although we don't have the problems stemming from the non-uniform Although we don't have the problems stemming from the non-uniform
client-string approach, there are a number of complexities in the client-string approach, there are a number of complexities in the
existing treatment of state management in the section entitled "Lock existing treatment of state management in the section entitled "Lock
skipping to change at page 6, line 31 skipping to change at page 15, line 34
o Migration is currently treated together with other sorts of o Migration is currently treated together with other sorts of
filesystem transitions including transitioning between replicas filesystem transitions including transitioning between replicas
without any NFS4ERR_MOVED errors. without any NFS4ERR_MOVED errors.
o There is separate handling and discussion of the cases of matching o There is separate handling and discussion of the cases of matching
and non-matching server scopes. and non-matching server scopes.
o In the case of matching server scopes, the text calls for an o In the case of matching server scopes, the text calls for an
unrealistic degree of transparency, suggesting that the source and unrealistic degree of transparency, suggesting that the source and
destination servers need to cooperate in stateid assignment. destination servers need to cooperate in stateid and client ID
assignment.
o In the case of non-matching server scopes, the text does not o In the case of non-matching server scopes, the text does not
mention the possibility of the transparent migration of state at mention the possibility of the transparent migration of state at
all, resulting in a functional regression from NFSV4.0 all, resulting in a functional regression from NFSV4.0
o The potential interaction between migration and trunking has not 5.1.2. Addressing pNFS Relationship with Migration
been addressed.
o There is insufficient attention to the question of how clients can
deal with the complexities of recovering from migration. As part
of this, the implications of the shift of lease migration
notification shifting from an error (NFS4ERR_LEASE_MOVED in
NFSv4.0) to status bit (SEQ4_STATUS_LEASE_MOVED in NFSv4.1) need
to be explored.
To summarize, there is a need for an NFSv4.1 treatment of Transparent
State Migration that is an extension of that in [RFC7931] and that
includes appropriate handling for NFSv4.1 features such as trunking.
4.1.2. Addressing pNFS relationship with migration
This is made difficult because, within the pNFS framework, migration This is made difficult because, within the pNFS framework, migration
might mean any of several things: might mean any of several things:
o Transfer of the MDS, leaving DS's as they are. o Transfer of the MDS, leaving DS's as they are.
This would be minimally disruptive to those using layouts but This would be minimally disruptive to those using layouts but
would require the pNFS control protocol being used to support the would require the pNFS control protocol being used to support the
DS being directed to a new MDS. DS being directed to a new MDS.
skipping to change at page 7, line 31 skipping to change at page 16, line 18
o Transfer of the filesystem to a new filesystem with both MDS and o Transfer of the filesystem to a new filesystem with both MDS and
DS's moving. DS's moving.
In such a transfer, an entirely different set of DS's will be at In such a transfer, an entirely different set of DS's will be at
the target location. There may even be no pNFS support on the the target location. There may even be no pNFS support on the
destination filesystem at all. destination filesystem at all.
Migration needs to support both the first and last of these models. Migration needs to support both the first and last of these models.
4.1.3. Addressing server_owner changes in NFSv4.1 5.1.3. Addressing Server_owner Changes in NFSv4.1
Section 2.10.5 of [RFC5661] states the following. Section 2.10.5 of [RFC5661] states the following.
The client should be prepared for the possibility that The client should be prepared for the possibility that
eir_server_owner values may be different on subsequent EXCHANGE_ID eir_server_owner values may be different on subsequent EXCHANGE_ID
requests made to the same network address, as a result of various requests made to the same network address, as a result of various
sorts of reconfiguration events. When this happens and the sorts of reconfiguration events. When this happens and the
changes result in the invalidation of previously valid forms of changes result in the invalidation of previously valid forms of
trunking, the client should cease to use those forms, either by trunking, the client should cease to use those forms, either by
dropping connections or by adding sessions. For a discussion of dropping connections or by adding sessions. For a discussion of
skipping to change at page 8, line 36 skipping to change at page 17, line 21
eir_server_owner.so_major_id changes, the client can use eir_server_owner.so_major_id changes, the client can use
filehandles it has and attempt reclaims. It may find that filehandles it has and attempt reclaims. It may find that
these are now stale but if NFS4ERR_STALE is not received, he these are now stale but if NFS4ERR_STALE is not received, he
can proceed to reclaim his opens. can proceed to reclaim his opens.
o When eir_server_scope and eir_server_owner.so_major_id remain o When eir_server_scope and eir_server_owner.so_major_id remain
the same, the client has to use the now-current values of the same, the client has to use the now-current values of
eir_server-owner.so_minor_id in deciding on appropriate forms eir_server-owner.so_minor_id in deciding on appropriate forms
of trunking. of trunking.
4.1.4. Addressing Confirmation Status of Migrated Client IDs in NFSv4.1 5.1.4. Addressing Confirmation Status of Migrated Client IDs in NFSv4.1
When a client ID is transferred between systems as a part of When a client ID is transferred between systems as a part of
migration, it is not always clear whether it should be considered migration, it has never been clear whether it should be considered
confirmed or unconfirmed on the target server. In the case in which confirmed or unconfirmed on the target server. In the case in which
an associated session is transferred together with the client ID, it an associated session is transferred together with the client ID, it
is clear that the transferred client ID needs to be considered is clear that the transferred client ID needs to be considered
confirmed, as the existence of an associated session is incompatible confirmed, as the existence of an associated session is incompatible
with an unconfirmed client ID. with an unconfirmed client ID.
The case in which a client ID is transferred without an associated The case in which a client ID is transferred without an associated
session is less clear-cut and there needs to be a choice between two session is less clear-cut, particularly since the treatment of
possibilities: EXCHANGE_ID in [RFC5661] assumes that CREATE_SESSION is the only
means by which a client id may be confirmed. While this assumption
is valid in the absence of Transparent State Migration,
implementation of migration means that if this assumption is
maintained, it is not clear how migrated client ID s can be a
accommodated. If this assumption were maintained, we would have to
choose between the following two alternatives, regarding whether the
client ID to be reported as confirmed when EXCHANGE_ID is used to
register an already-known client_owner with the server.
o Consider it unconfirmed, because of the lack of an associated o Report the client ID unconfirmed, because of the lack of an
session. This makes it simpler for the client to determine associated session. This makes it simpler for the client to
whether there is an associated session transferred at the same determine whether there is an associated session transferred at
time. However, it is inconsistent with the fact there are the same time. However, it is inconsistent with the fact there
stateids which have been transferred with the client ID. are stateids which have been transferred with the client ID.
o Consider it confirmed, because it was confirmed on the source o Report the client ID as confirmed, because it was confirmed on the
server and the transfer is not considered to have affected that. source server and the transfer is not considered to have affected
Although this makes it simpler for the client to determine whether that. Given the current description of EXCHANGE_ID in [RFC5661],
there is an associated session transferred at the same time, an some modification in the treatment of client id confirmation is
alternative is discussed in Section 4.1.5. called for. In particular, provision would have to be made to
enable the client id slot sequence id to be used by the client to
be determined.
A related issue concerns the potential use the SEQ4_STATUS flags to Although the first approach makes it simpler for the client to
determine whether all or some of the state present on the source has determine whether there is an associated session transferred at the
been transferred the destination server. This could be done using same time, it makes it more difficult to determine whether
either of the alternatives above but it is more in the spirit of the Transparent State Migration has occurred. Section 5.1.6.
second alternative. One potential use of these flags is discussed in
more detail in Section 4.2.2.
4.1.5. Addressing Session Migration in NFSv4.1 In any case, adjustments will be required to deal with the fact that
[RFC5661] currently assumes that a client id can only be confirmed by
issuing a CREATE_SESSION. In order to properly deal with the status
of migrated client ids, we have to distinguish among:
o The confirmation status as reported by EXCHANGE_ID.
o Whether the client id is considered confirmed as that term is used
in the many other cases in which the confirmation status of a
client ID affects how requests are handled.
o How the client is to determine the initial sequence id to be used
when doing operations such as CREATE_SESSION.
In [RFC5661] as it currently stands all of these are tied together
and it is not obvious how migrated client IDs could be accommodated
in this structure, and what changes are necessary to make this
possible. For more discussion of this issue, see Section 5.2.1.
5.1.5. Addressing Changes in Trunking Configuration
When the client us capable of finding out a set of network addresses
to use in accessing a server, it is always possible for that set to
change.
This sometimes requires that a network address previously used to
access a server becomes invalid for that purpose. This requires a
way of notifying the client and a way for the client to adapt to this
change by using a new set of network addresses to access the server.
his will involve recovery much like hat for migration although the
same server and file system is used throughout.
5.1.6. Addressing Session Migration in NFSv4.1
Some issues that need to be addressed regard the migration of Some issues that need to be addressed regard the migration of
sessions, in addition to client IDs and stateids sessions, in addition to client IDs and stateids
o It needs to be made clearer how the client can deal with the o It needs to be made clearer how the client can deal with the
possibility that sessions might or might not be transferred as possibility that sessions might or might not be transferred as
part of Transparent State Migration. part of Transparent State Migration.
o Rules need to be clarified regarding possible transfer of sessions o Rules need to be clarified regarding possible transfer of sessions
when either the source session is being used to access other file when either the source session is being used to access other file
systems on source server or there is already a session connecting systems on source server or there is already a session connecting
the client to the destination server. the client to the destination server.
o There needs to be more detail regarding how the protocol avoids o There needs to be more detail regarding how the protocol avoids
skipping to change at page 9, line 38 skipping to change at page 19, line 17
o Rules need to be clarified regarding possible transfer of sessions o Rules need to be clarified regarding possible transfer of sessions
when either the source session is being used to access other file when either the source session is being used to access other file
systems on source server or there is already a session connecting systems on source server or there is already a session connecting
the client to the destination server. the client to the destination server.
o There needs to be more detail regarding how the protocol avoids o There needs to be more detail regarding how the protocol avoids
situations in which the same session is subject to concurrent situations in which the same session is subject to concurrent
changes on two different servers at the same time. changes on two different servers at the same time.
4.2. Possible Resolutions for NFSv4.1 Issues 5.2. Possible Resolutions for NFSv4.1 Protocol Issues
The subsections below explore some ways of dealing with the issues
discussed in Section 4.1
First we introduce some terminology we will be using in these
sections:
o Location attributes include the fs_locations and fs_locations_info
attributes.
o Location entries are the individual file system locations in the
location attributes.
o Location elements are derived from location entries. If a
location entry specifies an IP address there is only a single
corresponding location element. Location entries that contain a
host name, are resolved using DNS, and may result in one or more
location elements. All location elements consist of a location
address which is the IP address of an interface to a server and an
fs name which is the location of the file system within the
server's pseudo-fs. The fs name is empty if the server has no
pseudo-fs and only a single exported file system at the root
filehandle.
o Two location elements are trunkable if they specify the same fs
name and the location addresses are such that trunking of the
location addresses can be used as shown by the server_owner values
returned.
4.2.1. Server Responsibilities in Effecting Transparent State Migration
The basic responsibility of the source server in effecting
Transparent State Migration is to make available to the destination
server a description of each piece of locking state associated with
the file system being migrated. In addition to client id string and
verifier, the source server needs to provide. for each stateid:
o The stateid including the current sequence value.
o The associated client ID.
o The handle of the associated file.
o The type of the lock, such as open, byte-range lock, delegation,
layout.
o For locks such as opens and byte-range locks, there will be
information about the owner(s) of the lock.
o For recallable/revocable lock types, the current recall status
needs to be included.
o For each lock type there will by type-specific information, such
as share and deny modes for opens and type and byte ranges for
byte-range locks and layouts.
A further server responsibility concerns locks that are revoked or
otherwise lost during the process of file system migration. Because
locks that appear to be lost during the process of migration will be
reclaimed by the client, the servers have to take steps to ensure
that locks revoked soon before or soon after migration are not
inadvertently allowed to be reclaimed in situations in which the
continuity of lock possession cannot be assured.
o For locks lost on the source but whose loss has not yet been
acknowledged by the client (by using FREE_STATEID), the
destination must be aware of this loss so that it can deny a
request to reclaim them.
o For locks lost on the destination after the state transfer but
before the client's RECLAIM_COMPLTE is done, the destination
server should note these and not allow them to be reclaimed.
A further responsibility of the servers concerns situations in which
stateid cannot be transferred transparently because it conflicts with
an existing stateid held by the client and associated with a
different file systems. In this case there are two valid choices:
o Treat the transfer, as in NFSv4.0, as one without Transparent
State Migration. In this case, conflicting locks cannot be
granted until the client does a RECLAIM_COMPLETE, after reclaiming
the lock it had, with the exception of reclaims denied because
they were attempts to reclaim locks that had been lost.
o Implement Transparent State Migration, except for the lock with
the conflicting stateid. In this case, the client will be aware
of a lost lock (through the SEQ4_STATUS flags) and be allowed to
reclaim it.
4.2.2. Determining Initial Migration Status in NFSv4.1
This section proposes a way in which a client which receives
NFS4ERR_MOVED can determine:
o Whether the NFS4ERR_MOVED indicates migration has occurred, or
whether it indicates another sort of file system transition as
discussed in Section 4.2.4
o In the case of migration, whether Transparent State Migration has
occurred.
o Whether any state has been lost during the process of Transparent
State Migration.
o Whether sessions have been transferred as part of Transparent
State Migration.
This is written assuming that the second option regarding client ID
confirmation status after migration (as discussed in Section 4.1.4)
is adopted. However that choice is not essential to the procedure
and could be changed.
The process begins by the client examining the location entries using
either of the location attributes. For those whose fs name matches
that currently being used, an EXCHANGE_ID is directed at the location
address and the server_owner and scope used to determine if the entry
is trunkable with that previously being used to access the file
system (i.e. that it represents another path to the same file system
and can share locking state with it). If it is, then this should be
treated as a transition from one set of paths to another, as
described in Section 4.2.4, rather than a migration event.
Otherwuse, if one or more of the EXCHANGE_ID operations above has
encountered a distinct server, then migration has occurred and the
procedure continues. If there were no location entries with a
matching fs name, then one with another fs name is selected, an
EXCHANGE_ID is done, and the procedure continues using the result of
that operation.
The determination of whether Transparent State Migration has occurred
is driven by the client ID returned and its confirmation status.
o If the client ID is an unconfirmed client ID not previously known
to the client, then Transparent State Migration has not occurred.
o If the client ID is a confirmed client ID previously known to the The subsections below explore some ways of dealing with clarifying
client, then any transferred state would have been merged with an the protocol to address issues discussed in Section 5.1
existing client ID representing the client to the destination
server. In this state merger case, Transparent State Migration
might or might not have occurred.
o If the client ID is a confirmed client ID not previously known to 5.2.1. Client ID Confirmation Issues
the client, then the client can conclude that the client ID was
transferred as part of Transparent State Migration. In this
transferred client ID case, Transparent State Migration has
occurred although some state may have been lost.
In the state merger case, it is possible that the server has not As mentioned previously [RFC5661], makes no provision for client IDs
attempted Transparent State Migration, in which case state may have that are confirmed other than through the use of CREATE_SESSION. For
been lost without it being reflected in the SEQ4_STATUS bits. To example Section 18.35 of [RFC5661] states:
determine whether this has happened, the client can use TEST_STATEID
to check whether the stateids created on the source server are still
accessible on the destination server. Once a single stateid is found
to have been successfully transferred, the client can conclude that
Transparent State Migration was begun and any failure to transport
all of the stateids will be reflected in the SEQ4_STATUS bits.
In any of the cases in which Transparent State Migration has The client uses the EXCHANGE_ID operation to register a particular
occurred, it is possible that a session was transferred as well. To client owner with the server. The client ID returned from this
deal with that possibility, clients can, after doing the EXCHANGE_ID, operation will be necessary for requests that create state on the
issue a BIND_CONN_TO_SESSION to connect the transferred session to a server and will serve as a parent object to sessions created by
connection to the new server. If that fails, it is an indication the client. In order to confirm the client ID it must first be
that the session was not transferred and that a new session needs to used, along with the returned eir_sequenceid, as arguments to
be created to take its place. CREATE_SESSION. If the flag EXCHGID4_FLAG_CONFIRMED_R is set in
the result, eir_flags, then eir_sequenceid MUST be ignored, as it
has no relevancy.
4.2.3. Client Response to Migration in NFSv4.1 In deciding how to address the status of migrated client IDs in the
case of Transparent State Migration, we should avoid giving undue
weight to the last sentence of the above simply because it is stated
in the form of a normative requirement. We should instead focus on
the reasons such terms (i.e. those defined by [RFC2119]) are to be
used, to state interoperability constraints. In this case, the
"MUST" applies to a conclusion based on the premise that a
CREATE_SESSION must have been done to assure that the client ID is
reliably known to the server.
Once the client has determined the initial migration status, it needs In that light, let us consider a possible replacement, that treats
to re-establish its lock state, if possible. To enable this to confirmation by means of CREATE_SESSION as one of a number of
happen without loss of the guarantees normally provided by locking, possible means and avoids some the undesirable consequences of
the destination server needs to implement a per-fs grace period in adherence to the current approach, originally conceived without
all cases in which lock state was lost, including those in which taking state migration into account.
Transparent State Migration was not implemented.
The following cases need to be dealt with: The client uses the EXCHANGE_ID operation to register a particular
client_owner with the server. However, when the client_owner has
been already been registered by other means (e.g. Transparent
State Migration), the client may still use EXCHANGE_ID to obtain
the client ID assigned previously.
o In a case in which Transparent State Migration has not occurred, The client ID returned from this operation will be associated with
the client can use the per-fs grace period provided by the the connection on which the EXHANGE_ID is received and will serve
destination server to reclaim locks that were held on the source as a parent object for sessions created by the client on this
server. connection or to which the connection is bound. As a result of
using those sessions to make requests involving the creation of
state, that state will become associated with the client ID
returned.
o In a cases in which Transparent State Migration has occurred, and In situations in which the registration of the client_owner has
no lock state was lost (as shown by SEQ4_STATUS flags), no lock not occurred previously, the client ID must first be used, along
reclaim is necessary. with the returned eir_sequenceid, in creating an associated
session using CREATE_SESSION.
o In a case in which Transparent State Migration has occurred, and If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the result,
some lock state was lost (as shown by SEQ4_STATUS flags), existing eir_flags, then it is an indication that the registration of the
stateids need to be checked for validity using TEST_STATEID, and client_owner has already occurred and that a further
reclaim used to re-establish any that were not transferred. CREATE_SESSION is not needed to confirm it. Of course, subsequent
CREATE_SESSION operations may be needed for other reasons.
For all of the cases above, RECLAIM_COMPLETE with an rca_one_fs value The value eir_seqenceid is used to establish an initial sequence
of true should be done before normal use of the file system including value associate with the client ID returned. In cases in which a
obtaining new locks for the file system. This applies even if no CREATE_SESSION has already been done, there is no need for this
locks were lost and needed to be reclaimed. value, since sequencing of such request has already been
established and the client has no need for this value and will
ignore it
4.2.4. Dealing with Multiple Location Entries 5.2.2. Dealing with Multiple Location Entries
The possibility that more than one server address may be present in The possibility that more than one server address may be present in
location attributes requires further clarification. This is location attributes requires further clarification. This is
particularly the case, given the potential role of trunking for particularly the case, given the potential role of trunking for
NFSv4.1, whose connection to migration needs to be clarified. NFSv4.1, whose connection to migration needs to be clarified.
The description of the location attributes in [RFC5661], while it The description of the location attributes in [RFC5661], while it
indicates that multiple address entries in these attributes may be indicates that multiple address entries in these attributes may be
used to indicate alternate paths to the file system, does so mainly used to indicate alternate paths to the file system, does so mainly
in the context of replication and does so without mentioning in the context of replication and does so without mentioning
skipping to change at page 14, line 36 skipping to change at page 21, line 34
There is no current mechanism by which the client can be notified There is no current mechanism by which the client can be notified
of a change in the set of available location for an fs. Given the of a change in the set of available location for an fs. Given the
client has at least one IP address available to access the client has at least one IP address available to access the
filesystem in question, periodic polling is an adequate mechanism filesystem in question, periodic polling is an adequate mechanism
for the client to find additional server addresses to use to for the client to find additional server addresses to use to
access the file system. access the file system.
o When the set of location elements contracts but none of the o When the set of location elements contracts but none of the
elements no longer usable were in fact being used by the client, elements no longer usable were in fact being used by the client,
then no migration is involved. Only if the client were to start then no migration is involved and no change in network addresses
using one of the unavailable elements will the client be notified is needed. Only if the client were to start using one of the
(via NFS4ERR_MOVED) of the need to not use those elements and to unavailable elements would the client be notified (via
use others provided by a location attribute. NFS4ERR_MOVED) of the need to not use those elements and to use
others provided by a location attribute.
When a specific server address being used becomes unavailable to When a specific server address being used becomes unavailable to
service a particular file system, NF4ERR_MOVED will be returned, and service a particular file system, NFS4ERR_MOVED will be returned, and
the client will respond based on the available locations. Whether the client will respond based on the available locations. Whether
continuity of locking state will be available depends on a number of continuity of locking state will be available depends on a number of
factors: factors:
o If there are still elements in use trunkable with the element that o If there are still elements in use trunkable with the element that
has become unavailable, there will still be a continuity of has become unavailable, there will still be a continuity of
locking state, even though Transparent State Migration per se has locking state, even though Transparent State Migration per se has
not occurred. If the in-use addresses are session-trunkable with not occurred. If the in-use addresses are session-trunkable with
the address becoming unavailable, only one connection is lost and the address becoming unavailable, only one connection is lost and
all existing sessions will remain available. If, on the other all existing sessions will remain available. If, on the other
skipping to change at page 15, line 25 skipping to change at page 22, line 24
of trunking in effect. This is also similar to the case in which of trunking in effect. This is also similar to the case in which
Transparent State Migration has occurred, even though there is no Transparent State Migration has occurred, even though there is no
migration, with the state remaining on the existing server. migration, with the state remaining on the existing server.
Note that this case, as well as the previous one, can be expected Note that this case, as well as the previous one, can be expected
in the case in which the server seeks to direct traffic with in the case in which the server seeks to direct traffic with
regard to particular file systems to choose addresses, in the regard to particular file systems to choose addresses, in the
interest of load balancing, to adjust to hardware availability interest of load balancing, to adjust to hardware availability
constraints, or for other reasons. constraints, or for other reasons.
o In other cases, migration has occurred and the client can use the o In other cases, migration has occurred and the client can
procedure described in Section 4.2.2 to determine whether determine whether Transparent State Migration occurred and whether
Transparent State Migration occurred and whether any locking state any locking state was lost during the transfer.
was lost during the transfer.
Whether migration has occurred or not, the client can use the
procedure described in Section 5.4.3 to recover access to existing
locking state and, in some cases, sessions.
One should note the following differences between migration with One should note the following differences between migration with
Transparent State Migration and the similar cases in which there is a Transparent State Migration and the similar cases in which there is a
continuity of locking state with no change in the server. continuity of locking state with no change in the server.
o When locks are lost (as indicated when using them or via the o When locks are lost (as indicated when using them or via the
SEQ4_STAUS flags) and migration has not been done, they are not to SEQ4_STATUS flags) and migration has not been done, they are not
be reclaimed. Instead such losses are treated as lock revocations to be reclaimed, except when SEQ4_STATUS_RESTART_RECLAIM_NEEDED is
and acknowledged using FREE_STATEID. set. Instead such losses are treated as lock revocations and
acknowledged using FREE_STATEID.
o When migration has not been done, there is no need for a o When migration has not been done, there is no need for a
RECLAIM_COMPLETE (with rca_one_fs set to true). RECLAIM_COMPLETE (with rca_one_fs set to true).
4.2.5. Client Recovery from Migration Events 5.2.3. Migration and pNFS
When pNFS is involved, the protocol is capable of supporting:
o Migration of the MDS, leaving DS's in place.
o Migration of the file system as a whole, including the MDS and
associated DS's.
o Replacement of one DS by another.
o Migration of a pNFS file system to one in which pNFS is not used.
o Migration of a file system not using pNFS to one in which layouts
are available.
Migration of the MDS function is directly supported by Transparent
State Migration. Layout state will normally be transparently
transferred, just as other state is. As a result, Transparent State
Migration provides a framework in which, given appropriate inter-MDS
data transfer, one MDS can be substituted for another.
Migration of the file system function as a whole can be accomplished
by recalling all layouts as part of the initial phase of the
migration process. As a result, IO will be done through the MDS
during the migration process, and new layouts can be granted once the
client is interacting with the new MDS. An MDS can also effect this
sort of transition by revoking all layouts as part of Transparent
State Migration, as long as the client is notified about the loss of
state.
In order to allow migration to a file system on which pNFS is not
supported, clients need to be prepared for a situation in which
layouts are not available or supported on the destination file system
and so direct IO requests to the destination server, rather than
depending on layouts being available.
Replacement of one DS by another is not addressed by migration as
such but can be effected by an MDS recalling layouts for the DS to be
replaced and issuing new ones to be served by the successor DS.
Migration may transfer a file system from a server which does not
support pNFS to one which does. In order to properly adapt to this
situation, clients which support pNFS, but function adequately in its
absence, should check for pNFS support when a file system is migrated
and be prepared to use pNFS when support is available.
5.3. Defining Server Responsibilities for NFSv4.1
The subsections below discuss the responsibilities of source and
destination servers in effecting the necessary transfer of
information to support Transparent State Migration.
5.3.1. Server Responsibilities in Effecting Transparent State Migration
The basic responsibility of the source server in effecting
Transparent State Migration is to make available to the destination
server a description of each piece of locking state associated with
the file system being migrated. In addition to client id string and
verifier, the source server needs to provide, for each stateid:
o The stateid including the current sequence value.
o The associated client ID.
o The handle of the associated file.
o The type of the lock, such as open, byte-range lock, delegation,
layout.
o For locks such as opens and byte-range locks, there will be
information about the owner(s) of the lock.
o For recallable/revocable lock types, the current recall status
needs to be included.
o For each lock type there will by type-specific information, such
as share and deny modes for opens and type and byte ranges for
byte-range locks and layouts.
A further server responsibility concerns locks that are revoked or
otherwise lost during the process of file system migration. Because
locks that appear to be lost during the process of migration will be
reclaimed by the client, the servers have to take steps to ensure
that locks revoked soon before or soon after migration are not
inadvertently allowed to be reclaimed in situations in which the
continuity of lock possession cannot be assured.
o For locks lost on the source but whose loss has not yet been
acknowledged by the client (by using FREE_STATEID), the
destination must be aware of this loss so that it can deny a
request to reclaim them.
o For locks lost on the destination after the state transfer but
before the client's RECLAIM_COMPLTE is done, the destination
server should note these and not allow them to be reclaimed.
An additional responsibility of the cooperating servers concerns
situations in which a stateid cannot be transferred transparently
because it conflicts with an existing stateid held by the client and
associated with a different file system. In this case there are two
valid choices:
o Treat the transfer, as in NFSv4.0, as one without Transparent
State Migration. In this case, conflicting locks cannot be
granted until the client does a RECLAIM_COMPLETE, after reclaiming
the locks it had, with the exception of reclaims denied because
they were attempts to reclaim locks that had been lost.
o Implement Transparent State Migration, except for the lock with
the conflicting stateid. In this case, the client will be aware
of a lost lock (through the SEQ4_STATUS flags) and be allowed to
reclaim it.
5.3.2. Synchronizing Session Transfer
When transferring state between the source and destination, the
issues discussed in Section 7.2 of [RFC7931] must still be attended
to. In this case, the use of NFS4ERR_DELAY is still necessary in
NFSv4.1, as it was in NFSv4.0, to prevent locking state changing
while it is being transferred.
There are a number of important differences in the NFS4.1 context:
o The absence of RELEASE_LOCKOWNER means that the one case in which
an operation could not be deferred by use of NFS4ERR_DELAY no
longer exists.
o Sequencing of operations is no longer done using owner-based
operation sequences numbers. Instead, sequencing is session-
based
As a result, when sessions are not transferred, the techniques
discussed in [RFC7931] are adequate and will not be further
discussed.
When sessions are transferred, there are a number of issues that pose
challenges since,
o A single session may be used to access multiple file systems, not
all of which are being transferred.
o Requests made on a session may, even if rejected, affect the state
of the session by advancing the sequence number associated with
the slot used.
As a result, when the filesystem state might otherwise be considered
unmodifiable, the client might have any number of in-flight requests,
each of which is capable of changing session state, which may be of a
number of types:
1. Those requests that were processed on the migrating file system,
before migration began.
2. Those requests which got the error NFS4ERR_DELAY because the file
system being accessed was in the process of being migrated.
3. Those requests which got the error NFS4ERR_MOVED because the file
system being accessed had been migrated.
4. Those requests that accessed the migrating file system, in order
to obtain location or status information.
5. Those requests that did not reference the migrating file system.
It should be noted that the history of any particular slot is likely
to include a number of these request classes. In the case in which a
session which is migrated is used by filesystems other than the one
migrated, requests of class 5 may be common and be the last request
processed, for many slots.
Since session state can change even after the locking state has been
fixed as part of the migration process, the session state known to
the client could be different from that on the destination server,
which necessarily reflects the session state on the source server, at
an earlier time. In deciding how to deal with this situation, it is
helpful to distinguish between two sorts of behavioral consequences
of the choice of initial sequence ID values.
o The error NFS4ERR_SEQ_MISORDERED is returned when the sequence ID
in a request is neither equal to the last one seen for the current
slot nor the next greater one.
In view of the difficulty of arriving at a mutually acceptable
value for the correct last sequence value at the point of
migration, it may be necessary for the server to show some degree
of forbearance, when the sequence ID is one that would be
considered unacceptable if session migration were not involved.
o Returning the cached reply for a previously executed request when
the sequence ID in the request matches the last value recorded for
the slot.
In the cases in which an error is returned and there is no
possibility of any non-idempotent operation having been executed,
it may not be necessary to adhere to this as strictly as might be
proper if session migration were not involved. For example, the
fact that the error NFS4ERR_DELAY was returned may not assist the
client in any material way, while the fact that NFS4ERR_MOVED was
returned by the source server may not be relevant when the request
was reissued, directed to the destination server.
One part of adapting to these sorts of issues would restrict
enforcement of normal slot sequence enforcement semantics until the
client itself, by issuing a request using a particular slot on the
destination server, established the new starting sequence for that
slot on the migrated session.
An important issue is that the specification needs to take note of
all potential COMPOUNDs, even if they might be unlikely in practice.
For example, a COMPOUND is allowed to access multiple file systems
and might perform non-idempotent operations in some of them before
accessing a file system being migrated. Also, a COMPOUND may return
considerable data in the response, before being rejected with
NFS4ERR_DELAY or NFS4ERR_MOVED, and may in addition be marked as
sa_cachethis.
Some possibilities that need to be considered to address the issues:
o Do not enforce any sequencing semantics for a particular slot
until the client has established the starting sequence for that
slot on the destination server.
o For each slot, do not return a cached reply returning
NFS4ERR_DELAY or NFS4ERR_MOVED until the client has established
the starting sequence for that slot on the destination server.
o Until the client has established the starting sequence for a
particular slot on the destination server, do not report
NFS4ERR_SEQ_MISORDERED or return a cached reply returning
NFS4ERR_DELAY or NFS4ERR_MOVED, where the reply consists solely of
a series of operations where the response is NFS4_OK until the
final error.
5.4. Defining Client Responsibilities for NFSv4.1
The subsections below discuss the responsibilities of the client in
dealing with transition to a new server (migration) and to use of new
network addresses in accessing existing servers.
5.4.1. Client Recovery from Migration Events
When a file system is migrated, there a number of migration-related When a file system is migrated, there a number of migration-related
status indications with which clients need to deal: status indications with which clients need to deal:
o If an attempt is made to use or return a filehandle within a file o If an attempt is made to use or return a filehandle within a file
system that has been migrated away from the server on which it was system that has been migrated away from the server on which it was
previously available, the error NFS4ERR_MOVED is returned. previously available, the error NFS4ERR_MOVED is returned.
This condition continues on subsequent attempts to access the file This condition continues on subsequent attempts to access the file
system in question. The only way the client can avoid the error system in question. The only way the client can avoid the error
skipping to change at page 16, line 28 skipping to change at page 28, line 37
notification by fetching a location attribute for the migrated notification by fetching a location attribute for the migrated
file system. When there are multiple migrated file systems, a file system. When there are multiple migrated file systems, a
location attribute for each such migrated file system needs to be location attribute for each such migrated file system needs to be
fetched, in order to clear the condition. Even after the fetched, in order to clear the condition. Even after the
condition is cleared, the client needs to respond by using the condition is cleared, the client needs to respond by using the
location information to access the destination server to ensure location information to access the destination server to ensure
that leases are not needlessly expired. that leases are not needlessly expired.
Unlike the case of NFSv4.0 in which the corresponding conditions are Unlike the case of NFSv4.0 in which the corresponding conditions are
both errors, in NFSv4.1 the client can, and often will, receive both both errors, in NFSv4.1 the client can, and often will, receive both
indications on the same request. As a result, the question of how to indications on the same request. As a result, implementations need
co-ordinate the necessary recovery actions when both indications to address the question of how to co-ordinate the necessary recovery
arrive simultaneously must be resolved. It should be noted that when actions when both indications arrive simultaneously. It should be
the server decides whether SEQ4_STATUS_LEASE_MOVED is ti be set, it noted that when the server decides whether SEQ4_STATUS_LEASE_MOVED is
has no way of knowing which file system will be referenced or whether to be set, it has no way of knowing which file system will be
NFS4ERR_MOVED will be returned. referenced or whether NFS4ERR_MOVED will be returned.
While it is true that, when only a single migrated file system is While it is true that, when only a single migrated file system is
involved, a single set of actions will clear both indications, the involved, a single set of actions will clear both indications, the
possibility of multiple migrated file systems calls for an approach possibility of multiple migrated file systems calls for an approach
in which there are separate recovery actions for each indication. In in which there are separate recovery actions for each indication. In
general, the response to neither indication can be subsumed within general, the response to neither indication can be subsumed within
the other since: the other since:
o If the client were to respond only to the MOVED indication, there o If the client were to respond only to the MOVED indication, there
would be no effective client response to a situation in which a would be no effective client response to a situation in which a
skipping to change at page 17, line 14 skipping to change at page 29, line 23
by a particular server, or when it happens that some servers, by a particular server, or when it happens that some servers,
after receiving migrated file systems have periods of after receiving migrated file systems have periods of
unavailability, such as occur as a result of server reboot. This unavailability, such as occur as a result of server reboot. This
can result in recovery for actively accessed migrated file systems can result in recovery for actively accessed migrated file systems
being unnecessarily delayed for long periods of time. being unnecessarily delayed for long periods of time.
Similar considerations apply to other arrangements in which one of Similar considerations apply to other arrangements in which one of
the indications, while not ignored per se, is subsumed within a the indications, while not ignored per se, is subsumed within a
single recovery process focused on recovery for the other indication. single recovery process focused on recovery for the other indication.
Generally speaking, client recovery for these indications should have Although clients are free to decide on their own approaches to
the following characteristics: recovery, we will explore below an approach with the following
characteristics:
o All instances of the MOVED indication should be dealt with o All instances of the MOVED indication, whether they involve
promptly, either by doing the necessary recovery directly, migration or not, should be dealt with promptly, either by doing
providing that it be done asynchronously, or ensuring that it is the necessary recovery directly, providing that it be done
already under way. asynchronously, or ensuring that it is already under way.
o All instances of the LEASE_MOVED indication should be dealt with o All instances of the LEASE_MOVED indication should be dealt with
asynchronously, in a migration discovery thread whose job is to asynchronously, in a migration discovery thread whose job is to
clear that indication by fetching the appropriate location clear that indication by fetching the appropriate location
attribute. Because this thread will only be fetching a location attribute. Because this thread will only be fetching a location
attribute and the fs_status attribute for the file systems attribute and the fs_status attribute for the file systems
referenced by the client, it cannot receive MOVED indications. referenced by the client, it cannot receive MOVED indications.
Some useful guidance regarding possible implementation of the Some useful guidance regarding possible implementation of a
migration discovery thread can be found in Section 4.2.6. migration discovery thread can be found in Section 5.4.2.
o When a migration discovery thread happens upon a migrated file o When a migration discovery thread happens upon a migrated file
system (i.e. not present and not a referral), the thread is likely system (i.e. not present and not a referral), the thread is likely
to have cleared one (out of an unknown number) of file systems to have cleared one (out of an unknown number) of file systems
whose migration needs to be responded to. The discovery thread whose migration needs to be responded to. The discovery thread
needs to schedule the appropriate migration recovery (as described needs to schedule the appropriate migration recovery (as described
in Section 4.2.3). This is necessary to ensure that migrated file in Section 5.4.3). This is necessary to ensure that migrated file
systems will be referenced on the destination server in order to systems will be referenced on the destination server in order to
avoid lease expiration avoid unnecessary lease expiration.
For many of the migrated file systems discovered in this way, the For many of the migrated file systems discovered in this way, the
client has not received any MOVED indication. In such cases, client has not received any MOVED indication. In such cases,
lease recovery needs to be scheduled but it should not interfere lease recovery needs to be scheduled but it should not interfere
with continuation of the migration discovery function. with continuation of the migration discovery function.
o When a migration discovery thread receives a LEASE_MOVED o When a migration discovery thread receives a LEASE_MOVED
indication, it takes no special action but continues its normal indication, it takes no special action but continues its normal
operation. On the other hand, if a LEASE_MOVED indication is not operation. On the other hand, if a LEASE_MOVED indication is not
received, it indicates that the thread has completed its work received, it indicates that the thread has completed its work
successfully. successfully.
4.2.6. The Migration Discovery Process 5.4.2. The Migration Discovery Process
As noted above, LEASE_MOVED indications are best dealt with in a As noted above, LEASE_MOVED indications are best dealt with in a
migration discovery thread. Because of this structure, migration discovery thread. Because of this structure,
o No action needs to be taken for such indications received by the o No action needs to be taken for such indications received by the
migration discovery threads, since continuation of that thread's migration discovery threads, since continuation of that thread's
work will address the issue. work will address the issue.
o For such indications received in other contexts, the generally o For such indications received in other contexts, the generally
appropriate response is to initiate or otherwise provide for the appropriate response is to initiate or otherwise provide for the
execution of a migration discovery thread for file systems execution of a migration discovery thread for file systems
associated with the server IP address returning the indication. associated with the server IP address returning the indication.
o In all cases in which the appropriate migration discovery thread o In all cases in which the appropriate migration discovery thread
is running, nothing further need be done to respond to LEASE_MOVED is running, nothing further needs to be done to respond to
indications. LEASE_MOVED indications.
This leaves a potential difficulty in situations in which the This leaves a potential difficulty in situations in which the
migration discovery thread is near to completion but is still migration discovery thread is near to completion but is still
operating. One should not ignore a LEASE_MOVED indication if the operating. One should not ignore a LEASE_MOVED indication if the
discovery thread is not able to respond to migrated file system discovery thread is not able to respond to migrated file system
without additional aid. A further difficulty in addressing such without additional aid. A further difficulty in addressing such
situation is that a LEASE_MOVED indication may reflect the server's situation is that a LEASE_MOVED indication may reflect the server's
state at the time the SEQUENCE operation was processed, which may be state at the time the SEQUENCE operation was processed, which may be
different from that in effect at the time the response is received. different from that in effect at the time the response is received.
skipping to change at page 19, line 25 skipping to change at page 31, line 34
o Otherwise, if there is any record that other requests saw a o Otherwise, if there is any record that other requests saw a
LEASE_MOVED indication, that record is cleared and the LEASE_MOVED indication, that record is cleared and the
verification request retried. The discovery thread remains in verification request retried. The discovery thread remains in
completion/verification state. completion/verification state.
o If there has been no LEASE_MOVED indication, the work of the o If there has been no LEASE_MOVED indication, the work of the
discovery thread is considered completed and it enters the non- discovery thread is considered completed and it enters the non-
operating state. operating state.
4.2.7. Synchronzing Session Transfer 5.4.3. Overview of Client Response to NFS4ERR_MOVED
When transferring state between the source and destination, the This section outlines a way in which a client that receives
issues discussed in Section 7.2 of [RFC7931] must still be attended NFS4ERR_MOVED can respond by using a new server or network address if
to. In this case, the use of NFS4ERR_DELAY is still necessary in one is available. As part of that process, it will determine:
NFSv4.1, as it was in NFSv4.0, to prevent locking state changing
while it is being transferred.
There are a number of important differences in the NFS4.1 context: o Whether the NFS4ERR_MOVED indicates migration has occurred, or
whether it indicates another sort of file system transition as
discussed in Section 5.2.2.
o The absence of RELEASE_LOCKOWNER means that the one case in which o In the case of migration, whether Transparent State Migration has
an operation could not be deferred by use of NFS4ERR_DELAY no occurred.
longer exists.
o Sequencing of operations is no longer done using owner-based o Whether any state has been lost during the process of Transparent
operation sequences numbers. Instead, sequencing is session- State Migration.
based
As a result, when sessions are not transferred, the techniques o Whether sessions have been transferred as part of Transparent
discussed in [RFC7931] are adequate and will not be further State Migration.
discussed.
When sessions are transferred, there are a number of issues that pose During the first phase of this process, the client proceeds to
challenges since, examine location entries to find the initial network address it will
use to continue access to the file system or its replacement. For
each location entry that the client examines, the process consists of
five steps:
o A single session may be used to access multiple file systems, not 1. Performing an EXCHANGE_ID is directed at the location address.
all of which are being transferred. This operation is used to register the client-owner with the
server, to obtain a client ID to be use subsequently to
communicate with it, to obtain tat client ID's confirmation
status and, to determine server_owner and scope for the purpose
of determining if the entry is trunkable with that previously
being used to access the file system (i.e. that it represents
another path to the same file system and can share locking state
with it).
o Requests made on a session, even if rejected may, affect the state 2. Making an initial determination of whether migration has
of the session by advancing the sequence number associated with occurred. The initial determination will be based on whether the
the slot used. EXCHANGE_ID results indicate that the current location element is
server-trunkable with that used to access the file system when
access was terminated by receiving NFS4ERR_MOVED. If it is, then
migration has not occurred and the transition is dealt with, at
least initially, as one involving continued access to the same
file system on the same server through a new network address.
As a result, when the filesystem state might otherwise be considered 3. Obtaining access to existing session state or creating new
unmodifiable, the client might have any number of in-flight requests, sessions. How this is done depends on the initial determination
each of which is capable of changing session state, which may be of a of whether migration has occurred and can be done as described in
number of types: Section 5.4.4 in the case of migration or as described in
Section 5.4.5 in the case of a network address transfer without
migration.
1. Those requests that were processed on the migrating file system, 4. Verification of the trunking relationship assumed in step 2 as
before migration began. discussed in Section 2.10.5.1 of [RFC5661]. Although this step
will generally confirm the initial determination, it is possible
for verification to fail with the result that an initial
determination that a network address shift (without migration)
has occurred may be invalidated and migration determined to have
occurred. There is no need to redo step 3 above, since it will
be possible to continue use of the session established already.
2. Those requests which got the error NFS4ERR_DELAY because the file 5. Obtaining access to existing locking state and/or reobtaining it.
system being accessed was in the process of being migrated. How this is done depends on the final determination of whether
migration has occurred and can be done as described in
Section 5.4.4 in the case of migration or as described in
Section 5.4.5 in the case of a network address transfer without
migration.
3. Those requests which got the error NFS4ERR_MOVED because the file Once the initial address has been determined, clients are free to
system being accessed had been migrated. apply an abbreviated process to find additional addresses trunkable
with it (clients may seek session-trunkable or server trunkable
addresses depending on whether they support clientid trunking).
During this later phase of the process, further location entries are
examined using the abbreviated procedure specified below:
4. Those requests that accessed the migrating file system, in order 1. Before the EXCHANGE_ID, the fs_name field is examined and if it
to obtain location or status information. does not match that currently being used, the entry is ignored.
otherwise, one proceeds as specified by step 1 above,.
5. Those requests that did not reference the migrating file system. 2. In the case that the network address is session-trunkable with
one used previously a BIND_CONN_TO_SESSION is used to access that
session using new network address. Otherwise, or if the bind
operation fails, a CREATE_SESSION is done.
It should be noted that the history of any particular slot is likely 3. The verification procedure referred to in step 4 above is used.
to include a number of these request classes. In the case in which a However, if it fails, the entry is ignored and the next available
session which is migrated is used by filesystems other than the one entry is used.
migrated, requests of class 5 may be common and be the last request
processed, for many slots.
Since session state can change even after the locking state has been 5.4.4. Obtaining Access to Sessions and State after Migration
fixed as part of the migration process, the session state known to
the client could be different from that on the destination server,
which necessarily reflects the session state on the source server, at
an earlier time. In deciding how to deal with this situation, it is
helpful to distinguish between two sorts of behavioral consequences
of the choice of initial sequence ID values.
o The error NFS4ERR_SEQ_MISORDERED is returned when the sequence ID In the event that migration has occurred, the determination of
in a request is neither equal to the last one seen for the current whether Transparent State Migration has occurred is driven by the
slot nor the next greater one. client ID returned by the EXCHANGE_ID and the reported confirmation
status.
In view of the difficulty of arriving at a mutually acceptable o If the client ID is an unconfirmed client ID not previously known
value for the correct last sequence a the point of migration, it to the client, then Transparent State Migration has not occurred.
may be necessary for the server to show some degree of
forbearance, when the sequence ID is one that would be considered
unacceptable if session migration were not involved.
o Returning the cached reply for a previously executed request when o If the client ID is a confirmed client ID previously known to the
the sequence ID in the request matches the last value recorded for client, then any transferred state would have been merged with an
the slot. existing client ID representing the client to the destination
server. In this state merger case, Transparent State Migration
might or might not have occurred and a determination as to whether
it has occurred is deferred until sessions are established and we
are ready to begin state recovery.
In the cases in which an error is returned and there is no o If the client ID is a confirmed client ID not previously known to
possibility of any non-idempotent operation having been executed, the client, then the client can conclude that the client ID was
it may not be necessary to adhere to this as strictly as might be transferred as part of Transparent State Migration. In this
proper if session migration were not involved. For example, the transferred client ID case, Transparent State Migration has
fact that the error NFS4ERR_DELAY was returned may not assist the occurred although some state may have been lost.
client in any material way, while the fact that NFS4ERR_MOVED was
returned by the source server may not be relevant when the request
was reissued, directed to the destination server.
One part of adapting to these sorts of issues would restrict Once the client ID has been obtained, it is necessary to obtain
enforcement of normal slot sequence enforcement semantics until the access to sessions to continue communication with the new server. In
client itself, by issuing a request using a particular slot on the any of the cases in which Transparent State Migration has occurred,
destination server, established the new starting sequence for that it is possible that a session was transferred as well. To deal with
slot on the migrated session. that possibility, clients can, after doing the EXCHANGE_ID, issue a
BIND_CONN_TO_SESSION to connect the transferred session to a
connection to the new server. If that fails, it is an indication
that the session was not transferred and that a new session needs to
be created to take its place.
An important issue is that the specification needs to take note of In some situations, it is possible for a BIND_CONN_TO_SESSION to
all potential COMPOUNDs, even if they might be unlikely in practice. succeed without session migration having occurred. If state merger
For example, a COMPOUND is allow to access multiple file systems and has taken place then the associated client ID may have already had a
might perform non-idempotent operations in some of them before set of existing sessions, with it being possible that the sessionid
accessing a file system being migrated. Also, a COMPOUND may return of a given session is the same as one that might have been migrated.
considerable data in the response, before being rejected with In that event, a BIND_CONN_TO_SESSION might succeed, even though
NFS4ERR_DELAY or NFS4ERR_MOVED, and may in addition be marked as there could have been no migration of the session with that
sa_cachethis. sessionid.
Some possibilities that need to be considered to address the issues: Once the client has determined the initial migration status, and
determined that there was a shift to a new server, it needs to re-
establish its lock state, if possible. To enable this to happen
without loss of the guarantees normally provided by locking, the
destination server needs to implement a per-fs grace period in all
cases in which lock state was lost, including those in which
Transparent State Migration was not implemented.
o Do not enforce any sequencing semantics for a particular slot Clients need to be deal with the following cases:
until the client has established the starting sequence for that
slot on the destination server.
o For each slot, do not return a cached reply returning o In the state merger case, it is possible that the server has not
NFS4ERR_DELAY or NFS4ERR_MOVED until the client has established attempted Transparent State Migration, in which case state may
the starting sequence for that slot on the destination server. have been lost without it being reflected in the SEQ4_STATUS bits.
To determine whether this has happened, the client can use
TEST_STATEID to check whether the stateids created on the source
server are still accessible on the destination server. Once a
single stateid is found to have been successfully transferred, the
client can conclude that Transparent State Migration was begun and
any failure to transport all of the stateids will be reflected in
the SEQ4_STATUS bits. Otherwise. Transparent State Migration has
not occurred.
o Until the client has established the starting sequence for a o In a case in which Transparent State Migration has not occurred,
particular slot on the destination server, do not report the client can use the per-fs grace period provided by the
NFS4ERR_SEQ_MISORDERED or return a cached reply returning destination server to reclaim locks that were held on the source
NFS4ERR_DELAY or NFS4ERR_MOVED, where the reply consists solely of server.
a series of operations where the response is NFS4_OK until the
final error.
4.2.8. Migration and pNFS o In a case in which Transparent State Migration has occurred, and
no lock state was lost (as shown by SEQ4_STATUS flags), no lock
reclaim is necessary.
When pNFS is involved, migration is capable of supporting: o In a case in which Transparent State Migration has occurred, and
some lock state was lost (as shown by SEQ4_STATUS flags), existing
stateids need to be checked for validity using TEST_STATEID, and
reclaim used to re-establish any that were not transferred.
o Migration of the MDS, leaving DS's in place. For all of the cases above, RECLAIM_COMPLETE with an rca_one_fs value
of true should be done before normal use of the file system including
obtaining new locks for the file system. This applies even if no
locks were lost and needed to be reclaimed.
o Migration of the file system as a whole, including the MDS and 5.4.5. Obtaining Access to Sessions and State after Network Address
associated DS's. Transfer
o Replacement of one DS by another. The case in which there is a transfer to a new network address
without migration is similar to that described in Section 5.4.4 in
that there is a need to obtain access to needed sessions and locking
state. However, the details are simpler and will vary depending on
the type of trunking between the address receiving NFS4ERR_MOVED and
that to which the transfer is to be made
o Migration of a pNFS file system to one in which pNFS is not used. To make a session available for use, a BIND_CONN_TO_SESSION should be
used to obtain access to the session previously in use. Only if this
fails, should a CREATE_SESSION be done. While this procedure mirrors
that in Section 5.4.4, there is an important difference in that
preservation of the session is not purely optional but depends on the
type of trunking.
o Migration of a file system not using pNFS to one in which layouts Access to appropriate locking state should need no actions beyond
are available. access to the session. However. the SEQ4_STATUS bits should be
checked for lost locking state, including the need to reclaim locks
after a server reboot.
Migration of the MDS function is directly supported by Transparent 5.5. Resolution of NFSv4.1 Issues
State Migration. Layout state will normally be transparently
transferred, just as other state is. As a result, Transparent State
Migration provides a framework in which, given appropriate inter-MDS
data transfer, one MDS can be substituted for another.
Migration of the file system function can be accomplished by One possibility is that addressing all of the NFSv4.1 issues would
recalling all layouts as part of the initial phase of the migration entail publication of a standards-track document updating [RFC5661].
process. As a result, IO will be done through the MDS during the
migration process, and new layouts can be granted once the client is
interacting with the new MDS. An MDS can also effect this sort of
transition by revoking all layouts as part of Transparent State
Migration, as long as the client is notified about the loss of state.
In order to allow migration to a file system on which pNFS is not Such a document would have three major elements:
supported, clients need to be prepared for a situation in layouts are
not available or supported on the destination file system and be
prepared to direct IO request to the destination server, rather than
depending on layouts being available.
Replacement of one DS by another is not addressed by migration as o A considerable expansion of the existing Section 11.4 explaining
such but can be effected by an MDS recalling layouts for the DS to be the various uses of the location attribute and the possible
replaced and issuing new ones to be served by the successor DS. interactions among these various uses. This, like the
corresponding replacement section for NFSv4.0 would be based on
our Section 3.2 above. Information regarding the specifics of
trunking discovery might appear in this section, in a new sub-
section. As part of this revision, the existing Section 11.4.2
would need to be revised to explain all the possible results of
NFS4ERR_MOVED including migration and a possible transparent
transition in which the network address changes but the server
does not.
Migration may transfer a file system from a server which does not o A revision of the existing section 18.35 (dealing with
support pNFS to one which does. In order to properly adapt to this EXCHANGE_ID) addressing the issues discussed in Section 5.2.1.
situation, clients which support pNFS, but function adequately in its
absence, should check for pNFS support when a file system is migrated
and be prepared to use pNFS when support is available.
5. Security Considerations o A major replacement of the existing Section 11.7, entitled
"Effecting File System Transitions", as discussed below.
In addition, there is a set of smaller changes necessary
o Update the existing Section 2.10.5 to clarify the proper response
to server_owner changes, as described in our Section 5.1.3.
o Replacement of the existing Section 15.1.2.4 to reflect the fact
that NFS4ERR_MOVED can occur when a file system is now accessible
at a different network address. A possible replacement text might
be:
The file system that contains the current filehandle object is
not accessible using the network address which has been used.
It may have been relocated, migrated to another server, be
accessible using another network address on the current server,
or it may have never been present. The client may obtain the
new file system location by obtaining the "fs_locations" or
"fs_locations_info" attribute for the current filehandle. For
further discussion, refer to Section 11.4.2
The replacement for the existing section 11.7 would maintain most
sections essentially as they are, only making minor changes to
include server-trunking in the discussion. However, in some cases
involving more significant changes to existing sub-sections, and
potential new sub-sections are listed below:
o The existing Section 11.7.1 needs to be modified to refer
explicitly to the previous discussion of trunking discovery.
In addition, the term "multi-home single-server namespace", used
nowhere else in [RFC5661], poses difficulties. From the
description given it appears that the case being referred to in
one in which two network addresses return server_owners with the
same major_id and different minor_id values, making the network
addresses server-trunkable without being session trunkable.
A better approach would be to refer to "server-trunking" as used
elsewhere in this document and use the replacement for the
existing Section 18.35 to identify clientid trunking as the means
to adapt to network addresses which are server-trunkable without
being session-trunkable and session trunking as the means to adapt
to network addresses which are session-trunkable.
o The existing Section 11.7.2 needs to be better connected to
trunking discovery. By calling these "transparent" transitions,
it obscures the fact that some (or all) of the "transitions" it is
discussing are not in fact transitions between servers or file
systems but merely changes the set of communication paths in use.
o The existing Section 11.7.2.1, needs to address more clearly the
case of server-trunkable addresses which are not session-
trunkable. As it is, it mentions the related concept of
clustering, but only deals explicitly with the case in which two
distinct servers share access to one or more file systems and does
not mention the case in which the network addresses can be used to
access a shared stateid space without being session-trunkable.
o The existing Section 11.7.2.2, while correct, needs to be part of
a general re-organization since the characteristics it lists as
necessary for a transparent transition will be of use in other
contexts, particularly as they apply to Transparent State
Migration as well. It make sense to move these to a new sub-
section within the equivalent of the Existing Section 11.7.
o The existing Section 11.7.7, needs the a major rework to deal with
its basic assumption, that existing state can only be made
available on the destination server if the source and destination
co-operate in state management and maintain a common client id
space. It is not clear how this can be done, other than for
servers working together so as to provide clientid trunking, a
case that is probably considered as a "transparent transition".
The section needs to modified to allow something along the lines
of NFSv4.0-style Transparent State Migration with the details
provided by a later section (see below).
A related issues concerns the sentence, "In the case of migration,
the servers involved in the migration of a file system SHOULD
transfer all server state from the original to the new server. It
is unclear why this is a "SHOULD" as the rest of the paragraph
essentially tells the client that it needs to be prepared for the
server not to do this. The equivalent is a "should" in [RFC7931],
and there is no reason to add to confusion by making a "SHOULD" in
NFSv4.1. also, there is no mention of the need to provide a fs-
specific grace period in the cases in which Transparent State
Migration is not made available.
o Adding a new section (at level of the existing Section 11.7.7)
about state transfer during migration. Although the phrase
"Transparent State Migration" is well established in the context
of NFSv4.0, the word "transparent" could cause confusion given the
existing use of the phrase "transparent transitions". A possible
title for the new section is "State Transfer during Migration"
The new section would present the NFSv4.1-equivalent of Transparent
State Migration as described in [RFC7931]. This would address the
issues presented in Section 5.1 along the lines suggested in Sections
5.2, 5.3, and 5.4.
6. Security Considerations
With regard to NFSv4.0, the Security Considerations section of With regard to NFSv4.0, the Security Considerations section of
[RFC7530] encourages clients to protect the integrity of the SECINFO [RFC7530] as modified by that of [RFC7931] takes proper care of
operation, any GETATTR operation for the fs_locations attribute. A migration-related issues. No change is needed.
needed change is to include the operations SETCLIENTID/
SETCLIENTID_CONFIRM as among those for which integrity protection is
recommended. A migration recovery event can use any or all of these
operations.
With regard to NFSv4.1, the Security Considerations section of With regard to NFSv4.1, the Security Considerations section of
[RFC5661] takes proper care of migration-related issues. No change [RFC5661] takes proper care of migration-related issues. No change
is needed. is needed.
6. IANA Considerations 7. IANA Considerations
This document does not require actions by IANA. This document does not require actions by IANA.
7. Normative References 8. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>. <http://www.rfc-editor.org/info/rfc2119>.
[RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1 "Network File System (NFS) Version 4 Minor Version 1
Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010,
<http://www.rfc-editor.org/info/rfc5661>. <http://www.rfc-editor.org/info/rfc5661>.
skipping to change at page 23, line 47 skipping to change at page 38, line 50
March 2015, <http://www.rfc-editor.org/info/rfc7530>. March 2015, <http://www.rfc-editor.org/info/rfc7530>.
[RFC7931] Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker, [RFC7931] Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker,
"NFSv4.0 Migration: Specification Update", RFC 7931, "NFSv4.0 Migration: Specification Update", RFC 7931,
DOI 10.17487/RFC7931, July 2016, DOI 10.17487/RFC7931, July 2016,
<http://www.rfc-editor.org/info/rfc7931>. <http://www.rfc-editor.org/info/rfc7931>.
Appendix A. Acknowledgements Appendix A. Acknowledgements
The editor and authors of this document gratefully acknowledge the The editor and authors of this document gratefully acknowledge the
contributions of Trond Myklebust of NetApp and Robert Thurlow of contributions of Trond Myklebust of Primary Data, Robert Thurlow of
Oracle. We also thank Tom Haynes of Primary Data and Spencer Shepler Oracle, and Andy Adamson of NetApp. We also thank Tom Haynes of
of Microsoft for their guidance and suggestions. Primary Data and Spencer Shepler of Microsoft for their guidance and
suggestions.
Special thanks go to members of the Oracle Solaris NFS team, Special thanks go to members of the Oracle Solaris NFS team,
especially Rick Mesta and James Wahlig, for their work implementing especially Rick Mesta and James Wahlig, for their work implementing
an NFSv4.0 migration prototype and identifying many of the issues an NFSv4.0 migration prototype and identifying many of the issues
documented here. documented here. Also, the work of Xuan Qi of Oracle with NFSv4.1
client and server prototypes was helpful.
Appendix B. History of this Document
The contents of successive versions of this document have changed
because new issues have been discovered, because there here have
changes in our understanding of how these features should interact,
and because some of the issues have been adequately addressed with
regard to certain protocol versions.
As a result, it may be helpful to understand the history of these
issues, which is complicated because multiple NFSv4 protocols have
been involved.
This history can be summarized as follows
o Initially, the focus was on the difficulties seen in NFSv4.0
implementations of Transparent State Migration, and on identifying
possible corrections to [RFC7530] that might address these issues.
At this point, treatment of NFSv4.1 was minimal.
o As examination of the issues continued, it became clear that the
use of the non-uniform client string model was a critical element
of the problem and further work proceeded on that basis.
During the period, treatment of NFSv4.1 was expanded but the fact
that NFSv4.1 had existing facilities for trunking detection was
taken as an indication that the problems would not be difficult to
address..
o As work proceeded on a standards-track document addressing the
NFSv4.0 issues, material that proposed changes to address the
issues became less relevant, since the effective vehicle for
addressing these issues became the standards-track document.
During this period, and subsequently, treatment of NFSv4.1
remained essentially unchanged.
o With the publication of [RFC7931], material regarding fixes for
the NSV4.0 became vestigial but the material was retained for a
while together with a shift from proposing those changes to
reporting that they had been made.
o Later, in response to experiences testing existing NFSv4.1
implementations of migration, the focus of the document shifted
decisively to NFSv4.1. As part of the analysis of migration
within NFSv4.1, it was realized that issues related to the
appearance of multiple addresses were fundamental to clearly
describing how migration would work and that changes in the set of
such addresses might or might not involve migration.
At this point, discussion of NFSV4.0 issues was further limited.
The issues seen were noted but the discussion of the resolution
was limited to explaining that they had been addressed by the
publication of [RFC7931].
o Finally, based on the results of work to provide NFSv4 with
trunking discovery facilities, a decision was made that this work
was most appropriately dealt with together with migration, for
reasons noted previously.
Since the trunking discovery facilities apply to all NFSv4 minor
versions, work was needed to define those for NFSv4.0as well,
together with the necessary interactions with migration.
Authors' Addresses Authors' Addresses
David Noveck (editor) David Noveck (editor)
NetApp NetApp
26 Locust Avenue 26 Locust Avenue
Lexington, MA 02421 Lexington, MA 02421
US US
Phone: +1 781 572 8038 Phone: +1 781 572 8038
 End of changes. 114 change blocks. 
503 lines changed or deleted 1288 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/