NFSv4                                                     D. Noveck, Ed.
Internet-Draft                                                    NetApp
Intended status: Informational                                 P. Shivam
Expires: August 12, September 30, 2017                                     C. Lever
                                                                B. Baker
                                                                  ORACLE
                                                        February 8,
                                                          March 29, 2017

  NFSv4 migration: Implementation Experience and Specification Issues
                  draft-ietf-nfsv4-migration-issues-11
                  draft-ietf-nfsv4-migration-issues-12

Abstract

   The migration feature of NFSv4 provides for moving responsibility for
   a single filesystem from one server to another, without disruption to
   clients.  Implementation experience has shown  A number of problems in the specification for of this feature
   in RFC7530.  This document explains
   the choices made to address these issues NFSv4.0 were resolved by updating the NFSv4.0 publication of RFC 7931.  In
   addition, there are specification in RFC7931 and those issues to be made resolved with regard
   to the NFSv4.1 specification, version of this feature which are discussed in order to properly address migration. this
   document.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on August 12, September 30, 2017.

Copyright Notice

   Copyright (c) 2017 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Conventions . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  NFSv4.0 Implementation Experience . Issues and Their Resolution . . . . . . . . . . . . .   3
     3.1.  Implementation  NFSv4.0 Issues  . . . . . . . . . . . . . . . . . .   3
       3.1.1.  Failure to Free Migrated State on Client Reboot . . .   4
       3.1.2.  Server Reboots Resulting in a Confused Lease
               Situation . . . . . . . . . . . . . . .   3
     3.2.  Resolution of NFSv4.0 Protocol Difficulties . . . . . . .   4
       3.1.3.  Client Complexity
   4.  Issues for NFSv4.1  . . . . . . . . . . . . . .   5
     3.2.  Sources of Protocol Difficulties  . . . . . . . . . . . .   7
       3.2.1.  Issues with nfs_client_id4 Generation and Use . . . .   7
       3.2.2.   5
     4.1.  Issues with Lease Proliferation . . . to Address for NFSv4.1 . . . . . . . .   9
   4.  Resolution of NFSv4.0 Protocol Difficulties . . . . . .   5
       4.1.1.  Addressing state merger in NFSv4.1  . . .   9
     4.1.  Changes Regarding nfs_client_id4 Client-string . . . . .   9
     4.2.  Changes Regarding Merged (vs. Synchronized) Leases .   6
       4.1.2.  Addressing pNFS relationship with migration . .  10
     4.3.  Other Changes to Migration-state Sections . . .   7
       4.1.3.  Addressing server_owner changes in NFSv4.1  . . . . .  11
       4.3.1.  Changes Regarding   7
       4.1.4.  Addressing Confirmation Status of Migrated
               Client ID Migration . . . . . IDs in NFSv4.1 . . .  12
       4.3.2.  Changes Regarding Callback Re-establishment . . . . .  12
       4.3.3.  NFS4ERR_LEASE_MOVED Rework . . . . . . . .   8
       4.1.5.  Addressing Session Migration in NFSv4.1 . . . . .  13
     4.4.  Changes to Other Sections . .   9
     4.2.  Possible Resolutions for NFSv4.1 Issues . . . . . . . . .   9
       4.2.1.  Server Responsibilities in Effecting Transparent
               State Migration . . . . .  13
       4.4.1.  Need for Additional Changes . . . . . . . . . . . . .  13
       4.4.2.  Callback Update .  10
       4.2.2.  Determining Initial Migration Status in NFSv4.1 . . .  11
       4.2.3.  Client Response to Migration in NFSv4.1 . . . . . . .  13
       4.2.4.  Dealing with Multiple Location Entries  . . . . . . .  13
       4.2.5.  Client Recovery from Migration Events .  14
       4.4.3.  clientid4 Handling . . . . . . .  15
       4.2.6.  The Migration Discovery Process . . . . . . . . . .  14
       4.4.4.  Handling of NFS4ERR_CLID_INUSE .  18
       4.2.7.  Synchronzing Session Transfer . . . . . . . . . .  16
   5.  Issues for NFSv4.1 . .  19
       4.2.8.  Migration and pNFS  . . . . . . . . . . . . . . . . .  22
   5.  Security Considerations . .  17
     5.1.  Addressing state merger in NFSv4.1 . . . . . . . . . . .  17
     5.2.  Addressing pNFS relationship with migration . . . . . .  23
   6.  IANA Considerations .  18
     5.3.  Addressing server owner changes in NFSv4.1 . . . . . . .  18
   6.  Security Considerations . . . . . . . . . . . . .  23
   7.  Normative References  . . . . . .  19
   7.  IANA Considerations . . . . . . . . . . . . . .  23
   Appendix A.  Acknowledgements . . . . . . .  20
   8.  Acknowledgements . . . . . . . . . . .  23
   Authors' Addresses  . . . . . . . . . . .  20
   9.  Normative References . . . . . . . . . . . . . . . . . . . .  20
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  20  24

1.  Introduction

   This document document. which deals with existing issues/problems in
   standards-track documents, is in the informational category, and
   while the facts it reports may have normative implications, any such
   normative significance reflects the readers' preferences.  For
   example, we may report that the reboot existing definition of a client with migrated migration for
   NFSv4.1 does not properly describe how migrating state results in is to be
   merged with existing state not being promptly cleared and that this will prevent granting
   of conflicting lock requests at least for the lease time, which is a
   fact. destination server.  While it is
   to be expected that client and server implementers will judge this to
   be a situation that is best avoided, the judgment as to how pressing
   this issue should be considered is a judgment for the reader, and
   eventually the nfsv4 working group to make.

   We do explore possible ways in which such issues can be avoided, with
   minimal negative effects, given that the working group has decided to
   address these issues, but the choice of exactly how to address these
   is best given effect in one or more standards-track documents and/or
   errata.

   This document focuses on NFSv4.0, NFSv4.1, since that is where the majority of
   implementation experience has been.  Nevertheless, there is
   discussion of analogous issues for
   NFSv4.0 have already been addressed by the implications publication of [RFC7931].
   Nevertheless, the NFSv4.0 experience for
   migration in NFSv4.1, as well as discussion history of other these issues with
   regard to in NFSv4.0 is presented,
   since understanding the treatment of migration similarities and differences between these
   protocols may be helpful in NFSv4.1. deciding how best to address remaining
   issues.

2.  Conventions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

   In the context of this informational document, these normative
   keywords will always occur in the context of a quotation, most often
   direct but sometimes indirect.  The context will make it clear
   whether the quotation is from:

   o  The previously current definitive definition of the NFSv4.0
      protocol [RFC7530].

   o  The current definitive definition of the NFSv4.1 protocol
      [RFC5661].

   o  A proposed or possible text to serve as a replacement for the
      current or previous definitive document text.  Sometimes, a number
      of possible alternative texts may be listed and benefits and
      detriments of each examined in turn.

3.  NFSv4.0 Implementation Experience Issues and Their Resolution

3.1.  Implementation  NFSv4.0 Issues

   Note that

   Many of the examples below reflect current experience which arises problems seen with Transparent State Migration derived
   from clients implementing the recommendation inability of servers to use different
   nfs_client_id4 id strings for determine whether two client IDs,
   issued on different server addresses, i.e.  using
   what is later referred servers, corresponded to herein as the "non-uniform client-string
   approach." same client.  This is simply because that is the experience implementers have had.
   The reader should not assume that
   difficulty derived in all cases, this practice is the
   source of turn from the difficulty.  It may be so in some cases but clearly it
   is not common practice, recommended by
   [RFC7530], in all cases.

3.1.1.  Failure to Free Migrated State on Client Reboot

   The following sort of situation has proved troublesome:

   o  A which each client C establishes a clientid4 C1 with server ABC specifying
      an nfs_client_id4 with id string value "C-ABC" and boot verifier
      0x111.

   o  The presented different client begins
   identification strings to access files in filesystem F on server ABC,
      resulting in generating stateids S1, S2, etc. under the lease for
      clientid C1.  It may also access files on other filesystems on different servers, rather than presenting
   the same server.

   o  The filesystem is migrated from server ABC identification string to server XYZ.  When
      transparent state migration is in effect, stateids S1 and S2 and
      clientid4 C1 are now available for use by client C at server XYZ.

   o  Client C reboots and attempts all servers.

   This practice, later referred to access data on server XYZ,
      whether in filesystem F or another.  It does a SETCLIENTID with an
      nfs_client_id4 with id as the "non-uniform" client string value "C-XYZ" and boot verifier
      0x112.  There is thus
   approach, derived from concern that, since NFSv4.0 provided no occasion means
   to free stateids S1 and S2 since
      they are associated with determine whether two IP addresses correspond to the server, a different
   single client name and so lease
      expiration is the only way that they can be gotten rid of.

   Note here that while it seems clear connected to us in this example that C-XYZ
   and C-ABC are from both might be confused by the same client, fact that
   state changes made via one IP address might unexpectedly affect the server has no way
   state maintained with respect to
   determine the structure second IP address, thought of the "opaque" id string.  In the protocol,
   it really is treated as opaque.  Only the client knows which
   nfs_client_id4 values designate the same client on a different
   server.

3.1.2.  Server Reboots Resulting in
   a Confused Lease Situation

   Further problems arise from scenarios like the following.

   o  Client C talks to separate server ABC using an nfs_client_id4

   To avoid this unexpected behavior, clients used the non-uniform
   client id string
      such as "C-ABC" and a boot verifier v1.  As a result, approach.  By doing so, a lease with
      clientid4 c.i is established: {v1, "C-ABC", c.i}.

   o  fs_a1 migrates from server ABC client connected to server XYZ along with its state.
      Now server XYZ also has a lease: {v1, "C-ABC", c.i}.

   o  Server ABC reboots.

   o  Client C talks two
   different servers (or to two IP addresses connected to the same
   server) appeared to be two different servers.  Since the server ABC using an nfs_client_id4 id string
      such as "C-ABC" and a boot verifier v1.  As a result, a lease with
      clientid4 c.j is established: {v1, "C-ABC", c.j}.

   o  fs_a2 migrates
   under the impression that two different clients are involved, state
   changes made on each distinct IP address cannot be reflected on
   another.

   However, by doing things this way, state migrated from server ABC to
   server XYZ.  Now server XYZ also
      has a lease: {v1, "C-ABC", c.j}. cannot be referred to the actual client which generated it,
   leading to confusion.

   In addition to this core problem, the following issues with regard to
   Transparent State Migration needed to be addressed:

   o  Now server XYZ has two leases that match {v1, "C-ABC", *}, when  Clarification regarding the protocol clearly assumes there can ability to merge state from different
      leases even though their expiration times might not be only one.

   Note that if precisely
      synchronized.

   o  Clarifying the treatment of client used "C" (rather than "C-ABC") as the IDs since it is not always
      clear when clientid4 and when nfs_client_id4 id string, was intended.

   o  Clarifying the exact same situation would arise.

   One logic of returning NFS4ERR_LEASE_MOVED.

   o  Clarifying the first cases in which this sort handling NFS4ERR_CLID_INUSE.

3.2.  Resolution of situation has resulted
   in difficulties is in connection with doing a SETCLIENTID for
   callback update. NFSv4.0 Protocol Difficulties

   The SETCLIENTID for callback update only includes client string identification issue was addressed in [RFC7931] as
   follows:

   o  Defining both the nfs_client_id4,
   assuming there can only be one such with a given nfs_client_id4
   value.  If there were multiple, confirmed uniform and non-uniform client records with
   identical nfs_client_id4 id string values, there would be no
      approaches as valid choices but indicating that the latter posed
      difficulties for Transparent Stare Migration.

   o  Providing a way that clients could use to
   map the callback update request determine whether two IP
      addresses are connected to the correct client record.  Apart
   from same server.

   o  Allowing clients using the migration handling specified in [RFC7530], such uniform approach to avoid negative
      consequences due to otherwise unexpected behavior since behavior
      that is a situation
   cannot arise.

3.1.3.  Client Complexity Issues

   Consider the following situation: consequence of known trunking relationships is not
      unexpected.

   o  There are  As a set of clients C1 through Cn accessing result, servers S1
      through Sm.  Each server manages some significant number migrating state are aware of
      filesystems with the filesystem count L being significantly
      greater than m.

   o  Each fact that
      the same client Cx will access a subset is associated with two different items of the servers and so will
      have up to m clientids, which we will call Cxy for server Sy.

   o  Now assume state
      even when that for load-balancing or state was originally created on two different
      servers.

   Since all of the other operational reasons,
      numbers issues noted in Section 3.1 were also
   addressed, publication of filesystems are migrated among [RFC7931] updating [RFC7530] addressed all
   known issues with Transparent State Migration in NFSv4.0.

4.  Issues for NFSv4.1

4.1.  Issues to Address for NFSv4.1

   Because NFSv4.1 embraces the servers.  As uniform client-string approach, as
   advised by section 2.4 of [RFC5661], addressing migration issues is
   simpler, in that a
      result, each client-server pair will have up to m clientids and
      each shift in client will have up id string models is not required.
   Instead, NFSv4 returns information in the EXCHANGE_ID response to m**2 clientids.  If we add
   enable trunking relationships to be determined by the
      possibility client.

   The other necessary part of server reboot, addressing migration issues, providing
   for the only bound on a client's
      clientid count server's merger of leases that relate to the same client, is L.

   Now, instead
   not currently addressed by [RFC5661] and changes need to be made to
   make it clear that state needs to be appropriately merged as part of a clientid4 identifying
   migration, to avoid multiple client IDs between a client-server pair, we have
   many more entities for the client to deal with. pair.

   In addition, it
   isn't clear how there are a number of new state is features within NFSv4.1 whose
   relationship with migration needs to be incorporated in this structure. clarified.  Some examples:

   o  The limitations interaction of the migrated state (inability trunking with migration and other aspects of
      multi-server namespace needs to be freed on
   reboot) would argue against adding more such state but trying clarified.

   o  There needs to
   avoid that would run into its own difficulties.  For example, a
   single lockowner string presented under two different clientids would
   appear as two different entities.

   Thus we have to choose between: be some clarification of how migration, and
      particularly Transparent State Migration, should interact with
      pNFS layouts.

   o  indefinite prolongation  The current discussion (in [RFC5661]), of foreign clientids even after all
      transferred state the possibility of
      server_owner changes is gone. incomplete and confusing.

   o  having multiple requests for the same lockowner-string-named
      entity carried on in parallel  The expected confirmation status of client IDs transferred by separate identically named
      lockowners under different clientid4's
      Transparent State Migration needs to be clarified.

   o  Adding serialization at the lock-owner string level, in addition  There are a number of issues related to that at the lockowner level.

   In any case, we have gone (in adding migration as it was described)
   from a situation of sessions
      that need to be addressed

   Discussion of how to resolve these issues will appear in which

   o  Each client has a single clientid4/lease for each server it talks
      to.

   o  Each client the sections
   below.

4.1.1.  Addressing state merger in NFSv4.1

   The existing treatment of state transfer in [RFC5661], has a single nfs_client_id4 for each server similar
   problems to that in [RFC7530] in that it talks
      to.

   o  Every assumes that the state id can be mapped to an associated lease based for
   multiple filesystems formerly on the
      server different servers will not be merged
   so that it was obtained from.

   To one in which

   o  Each client may have multiple clientid4's for appears under a single server.

   o  For each stateid, the common client must separately record ID.  We've already
   seen the clientid4 reasons that it this is assigned to, or it must manage separate "state blobs"
      for each fsid and map those to clientid4's.

   o  Before doing an operation that can result in a stateid, the client
      must either find a "state blob" based on fsid or create a new one,
      possibly problem with a new clientid4.

   o  There may be multiple clientid4's all connected regard to NFSv4.0.

   Although we don't have the same server
      and using the same nfs_clientid4.

   This sort problems stemming from the non-uniform
   client-string approach, there are a number of additional client complexity is troublesome complexities in the
   existing treatment of state management in the section entitled "Lock
   State and needs File System Transitions" in [RFC5661] that make this non-
   trivial to
   be eliminated.

3.2.  Sources of Protocol Difficulties

3.2.1.  Issues address:

   o  Migration is currently treated together with nfs_client_id4 Generation other sorts of
      filesystem transitions including transitioning between replicas
      without any NFS4ERR_MOVED errors.

   o  There is separate handling and discussion of the cases of matching
      and Use non-matching server scopes.

   o  In [RFC7530], the section entitled "Client ID" says:

      The second field, id is a variable length string that uniquely
      defines case of matching server scopes, the client.

   There are two possible interpretations text calls for an
      unrealistic degree of transparency, suggesting that the phrase "uniquely
   defines" source and
      destination servers need to cooperate in the above: stateid assignment.

   o  The relation between strings and clients is  In the case of non-matching server scopes, the text does not
      mention the possibility of the transparent migration of state at
      all, resulting in a function functional regression from such
      strings to clients so that each string designates a single client. NFSV4.0

   o  The relation potential interaction between strings migration and clients trunking has not
      been addressed.

   o  There is a bijection between
      such strings and insufficient attention to the question of how clients so that each string designates a single
      client and each client is named by a single string.

   The first interpretation would make these client-strings like phone
   numbers (a single person can have several) while
      deal with the second would
   make them like social security numbers.

   Debate about complexities of recovering from migration.  As part
      of this, the possible meanings implications of the shift of "uniquely defines" lease migration
      notification shifting from an error (NFS4ERR_LEASE_MOVED in this
   context is quite possible but not very helpful.  The following points
   should
      NFSv4.0) to status bit (SEQ4_STATUS_LEASE_MOVED in NFSv4.1) need
      to be noted though:

   o  The second interpretation explored.

   To summarize, there is more consistent with the way
      "uniquely defines" a need for an NFSv4.1 treatment of Transparent
   State Migration that is used elsewhere an extension of that in the spec.

   o  The spec as now written intends the first interpretation (or is
      internally inconsistent).  In fact, it recommends, although non-
      normatively, [RFC7931] and that a single client have at least as many client-
      strings
   includes appropriate handling for NFSv4.1 features such as server addresses that it interacts with.  It says, in trunking.

4.1.2.  Addressing pNFS relationship with migration

   This is made difficult because, within the third bullet point regarding construction pNFS framework, migration
   might mean any of several things:

   o  Transfer of the string (which
      we shall henceforth refer to MDS, leaving DS's as client-string-BP3):

         The string should they are.

      This would be different for each server network address
         that minimally disruptive to those using layouts but
      would require the client accesses, rather than common pNFS control protocol being used to all server
         network addresses.

   o  If internode interactions are limited support the
      DS being directed to those between a client new MDS.

   o  Transfer of a DS, leaving everything else in place.

      Such a transfer can be handled without using migration at all.
      The server can recall/revoke layouts, and its servers, there is no occasion for servers issue new ones, as
      appropriate.

   o  Transfer of the filesystem to be concerned a new filesystem with the question both MDS and
      DS's moving.

      In such a transfer, an entirely different set of whether two client-strings designate DS's will be at
      the same
      client, so that there is target location.  There may even be no occasion for pNFS support on the difference in
      interpretation to matter.

   o  When transparent migration of client state occurs between two
      servers, it becomes important
      destination filesystem at all.

   Migration needs to determine when state on two
      different servers is for support both the same client or not, first and this
      distinction becomes very important.

   Given last of these models.

4.1.3.  Addressing server_owner changes in NFSv4.1

   Section 2.10.5 of [RFC5661] states the need following.

      The client should be prepared for the server to possibility that
      eir_server_owner values may be aware different on subsequent EXCHANGE_ID
      requests made to the same network address, as a result of various
      sorts of reconfiguration events.  When this happens and the
      changes result in the invalidation of previously valid forms of
      trunking, the client identity with
   regard should cease to migrated state, use those forms, either client-string construction rules
   will have to change or there will be a need to get around current
   issues, by
      dropping connections or perhaps by adding sessions.  For a combination discussion of these two will be required.
   Later sections will examine the options and propose a solution.

   One consideration that may indicate that this cannot remain exactly
      lock reclaim as it has been derives from the fact that the current explanation for relates to such reconfiguration events, see
      Section 8.4.2.1.

   While this behavior paragraph is not correct.  In [RFC7530], the section entitled
   "Client ID" says:

      The reason literally true in that such reconfiguration
   events can happen and clients have to deal with them, it is confusing
   in that it may not can be possible for the client read as suggesting that clients have to tell
      if the same server deal with
   them without disruption, which in general is impossible.

   A clearer alternative would be:

      It is listening always possible that, as a result of various sorts of
      reconfiguration events, eir_server_scope and eir_server_owner
      values may be different on multiple network addresses.  If
      the client issues SETCLIENTID with subsequent EXCHANGE_ID requests made to
      the same id string to each network address of address.

      In most cases such a server, the server reconfiguration events will think it is the
      same client, be disruptive and each successive SETCLIENTID will cause the
      indicate that an IP address formerly connected to one server is
      now connected to begin the process of removing the client's previous leased
      state.

   In point an entirely different one.

      Some guidelines on client handling of fact, a "SETCLIENTID with such situations follow:

      o  When eir_server_scope changes, the same id string" sent to
   multiple network addresses will client has no assurance that
         any id's it obtained previously (e.g. file handles) can be treated as all from
         validly used on the same
   client but will not "cause new server, and, even if the new server
         accepts them, there is no assurance that this is not due to begin the process of
   removing the client's previous leased state" unless the server
   believes
         accident.  Thus it is best to treat all such state as lost/
         stale although a different instance of the same client, i.e. if client may assume that the
   id string probability of
         inadvertent acceptance is low and treat this situation as
         within the next case.

      o  When eir_server_scope remains the same and there is a different boot verifier.  If
         eir_server_owner.so_major_id changes, the client does not reboot, the verifier should not change.  If can use
         filehandles it does
   reboot, the verifier will change, has and it is appropriate attempt reclaims.  It may find that
         these are now stale but if NFS4ERR_STALE is not received, he
         can proceed to reclaim his opens.

      o  When eir_server_scope and eir_server_owner.so_major_id remain
         the
   server "begin same, the process of removing client has to use the client's previous leased
   state.

   The situation now-current values of multiple SETCLIENTID requests received by a server
         eir_server-owner.so_minor_id in deciding on multiple network addresses is exactly the same, from the protocol
   design point appropriate forms
         of trunking.

4.1.4.  Addressing Confirmation Status of view, Migrated Client IDs in NFSv4.1

   When a client ID is transferred between systems as when multiple (i.e. duplicate) SETCLIENTID
   requests are received by the server on a single network address.  The
   same protocol mechanisms that prevent erroneous state deletion in the
   latter case prevent part of
   migration, it in the former case.  There is no reason for
   special handling of not always clear whether it should be considered
   confirmed or unconfirmed on the multiple-network-appearance case, target server.  In the case in this
   regard.

3.2.2.  Issues which
   an associated session is transferred together with Lease Proliferation

   It the client ID, it
   is often felt clear that this the transferred client ID needs to be considered
   confirmed, as the existence of an associated session is incompatible
   with an unconfirmed client ID.

   The case in which a consequence of the client-string
   construction issues, client ID is transferred without an associated
   session is less clear-cut and there needs to be a choice between two
   possibilities:

   o  Consider it is certainly the case that unconfirmed, because of the two are
   closely connected in that non-uniform client-strings make lack of an associated
      session.  This makes it
   impossible simpler for the server client to appropriately combine leases from determine
      whether there is an associated session transferred at the same client.
      time.  However, even where it is inconsistent with the server could combine leases from fact there are
      stateids which have been transferred with the same
   client, client ID.

   o  Consider it needs to be clear how confirmed, because it was confirmed on the source
      server and when the transfer is not considered to have affected that.
      Although this makes it will do so, so that simpler for the client will be prepared.  These issues will have to be addressed determine whether
      there is an associated session transferred at
   various places the same time, an
      alternative is discussed in Section 4.1.5.

   A related issue concerns the protocol specification. potential use the SEQ4_STATUS flags to
   determine whether all or some of the state present on the source has
   been transferred the destination server.  This could be enough only if we are prepared to do away with done using
   either of the
   "should" recommending non-uniform client-strings and replace it with
   a "should not" or even a "SHOULD NOT".  Current client implementation
   patterns make this an unpalatable choice for use as a general
   solution, but alternatives above but it is reasonable to "RECOMMEND" this choice for a well-
   defined subset of clients.  One alternative would be to create a way
   for more in the server to infer from client behavior which leases are held by spirit of the same client and
   second alternative.  One potential use this information to do appropriate lease
   mergers.  Prototyping and detailed specification work has shown that
   this could be done but the resulting complexity of these flags is such discussed in
   more detail in Section 4.2.2.

4.1.5.  Addressing Session Migration in NFSv4.1

   Some issues that a better
   choice is need to "RECOMMEND" use of the uniform client-string approach
   for clients supporting be addressed regard the migration feature.

   Because of the discussion of client-string construction
   sessions, in [RFC7530],
   most existing clients implement the non-uniform client-string
   approach.  As a result, existing servers may not have been tested
   with clients implementing uniform client-strings.  As a consequence,
   care must be taken addition to preserve interoperability between UCS-capable
   clients and servers that don't tolerate uniform client strings for
   one reason or another.

4.  Resolution of NFSv4.0 Protocol Difficulties

   This section lists the changes that were necessary to resolve the
   difficulties mentioned above.  Such changes, along with other
   clarifications found to be desirable during drafting IDs and review are
   contained in [RFC7931].

4.1.  Changes Regarding nfs_client_id4 Client-string stateids

   o  It was decided needs to replace client-string-BP3 with the following text:

      The string MAY be different for each server network address that made clearer how the client accesses, rather than common to all server network
      addresses.

   In addition, given can deal with the importance
      possibility that sessions might or might not be transferred as
      part of the issue Transparent State Migration.

   o  Rules need to be clarified regarding possible transfer of client identity and sessions
      when either the fact that both client string-approaches are source session is being used to be considered
   valid, access other file
      systems on source server or there is already a greatly expanded treatment of session connecting
      the client identity was desirable.
   It had to the following major elements. destination server.

   o  Fully describing  There needs to be more detail regarding how the consequences of making protocol avoids
      situations in which the string same session is subject to concurrent
      changes on two different
      for each network address (the non-uniform client-string approach)
      and of making it servers at the same time.

4.2.  Possible Resolutions for all network addresses (the uniform
      client string approach).

   o  Giving helpful guidance about the factors that might affect client
      implementation choice between these approaches.

   o  Describing NFSv4.1 Issues

   The subsections below explore some ways of dealing with the compatibility issues that might cause servers to
   discussed in Section 4.1

   First we introduce some terminology we will be
      incompatible with using in these
   sections:

   o  Location attributes include the uniform approach fs_locations and give guidance about
      dealing with these. fs_locations_info
      attributes.

   o  Describing how a client using  Location entries are the uniform approach might use
      server behavior to determine server address trunking patterns. individual file system locations in the
      location attributes.

   o  Presenting  Location elements are derived from location entries.  If a clearer
      location entry specifies an IP address there is only a single
      corresponding location element.  Location entries that contain a
      host name, are resolved using DNS, and may result in one or more complete set of recommendations to
      guide client string construction.

4.2.  Changes Regarding Merged (vs.  Synchronized) Leases

   In [RFC7530], the section entitled "Migration and State" says:

      As part
      location elements.  All location elements consist of a location
      address which is the transfer IP address of information between servers, leases
      would be transferred as well.  The leases being transferred an interface to the
      new server will typically have a different expiration time from
      those for server and an
      fs name which is the same client, previously on location of the old server.  To
      maintain file system within the
      server's pseudo-fs.  The fs name is empty if the property that all leases on a given server for has no
      pseudo-fs and only a
      given client expire single exported file system at the root
      filehandle.

   o  Two location elements are trunkable if they specify the same time, fs
      name and the server should advance location addresses are such that trunking of the expiration time to
      location addresses can be used as shown by the later server_owner values
      returned.

4.2.1.  Server Responsibilities in Effecting Transparent State Migration

   The basic responsibility of the leases being transferred
      or the leases already present.  This allows the client source server in effecting
   Transparent State Migration is to maintain
      lease renewal of both classes without special effort:

   There are make available to the destination
   server a number description of problems each piece of locking state associated with this
   the file system being migrated.  In addition to client id string and any resolution of our
   difficulties must address them somehow.
   verifier, the source server needs to provide.  for each stateid:

   o  [RFC7530] recommends that  The stateid including the current sequence value.

   o  The associated client make it essentially
      impossible to determine when two leases are from "the same
      client". ID.

   o  It is not appropriate to speak  The handle of "maintain[ing] the property that
      all leases on a given server for a given client expire at associated file.

   o  The type of the same
      time", since this is not a property that holds even in lock, such as open, byte-range lock, delegation,
      layout.

   o  For locks such as opens and byte-range locks, there will be
      information about the absence owner(s) of migration. the lock.

   o  For recallable/revocable lock types, the current recall status
      needs to be included.

   o  For each lock type there will by type-specific information, such
      as share and deny modes for opens and type and byte ranges for
      byte-range locks and layouts.

   A further server listening on multiple network addresses
      may have responsibility concerns locks that are revoked or
   otherwise lost during the same client process of file system migration.  Because
   locks that appear as multiple clients with no way to
      recognize the client as the same.

   o  Even if be lost during the client identity issue could process of migration will be resolved, advancing
   reclaimed by the
      lease time at client, the point of servers have to take steps to ensure
   that locks revoked soon before or soon after migration would are not maintain the
      desired synchronization property.  The leases would
   inadvertently allowed to be
      synchronized until one of them was renewed, after reclaimed in situations in which they would the
   continuity of lock possession cannot be unsynchronized again.

   To avoid assured.

   o  For locks lost on the source but whose loss has not yet been
      acknowledged by the client complexity, we need to have no more than one lease
   between a single client and a single server.  This requires merger (by using FREE_STATEID), the
      destination must be aware of
   leases since there is no real help from synchronizing them at this loss so that it can deny a
   single instant.
      request to reclaim them.

   o  For locks lost on the uniform approach, destination after the state transfer but
      before the client's RECLAIM_COMPLTE is done, the destination
      server would simply merge
   leases as part should note these and not allow them to be reclaimed.

   A further responsibility of state transfer, since two leases with the same
   nfs_client_id4 values must servers concerns situations in which
   stateid cannot be for transferred transparently because it conflicts with
   an existing stateid held by the same client.

   We have made client and associated with a
   different file systems.  In this case there are two valid choices:

   o  Treat the following decisions transfer, as far in NFSv4.0, as proposed normative
   statements regarding for state merger.  They reflect one without Transparent
      State Migration.  In this case, conflicting locks cannot be
      granted until the facts that
   we want to allow full migration support in client does a RECLAIM_COMPLETE, after reclaiming
      the simplest way possible
   and that we can't say MUST since we have older clients and servers to
   deal with.

   o  Clients MAY use lock it had, with the uniform client-string approach and are well-
      advised to do so if exception of reclaims denied because
      they are concerned about getting good
      migration support.

   o  Servers SHOULD provide automatic lease merger during state
      migration so were attempts to reclaim locks that clients using the uniform id approach get had been lost.

   o  Implement Transparent State Migration, except for the
      support automatically.

   If servers obey lock with
      the SHOULD and clients choose to adopt conflicting stateid.  In this case, the uniform id
   approach, having more than a single lease for a given client-server
   pair client will be a transient situation, cleaned up as part of adapting to
   use aware
      of migrated state.

   Since clients and servers will be a mixture of old and new lost lock (through the SEQ4_STATUS flags) and
   because nothing is a MUST we have be allowed to ensure that no combination will
   show worse behavior than is exhibited by current (i.e. old) clients
   and servers.

4.3.  Other Changes to Migration-state Sections
4.3.1.  Changes Regarding Client ID
      reclaim it.

4.2.2.  Determining Initial Migration

   In [RFC7530], the section entitled "Migration and State" says:

      In the case of migration, the servers involved Status in the migration of
      a filesystem SHOULD transfer all server state from the original to
      the new server. NFSv4.1

   This must be done in section proposes a way that is transparent to
      the client.  This state transfer will ease the client's transition
      when in which a filesystem migration occurs.  If client which receives
   NFS4ERR_MOVED can determine:

   o  Whether the servers are successful NFS4ERR_MOVED indicates migration has occurred, or
      whether it indicates another sort of file system transition as
      discussed in transferring all state, the client will continue to use
      stateids assigned by Section 4.2.4

   o  In the original server.  Therefore case of migration, whether Transparent State Migration has
      occurred.

   o  Whether any state has been lost during the new
      server must recognize these stateids process of Transparent
      State Migration.

   o  Whether sessions have been transferred as valid. part of Transparent
      State Migration.

   This holds true
      for is written assuming that the second option regarding client ID as well.  Since responsibility for an entire
      filesystem is transferred with a
   confirmation status after migration event, there (as discussed in Section 4.1.4)
   is no
      possibility adopted.  However that conflicts will arise on the new server as a
      result of the transfer of locks.

   This poses some difficulties, mostly because the part about "client
   ID" choice is not clear:

   o  It isn't clear what part of the paragraph the "this" in the
      statement "this holds true ..." is meant essential to signify.

   o the procedure
   and could be changed.

   The phrase "the process begins by the client ID" examining the location entries using
   either of the location attributes.  For those whose fs name matches
   that currently being used, an EXCHANGE_ID is ambiguous, possibly indicating directed at the
      clientid4 location
   address and possibly indicating the nfs_client_id4.

   o  If server_owner and scope used to determine if the text means entry
   is trunkable with that previously being used to suggest access the file
   system (i.e. that it represents another path to the same clientid4 must be used,
      the logic is not clear since the issue is not the same file system
   and can share locking state with it).  If it is, then this should be
   treated as for
      stateids a transition from one set of which there might be many.  Adapting paths to the change of
      a single clientid, as might happen another, as
   described in Section 4.2.4, rather than a part migration event.

   Otherwuse, if one or more of lease migration, the EXCHANGE_ID operations above has
   encountered a distinct server, then migration has occurred and the
   procedure continues.  If there were no location entries with a
   matching fs name, then one with another fs name is relatively easy for selected, an
   EXCHANGE_ID is done, and the client.

   We have decided procedure continues using the result of
   that it operation.

   The determination of whether Transparent State Migration has occurred
   is best to address this issue as follows:

   o  Make it clear that both clientid4 and nfs_client_id4 (including
      both id string driven by the client ID returned and boot verifier) are to be transferred. its confirmation status.

   o  Indicate that the initial transfer will result in  If the same
      clientid4 after transfer but this client ID is an unconfirmed client ID not guaranteed since there
      may conflict previously known
      to the client, then Transparent State Migration has not occurred.

   o  If the client ID is a confirmed client ID previously known to the
      client, then any transferred state would have been merged with an
      existing clientid4 on client ID representing the client to the destination server
      and because lease
      server.  In this state merger can result in case, Transparent State Migration
      might or might not have occurred.

   o  If the client ID is a change of confirmed client ID not previously known to
      the clientid4.

4.3.2.  Changes Regarding Callback Re-establishment

   In [RFC7530], client, then the section entitled "Migration and State" says:

      A client SHOULD re-establish new callback information with can conclude that the new
      server as soon client ID was
      transferred as possible, according to sequences described in
      sections "Operation 35: SETCLIENTID - Negotiate Client ID" and
      "Operation 36: SETCLIENTID_CONFIRM - Confirm Client ID".  This
      ensures that server operations are not blocked by the inability to
      recall delegations.

   The above will need to be fixed to reflect the possibility of merging part of leases,

4.3.3.  NFS4ERR_LEASE_MOVED Rework Transparent State Migration.  In [RFC7530], the section entitled "Notification of Migrated Lease"
   says:

      Upon receiving the NFS4ERR_LEASE_MOVED error, a this
      transferred client ID case, Transparent State Migration has
      occurred although some state may have been lost.

   In the state merger case, it is possible that
      supports filesystem migration MUST probe all filesystems from that the server on has not
   attempted Transparent State Migration, in which case state may have
   been lost without it holds open state.  Once being reflected in the client SEQ4_STATUS bits.  To
   determine whether this has
      successfully probed all those filesystems which are migrated, happened, the client can use TEST_STATEID
   to check whether the stateids created on the source server MUST resume normal handling of stateful requests from that
      client.

   There is are still
   accessible on the destination server.  Once a lack of clarity that is prompted by ambiguity about what
   exactly probing single stateid is and what found
   to have been successfully transferred, the interlock between client can conclude that
   Transparent State Migration was begun and server
   must be.  This has led any failure to some worry about the scalability transport
   all of the
   probing process, and although the time required does scale linearly
   with stateids will be reflected in the number SEQ4_STATUS bits.

   In any of filesystems that the client may have state for
   with respect to cases in which Transparent State Migration has
   occurred, it is possible that a given server, the actual process can be done
   efficiently. session was transferred as well.  To address these issues,
   deal with that possibility, clients can, after doing the text above had to be rewritten to be
   more clear and to give suggestions about how EXCHANGE_ID,
   issue a BIND_CONN_TO_SESSION to do connect the required
   scanning efficiently.

4.4.  Changes to Other Sections

4.4.1.  Need for Additional Changes

   There are transferred session to a number of cases in which certain sections, not
   specifically related
   connection to migration, require additional clarification.
   This is generally because text the new server.  If that fails, it is clear in a context in which
   leases an indication
   that the session was not transferred and clientids are created in one place and live there forever
   may need further refinement in the more dynamic environment that
   arises as part of migration.

   Some examples:

   o  Some people are under the impression that updating callback
      endpoint information for an existing client, as used during
      migration, may cause the destination server to free existing
      state.  There need a new session needs to
   be additions created to clarify the situation.

   o  The handling of the sets of clientid4's maintained by each server
      needs take its place.

4.2.3.  Client Response to be clarified.  In particular, the issue of how Migration in NFSv4.1

   Once the client
      adapts to has determined the presumably independent and uncoordinated clientid4
      sets initial migration status, it needs
   to be clearly addressed

   o  Statements regarding handling of invalid clientid4's need re-establish its lock state, if possible.  To enable this to be
      clarified and/or refined in light
   happen without loss of the possibilities that arise
      due to lease motion and merger.

   o  Confusion and lack of clarity about NFS4ERR_CLID_INUSE.

4.4.2.  Callback Update

   Some changes are necessary to reduce confusion about guarantees normally provided by locking,
   the process of
   callback information update and in particular destination server needs to make it clear that
   no state is freed as implement a result:

   o  Make it clear that after migration there are confirmed entries for
      transferred clientid4/nfs_client_id4 pairs.

   o  Be explicit in the sections headed "otherwise," in the
      descriptions of SETCLIENTID and SETCLIENTID_CONFIRM, that these
      don't apply per-fs grace period in the
   all cases we are concerned about.

4.4.3.  clientid4 Handling

   To address both of the clientid4-related issues mentioned in
   Section 4.4.1, it which lock state was necessary lost, including those in which
   Transparent State Migration was not implemented.

   The following cases need to replace the last three paragraphs
   of the section entitled "Client ID" with the following:

      Once be dealt with:

   o  In a SETCLIENTID and SETCLIENTID_CONFIRM sequence case in which Transparent State Migration has
      successfully completed, the client uses not occurred,
      the shorthand client
      identifier, of type clientid4, instead of can use the longer and less
      compact nfs_client_id4 structure.  This shorthand client
      identifier (a client ID) is assigned per-fs grace period provided by the
      destination server and should be
      chosen so to reclaim locks that it will not conflict with a client ID previously
      assigned by same were held on the source
      server.  This applies across server restarts or
      reboots.

      Distinct servers MAY assign clientid4's independently,

   o  In a cases in which Transparent State Migration has occurred, and will
      generally do so.  Therefore,
      no lock state was lost (as shown by SEQ4_STATUS flags), no lock
      reclaim is necessary.

   o  In a client case in which Transparent State Migration has occurred, and
      some lock state was lost (as shown by SEQ4_STATUS flags), existing
      stateids need to be prepared checked for validity using TEST_STATEID, and
      reclaim used to deal
      with multiple instances re-establish any that were not transferred.

   For all of the same clientid4 cases above, RECLAIM_COMPLETE with an rca_one_fs value received on
      distinct IP addresses, denoting separate entities.  When trunking
   of server IP addresses is not a consideration, a client true should
      keep track be done before normal use of (IP-address, clientid4) pairs, so the file system including
   obtaining new locks for the file system.  This applies even if no
   locks were lost and needed to be reclaimed.

4.2.4.  Dealing with Multiple Location Entries

   The possibility that each pair more than one server address may be present in
   location attributes requires further clarification.  This is
      distinct.  In
   particularly the case, given the face potential role of possible trunking of server IP
      addresses, the client will use the receipt for
   NFSv4.1, whose connection to migration needs to be clarified.

   The description of the same clientid4
      from multiple IP-addresses, as an indication location attributes in [RFC5661], while it
   indicates that the two IP-
      addresses multiple address entries in these attributes may be trunked and proceed
   used to indicate alternate paths to determine, from the
      observed server behavior whether the two addresses are file system, does so mainly
   in fact
      trunked.

      When a clientid4 is presented to a server the context of replication and that clientid4 is does so without mentioning
   trunking.  The discussion of migration does not recognized, discuss the server
   possibility of multiple location entries or trunking, which we will
   explore here.

   We will reject cover cases in which multiple addresses appear directly in
   the request with attributes as well as those in which the error
      NFS4ERR_STALE_CLIENTID.  This can occur for multiple addresses
   result because a number single location entry is expanded into multiple
   location elements using addresses provided by DNS.

   When the set of reasons:

      *  A server reboot causing loss of the server's knowledge of the
         client

      *  Client error sending an incorrect clientid4 or a valid
         clientid4 location elements by which a file system may be
   accessed changes, migration need not be involved.  Some cases to
   consider:

   o  When the wrong server.

      *  Loss set of lease state due to lease expiration.

      *  Client or server error causing the server to believe that location elements expands, migration is not
      involved.  In the
         client has rebooted (i.e. receiving a SETCLIENTID with an
         nfs_client_id4 case in which has a matching id string and a non-
         matching boot verifier).

      *  Migration of all state under the associated lease causes its
         non-existence to be recognized on additional elements are not
      trunkable with ones previously being used, the source server.

      *  Merger new elements serve
      as additional access locations, available in case of state under the associated lease failure
      of server addresses being used.  When additional elements are
      trunkable with another lease
         under a different clientid causes those currently being used the clientid4 serving as client may use the
         source
      additional addresses just as they might have if they had been
      available when use of the merge to cease being recognized on its server.

      In file system began.

      There is no current mechanism by which the event client can be notified
      of a server reboot, or loss change in the set of lease state due available location for an fs.  Given the
      client has at least one IP address available to
      lease expiration, access the
      filesystem in question, periodic polling is an adequate mechanism
      for the client must obtain a new clientid4 by to find additional server addresses to use to
      access the file system.

   o  When the set of location elements contracts but none of the SETCLIENTID operation and
      elements no longer usable were in fact being used by the client,
      then proceed no migration is involved.  Only if the client were to any other necessary
      recovery for start
      using one of the server reboot case (See unavailable elements will the section entitled
      "Server Failure and Recovery").  In cases of server or client
      error resulting in this error, use be notified
      (via NFS4ERR_MOVED) of SETCLIENTID the need to establish not use those elements and to
      use others provided by a
      new lease is desirable as well.

      In location attribute.

   When a specific server address being used becomes unavailable to
   service a particular file system, NF4ERR_MOVED will be returned, and
   the last two cases, different recovery procedures client will respond based on the available locations.  Whether
   continuity of locking state will be available depends on a number of
   factors:

   o  If there are required.
      Note that in cases still elements in which use trunkable with the element that
      has become unavailable, there is any uncertainty about which
      sort will still be a continuity of handling is applicable,
      locking state, even though Transparent State Migration per se has
      not occurred.  If the distinguishing characteristic
      is that in reboot-like cases, in-use addresses are session-trunkable with
      the clientid4 address becoming unavailable, only one connection is lost and
      all associated
      stateids cease to exist while in migration-related cases, existing sessions will remain available.  If, on the
      clientid4 ceases to exist while other
      hand, the stateids in-use addresses are still valid.

      The client must also employ only clientid-trunkable with the SETCLIENTID operation when it
      receives a NFS4ERR_STALE_STATEID error using a stateid derived
      from its current clientid4, since this indicates
      address becoming unavailable, a situation, such session can be lost.  However,
      that session can be made available on those other nodes, just as server reboot which
      they it would have been if Transparent State Migration were in
      effect, even though no migration has invalidated occurred.

   o  Otherwise, if there are available addresses trunkable with the existing clientid4 and
      associated stateids (see one
      that has become unavailable, the section entitled "lock-owner" for
      details).

      See client has access to existing
      locking state once it establishes a connection with the detailed descriptions of SETCLIENTID and
      SETCLIENTID_CONFIRM for new
      addresses, using a complete specification of new or existing session depending on the
      operations.

4.4.4.  Handling type
      of NFS4ERR_CLID_INUSE

   It appears trunking in effect.  This is also similar to be the intention that only a single principal be used
   for client establishment between any client-server pair.  However:

   o  There case in which
      Transparent State Migration has occurred, even though there is no explicit statement to
      migration, with the state remaining on the existing server.

      Note that this effect.

   o  The error that indicates a principal conflict has a name which
      does not clarify this issue: NFS4ERR_CLID_INUSE.

   o  The definition of case, as well as the error is also not very helpful: "The
      SETCLIENTID operation has found that a client id is already previous one, can be expected
      in the case in use
      by another client".

   As a result, servers exist which reject a SETCLIENTID simply because
   there already exists a clientid for the same client, established
   using a different IP address.  Although this is generally understood server seeks to be erroneous, such servers still exist and the spec should make direct traffic with
      regard to particular file systems to choose addresses, in the correct behavior clear.

   Although
      interest of load balancing, to adjust to hardware availability
      constraints, or for other reasons.

   o  In other cases, migration has occurred and the error name cannot be changed, client can use the following changes
   should be made
      procedure described in Section 4.2.2 to avoid confusion:

   o  The definition of determine whether
      Transparent State Migration occurred and whether any locking state
      was lost during the error transfer.

   One should be changed to read as follows:

         The SETCLIENTID operation has found that note the specified
         nfs_client_id4 was previously presented following differences between migration with a different
         principal
   Transparent State Migration and that client instance currently holds an active
         lease.  A server MAY return this error if the same principal similar cases in which there is
         used but a
   continuity of locking state with no change in authentication flavor gives good reason to
         reject the new SETCLIENTID operation as not bona fide. server.

   o  In the description of SETCLIENTID, the phrase "then  When locks are lost (as indicated when using them or via the server
      returns a NFS4ERR_CLID_INUSE error" should
      SEQ4_STAUS flags) and migration has not been done, they are not to
      be expanded reclaimed.  Instead such losses are treated as lock revocations
      and acknowledged using FREE_STATEID.

   o  When migration has not been done, there is no need for a
      RECLAIM_COMPLETE (with rca_one_fs set to read
      "then the server returns true).

4.2.5.  Client Recovery from Migration Events

   When a NFS4ERR_CLID_INUSE error, since use of file system is migrated, there a single client number of migration-related
   status indications with multiple principals is not allowed."

5.  Issues for NFSv4.1

   Because NFSv4.1 embraces the uniform client-string approach, as
   advised by section 2.4 of [RFC5661], addressing migration issues is
   simpler.

   Nevertheless, there are some issues that will have which clients need to be addressed.
   Some examples: deal:

   o  The other necessary part of addressing migration issues, providing
      for the server's merger of leases that relate to the same client,  If an attempt is not currently addressed by NFSv4.1 and changes need to be made to make it clear that state needs to be appropriately merged as
      part of migration, to avoid multiple clientids between use or return a filehandle within a client- file
      system that has been migrated away from the server pair.

   o  There needs to be some clarification of how migration, and
      particularly transparent state migration, should interact with
      pNFS layouts.

   o  The current discussion (in [RFC5661]), of on which it was
      previously available, the possibility of
      server_owner changes error NFS4ERR_MOVED is incomplete and confusing.

   Discussion of how returned.

      This condition continues on subsequent attempts to resolve these issues will appear in access the sections
   below.

5.1.  Addressing state merger file
      system in NFSv4.1 question.  The existing treatment of state transfer in [RFC5661], has similar
   problems only way the client can avoid the error
      is to that in [RFC7530] cease accessing the filesystem in that question at its old server
      location and access it assumes that the state for
   multiple filesystems instead on different servers will not be merged the server to so
   that which it appears under has been
      migrated.

   o  Whenever a single common clientid.  We've already seen
   the reasons that this SEQUENCE operation is sent by a problem, with regard client to NFSv4.0.

   Although we don't have the problems stemming from the non-uniform
   client-string approach, there are a number of complexities in the
   existing treatment of server
      which generated state management in the section entitled "Lock
   State and File System Transitions" in [RFC5661] held on that make this non-
   trivial to address:

   o  Migration client which is currently treated together associated with other sorts of
      filesystem transitions including transitioning between replicas
      without any NFS4ERR_MOVED errors.

   o  There is separate handling and discussion of
      a file system that has been migrated away from the cases of matching
      and non-matching server scopes.

   o  In on which
      it was previously available, the case of matching server scopes, status bit
      SEQ4_STATUS_LEASE_MOVED is set in the text calls for an
      impossible degree of transparency.

   o  In response.

      This condition continues until the case of non-matching server scopes, client acknowledges the text does not
      mention transparent state migration at all, resulting in
      notification by fetching a
      functional regression from NFSV4.0

5.2.  Addressing pNFS relationship with migration

   This is made difficult because, within the PNFS framework, migration
   might mean any of several things:

   o  Transfer of location attribute for the MDS, leaving DS's alone.

      This would migrated
      file system.  When there are multiple migrated file systems, a
      location attribute for each such migrated file system needs to be minimally disruptive
      fetched, in order to those clear the condition.  Even after the
      condition is cleared, the client needs to respond by using layouts but
      would require the pNFS control protocol
      location information to support access the DS being
      directed destination server to a new MDS.

   o  Transfer ensure
      that leases are not needlessly expired.

   Unlike the case of a DS, leaving everything else NFSv4.0 in place.

      Such a transfer can be handled without using migration at all.
      The server can recall/revoke layouts, as appropriate.

   o  Transfer of which the filesystem to a new filesystem with corresponding conditions are
   both MDS errors, in NFSv4.1 the client can, and
      DS's moving.

      In such often will, receive both
   indications on the same request.  As a transfer, an entirely different set result, the question of DS's will how to
   co-ordinate the necessary recovery actions when both indications
   arrive simultaneously must be at resolved.  It should be noted that when
   the target location.  There may even server decides whether SEQ4_STATUS_LEASE_MOVED is ti be set, it
   has no pNFS support on the
      destination filesystem at all.

   Migration needs to support way of knowing which file system will be referenced or whether
   NFS4ERR_MOVED will be returned.

   While it is true that, when only a single migrated file system is
   involved, a single set of actions will clear both indications, the first and last
   possibility of these models.

5.3.  Addressing server owner changes multiple migrated file systems calls for an approach
   in NFSv4.1

   Section 2.10.5 of [RFC5661] states the following.

      The client should be prepared which there are separate recovery actions for each indication.  In
   general, the possibility that
      eir_server_owner values may response to neither indication can be different on subsequent EXCHANGE_ID
      requests made subsumed within
   the other since:

   o  If the client were to respond only to the same network address, as MOVED indication, there
      would be no effective client response to a result of various
      sorts of reconfiguration events.  When this happens and the
      changes result situation in which a
      file system was not being actively accessed at the invalidation of previously valid forms of
      trunking, time migration
      occurred.  As a result, leases on the destination server might be
      needlessly expired.

   o  If the client should cease were to respond only to the LEASE_MOVED indication,
      recovery for migrated file systems in active use those forms, either by
      dropping connections or by adding sessions.  For could be deferred
      in order to accomplish recovery for others not being actively
      accessed.  The consequences of this choice can pose particular
      problems when there are a discussion large number of
      lock reclaim as file systems supported
      by a particular server, or when it relates to happens that some servers,
      after receiving migrated file systems have periods of
      unavailability, such reconfiguration events, see
      Section 8.4.2.1.

   While this paragraph is literally true in that such reconfiguration
   events as occur as a result of server reboot.  This
      can happen and clients have result in recovery for actively accessed migrated file systems
      being unnecessarily delayed for long periods of time.

   Similar considerations apply to deal with them, it is confusing other arrangements in which one of
   the indications, while not ignored per se, is subsumed within a
   single recovery process focused on recovery for the other indication.

   Generally speaking, client recovery for these indications should have
   the following characteristics:

   o  All instances of the MOVED indication should be dealt with
      promptly, either by doing the necessary recovery directly,
      providing that it can be read as suggesting done asynchronously, or ensuring that clients have to deal it is
      already under way.

   o  All instances of the LEASE_MOVED indication should be dealt with
   them without disruption, which
      asynchronously, in general is impossible.

   A clearer alternative would be:

      It a migration discovery thread whose job is always to
      clear that indication by fetching the appropriate location
      attribute.  Because this thread will only be fetching a location
      attribute and the fs_status attribute for the file systems
      referenced by the client, it cannot receive MOVED indications.
      Some useful guidance regarding possible that, as implementation of the
      migration discovery thread can be found in Section 4.2.6.

   o  When a result migration discovery thread happens upon a migrated file
      system (i.e. not present and not a referral), the thread is likely
      to have cleared one (out of various sorts an unknown number) of
      reconfiguration events, eir_server_scope and eir_server_owner
      values may file systems
      whose migration needs to be different responded to.  The discovery thread
      needs to schedule the appropriate migration recovery (as described
      in Section 4.2.3).  This is necessary to ensure that migrated file
      systems will be referenced on subsequent EXCHANGE_ID requests made the destination server in order to
      avoid lease expiration

      For many of the same network address. migrated file systems discovered in this way, the
      client has not received any MOVED indication.  In most cases such reconfiguration events will cases,
      lease recovery needs to be disruptive scheduled but it should not interfere
      with continuation of the migration discovery function.

   o  When a migration discovery thread receives a LEASE_MOVED
      indication, it takes no special action but continues its normal
      operation.  On the other hand, if a LEASE_MOVED indication is not
      received, it indicates that the thread has completed its work
      successfully.

4.2.6.  The Migration Discovery Process

   As noted above, LEASE_MOVED indications are best dealt with in a
   migration discovery thread.  Because of this structure,

   o  No action needs to be taken for such indications received by the
      migration discovery threads, since continuation of that thread's
      work will address the issue.

   o  For such indications received in other contexts, the generally
      appropriate response is to initiate or otherwise provide for the
      execution of a migration discovery thread for file systems
      associated with the server IP address returning the indication.

   o  In all cases in which the appropriate migration discovery thread
      is running, nothing further need be done to respond to LEASE_MOVED
      indications.

   This leaves a potential difficulty in situations in which the
   migration discovery thread is near to completion but is still
   operating.  One should not ignore a LEASE_MOVED indication if the
   discovery thread is not able to respond to migrated file system
   without additional aid.  A further difficulty in addressing such
   situation is that a LEASE_MOVED indication may reflect the server's
   state at the time the SEQUENCE operation was processed, which may be
   different from that in effect at the time the response is received.

   A useful approach to this issue involves the use of separate
   externally-visible discovery thread states representing non-
   operation, normal operation, and completion/verification of migration
   discovery processing.

   Within that framework, discovery thread processing would proceed as
   follows.

   o  While in the normal-operation state, the thread would fetch, for
      successive file systems known to the client on the server being
      worked on, a location attribute plus the fs_status attribute.

   o  If the fs_status attribute indicates that the file system is a
      migrated one (i.e. fss_absent is true and fss_type !=
      STATUS4_REFERRAL) and thus that it is likely that the fetch of the
      location attribute has cleared one the file systems contributing
      to the LEASE_MOVED indication.

   o  In cases in which that happened, the thread cannot know whether
      the LEASE_MOVED indication has been cleared and so it enters the
      completion/verification state and proceeds to issue a COMPOUND to
      see if the LEASE_MOVED indication has been cleared.

   o  When the discovery thread is in the completion/verification state,
      if others get a LEASE_MOVED indication they note this fact and it
      is used when the request completes, as described below.

   When the request used in the completion/verification state completes:

   o  If a LEASE_MOVED indication is returned, the discovery thread
      resumes its normal work.

   o  Otherwise, if there is any record that other requests saw a
      LEASE_MOVED indication, that record is cleared and the
      verification request retried.  The discovery thread remains in
      completion/verification state.

   o  If there has been no LEASE_MOVED indication, the work of the
      discovery thread is considered completed and it enters the non-
      operating state.

4.2.7.  Synchronzing Session Transfer

   When transferring state between the source and destination, the
   issues discussed in Section 7.2 of [RFC7931] must still be attended
   to.  In this case, the use of NFS4ERR_DELAY is still necessary in
   NFSv4.1, as it was in NFSv4.0, to prevent locking state changing
   while it is being transferred.

   There are a number of important differences in the NFS4.1 context:

   o  The absence of RELEASE_LOCKOWNER means that the one case in which
      an operation could not be deferred by use of NFS4ERR_DELAY no
      longer exists.

   o  Sequencing of operations is no longer done using owner-based
      operation sequences numbers.  Instead, sequencing is session-
      based

   As a result, when sessions are not transferred, the techniques
   discussed in [RFC7931] are adequate and will not be further
   discussed.

   When sessions are transferred, there are a number of issues that pose
   challenges since,

   o  A single session may be used to access multiple file systems, not
      all of which are being transferred.

   o  Requests made on a session, even if rejected may, affect the state
      of the session by advancing the sequence number associated with
      the slot used.

   As a result, when the filesystem state might otherwise be considered
   unmodifiable, the client might have any number of in-flight requests,
   each of which is capable of changing session state, which may be of a
   number of types:

   1.  Those requests that were processed on the migrating file system,
       before migration began.

   2.  Those requests which got the error NFS4ERR_DELAY because the file
       system being accessed was in the process of being migrated.

   3.  Those requests which got the error NFS4ERR_MOVED because the file
       system being accessed had been migrated.

   4.  Those requests that accessed the migrating file system, in order
       to obtain location or status information.

   5.  Those requests that did not reference the migrating file system.

   It should be noted that the history of any particular slot is likely
   to include a number of these request classes.  In the case in which a
   session which is migrated is used by filesystems other than the one
   migrated, requests of class 5 may be common and be the last request
   processed, for many slots.

   Since session state can change even after the locking state has been
   fixed as part of the migration process, the session state known to
   the client could be different from that on the destination server,
   which necessarily reflects the session state on the source server, at
   an earlier time.  In deciding how to deal with this situation, it is
   helpful to distinguish between two sorts of behavioral consequences
   of the choice of initial sequence ID values.

   o  The error NFS4ERR_SEQ_MISORDERED is returned when the sequence ID
      in a request is neither equal to the last one seen for the current
      slot nor the next greater one.

      In view of the difficulty of arriving at a mutually acceptable
      value for the correct last sequence a the point of migration, it
      may be necessary for the server to show some degree of
      forbearance, when the sequence ID is one that would be considered
      unacceptable if session migration were not involved.

   o  Returning the cached reply for a previously executed request when
      the sequence ID in the request matches the last value recorded for
      the slot.

      In the cases in which an error is returned and there is no
      possibility of any non-idempotent operation having been executed,
      it may not be necessary to adhere to this as strictly as might be
      proper if session migration were not involved.  For example, the
      fact that the error NFS4ERR_DELAY was returned may not assist the
      client in any material way, while the fact that NFS4ERR_MOVED was
      returned by the source server may not be relevant when the request
      was reissued, directed to the destination server.

   One part of adapting to these sorts of issues would restrict
   enforcement of normal slot sequence enforcement semantics until the
   client itself, by issuing a request using a particular slot on the
   destination server, established the new starting sequence for that
   slot on the migrated session.

   An important issue is that the specification needs to take note of
   all potential COMPOUNDs, even if they might be unlikely in practice.
   For example, a COMPOUND is allow to access multiple file systems and
   might perform non-idempotent operations in some of them before
   accessing a file system being migrated.  Also, a COMPOUND may return
   considerable data in the response, before being rejected with
   NFS4ERR_DELAY or NFS4ERR_MOVED, and may in addition be marked as
   sa_cachethis.

   Some possibilities that need to be considered to address the issues:

   o  Do not enforce any sequencing semantics for a particular slot
      until the client has established the starting sequence for that
      slot on the destination server.

   o  For each slot, do not return a cached reply returning
      NFS4ERR_DELAY or NFS4ERR_MOVED until the client has established
      the starting sequence for that slot on the destination server.

   o  Until the client has established the starting sequence for a
      particular slot on the destination server, do not report
      NFS4ERR_SEQ_MISORDERED or return a cached reply returning
      NFS4ERR_DELAY or NFS4ERR_MOVED, where the reply consists solely of
      a series of operations where the response is NFS4_OK until the
      final error.

4.2.8.  Migration and pNFS

   When pNFS is involved, migration is capable of supporting:

   o  Migration of the MDS, leaving DS's in place.

   o  Migration of the file system as a whole, including the MDS and
      indicate that an IP address formerly connected
      associated DS's.

   o  Replacement of one DS by another.

   o  Migration of a pNFS file system to one server in which pNFS is
      now connected not used.

   o  Migration of a file system not using pNFS to an entirely different one.

      Some guidelines on client handling one in which layouts
      are available.

   Migration of the MDS function is directly supported by Transparent
   State Migration.  Layout state will normally be transparently
   transferred, just as other state is.  As a result, Transparent State
   Migration provides a framework in which, given appropriate inter-MDS
   data transfer, one MDS can be substituted for another.

   Migration of such situations follow:

      *  When eir_server_scope changes, the client has no assurance that
         any id's it obtained previously (e.g. file handles) system function can be
         validly used on accomplished by
   recalling all layouts as part of the new server, and, even if initial phase of the migration
   process.  As a result, IO will be done through the MDS during the
   migration process, and new server
         accepts them, there layouts can be granted once the client is no assurance that
   interacting with the new MDS.  An MDS can also effect this sort of
   transition by revoking all layouts as part of Transparent State
   Migration, as long as the client is not due notified about the loss of state.

   In order to
         accident.  Thus it allow migration to a file system on which pNFS is best not
   supported, clients need to treat all such state as lost/
         stale although be prepared for a client may assume that situation in layouts are
   not available or supported on the probability destination file system and be
   prepared to direct IO request to the destination server, rather than
   depending on layouts being available.

   Replacement of
         inadvertent acceptance one DS by another is low and treat this situation not addressed by migration as
         within the next case.

      *  When eir_server_scope remains
   such but can be effected by an MDS recalling layouts for the same DS to be
   replaced and
         eir_server_owner.so_major_id changes, issuing new ones to be served by the client can use
         filehandles it has and attempt reclaims.  It successor DS.

   Migration may find that
         these are now stale but if NFS4ERR_STALE is transfer a file system from a server which does not received, he
         can proceed
   support pNFS to reclaim his opens.

      *  When eir_server_scope one which does.  In order to properly adapt to this
   situation, clients which support pNFS, but function adequately in its
   absence, should check for pNFS support when a file system is migrated
   and eir_server_owner.so_major_id remain
         the same, the client has be prepared to use the now-current values of
         eir_server-owner.so_minor_id in deciding on appropriate forms
         of trunking.

6. pNFS when support is available.

5.  Security Considerations

   With regard to NFSv4.0, the Security Considerations section of
   [RFC7530] encourages clients to protect the integrity of the SECINFO
   operation, any GETATTR operation for the fs_locations attribute.  A
   needed change is to include the operations SETCLIENTID/
   SETCLIENTID_CONFIRM as among those for which integrity protection is
   recommended.  A migration recovery event can use any or all of these
   operations.

   With regard to NFSv4.1, the Security Considerations section of
   [RFC5661] takes proper care of migration-related issues.  No change
   is needed.

7.

6.  IANA Considerations

   This document does not require actions by IANA.

8.  Acknowledgements

   The editor and authors of this document gratefully acknowledge the
   contributions of Trond Myklebust of NetApp and Robert Thurlow of
   Oracle.  We also thank Tom Haynes of NetApp and Spencer Shepler of
   Microsoft for their guidance and suggestions.

   Special thanks go to members of the Oracle Solaris NFS team,
   especially Rick Mesta and James Wahlig, for their work implementing
   an NFSv4.0 migration prototype and identifying many of the issues
   documented here.

9.

7.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <http://www.rfc-editor.org/info/rfc2119>.

   [RFC5661]  Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
              "Network File System (NFS) Version 4 Minor Version 1
              Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010,
              <http://www.rfc-editor.org/info/rfc5661>.

   [RFC7530]  Haynes, T., Ed. and D. Noveck, Ed., "Network File System
              (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530,
              March 2015, <http://www.rfc-editor.org/info/rfc7530>.

   [RFC7931]  Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker,
              "NFSv4.0 Migration: Specification Update", RFC 7931,
              DOI 10.17487/RFC7931, July 2016,
              <http://www.rfc-editor.org/info/rfc7931>.

Appendix A.  Acknowledgements

   The editor and authors of this document gratefully acknowledge the
   contributions of Trond Myklebust of NetApp and Robert Thurlow of
   Oracle.  We also thank Tom Haynes of Primary Data and Spencer Shepler
   of Microsoft for their guidance and suggestions.

   Special thanks go to members of the Oracle Solaris NFS team,
   especially Rick Mesta and James Wahlig, for their work implementing
   an NFSv4.0 migration prototype and identifying many of the issues
   documented here.

Authors' Addresses

   David Noveck (editor)
   NetApp
   26 Locust Avenue
   Lexington, MA  02421
   US

   Phone: +1 781 572 8038
   Email: davenoveck@gmail.com

   Piyush Shivam
   Oracle Corporation
   5300 Riata Park Ct.
   Austin, TX  78727
   US

   Phone: +1 512 401 1019
   Email: piyush.shivam@oracle.com

   Charles Lever
   Oracle Corporation
   1015 Granger Avenue
   Ann Arbor, MI  48104
   US

   Phone: +1 248 614 5091
   Email: chuck.lever@oracle.com

   Bill Baker
   Oracle Corporation
   5300 Riata Park Ct.
   Austin, TX  78727
   US

   Phone: +1 512 401 1081
   Email: bill.baker@oracle.com