Tech-invite3GPPspaceIETF RFCsSIP
in Index   Prev   Next

RFC 7931

NFSv4.0 Migration: Specification Update

Pages: 55
Proposed Standard
Updates:  7530
Part 1 of 2 – Pages 1 to 29
None   None   Next

Top   ToC   RFC7931 - Page 1
Internet Engineering Task Force (IETF)                    D. Noveck, Ed.
Request for Comments: 7931                                           HPE
Updates: 7530                                                  P. Shivam
Category: Standards Track                                       C. Lever
ISSN: 2070-1721                                                 B. Baker
                                                               July 2016

                NFSv4.0 Migration: Specification Update


The migration feature of NFSv4 allows the transfer of responsibility for a single file system from one server to another without disruption to clients. Recent implementation experience has shown problems in the existing specification for this feature in NFSv4.0. This document identifies the problem areas and provides revised specification text that updates the NFSv4.0 specification in RFC 7530. Status of This Memo This is an Internet Standards Track document. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at
Top   ToC   RFC7931 - Page 2
Copyright Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.
Top   ToC   RFC7931 - Page 3

Table of Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 3.2. Data Type Definitions . . . . . . . . . . . . . . . . . . 5 4. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 5. Client Identity Definition . . . . . . . . . . . . . . . . . 7 5.1. Differences from Replaced Sections . . . . . . . . . . . 7 5.2. Client Identity Data Items . . . . . . . . . . . . . . . 8 5.2.1. Client Identity Structure . . . . . . . . . . . . . . 9 5.2.2. Client Identity Shorthand . . . . . . . . . . . . . . 11 5.3. Server Release of Client ID . . . . . . . . . . . . . . . 13 5.4. Client ID String Approaches . . . . . . . . . . . . . . . 14 5.5. Non-uniform Client ID String Approach . . . . . . . . . . 16 5.6. Uniform Client ID String Approach . . . . . . . . . . . . 16 5.7. Mixing Client ID String Approaches . . . . . . . . . . . 18 5.8. Trunking Determination when Using Uniform Client ID Strings . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.9. Client ID String Construction Details . . . . . . . . . . 26 6. Locking and Multi-Server Namespace . . . . . . . . . . . . . 28 6.1. Lock State and File System Transitions . . . . . . . . . 28 6.1.1. Migration and State . . . . . . . . . . . . . . . . . 29 Migration and Client IDs . . . . . . . . . . . . 31 Migration and State Owner Information . . . . . . 32 6.1.2. Replication and State . . . . . . . . . . . . . . . . 36 6.1.3. Notification of Migrated Lease . . . . . . . . . . . 36 6.1.4. Migration and the lease_time Attribute . . . . . . . 39 7. Server Implementation Considerations . . . . . . . . . . . . 39 7.1. Relation of Locking State Transfer to Other Aspects of File System Motion . . . . . . . . . . . . . . . . . . . 39 7.2. Preventing Locking State Modification during Transfer . . 41 8. Additional Changes . . . . . . . . . . . . . . . . . . . . . 44 8.1. Summary of Additional Changes from Previous Documents . . 45 8.2. NFS4ERR_CLID_INUSE Definition . . . . . . . . . . . . . . 45 8.3. NFS4ERR_DELAY Return from RELEASE_LOCKOWNER . . . . . . . 45 8.4. Operation 35: SETCLIENTID -- Negotiate Client ID . . . . 46 8.5. Security Considerations for Inter-server Information Transfer . . . . . . . . . . . . . . . . . . . . . . . . 51 8.6. Security Considerations Revision . . . . . . . . . . . . 51 9. Security Considerations . . . . . . . . . . . . . . . . . . . 52 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 52 10.1. Normative References . . . . . . . . . . . . . . . . . . 52 10.2. Informative References . . . . . . . . . . . . . . . . . 52 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 53 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 54
Top   ToC   RFC7931 - Page 4

1. Introduction

This Standards Track document corrects the existing definitive specification of the NFSv4.0 protocol described in [RFC7530]. Given this fact, one should take the current document into account when learning about NFSv4.0, particularly if one is concerned with issues that relate to: o File system migration, particularly when it involves transparent state migration. o The construction and interpretation of the nfs_client_id4 structure and particularly the requirements on the id string within it, referred to below as a "client ID string".

2. Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

3. Definitions

3.1. Terminology

The following definitions are included to provide an appropriate context for the reader. This section is derived from Section 1.5 of [RFC7530] but has been adapted to the needs of this document. Boot Instance Id: A boot instance id is an identifier, such as a boot time, allowing two different instances of the same client to be reliably distinguished. A boot instance id is opaque to the server and is often used as the verifier field in the nfs_client_id4 structure, which identifies the client to the server. Client: A client is an entity that accesses the NFS server's resources. The client may be an application that contains the logic to access the NFS server directly. The client may also be the traditional operating system client that provides remote file system services for a set of applications. With reference to byte-range locking, the client is also the entity that maintains a set of locks on behalf of one or more applications. This client is responsible for crash or failure recovery for those locks it manages.
Top   ToC   RFC7931 - Page 5
      Note that multiple clients may share the same transport and
      connection, and multiple clients may exist on the same network

   Client ID:  A client ID is a 64-bit quantity (in the form of a
      clientid4) used as a unique, shorthand reference to a particular
      client instance, identified by a client-supplied verifier (in the
      form of a boot instance id) and client ID string.  The server is
      responsible for supplying the client ID.

   File System:  A file system is the collection of objects on a server
      that share the same fsid attribute (see Section of

   Grace Period:  A grace period is an interval of time during which the
      server will only grant locking requests to reclaim existing locks
      but not those that create new locks.  This gives clients an
      opportunity to re-establish locking state in response to a
      potentially disruptive event.  The grace period may be general to
      help deal with server reboot, or it may be specific to a file
      system to deal with file system migration when transparent state
      migration is not provided.

   Lease:  A lease is an interval of time defined by the server for
      which the client is irrevocably granted a lock.  At the end of a
      lease period, the lock may be revoked if the lease has not been
      extended.  The lock must be revoked if a conflicting lock has been
      granted after the lease interval.

      All leases granted by a server have the same fixed duration.  Note
      that the fixed interval duration was chosen to alleviate the
      expense a server would have in maintaining state about variable-
      length leases across server failures.

   Lock:  The term "lock" is used to refer to record (byte-range) locks
      as well as share reservations unless specifically stated

   Lock-Owner:  Each byte-range lock is associated with a specific lock-
      owner and an open-owner.  The lock-owner consists of a client ID
      and an opaque owner string.  The client presents this to the
      server to establish the ownership of the byte-range lock as

   Open-Owner:  Each open file is associated with a specific open-owner,
      which consists of a client ID and an opaque owner string.  The
      client presents this to the server to establish the ownership of
      the open as needed.
Top   ToC   RFC7931 - Page 6
   Server:  A server is an entity responsible for coordinating client
      access to a set of file systems.

   Stateid:  A stateid is a 128-bit quantity returned by a server that
      uniquely identifies the open and locking states provided by the
      server for a specific open-owner or lock-owner/open-owner pair for
      a specific file and type of lock.

   Trunking:  A situation in which multiple physical addresses are
      connected to the same logical server.

   Verifier:  A verifier is a quantity, in the form of a verifier4, that
      allows one party to an interaction to be aware of a
      reinitialization or other significant change to the state of the
      other party.  In [RFC7530], this term most often designates the
      verifier field of an nfs_client_id4, in which a boot instance id
      is placed to allow the server to determine when there has been a
      client reboot, making it necessary to eliminate locking state
      associated with the previous instance of the same client.

3.2. Data Type Definitions

This section contains a table that shows where data types referred to in this document are defined. +-----------------+--------------------------------+ | Item | Section | +-----------------+--------------------------------+ | cb_client4 | Section 2.2.11 in [RFC7530] | | clientaddr4 | Section 2.2.10 in [RFC7530] | | clientid4 | Section 2.1 in [RFC7530] | | lock_owner4 | Section 2.2.14 in [RFC7530] | | nfs_client_id4 | Section 5.2.1 (this document) | | open_owner4 | Section 2.2.13 in [RFC7530] | | verifier4 | Section 2.1 in [RFC7530] | +-----------------+--------------------------------+

4. Background

Implementation experience with transparent state migration has exposed a number of problems with the then existing specifications of this feature in [RFC7530] and predecessors. The symptoms were: o After migration of a file system, a reboot of the associated client was not appropriately dealt with, in that the state associated with the rebooting client was not promptly freed.
Top   ToC   RFC7931 - Page 7
   o  Situations can arise whereby a given server has multiple leases
      with the same nfs_client_id4 (consisting of id and verifier
      fields), when the protocol clearly assumes there can be only one.

   o  Excessive client implementation complexity since clients have to
      deal with situations in which a single client can wind up with its
      locking state with a given server divided among multiple leases
      each with its own clientid4.

   An analysis of these symptoms leads to the conclusion that existing
   specifications have erred.  They assume that locking state, including
   both state ids and clientid4s, should be transferred as part of
   transparent state migration.  The troubling symptoms arise from the
   failure to describe how migrating state is to be integrated with
   existing client definition structures on the destination server.

   The need for the server to appropriately merge stateids associated
   with a common client boot instance encounters a difficult problem.
   The issue is that the common client practice with regard to the
   presentation of unique strings specifying client identity makes it
   essentially impossible for the client to determine whether or not two
   stateids, originally generated on different servers, are referable to
   the same client.  This practice is allowed and endorsed by the
   existing NFSv4.0 specification [RFC7530].

   However, upon the prototyping of clients implementing an alternative
   approach, it has been found that there exist servers that do not work
   well with these new clients.  It appears that current circumstances,
   in which a particular client implementation pattern had been adopted
   universally, have resulted in some servers not being able to
   interoperate against alternate client implementation patterns.  As a
   result, we have a situation that requires careful attention to
   untangling compatibility issues.

   This document updates the existing NFSv4.0 specification [RFC7530] as

   o  It makes clear that NFSv4.0 supports multiple approaches to the
      construction of client ID strings, including those formerly
      endorsed by existing NFSV4.0 specifications and those currently
      being widely deployed.

   o  It explains how clients can effectively use client ID strings that
      are presented to multiple servers.
Top   ToC   RFC7931 - Page 8
   o  It addresses the potential compatibility issues that might arise
      for clients adopting a previously non-favored client ID string
      construction approach including the existence of servers that have
      problems with the new approach.

   o  It gives some guidance regarding the factors that might govern
      clients' choice of a client ID string construction approach and
      recommends that clients construct client ID strings in a manner
      that supports lease merger if they intend to support transparent
      state migration.

   o  It specifies how state is to be transparently migrated, including
      defining how state that arrives at a new server as part of
      migration is to be merged into existing leases for clients
      connected to the target server.

   o  It makes further clarifications and corrections to address cases
      where the specification text does not take proper account of the
      issues raised by state migration or where it has been found that
      the existing text is insufficiently clear.  This includes a
      revised definition of the SETCLIENTID operation in Section 8.4,
      which replaces Section 16.33 in [RFC7530].

   For a more complete explanation of the choices made in addressing
   these issues, see [INFO-MIGR].

5. Client Identity Definition

This section is a replacement for Sections 9.1.1 and 9.1.2 in [RFC7530]. The replaced sections are named "Client ID" and "Server Release of Client ID", respectively. It supersedes the replaced sections.

5.1. Differences from Replaced Sections

Because of the need for greater attention to and careful description of this area, this section is much larger than the sections it replaces. The principal changes/additions made by this section are: o It corrects inconsistencies regarding the possible role or non- role of the client IP address in construction of client ID strings. o It clearly addresses the need to maintain a non-volatile record across reboots of client ID strings or any changeable values that are used in their construction.
Top   ToC   RFC7931 - Page 9
   o  It provides a more complete description of circumstances leading
      to clientid4 invalidity and the appropriate recovery actions.

   o  It presents, as valid alternatives, two approaches to client ID
      string construction (named "uniform" and "non-uniform") and gives
      some implementation guidance to help implementers choose one or
      the other of these.

   o  It adds a discussion of issues involved for clients in interacting
      with servers whose behavior is not consistent with use of uniform
      client ID strings.

   o  It adds a description of how server behavior might be used by the
      client to determine when multiple server IP addresses correspond
      to the same server.

5.2. Client Identity Data Items

The NFSv4 protocol contains a number of protocol entities to identify clients and client-based entities for locking-related purposes: o The nfs_client_id4 structure, which uniquely identifies a specific client boot instance. That identification is presented to the server by doing a SETCLIENTID operation. The SETCLIENTID operation is described in Section 8.4, which modifies a description in Section 16.33 of [RFC7530]. o The clientid4, which is returned by the server upon completion of a successful SETCLIENTID operation. This id is used by the client to identify itself when doing subsequent locking-related operations. A clientid4 is associated with a particular lease whereby a client instance holds state on a server instance and may become invalid due to client reboot, server reboot, or other circumstances. o Opaque arrays, which are used together with the clientid4 to designate within-client entities (e.g., processes) as the owners of opens (open-owners) and owners of byte-range locks (lock- owners).
Top   ToC   RFC7931 - Page 10

5.2.1. Client Identity Structure

The basis of the client identification infrastructure is encapsulated in the following data structure, which also appears in Section 9.1.1 of [RFC7530]: struct nfs_client_id4 { verifier4 verifier; opaque id<NFS4_OPAQUE_LIMIT>; }; The nfs_client_id4 structure uniquely defines a particular client boot instance as follows: o The id field is a variable-length string that uniquely identifies a specific client. Although it is described here as a string and is often referred to as a "client string", it should be understood that the protocol defines this as opaque data. In particular, those receiving such an id should not assume that it will be in the UTF-8 encoding. Servers MUST NOT reject an nfs_client_id4 simply because the id string does not follow the rules of UTF-8 encoding. The encoding and decoding processes for this field (e.g., use of network byte order) need to result in the same internal representation whatever the endianness of the originating and receiving machines. o The verifier field contains a client boot instance identifier that is used by the server to detect client reboots. Only if the boot instance is different from that which the server has previously recorded in connection with the client (as identified by the id field) does the server cancel the client's leased state. This cancellation occurs once it receives confirmation of the new nfs_clientd4 via SETCLIENTID_CONFIRM. The SETCLIENTID_CONFIRM operation is described in Section 16.34 of [RFC7530]. In order to prevent the possibility of malicious destruction of the locking state associated with a client, the server MUST NOT cancel a client's leased state if the principal that established the state for a given id string is not the same as the principal issuing the SETCLIENTID.
Top   ToC   RFC7931 - Page 11
   There are several considerations for how the client generates the id

   o  The string should be unique so that multiple clients do not
      present the same string.  The consequences of two clients
      presenting the same string range from one client getting an error
      to one client having its leased state abruptly and unexpectedly

   o  The string should be selected so that subsequent incarnations
      (e.g., reboots) of the same client cause the client to present the
      same string.  The implementer is cautioned against an approach
      that requires the string to be recorded in a local file because
      this precludes the use of the implementation in an environment
      where there is no local disk and all file access is from an NFSv4

   o  The string MAY be different for each server network address that
      the client accesses rather than common to all server network

      The considerations that might influence a client to use different
      strings for different network server addresses are explained in
      Section 5.4.

   o  The algorithm for generating the string should not assume that the
      clients' network addresses will remain the same for any set period
      of time.  Even while the client is still running in its current
      incarnation, changes might occur between client incarnations.

      Changes to the client ID string due to network address changes
      would result in successive SETCLIENTID operations for the same
      client appearing as from different clients, interfering with the
      use of the nfs_client_id4 verifier field to cancel state
      associated with previous boot instances of the same client.

      The difficulty is more severe if the client address is the only
      client-based information in the client ID string.  In such a case,
      there is a real risk that after the client gives up the network
      address, another client, using the same algorithm, would generate
      a conflicting id string.  This would be likely to cause an
      inappropriate loss of locking state.  See Section 5.9 for detailed
      guidance regarding client ID string construction.
Top   ToC   RFC7931 - Page 12

5.2.2. Client Identity Shorthand

Once a SETCLIENTID and SETCLIENTID_CONFIRM sequence has successfully completed, the client uses the shorthand client identifier, of type clientid4, instead of the longer and less compact nfs_client_id4 structure. This shorthand client identifier (a client ID) is assigned by the server and should be chosen so that it will not conflict with a client ID previously assigned by the same server and, to the degree practicable, by other servers as well. This applies across server restarts or reboots. Establishment of the client ID by a new incarnation of the client also has the effect of immediately breaking any leased state that a previous incarnation of the client might have had on the server, as opposed to forcing the new client incarnation to wait for the leases to expire. Breaking the lease state amounts to the server removing all locks, share reservations, and delegation states not requested using the CLAIM_DELEGATE_PREV claim type associated with a client having the same identity. For a discussion of delegation state recovery, see Section 10.2.1 of [RFC7530]. Note that the SETCLIENTID and SETCLIENTID_CONFIRM operations have a secondary purpose of establishing the information the server needs to make callbacks to the client for the purpose of supporting delegations. The client is able to change this information via SETCLIENTID and SETCLIENTID_CONFIRM within the same incarnation of the client without causing removal of the client's leased state. Distinct servers MAY assign clientid4s independently, and they will generally do so. Therefore, a client has to be prepared to deal with multiple instances of the same clientid4 value received on distinct IP addresses, denoting separate entities. When trunking of server IP addresses is not a consideration, a client should keep track of <IP-address, clientid4> pairs, so that each pair is distinct. For a discussion of how to address the issue in the face of possible trunking of server IP addresses, see Section 5.4. Owners of opens and owners of byte-range locks are separate entities and remain separate even if the same opaque arrays are used to designate owners of each. The protocol distinguishes between open- owners (represented by open_owner4 structures) and lock-owners (represented by lock_owner4 structures). Both sorts of owners consist of a clientid4 and an opaque owner string. For each client, there is a set of distinct owner values used with that client which constitutes the set of known owners of that type, for the given client.
Top   ToC   RFC7931 - Page 13
   Each open is associated with a specific open-owner while each byte-
   range lock is associated with a lock-owner and an open-owner, the
   latter being the open-owner associated with the open file under which
   the LOCK operation was done.

   When a clientid4 is presented to a server and that clientid4 is not
   valid, the server will reject the request with an error that depends
   on the reason for clientid4 invalidity.  The error
   NFS4ERR_ADMIN_REVOKED is returned when the invalidation is the result
   of administrative action.  When the clientid4 is unrecognizable, the
   error NFS4ERR_STALE_CLIENTID or NFS4ERR_EXPIRED may be returned.  An
   unrecognizable clientid4 can occur for a number of reasons:

   o  A server reboot causing loss of the server's knowledge of the
      client.  (Always returns NFS4ERR_STALE_CLIENTID.)

   o  Client error sending an incorrect clientid4 or a valid clientid4
      to the wrong server.  (May return either error.)

   o  Loss of lease state due to lease expiration.  (Always returns

   o  Client or server error causing the server to believe that the
      client has rebooted (i.e., receiving a SETCLIENTID with an
      nfs_client_id4 that has a matching id string and a non-matching
      boot instance id as the verifier).  (May return either error.)

   o  Migration of all state under the associated lease causes its non-
      existence to be recognized on the source server.  (Always returns

   o  Merger of state under the associated lease with another lease
      under a different client ID causes the clientid4 serving as the
      source of the merge to cease being recognized on its server.
      (Always returns NFS4ERR_STALE_CLIENTID.)

   In the event of a server reboot, loss of lease state due to lease
   expiration, or administrative revocation of a clientid4, the client
   must obtain a new clientid4 by use of the SETCLIENTID operation and
   then proceed to any other necessary recovery for the server reboot
   case (see Section 9.6.2 in [RFC7530]).  In cases of server or client
   error resulting in a clientid4 becoming unusable, use of SETCLIENTID
   to establish a new lease is desirable as well.

   In cases in which loss of server knowledge of a clientid4 is the
   result of migration, different recovery procedures are required.  See
   Section 6.1.1 for details.  Note that in cases in which there is any
   uncertainty about which sort of handling is applicable, the
Top   ToC   RFC7931 - Page 14
   distinguishing characteristic is that in reboot-like cases, the
   clientid4 and all associated stateids cease to exist while in
   migration-related cases, the clientid4 ceases to exist while the
   stateids are still valid.

   The client must also employ the SETCLIENTID operation when it
   receives an NFS4ERR_STALE_STATEID error using a stateid derived from
   its current clientid4, since this indicates a situation, such as a
   server reboot that has invalidated the existing clientid4 and
   associated stateids (see Section 9.1.5 in [RFC7530] for details).

   See the detailed descriptions of SETCLIENTID (in Section 8.4) and
   SETCLIENTID_CONFIRM (in Section 16.34 of [RFC7530]) for a complete
   specification of these operations.

5.3. Server Release of Client ID

If the server determines that the client holds no associated state for its clientid4, the server may choose to release that clientid4. The server may make this choice for an inactive client so that resources are not consumed by those intermittently active clients. If the client contacts the server after this release, the server must ensure the client receives the appropriate error so that it will use the SETCLIENTID/SETCLIENTID_CONFIRM sequence to establish a new identity. It should be clear that the server must be very hesitant to release a client ID since the resulting work on the client to recover from such an event will be the same burden as if the server had failed and restarted. Typically, a server would not release a client ID unless there had been no activity from that client for many minutes. Note that if the id string in a SETCLIENTID request is properly constructed, and if the client takes care to use the same principal for each successive use of SETCLIENTID, then, barring an active denial-of-service attack, NFS4ERR_CLID_INUSE should never be returned. However, client bugs, server bugs, or perhaps a deliberate change of the principal owner of the id string (such as may occur in the case in which a client changes security flavors, and under the new flavor, there is no mapping to the previous owner) will in rare cases result in NFS4ERR_CLID_INUSE. In situations in which there is an apparent change of principal, when the server gets a SETCLIENTID specifying a client ID string for which the server has a clientid4 that currently has no state, or for which it has state, but where the lease has expired, the server MUST allow the SETCLIENTID rather than returning NFS4ERR_CLID_INUSE. The server
Top   ToC   RFC7931 - Page 15
   MUST then confirm the new client ID if followed by the appropriate

5.4. Client ID String Approaches

One particular aspect of the construction of the nfs_client_id4 string has proved recurrently troublesome. The client has a choice of: o Presenting the same id string to multiple server addresses. This is referred to as the "uniform client ID string approach" and is discussed in Section 5.6. o Presenting different id strings to multiple server addresses. This is referred to as the "non-uniform client ID string approach" and is discussed in Section 5.5. Note that implementation considerations, including compatibility with existing servers, may make it desirable for a client to use both approaches, based on configuration information, such as mount options. This issue will be discussed in Section 5.7. Construction of the client ID string has arisen as a difficult issue because of the way in which the NFS protocols have evolved. It is useful to consider two points in that evolution. o NFSv3 as a stateless protocol had no need to identify the state shared by a particular client-server pair (see [RFC1813]). Thus, there was no need to consider the question of whether a set of requests come from the same client or whether two server IP addresses are connected to the same server. As the environment was one in which the user supplied the target server IP address as part of incorporating the remote file system in the client's file namespace, there was no occasion to take note of server trunking. Within a stateless protocol, the situation was symmetrical. The client has no server identity information, and the server has no client identity information. o NFSv4.1 is a stateful protocol with full support for client and server identity determination (see [RFC5661]). This enables the server to be aware when two requests come from the same client (they are on sessions sharing a clientid4) and the client to be aware when two server IP addresses are connected to the same server. Section of [RFC5661] explains how the client is able to assure itself that the connections are to the same logical server.
Top   ToC   RFC7931 - Page 16
   NFSv4.0 is unfortunately halfway between these two.  It introduced
   new requirements such as the need to identify specific clients and
   client instances without addressing server identity issues.  The two
   client ID string approaches have arisen in attempts to deal with the
   changing requirements of the protocol as implementation has
   proceeded, and features that were not very substantial in early
   implementations of NFSv4.0 became more substantial as implementation

   o  In the absence of any implementation of features related to
      fs_locations (replication, referral, and migration), the situation
      is very similar to that of NFSv3 (see Section 8.1 and the
      subsections within Section 8.4 of [RFC7530] for discussion of
      these features).  In this case, locking state has been added, but
      there is no need for concern about the provision of accurate
      client and server identity determination.  This is the situation
      that gave rise to the non-uniform client ID string approach.

   o  In the presence of replication and referrals, the client may have
      occasion to take advantage of knowledge of server trunking
      information.  Even more important, transparent state migration, by
      transferring state among servers, causes difficulties for the non-
      uniform client ID string approach, in that the two different
      client ID strings sent to different IP addresses may wind up being
      processed by the same logical server, adding confusion.

   o  A further consideration is that client implementations typically
      provide NFSv4.1 by augmenting their existing NFSv4.0
      implementation, not by providing two separate implementations.
      Thus, the more NFSv4.0 and NFSv4.1 can work alike, the less
      complex the clients are.  This is a key reason why those
      implementing NFSv4.0 clients might prefer using the uniform client
      string model, even if they have chosen not to provide
      fs_locations-related features in their NFSv4.0 client.

   Both approaches have to deal with the asymmetry in client and server
   identity information between client and server.  Each seeks to make
   the client's and the server's views match.  In the process, each
   encounters some combination of inelegant protocol features and/or
   implementation difficulties.  The choice of which to use is up to the
   client implementer, and the sections below try to give some useful
Top   ToC   RFC7931 - Page 17

5.5. Non-uniform Client ID String Approach

The non-uniform client ID string approach is an attempt to handle these matters in NFSv4.0 client implementations in as NFSv3-like a way as possible. For a client using the non-uniform approach, all internal recording of clientid4 values is to include, whether explicitly or implicitly, the server IP address so that one always has an <IP-address, clientid4> pair. Two such pairs from different servers are always distinct even when the clientid4 values are the same, as they may occasionally be. In this approach, such equality is always treated as simple happenstance. Making the client ID string different on different server IP addresses results in a situation in which a server has no way of tying together information from the same client, when the client accesses multiple server IP addresses. As a result, it will treat a single client as multiple clients with separate leases for each server network address. Since there is no way in the protocol for the client to determine if two network addresses are connected to the same server, the resulting lack of knowledge is symmetrical and can result in simpler client implementations in which there is a single clientid4/lease per server network address. Support for migration, particularly with transparent state migration, is more complex in the case of non-uniform client ID strings. For example, migration of a lease can result in multiple leases for the same client accessing the same server addresses, vitiating many of the advantages of this approach. Therefore, client implementations that support migration with transparent state migration are likely to experience difficulties using the non-uniform client ID string approach and should not do so, except where it is necessary for compatibility with existing server implementations (for details of arranging use of multiple client ID string approaches, see Section 5.7).

5.6. Uniform Client ID String Approach

When the client ID string is kept uniform, the server has the basis to have a single clientid4/lease for each distinct client. The problem that has to be addressed is the lack of explicit server identity information, which was made available in NFSv4.1. When the same client ID string is given to multiple IP addresses, the client can determine whether two IP addresses correspond to a single server, based on the server's behavior. This is the inverse of the strategy adopted for the non-uniform approach in which different
Top   ToC   RFC7931 - Page 18
   server IP addresses are told about different clients, simply to
   prevent a server from manifesting behavior that is inconsistent with
   there being a single server for each IP address, in line with the
   traditions of NFS.  So, to compare:

   o  In the non-uniform approach, servers are told about different
      clients because, if the server were to use accurate client
      identity information, two IP addresses on the same server would
      behave as if they were talking to the same client, which might
      prove disconcerting to a client not expecting such behavior.

   o  In the uniform approach, the servers are told about there being a
      single client, which is, after all, the truth.  Then, when the
      server uses this information, two IP addresses on the same server
      will behave as if they are talking to the same client, and this
      difference in behavior allows the client to infer the server IP
      address trunking configuration, even though NFSv4.0 does not
      explicitly provide this information.

      The approach given in the section below shows one example of how
      this might be done.

   The uniform client ID string approach makes it necessary to exercise
   more care in the definition of the boot instance id sent as the
   verifier field in an nfs_client_id4:

   o  In [RFC7530], the client is told to change the verifier field
      value when reboot occurs, but there is no explicit statement as to
      the converse, so that any requirement to keep the verifier field
      constant unless rebooting is only present by implication.

   o  Many existing clients change the boot instance id every time they
      destroy and recreate the data structure that tracks an
      <IP-address, clientid4> pair.  This might happen if the last mount
      of a particular server is removed, and then a fresh mount is
      created.  Also, note that this might result in each <IP-address,
      clientid4> pair having its own boot instance id that is
      independent of the others.

   o  Within the uniform client ID string approach, an nfs_client_id4
      designates a globally known client instance, so that the verifier
      field should change if and only if a new client instance is
      created, typically as a result of a reboot.

      Clients using the uniform client ID string approach are therefore
      well advised to use a verifier established only once for each
      reboot, typically at reboot time.
Top   ToC   RFC7931 - Page 19
   The following are advantages for the implementation of using the
   uniform client ID string approach:

   o  Clients can take advantage of server trunking (and clustering with
      single-server-equivalent semantics) to increase bandwidth or

   o  There are advantages in state management so that, for example, one
      never has a delegation under one clientid4 revoked because of a
      reference to the same file from the same client under a different

   o  The uniform client ID string approach allows the server to do any
      necessary automatic lease merger in connection with transparent
      state migration, without requiring any client involvement.  This
      consideration is of sufficient weight to cause us to recommend use
      of the uniform client ID string approach for clients supporting
      transparent state migration.

   The following implementation considerations might cause issues for
   client implementations.

   o  This approach is considerably different from the non-uniform
      approach, which most client implementations have been following.
      Until substantial implementation experience is obtained with this
      approach, reluctance to embrace something so new is to be

   o  Mapping between server network addresses and leases is more
      complicated in that it is no longer a one-to-one mapping.

   Another set of relevant considerations relate to privacy concerns,
   which users of the client might have in that use of the uniform
   client ID string approach would enable multiple servers acting in
   concert to determine when multiple requests received at different
   times derive from the same NFSv4.0 client.  For example, this might
   enable determination that multiple distinct user identities in fact
   are likely to correspond to requests made by the same person, even
   when those requests are directed to different servers.

   How to balance these considerations depends on implementation goals.

5.7. Mixing Client ID String Approaches

As noted above, a client that needs to use the uniform client ID string approach (e.g., to support migration) may also need to support existing servers with implementations that do not work properly in this case.
Top   ToC   RFC7931 - Page 20
   Some examples of such server issues include:

   o  Some existing NFSv4.0 server implementations of IP address
      failover depend on clients' use of a non-uniform client ID string
      approach.  In particular, when a server supports both its own IP
      address and one failed over from a partner server, it may have
      separate sets of state applicable to the two IP addresses, owned
      by different servers but residing on a single one.

      In this situation, some servers have relied on clients' use of the
      non-uniform client ID string approach, as suggested but not
      mandated by [RFC7530], to keep these sets of state separate, and
      they will have problems handling clients using the uniform client
      ID string approach, in that such clients will see changes in
      trunking relationships whenever server failover and giveback

   o  Some existing servers incorrectly return NFS4ERR_CLID_INUSE simply
      because there already exists a clientid4 for the same client,
      established using a different IP address.  This causes difficulty
      for a multihomed client using the uniform client ID string

      Although this behavior is not correct, such servers still exist,
      and this specification should give clients guidance about dealing
      with the situation, as well as making the correct behavior clear.

   In order to support use of these sorts of servers, the client can use
   different client ID string approaches for different mounts, in order
   to assure that:

   o  The uniform client ID string approach is used when accessing
      servers that may return NFS4ERR_MOVED and when the client wishes
      to enable transparent state migration.

   o  The non-uniform client ID string approach is used when accessing
      servers whose implementations make them incompatible with the
      uniform client ID string approach.

   Since the client cannot easily determine which of the above are true,
   implementations are likely to rely on user-specified mount options to
   select the appropriate approach to use, in cases in which a client
   supports simultaneous use of multiple approaches.  Choice of a
   default to use in such cases is up to the client implementation.
Top   ToC   RFC7931 - Page 21
   In the case in which the same server has multiple mounts, and both
   approaches are specified for the same server, the client could have
   multiple clientid4s corresponding to the same server, one for each
   approach, and would then have to keep these separate.

5.8. Trunking Determination when Using Uniform Client ID Strings

This section provides an example of how trunking determination could be done by a client following the uniform client ID string approach (whether this is used for all mounts or not). Clients need not follow this procedure, but implementers should make sure that the issues dealt with by this procedure are all properly addressed. It is best to clarify here the various possible purposes of trunking determination and the corresponding requirements as to server behavior. The following points should be noted: o The primary purpose of the trunking determination algorithm is to make sure that, if the server treats client requests on two IP addresses as part of the same client, the client will not be surprised and encounter disconcerting server behavior, as mentioned in Section 5.6. Such behavior could occur if the client were unaware that all of its client requests for the two IP addresses were being handled as part of a single client talking to a single server. o A second purpose is to be able to use knowledge of trunking relationships for better performance, etc. o If a server were to give out distinct clientid4s in response to receiving the same nfs_client_id4 on different network addresses, and acted as if these were separate clients, the primary purpose of trunking determination would be met, as long as the server did not treat them as part of the same client. In this case, the server would be acting, with regard to that client, as if it were two distinct servers. This would interfere with the secondary purpose of trunking determination, but there is nothing the client can do about that. o Suppose a server were to give such a client two different clientid4s but act as if they were one. That is the only way that the server could behave in a way that would defeat the primary purpose of the trunking determination algorithm. Servers MUST NOT behave that way. For a client using the uniform approach, clientid4 values are treated as important information in determining server trunking patterns.
Top   ToC   RFC7931 - Page 22
   For two different IP addresses to return the same clientid4 value is
   a necessary, though not a sufficient condition for them to be
   considered as connected to the same server.  As a result, when two
   different IP addresses return the same clientid4, the client needs to
   determine, using the procedure given below or otherwise, whether the
   IP addresses are connected to the same server.  For such clients, all
   internal recording of clientid4 values needs to include, whether
   explicitly or implicitly, identification of the server from which the
   clientid4 was received so that one always has a (server, clientid4)
   pair.  Two such pairs from different servers are always considered
   distinct even when the clientid4 values are the same, as they may
   occasionally be.

   In order to make this approach work, the client must have certain
   information accessible for each nfs_client_id4 used by the uniform
   approach (only one in general).  The client needs to maintain a list
   of all server IP addresses, together with the associated clientid4
   values, SETCLIENTID principals, and authentication flavors.  As a
   part of the associated data structures, there should be the ability
   to mark a server IP structure as having the same server as another
   and to mark an IP address as currently unresolved.  One way to do
   this is to allow each such entry to point to another with the pointer
   value being one of:

   o  A pointer to another entry for an IP address associated with the
      same server, where that IP address is the first one referenced to
      access that server.

   o  A pointer to the current entry if there is no earlier IP address
      associated with the same server, i.e., where the current IP
      address is the first one referenced to access that server.  The
      text below refers to such an IP address as the lead IP address for
      a given server.

   o  The value NULL if the address's server identity is currently

   In order to keep the above information current, in the interests of
   the most effective trunking determination, RENEWs should be
   periodically done on each server.  However, even if this is not done,
   the primary purpose of the trunking determination algorithm, to
   prevent confusion due to trunking hidden from the client, will be

   Given this apparatus, when a SETCLIENTID is done and a clientid4
   returned, the data structure can be searched for a matching clientid4
   and if such is found, further processing can be done to determine
   whether the clientid4 match is accidental, or the result of trunking.
Top   ToC   RFC7931 - Page 23
   In this algorithm, when SETCLIENTID is done initially, it will use
   the common nfs_client_id4 and specify the current target IP address
   as callback.cb_location within the callback parameters.  We call the
   clientid4 and SETCLIENTID verifier returned by this operation XC and
   XV, respectively.

   This choice of callback parameters is provisional and reflects the
   client's preferences in the event that the IP address is not trunked
   with other IP addresses.  The algorithm is constructed so that only
   the appropriate callback parameters, reflecting observed trunking
   patterns, are actually confirmed.

   Note that when the client has done previous SETCLIENTIDs to any IP
   addresses, with more than one principal or authentication flavor, one
   has the possibility of receiving NFS4ERR_CLID_INUSE, since it is not
   yet known which of the connections with existing IP addresses might
   be trunked with the current one.  In the event that the SETCLIENTID
   fails with NFS4ERR_CLID_INUSE, one must try all other combinations of
   principals and authentication flavors currently in use, and
   eventually one will be correct and not return NFS4ERR_CLID_INUSE.

   Note that at this point, no SETCLIENTID_CONFIRM has yet been done.
   This is because the SETCLIENTID just done has either established a
   new clientid4 on a previously unknown server or changed the callback
   parameters on a clientid4 associated with some already known server.
   Given it is undesirable to confirm something that should not happen,
   what is to be done next depends on information about existing

   o  If no matching clientid4 is found, the IP address X and clientid4
      XC are added to the list and considered as having no existing
      known IP addresses trunked with it.  The IP address is marked as a
      lead IP address for a new server.  A SETCLIENTID_CONFIRM is done
      using XC and XV.

   o  If a matching clientid4 is found that is marked unresolved,
      processing on the new IP address is suspended.  In order to
      simplify processing, there can only be one unresolved IP address
      for any given clientid4.

   o  If one or more matching clientid4s are found, none of which are
      marked unresolved, the new IP address X is entered and marked
      unresolved.  A SETCLIENTID_CONFIRM is done to X using XC and XV.

   When, as a result of encountering the last of the three cases shown
   above, an unresolved IP address exists, further processing is
   required.  After applying the steps below to each of the lead IP
   addresses with a matching clientid4, the address will have been
Top   ToC   RFC7931 - Page 24
   resolved: It may have been determined to be part of an already known
   server as a new IP address to be added to an existing set of IP
   addresses for that server.  Otherwise, it will be recognized as a new
   server.  At the point at which this determination is made, the
   unresolved indication is cleared and any suspended SETCLIENTID
   processing is restarted.

   For each lead IP address IPn with a clientid4 matching XC, the
   following steps are done.  Because the Remote Procedure Call (RPC) to
   do a SETCLIENTID could take considerable time, it is desirable for
   the client to perform these operations in parallel.  Note that
   because the clientid4 is a 64-bit value, the number of such IP
   addresses that would need to be tested is expected to be quite small,
   even when the client is interacting with many NFSv4.0 servers.  Thus,
   while parallel processing is desirable, it is not necessary.

   o  If the principal for IPn does not match that for X, the IP address
      is skipped, since it is impossible for IPn and X to be trunked in
      these circumstances.  If the principal does match but the
      authentication flavor does not, the authentication flavor already
      used should be used for address X as well.  This will avoid any
      possibility that NFS4ERR_CLID_INUSE will be returned for the
      SETCLIENTID and SETCLIENTID_CONFIRM to be done below, as long as
      the server(s) at IP addresses IPn and X is correctly implemented.

   o  A SETCLIENTID is done to update the callback parameters to reflect
      the possibility that X will be marked as associated with the
      server whose lead IP address is IPn.  The specific callback
      parameters chosen, in terms of cb_client4 and callback_ident, are
      up to the client and should reflect its preferences as to callback
      handling for the common clientid4, in the event that X and IPn are
      trunked together.  When a SETCLIENTID is done on IP address IPn, a
      setclientid_confirm value (in the form of a verifier4) is
      returned, which will be referred to as SCn.

      Note that the NFSv4.0 specification requires the server to make
      sure that such verifiers are very unlikely to be regenerated.
      Given that it is already highly unlikely that the clientid4 XC is
      duplicated by distinct servers, the probability that SCn is
      duplicated as well has to be considered vanishingly small.  Note
      also that the callback update procedure can be repeated multiple
      times to reduce the probability of further spurious matches.

   o  The setclientid_confirm value SCn is saved for later use in
      confirming the SETCLIENTID done to IPn.
Top   ToC   RFC7931 - Page 25
   Once the SCn values are gathered up by the procedure above, they are
   each tested by being used as the verifier for a SETCLIENTID_CONFIRM
   operation directed to the original IP address X, whose trunking
   relationships are to be determined.  These RPC operations may be done
   in parallel.

   There are a number of things that should be noted at this point.

   o  The SETCLIENTID operations done on the various IPn addresses in
      the procedure above will never be confirmed by SETCLIENTID_CONFIRM
      operations directed to the various IPn addresses.  If these
      callback updates are to be confirmed, they will be confirmed by
      SETCLIENTID_CONFIRM operations directed at the original IP address
      X, which can only happen if SCn was generated by an IPn that was
      trunked with X, allowing the SETCLIENTID to be successfully
      confirmed and allowing us to infer the existence of that trunking

   o  The number of successful SETCLIENTID_CONFIRM operations done
      should never be more than one.  If both SCn and SCm are accepted
      by X, then it indicates that both IPn and IPm are trunked with X,
      but that is only possible if IPn and IPm are trunked together.
      Since these two addresses were earlier recognized as not trunked
      together, this should be impossible, if the servers in question
      are implemented correctly.

   Further processing depends on the success or failure of the various
   SETCLIENTD_CONFIRM operations done in the step above.

   o  If the setclientid_confirm value generated by a particular IPn is
      accepted on X, then X and IPn are recognized as connected to the
      same server, and the entry for X is marked as associated with IPn.

   o  If none of the confirm operations are accepted, then X is
      recognized as a distinct server.  Its callback parameters will
      remain as the ones established by the original SETCLIENTID.

   In either of the cases, the entry is considered resolved and
   processing can be restarted for IP addresses whose clientid4 matched
   XC but whose resolution had been deferred.

   The procedure described above must be performed so as to exclude the
   possibility that multiple SETCLIENTIDs done to different server IP
   addresses and returning the same clientid4 might "race" in such a
   fashion that there is no explicit determination of whether they
   correspond to the same server.  The following possibilities for
   serialization are all valid, and implementers may choose among them
Top   ToC   RFC7931 - Page 26
   based on a tradeoff between performance and complexity.  They are
   listed in order of increasing parallelism:

   o  An NFSv4.0 client might serialize all instances of SETCLIENTID/
      SETCLIENTID_CONFIRM processing, either directly or by serializing
      mount operations involving use of NFSv4.0.  While doing so will
      prevent the races mentioned above, this degree of serialization
      can cause performance issues when there is a high volume of mount

   o  One might instead serialize the period of processing that begins
      when the clientid4 received from the server is processed and ends
      when all trunking determination for that server is completed.
      This prevents the races mentioned above, without adding to delay
      except when trunking determination is common.

   o  One might avoid much of the serialization implied above, by
      allowing trunking determination for distinct clientid4 values to
      happen in parallel, with serialization of trunking determination
      happening independently for each distinct clientid4 value.

   The procedure above has made no explicit mention of the possibility
   that server reboot can occur at any time.  To address this
   possibility, the client should make sure the following steps are

   o  When a SETCLIENTID_CONFIRM is rejected by a given IPn, the client
      should be aware of the possibility that the rejection is due to XC
      (rather than XV) being invalid.  This situation can be addressed
      by doing a RENEW specifying XC directed to the IP address X.  If
      that operation succeeds, then the rejection is to be acted on
      normally since either XV is invalid on IPn or XC has become
      invalid on IPn while it is valid on X, showing that IPn and X are
      not trunked.  If, on the other hand, XC is not valid on X, then
      the trunking detection process should be restarted once a new
      client ID is established on X.

   o  In the event of a reboot detected on any server-lead IP, the set
      of IP addresses associated with the server should not change, and
      state should be re-established for the lease as a whole, using all
      available connected server IP addresses.  It is prudent to verify
      connectivity by doing a RENEW using the new clientid4 on each such
      server address before using it, however.

   Another situation not discussed explicitly above is the possibility
   that a SETCLIENTID done to one of the IPn addresses might take so
   long that it is necessary to time out the operation, to prevent
   unacceptably delaying the MOUNT operation.  One simple possibility is
Top   ToC   RFC7931 - Page 27
   to simply fail the MOUNT at this point.  Because the average number
   of IP addresses that might have to be tested is quite small, this
   will not greatly increase the probability of MOUNT failure.  Other
   possible approaches are:

   o  If the IPn has sufficient state in existence, the existing
      stateids and sequence values might be validated by being used on
      IP address X.  In the event of success, X and IPn should be
      considered trunked together.

      What constitutes "sufficient" state in this context is an
      implementation decision that is affected by the implementer's
      willingness to fail the MOUNT in an uncertain case and the
      strength of the state verification procedure implemented.

   o  If IPn has no locking state in existence, X could be recorded as a
      lead IP address on a provisional basis, subject to trunking being
      tested again, once IPn starts becoming responsive.  To avoid
      confusion between IPn and X, and the need to merge distinct state
      corpora for X and IPn at a later point, this retest of trunking
      should occur after RENEWs on IPn are responded to and before
      establishing any new state for either IPn as a separate server or
      for IPn considered as a server address trunked with X.

   o  The client locking-related code could be made more tolerant of
      what would otherwise be considered anomalous results due to an
      unrecognized trunking relationship.  The client could use the
      appearance of behavior explainable by a previously unknown
      trunking relationship as the cue to consider the addresses as

      This choice has a lot of complexity associated with it, and it is
      likely that few implementations will use it.  When the set of
      locking state on IPn is small (e.g., a single stateid) but not
      empty, most client implementations are likely to either fail the
      MOUNT or implement a more stringent verification procedure using
      the existing stateid on IPn as a basis to generate further state
      as raw material for the trunking verification process.

5.9. Client ID String Construction Details

This section gives more detailed guidance on client ID string construction. The guidance in this section will cover cases in which either the uniform or the non-uniform approach to the client ID string is used.
Top   ToC   RFC7931 - Page 28
   Note that among the items suggested for inclusion, there are many
   that may conceivably change.  In order for the client ID string to
   remain valid in such circumstances, the client SHOULD either:

   o  Use a saved copy of such value rather than the changeable value
      itself, or

   o  Save the constructed client ID string rather than constructing it
      anew at SETCLIENTID time, based on unchangeable parameters and
      saved copies of changeable data items.

   A file is not always a valid choice to store such information, given
   the existence of diskless clients.  In such situations, whatever
   facilities exist for a client to store configuration information such
   as boot arguments should be used.

   Given the considerations listed in Section 5.2.1, an id string would
   be one that includes as its basis:

   o  An identifier uniquely associated with the node on which the
      client is running.

   o  For a user-level NFSv4.0 client, it should contain additional
      information to distinguish the client from a kernel-based client
      and from other user-level clients running on the same node, such
      as a universally unique identifier (UUID).

   o  Where the non-uniform approach is to be used, the IP address of
      the server.

   o  Additional information that tends to be unique, such as one or
      more of:

      *  The timestamp of when the NFSv4 software was first installed on
         the client (though this is subject to the previously mentioned
         caution about using information that is stored in a file,
         because the file might only be accessible over NFSv4).

      *  A true random number, generally established once and saved.

   With regard to the identifier associated with the node on which the
   client is running, the following possibilities are likely candidates.

   o  The client machine's serial number.

   o  The client's IP address.  Note that this SHOULD be treated as a
      changeable value.
Top   ToC   RFC7931 - Page 29
   o  A Media Access Control (MAC) address.  Note that this also should
      be considered a changeable value because of the possibility of
      configuration changes.

   Privacy concerns may be an issue if some of the items above (e.g.,
   machine serial number and MAC address) are used.  When it is
   necessary to use such items to ensure uniqueness, application of a
   one-way hash function is desirable.  When the non-uniform approach is
   used, that hash function should be applied to all of the components
   chosen as a unit rather than to particular individual elements.

(page 29 continued on part 2)

Next Section