tech-invite   World Map     

IETF     RFCs     Groups     SIP     ABNFs    |    3GPP     Specs     Gloss.     Arch.     IMS     UICC    |    Misc.    |    search     info

RFC 5661

 
 
 

Network File System (NFS) Version 4 Minor Version 1 Protocol

Part 11 of 20, p. 309 to 334
Prev RFC Part       Next RFC Part

 


prevText      Top      Up      ToC       Page 309 
13.  NFSv4.1 as a Storage Protocol in pNFS: the File Layout Type

   This section describes the semantics and format of NFSv4.1 file-based
   layouts for pNFS.  NFSv4.1 file-based layouts use the
   LAYOUT4_NFSV4_1_FILES layout type.  The LAYOUT4_NFSV4_1_FILES type
   defines striping data across multiple NFSv4.1 data servers.

13.1.  Client ID and Session Considerations

   Sessions are a REQUIRED feature of NFSv4.1, and this extends to both
   the metadata server and file-based (NFSv4.1-based) data servers.

   The role a server plays in pNFS is determined by the result it
   returns from EXCHANGE_ID.  The roles are:

   o  Metadata server (EXCHGID4_FLAG_USE_PNFS_MDS is set in the result
      eir_flags).

   o  Data server (EXCHGID4_FLAG_USE_PNFS_DS).

   o  Non-metadata server (EXCHGID4_FLAG_USE_NON_PNFS).  This is an
      NFSv4.1 server that does not support operations (e.g., LAYOUTGET)
      or attributes that pertain to pNFS.

   The client MAY request zero or more of EXCHGID4_FLAG_USE_NON_PNFS,
   EXCHGID4_FLAG_USE_PNFS_DS, or EXCHGID4_FLAG_USE_PNFS_MDS, even though
   some combinations (e.g., EXCHGID4_FLAG_USE_NON_PNFS |
   EXCHGID4_FLAG_USE_PNFS_MDS) are contradictory.  However, the server
   MUST only return the following acceptable combinations:

        +--------------------------------------------------------+
        | Acceptable Results from EXCHANGE_ID                    |
        +--------------------------------------------------------+
        | EXCHGID4_FLAG_USE_PNFS_MDS                             |
        | EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS |
        | EXCHGID4_FLAG_USE_PNFS_DS                              |
        | EXCHGID4_FLAG_USE_NON_PNFS                             |
        | EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS |
        +--------------------------------------------------------+

   As the above table implies, a server can have one or two roles.  A
   server can be both a metadata server and a data server, or it can be
   both a data server and non-metadata server.  In addition to returning
   two roles in the EXCHANGE_ID's results, and thus serving both roles

Top      Up      ToC       Page 310 
   via a common client ID, a server can serve two roles by returning a
   unique client ID and server owner for each role in each of two
   EXCHANGE_ID results, with each result indicating each role.

   In the case of a server with concurrent pNFS roles that are served by
   a common client ID, if the EXCHANGE_ID request from the client has
   zero or a combination of the bits set in eia_flags, the server result
   should set bits that represent the higher of the acceptable
   combination of the server roles, with a preference to match the roles
   requested by the client.  Thus, if a client request has
   (EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS |
   EXCHGID4_FLAG_USE_PNFS_DS) flags set, and the server is both a
   metadata server and a data server, serving both the roles by a common
   client ID, the server SHOULD return with (EXCHGID4_FLAG_USE_PNFS_MDS
   | EXCHGID4_FLAG_USE_PNFS_DS) set.

   In the case of a server that has multiple concurrent pNFS roles, each
   role served by a unique client ID, if the client specifies zero or a
   combination of roles in the request, the server results SHOULD return
   only one of the roles from the combination specified by the client
   request.  If the role specified by the server result does not match
   the intended use by the client, the client should send the
   EXCHANGE_ID specifying just the interested pNFS role.

   If a pNFS metadata client gets a layout that refers it to an NFSv4.1
   data server, it needs a client ID on that data server.  If it does
   not yet have a client ID from the server that had the
   EXCHGID4_FLAG_USE_PNFS_DS flag set in the EXCHANGE_ID results, then
   the client needs to send an EXCHANGE_ID to the data server, using the
   same co_ownerid as it sent to the metadata server, with the
   EXCHGID4_FLAG_USE_PNFS_DS flag set in the arguments.  If the server's
   EXCHANGE_ID results have EXCHGID4_FLAG_USE_PNFS_DS set, then the
   client may use the client ID to create sessions that will exchange
   pNFS data operations.  The client ID returned by the data server has
   no relationship with the client ID returned by a metadata server
   unless the client IDs are equal, and the server owners and server
   scopes of the data server and metadata server are equal.

   In NFSv4.1, the session ID in the SEQUENCE operation implies the
   client ID, which in turn might be used by the server to map the
   stateid to the right client/server pair.  However, when a data server
   is presented with a READ or WRITE operation with a stateid, because
   the stateid is associated with a client ID on a metadata server, and
   because the session ID in the preceding SEQUENCE operation is tied to
   the client ID of the data server, the data server has no obvious way
   to determine the metadata server from the COMPOUND procedure, and

Top      Up      ToC       Page 311 
   thus has no way to validate the stateid.  One RECOMMENDED approach is
   for pNFS servers to encode metadata server routing and/or identity
   information in the data server filehandles as returned in the layout.

   If metadata server routing and/or identity information is encoded in
   data server filehandles, when the metadata server identity or
   location changes, the data server filehandles it gave out will become
   invalid (stale), and so the metadata server MUST first recall the
   layouts.  Invalidating a data server filehandle does not render the
   NFS client's data cache invalid.  The client's cache should map a
   data server filehandle to a metadata server filehandle, and a
   metadata server filehandle to cached data.

   If a server is both a metadata server and a data server, the server
   might need to distinguish operations on files that are directed to
   the metadata server from those that are directed to the data server.
   It is RECOMMENDED that the values of the filehandles returned by the
   LAYOUTGET operation be different than the value of the filehandle
   returned by the OPEN of the same file.

   Another scenario is for the metadata server and the storage device to
   be distinct from one client's point of view, and the roles reversed
   from another client's point of view.  For example, in the cluster
   file system model, a metadata server to one client might be a data
   server to another client.  If NFSv4.1 is being used as the storage
   protocol, then pNFS servers need to encode the values of filehandles
   according to their specific roles.

13.1.1.  Sessions Considerations for Data Servers

   Section 2.10.11.2 states that a client has to keep its lease renewed
   in order to prevent a session from being deleted by the server.  If
   the reply to EXCHANGE_ID has just the EXCHGID4_FLAG_USE_PNFS_DS role
   set, then (as noted in Section 13.6) the client will not be able to
   determine the data server's lease_time attribute because GETATTR will
   not be permitted.  Instead, the rule is that any time a client
   receives a layout referring it to a data server that returns just the
   EXCHGID4_FLAG_USE_PNFS_DS role, the client MAY assume that the
   lease_time attribute from the metadata server that returned the
   layout applies to the data server.  Thus, the data server MUST be
   aware of the values of all lease_time attributes of all metadata
   servers for which it is providing I/O, and it MUST use the maximum of
   all such lease_time values as the lease interval for all client IDs
   and sessions established on it.

   For example, if one metadata server has a lease_time attribute of 20
   seconds, and a second metadata server has a lease_time attribute of
   10 seconds, then if both servers return layouts that refer to an

Top      Up      ToC       Page 312 
   EXCHGID4_FLAG_USE_PNFS_DS-only data server, the data server MUST
   renew a client's lease if the interval between two SEQUENCE
   operations on different COMPOUND requests is less than 20 seconds.

13.2.  File Layout Definitions

   The following definitions apply to the LAYOUT4_NFSV4_1_FILES layout
   type and may be applicable to other layout types.

   Unit.  A unit is a fixed-size quantity of data written to a data
      server.

   Pattern.  A pattern is a method of distributing one or more equal
      sized units across a set of data servers.  A pattern is iterated
      one or more times.

   Stripe.  A stripe is a set of data distributed across a set of data
      servers in a pattern before that pattern repeats.

   Stripe Count.  A stripe count is the number of units in a pattern.

   Stripe Width.  A stripe width is the size of a stripe in bytes.  The
      stripe width = the stripe count * the size of the stripe unit.

   Hereafter, this document will refer to a unit that is a written in a
   pattern as a "stripe unit".

   A pattern may have more stripe units than data servers.  If so, some
   data servers will have more than one stripe unit per stripe.  A data
   server that has multiple stripe units per stripe MAY store each unit
   in a different data file (and depending on the implementation, will
   possibly assign a unique data filehandle to each data file).

13.3.  File Layout Data Types

   The high level NFSv4.1 layout types are nfsv4_1_file_layouthint4,
   nfsv4_1_file_layout_ds_addr4, and nfsv4_1_file_layout4.

   The SETATTR operation supports a layout hint attribute
   (Section 5.12.4).  When the client sets a layout hint (data type
   layouthint4) with a layout type of LAYOUT4_NFSV4_1_FILES (the
   loh_type field), the loh_body field contains a value of data type
   nfsv4_1_file_layouthint4.

Top      Up      ToC       Page 313 
   const NFL4_UFLG_MASK            = 0x0000003F;
   const NFL4_UFLG_DENSE           = 0x00000001;
   const NFL4_UFLG_COMMIT_THRU_MDS = 0x00000002;
   const NFL4_UFLG_STRIPE_UNIT_SIZE_MASK
                                   = 0xFFFFFFC0;

   typedef uint32_t nfl_util4;

   enum filelayout_hint_care4 {
           NFLH4_CARE_DENSE        = NFL4_UFLG_DENSE,

           NFLH4_CARE_COMMIT_THRU_MDS
                                   = NFL4_UFLG_COMMIT_THRU_MDS,

           NFLH4_CARE_STRIPE_UNIT_SIZE
                                   = 0x00000040,

           NFLH4_CARE_STRIPE_COUNT = 0x00000080
   };

   /* Encoded in the loh_body field of data type layouthint4: */

   struct nfsv4_1_file_layouthint4 {
           uint32_t        nflh_care;
           nfl_util4       nflh_util;
           count4          nflh_stripe_count;
   };

   The generic layout hint structure is described in Section 3.3.19.
   The client uses the layout hint in the layout_hint (Section 5.12.4)
   attribute to indicate the preferred type of layout to be used for a
   newly created file.  The LAYOUT4_NFSV4_1_FILES layout-type-specific
   content for the layout hint is composed of three fields.  The first
   field, nflh_care, is a set of flags indicating which values of the
   hint the client cares about.  If the NFLH4_CARE_DENSE flag is set,
   then the client indicates in the second field, nflh_util, a
   preference for how the data file is packed (Section 13.4.4), which is
   controlled by the value of the expression nflh_util & NFL4_UFLG_DENSE
   ("&" represents the bitwise AND operator).  If the
   NFLH4_CARE_COMMIT_THRU_MDS flag is set, then the client indicates a
   preference for whether the client should send COMMIT operations to
   the metadata server or data server (Section 13.7), which is
   controlled by the value of nflh_util & NFL4_UFLG_COMMIT_THRU_MDS.  If
   the NFLH4_CARE_STRIPE_UNIT_SIZE flag is set, the client indicates its
   preferred stripe unit size, which is indicated in nflh_util &
   NFL4_UFLG_STRIPE_UNIT_SIZE_MASK (thus, the stripe unit size MUST be a
   multiple of 64 bytes).  The minimum stripe unit size is 64 bytes.  If

Top      Up      ToC       Page 314 
   the NFLH4_CARE_STRIPE_COUNT flag is set, the client indicates in the
   third field, nflh_stripe_count, the stripe count.  The stripe count
   multiplied by the stripe unit size is the stripe width.

   When LAYOUTGET returns a LAYOUT4_NFSV4_1_FILES layout (indicated in
   the loc_type field of the lo_content field), the loc_body field of
   the lo_content field contains a value of data type
   nfsv4_1_file_layout4.  Among other content, nfsv4_1_file_layout4 has
   a storage device ID (field nfl_deviceid) of data type deviceid4.  The
   GETDEVICEINFO operation maps a device ID to a storage device address
   (type device_addr4).  When GETDEVICEINFO returns a device address
   with a layout type of LAYOUT4_NFSV4_1_FILES (the da_layout_type
   field), the da_addr_body field contains a value of data type
   nfsv4_1_file_layout_ds_addr4.

   typedef netaddr4 multipath_list4<>;

   /*
    * Encoded in the da_addr_body field of
    * data type device_addr4:
    */
   struct nfsv4_1_file_layout_ds_addr4 {
           uint32_t        nflda_stripe_indices<>;
           multipath_list4 nflda_multipath_ds_list<>;
   };

   The nfsv4_1_file_layout_ds_addr4 data type represents the device
   address.  It is composed of two fields:

   1.  nflda_multipath_ds_list: An array of lists of data servers, where
       each list can be one or more elements, and each element
       represents a data server address that may serve equally as the
       target of I/O operations (see Section 13.5).  The length of this
       array might be different than the stripe count.

   2.  nflda_stripe_indices: An array of indices used to index into
       nflda_multipath_ds_list.  The value of each element of
       nflda_stripe_indices MUST be less than the number of elements in
       nflda_multipath_ds_list.  Each element of nflda_multipath_ds_list
       SHOULD be referred to by one or more elements of
       nflda_stripe_indices.  The number of elements in
       nflda_stripe_indices is always equal to the stripe count.

Top      Up      ToC       Page 315 
   /*
    * Encoded in the loc_body field of
    * data type layout_content4:
    */
   struct nfsv4_1_file_layout4 {
            deviceid4      nfl_deviceid;
            nfl_util4      nfl_util;
            uint32_t       nfl_first_stripe_index;
            offset4        nfl_pattern_offset;
            nfs_fh4        nfl_fh_list<>;
   };

   The nfsv4_1_file_layout4 data type represents the layout.  It is
   composed of the following fields:

   1.  nfl_deviceid: The device ID that maps to a value of type
       nfsv4_1_file_layout_ds_addr4.

   2.  nfl_util: Like the nflh_util field of data type
       nfsv4_1_file_layouthint4, a compact representation of how the
       data on a file on each data server is packed, whether the client
       should send COMMIT operations to the metadata server or data
       server, and the stripe unit size.  If a server returns two or
       more overlapping layouts, each stripe unit size in each
       overlapping layout MUST be the same.

   3.  nfl_first_stripe_index: The index into the first element of the
       nflda_stripe_indices array to use.

   4.  nfl_pattern_offset: This field is the logical offset into the
       file where the striping pattern starts.  It is required for
       converting the client's logical I/O offset (e.g., the current
       offset in a POSIX file descriptor before the read() or write()
       system call is sent) into the stripe unit number (see
       Section 13.4.1).

       If dense packing is used, then nfl_pattern_offset is also needed
       to convert the client's logical I/O offset to an offset on the
       file on the data server corresponding to the stripe unit number
       (see Section 13.4.4).

       Note that nfl_pattern_offset is not always the same as lo_offset.
       For example, via the LAYOUTGET operation, a client might request
       a layout starting at offset 1000 of a file that has its striping
       pattern start at offset zero.

Top      Up      ToC       Page 316 
   5.  nfl_fh_list: An array of data server filehandles for each list of
       data servers in each element of the nflda_multipath_ds_list
       array.  The number of elements in nfl_fh_list depends on whether
       sparse or dense packing is being used.

       *  If sparse packing is being used, the number of elements in
          nfl_fh_list MUST be one of three values:

          +  Zero.  This means that filehandles used for each data
             server are the same as the filehandle returned by the OPEN
             operation from the metadata server.

          +  One.  This means that every data server uses the same
             filehandle: what is specified in nfl_fh_list[0].

          +  The same number of elements in nflda_multipath_ds_list.
             Thus, in this case, when sending an I/O operation to any
             data server in nflda_multipath_ds_list[X], the filehandle
             in nfl_fh_list[X] MUST be used.

          See the discussion on sparse packing in Section 13.4.4.


       *  If dense packing is being used, the number of elements in
          nfl_fh_list MUST be the same as the number of elements in
          nflda_stripe_indices.  Thus, when sending an I/O operation to
          any data server in
          nflda_multipath_ds_list[nflda_stripe_indices[Y]], the
          filehandle in nfl_fh_list[Y] MUST be used.  In addition, any
          time there exists i and j, (i != j), such that the
          intersection of
          nflda_multipath_ds_list[nflda_stripe_indices[i]] and
          nflda_multipath_ds_list[nflda_stripe_indices[j]] is not empty,
          then nfl_fh_list[i] MUST NOT equal nfl_fh_list[j].  In other
          words, when dense packing is being used, if a data server
          appears in two or more units of a striping pattern, each
          reference to the data server MUST use a different filehandle.

          Indeed, if there are multiple striping patterns, as indicated
          by the presence of multiple objects of data type layout4
          (either returned in one or multiple LAYOUTGET operations), and
          a data server is the target of a unit of one pattern and
          another unit of another pattern, then each reference to each
          data server MUST use a different filehandle.

          See the discussion on dense packing in Section 13.4.4.

   The details on the interpretation of the layout are in Section 13.4.

Top      Up      ToC       Page 317 
13.4.  Interpreting the File Layout

13.4.1.  Determining the Stripe Unit Number

   To find the stripe unit number that corresponds to the client's
   logical file offset, the pattern offset will also be used.  The i'th
   stripe unit (SUi) is:

       relative_offset = file_offset - nfl_pattern_offset;
       SUi = floor(relative_offset / stripe_unit_size);

13.4.2.  Interpreting the File Layout Using Sparse Packing

   When sparse packing is used, the algorithm for determining the
   filehandle and set of data-server network addresses to write stripe
   unit i (SUi) to is:


      stripe_count = number of elements in nflda_stripe_indices;

      j = (SUi + nfl_first_stripe_index) % stripe_count;

      idx = nflda_stripe_indices[j];

      fh_count = number of elements in nfl_fh_list;
      ds_count = number of elements in nflda_multipath_ds_list;

      switch (fh_count) {
        case ds_count:
          fh = nfl_fh_list[idx];
          break;

        case 1:
          fh = nfl_fh_list[0];
          break;

        case 0:
          fh = filehandle returned by OPEN;
          break;

        default:
          throw a fatal exception;
          break;
      }

      address_list = nflda_multipath_ds_list[idx];

Top      Up      ToC       Page 318 
   The client would then select a data server from address_list, and
   send a READ or WRITE operation using the filehandle specified in fh.

   Consider the following example:

   Suppose we have a device address consisting of seven data servers,
   arranged in three equivalence (Section 13.5) classes:

      { A, B, C, D }, { E }, { F, G }

   where A through G are network addresses.

   Then

      nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }

   i.e.,

      nflda_multipath_ds_list[0] = { A, B, C, D }

      nflda_multipath_ds_list[1] = { E }

      nflda_multipath_ds_list[2] = { F, G }

   Suppose the striping index array is:

      nflda_stripe_indices<> = { 2, 0, 1, 0 }

   Now suppose the client gets a layout that has a device ID that maps
   to the above device address.  The initial index contains

      nfl_first_stripe_index = 2,

   and the filehandle list is

      nfl_fh_list = { 0x36, 0x87, 0x67 }.

   If the client wants to write to SU0, the set of valid { network
   address, filehandle } combinations for SUi are determined by:

      nfl_first_stripe_index = 2

Top      Up      ToC       Page 319 
   So

      idx = nflda_stripe_indices[(0 + 2) % 4]

         = nflda_stripe_indices[2]

         = 1

   So

      nflda_multipath_ds_list[1] = { E }

   and

      nfl_fh_list[1] = { 0x87 }

   The client can thus write SU0 to { 0x87, { E } }.

   The destinations of the first 13 storage units are:

                    +-----+------------+--------------+
                    | SUi | filehandle | data servers |
                    +-----+------------+--------------+
                    | 0   | 87         | E            |
                    | 1   | 36         | A,B,C,D      |
                    | 2   | 67         | F,G          |
                    | 3   | 36         | A,B,C,D      |
                    | 4   | 87         | E            |
                    | 5   | 36         | A,B,C,D      |
                    | 6   | 67         | F,G          |
                    | 7   | 36         | A,B,C,D      |
                    | 8   | 87         | E            |
                    | 9   | 36         | A,B,C,D      |
                    | 10  | 67         | F,G          |
                    | 11  | 36         | A,B,C,D      |
                    | 12  | 87         | E            |
                    +-----+------------+--------------+

13.4.3.  Interpreting the File Layout Using Dense Packing

   When dense packing is used, the algorithm for determining the
   filehandle and set of data server network addresses to write stripe
   unit i (SUi) to is:

Top      Up      ToC       Page 320 
      stripe_count = number of elements in nflda_stripe_indices;

      j = (SUi + nfl_first_stripe_index) % stripe_count;

      idx = nflda_stripe_indices[j];

      fh_count = number of elements in nfl_fh_list;
      ds_count = number of elements in nflda_multipath_ds_list;

      switch (fh_count) {
        case stripe_count:
          fh = nfl_fh_list[j];
          break;

        default:
          throw a fatal exception;
          break;
      }

      address_list = nflda_multipath_ds_list[idx];


   The client would then select a data server from address_list, and
   send a READ or WRITE operation using the filehandle specified in fh.

   Consider the following example (which is the same as the sparse
   packing example, except for the filehandle list):

   Suppose we have a device address consisting of seven data servers,
   arranged in three equivalence (Section 13.5) classes:

      { A, B, C, D }, { E }, { F, G }

   where A through G are network addresses.

   Then

      nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }

   i.e.,

      nflda_multipath_ds_list[0] = { A, B, C, D }

      nflda_multipath_ds_list[1] = { E }

      nflda_multipath_ds_list[2] = { F, G }

Top      Up      ToC       Page 321 
   Suppose the striping index array is:

      nflda_stripe_indices<> = { 2, 0, 1, 0 }

   Now suppose the client gets a layout that has a device ID that maps
   to the above device address.  The initial index contains

      nfl_first_stripe_index = 2,

   and

      nfl_fh_list = { 0x67, 0x37, 0x87, 0x36 }.

   The interesting examples for dense packing are SU1 and SU3 because
   each stripe unit refers to the same data server list, yet each stripe
   unit MUST use a different filehandle.  If the client wants to write
   to SU1, the set of valid { network address, filehandle } combinations
   for SUi are determined by:

      nfl_first_stripe_index = 2

   So

      j = (1 + 2) % 4 = 3

         idx = nflda_stripe_indices[j]

         = nflda_stripe_indices[3]

         = 0

   So

      nflda_multipath_ds_list[0] = { A, B, C, D }

   and

      nfl_fh_list[3] = { 0x36 }

   The client can thus write SU1 to { 0x36, { A, B, C, D } }.

   For SU3, j = (3 + 2) % 4 = 1, and nflda_stripe_indices[1] = 0.  Then
   nflda_multipath_ds_list[0] = { A, B, C, D }, and nfl_fh_list[1] =
   0x37.  The client can thus write SU3 to { 0x37, { A, B, C, D } }.

Top      Up      ToC       Page 322 
   The destinations of the first 13 storage units are:

                    +-----+------------+--------------+
                    | SUi | filehandle | data servers |
                    +-----+------------+--------------+
                    | 0   | 87         | E            |
                    | 1   | 36         | A,B,C,D      |
                    | 2   | 67         | F,G          |
                    | 3   | 37         | A,B,C,D      |
                    | 4   | 87         | E            |
                    | 5   | 36         | A,B,C,D      |
                    | 6   | 67         | F,G          |
                    | 7   | 37         | A,B,C,D      |
                    | 8   | 87         | E            |
                    | 9   | 36         | A,B,C,D      |
                    | 10  | 67         | F,G          |
                    | 11  | 37         | A,B,C,D      |
                    | 12  | 87         | E            |
                    +-----+------------+--------------+

13.4.4.  Sparse and Dense Stripe Unit Packing

   The flag NFL4_UFLG_DENSE of the nfl_util4 data type (field nflh_util
   of the data type nfsv4_1_file_layouthint4 and field nfl_util of data
   type nfsv4_1_file_layout_ds_addr4) specifies how the data is packed
   within the data file on a data server.  It allows for two different
   data packings: sparse and dense.  The packing type determines the
   calculation that will be made to map the client-visible file offset
   to the offset within the data file located on the data server.

   If nfl_util & NFL4_UFLG_DENSE is zero, this means that sparse packing
   is being used.  Hence, the logical offsets of the file as viewed by a
   client sending READs and WRITEs directly to the metadata server are
   the same offsets each data server uses when storing a stripe unit.
   The effect then, for striping patterns consisting of at least two
   stripe units, is for each data server file to be sparse or "holey".
   So for example, suppose there is a pattern with three stripe units,
   the stripe unit size is 4096 bytes, and there are three data servers
   in the pattern.  Then, the file in data server 1 will have stripe
   units 0, 3, 6, 9, ... filled; data server 2's file will have stripe
   units 1, 4, 7, 10, ... filled; and data server 3's file will have
   stripe units 2, 5, 8, 11, ... filled.  The unfilled stripe units of
   each file will be holes; hence, the files in each data server are
   sparse.

   If sparse packing is being used and a client attempts I/O to one of
   the holes, then an error MUST be returned by the data server.  Using
   the above example, if data server 3 received a READ or WRITE

Top      Up      ToC       Page 323 
   operation for block 4, the data server would return
   NFS4ERR_PNFS_IO_HOLE.  Thus, data servers need to understand the
   striping pattern in order to support sparse packing.

   If nfl_util & NFL4_UFLG_DENSE is one, this means that dense packing
   is being used, and the data server files have no holes.  Dense
   packing might be selected because the data server does not
   (efficiently) support holey files or because the data server cannot
   recognize read-ahead unless there are no holes.  If dense packing is
   indicated in the layout, the data files will be packed.  Using the
   same striping pattern and stripe unit size that were used for the
   sparse packing example, the corresponding dense packing example would
   have all stripe units of all data files filled as follows:

   o  Logical stripe units 0, 3, 6, ... of the file would live on stripe
      units 0, 1, 2, ... of the file of data server 1.

   o  Logical stripe units 1, 4, 7, ... of the file would live on stripe
      units 0, 1, 2, ... of the file of data server 2.

   o  Logical stripe units 2, 5, 8, ... of the file would live on stripe
      units 0, 1, 2, ... of the file of data server 3.

   Because dense packing does not leave holes on the data servers, the
   pNFS client is allowed to write to any offset of any data file of any
   data server in the stripe.  Thus, the data servers need not know the
   file's striping pattern.

   The calculation to determine the byte offset within the data file for
   dense data server layouts is:

      stripe_width = stripe_unit_size * N;
         where N = number of elements in nflda_stripe_indices.

      relative_offset = file_offset - nfl_pattern_offset;

      data_file_offset = floor(relative_offset / stripe_width)
         * stripe_unit_size
         + relative_offset % stripe_unit_size

   If dense packing is being used, and a data server appears more than
   once in a striping pattern, then to distinguish one stripe unit from
   another, the data server MUST use a different filehandle.  Let's
   suppose there are two data servers.  Logical stripe units 0, 3, 6 are
   served by data server 1; logical stripe units 1, 4, 7 are served by
   data server 2; and logical stripe units 2, 5, 8 are also served by
   data server 2.  Unless data server 2 has two filehandles (each
   referring to a different data file), then, for example, a write to

Top      Up      ToC       Page 324 
   logical stripe unit 1 overwrites the write to logical stripe unit 2
   because both logical stripe units are located in the same stripe unit
   (0) of data server 2.

13.5.  Data Server Multipathing

   The NFSv4.1 file layout supports multipathing to multiple data server
   addresses.  Data-server-level multipathing is used for bandwidth
   scaling via trunking (Section 2.10.5) and for higher availability of
   use in the case of a data-server failure.  Multipathing allows the
   client to switch to another data server address which may be that of
   another data server that is exporting the same data stripe unit,
   without having to contact the metadata server for a new layout.

   To support data server multipathing, each element of the
   nflda_multipath_ds_list contains an array of one more data server
   network addresses.  This array (data type multipath_list4) represents
   a list of data servers (each identified by a network address), with
   the possibility that some data servers will appear in the list
   multiple times.

   The client is free to use any of the network addresses as a
   destination to send data server requests.  If some network addresses
   are less optimal paths to the data than others, then the MDS SHOULD
   NOT include those network addresses in an element of
   nflda_multipath_ds_list.  If less optimal network addresses exist to
   provide failover, the RECOMMENDED method to offer the addresses is to
   provide them in a replacement device-ID-to-device-address mapping, or
   a replacement device ID.  When a client finds that no data server in
   an element of nflda_multipath_ds_list responds, it SHOULD send a
   GETDEVICEINFO to attempt to replace the existing device-ID-to-device-
   address mappings.  If the MDS detects that all data servers
   represented by an element of nflda_multipath_ds_list are unavailable,
   the MDS SHOULD send a CB_NOTIFY_DEVICEID (if the client has indicated
   it wants device ID notifications for changed device IDs) to change
   the device-ID-to-device-address mappings to the available data
   servers.  If the device ID itself will be replaced, the MDS SHOULD
   recall all layouts with the device ID, and thus force the client to
   get new layouts and device ID mappings via LAYOUTGET and
   GETDEVICEINFO.

   Generally, if two network addresses appear in an element of
   nflda_multipath_ds_list, they will designate the same data server,
   and the two data server addresses will support the implementation of
   client ID or session trunking (the latter is RECOMMENDED) as defined
   in Section 2.10.5.  The two data server addresses will share the same
   server owner or major ID of the server owner.  It is not always

Top      Up      ToC       Page 325 
   necessary for the two data server addresses to designate the same
   server with trunking being used.  For example, the data could be
   read-only, and the data consist of exact replicas.

13.6.  Operations Sent to NFSv4.1 Data Servers

   Clients accessing data on an NFSv4.1 data server MUST send only the
   NULL procedure and COMPOUND procedures whose operations are taken
   only from two restricted subsets of the operations defined as valid
   NFSv4.1 operations.  Clients MUST use the filehandle specified by the
   layout when accessing data on NFSv4.1 data servers.

   The first of these operation subsets consists of management
   operations.  This subset consists of the BACKCHANNEL_CTL,
   BIND_CONN_TO_SESSION, CREATE_SESSION, DESTROY_CLIENTID,
   DESTROY_SESSION, EXCHANGE_ID, SECINFO_NO_NAME, SET_SSV, and SEQUENCE
   operations.  The client may use these operations in order to set up
   and maintain the appropriate client IDs, sessions, and security
   contexts involved in communication with the data server.  Henceforth,
   these will be referred to as data-server housekeeping operations.

   The second subset consists of COMMIT, READ, WRITE, and PUTFH.  These
   operations MUST be used with a current filehandle specified by the
   layout.  In the case of PUTFH, the new current filehandle MUST be one
   taken from the layout.  Henceforth, these will be referred to as
   data-server I/O operations.  As described in Section 12.5.1, a client
   MUST NOT send an I/O to a data server for which it does not hold a
   valid layout; the data server MUST reject such an I/O.

   Unless the server has a concurrent non-data-server personality --
   i.e., EXCHANGE_ID results returned (EXCHGID4_FLAG_USE_PNFS_DS |
   EXCHGID4_FLAG_USE_PNFS_MDS) or (EXCHGID4_FLAG_USE_PNFS_DS |
   EXCHGID4_FLAG_USE_NON_PNFS) see Section 13.1 -- any attempted use of
   operations against a data server other than those specified in the
   two subsets above MUST return NFS4ERR_NOTSUPP to the client.

   When the server has concurrent data-server and non-data-server
   personalities, each COMPOUND sent by the client MUST be constructed
   so that it is appropriate to one of the two personalities, and it
   MUST NOT contain operations directed to a mix of those personalities.
   The server MUST enforce this.  To understand the constraints,
   operations within a COMPOUND are divided into the following three
   classes:

   1.  An operation that is ambiguous regarding its personality
       assignment.  This includes all of the data-server housekeeping
       operations.  Additionally, if the server has assigned filehandles
       so that the ones defined by the layout are the same as those used

Top      Up      ToC       Page 326 
       by the metadata server, all operations using such filehandles are
       within this class, with the following exception.  The exception
       is that if the operation uses a stateid that is incompatible with
       a data-server personality (e.g., a special stateid or the stateid
       has a non-zero "seqid" field, see Section 13.9.1), the operation
       is in class 3, as described below.  A COMPOUND containing
       multiple class 1 operations (and operations of no other class)
       MAY be sent to a server with multiple concurrent data server and
       non-data-server personalities.

   2.  An operation that is unambiguously referable to the data-server
       personality.  This includes data-server I/O operations where the
       filehandle is one that can only be validly directed to the data-
       server personality.

   3.  An operation that is unambiguously referable to the non-data-
       server personality.  This includes all COMPOUND operations that
       are neither data-server housekeeping nor data-server I/O
       operations, plus data-server I/O operations where the current fh
       (or the one to be made the current fh in the case of PUTFH) is
       only valid on the metadata server or where a stateid is used that
       is incompatible with the data server, i.e., is a special stateid
       or has a non-zero seqid value.

   When a COMPOUND first executes an operation from class 3 above, it
   acts as a normal COMPOUND on any other server, and the data-server
   personality ceases to be relevant.  There are no special restrictions
   on the operations in the COMPOUND to limit them to those for a data
   server.  When a PUTFH is done, filehandles derived from the layout
   are not valid.  If their format is not normally acceptable, then
   NFS4ERR_BADHANDLE MUST result.  Similarly, current filehandles for
   other operations do not accept filehandles derived from layouts and
   are not normally usable on the metadata server.  Using these will
   result in NFS4ERR_STALE.

   When a COMPOUND first executes an operation from class 2, which would
   be PUTFH where the filehandle is one from a layout, the COMPOUND
   henceforth is interpreted with respect to the data-server
   personality.  Operations outside the two classes discussed above MUST
   result in NFS4ERR_NOTSUPP.  Filehandles are validated using the rules
   of the data server, resulting in NFS4ERR_BADHANDLE and/or
   NFS4ERR_STALE even when they would not normally do so when addressed
   to the non-data-server personality.  Stateids must obey the rules of
   the data server in that any use of special stateids or stateids with
   non-zero seqid values must result in NFS4ERR_BAD_STATEID.

Top      Up      ToC       Page 327 
   Until the server first executes an operation from class 2 or class 3,
   the client MUST NOT depend on the operation being executed by either
   the data-server or the non-data-server personality.  The server MUST
   pick one personality consistently for a given COMPOUND, with the only
   possible transition being a single one when the first operation from
   class 2 or class 3 is executed.

   Because of the complexity induced by assigning filehandles so they
   can be used on both a data server and a metadata server, it is
   RECOMMENDED that where the same server can have both personalities,
   the server assign separate unique filehandles to both personalities.
   This makes it unambiguous for which server a given request is
   intended.

   GETATTR and SETATTR MUST be directed to the metadata server.  In the
   case of a SETATTR of the size attribute, the control protocol is
   responsible for propagating size updates/truncations to the data
   servers.  In the case of extending WRITEs to the data servers, the
   new size must be visible on the metadata server once a LAYOUTCOMMIT
   has completed (see Section 12.5.4.2).  Section 13.10 describes the
   mechanism by which the client is to handle data-server files that do
   not reflect the metadata server's size.

13.7.  COMMIT through Metadata Server

   The file layout provides two alternate means of providing for the
   commit of data written through data servers.  The flag
   NFL4_UFLG_COMMIT_THRU_MDS in the field nfl_util of the file layout
   (data type nfsv4_1_file_layout4) is an indication from the metadata
   server to the client of the REQUIRED way of performing COMMIT, either
   by sending the COMMIT to the data server or the metadata server.
   These two methods of dealing with the issue correspond to broad
   styles of implementation for a pNFS server supporting the file layout
   type.

   o  When the flag is FALSE, COMMIT operations MUST to be sent to the
      data server to which the corresponding WRITE operations were sent.
      This approach is sometimes useful when file striping is
      implemented within the pNFS server (instead of the file system),
      with the individual data servers each implementing their own file
      systems.

   o  When the flag is TRUE, COMMIT operations MUST be sent to the
      metadata server, rather than to the individual data servers.  This
      approach is sometimes useful when file striping is implemented
      within the clustered file system that is the backend to the pNFS
      server.  In such an implementation, each COMMIT to each data
      server might result in repeated writes of metadata blocks to the

Top      Up      ToC       Page 328 
      detriment of write performance.  Sending a single COMMIT to the
      metadata server can be more efficient when there exists a
      clustered file system capable of implementing such a coordinated
      COMMIT.

      If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is TRUE, then in order to
      maintain the current NFSv4.1 commit and recovery model, the data
      servers MUST return a common writeverf verifier in all WRITE
      responses for a given file layout, and the metadata server's
      COMMIT implementation must return the same writeverf.  The value
      of the writeverf verifier MUST be changed at the metadata server
      or any data server that is referenced in the layout, whenever
      there is a server event that can possibly lead to loss of
      uncommitted data.  The scope of the verifier can be for a file or
      for the entire pNFS server.  It might be more difficult for the
      server to maintain the verifier at the file level, but the benefit
      is that only events that impact a given file will require recovery
      action.

   Note that if the layout specified dense packing, then the offset used
   to a COMMIT to the MDS may differ than that of an offset used to a
   COMMIT to the data server.

   The single COMMIT to the metadata server will return a verifier, and
   the client should compare it to all the verifiers from the WRITEs and
   fail the COMMIT if there are any mismatched verifiers.  If COMMIT to
   the metadata server fails, the client should re-send WRITEs for all
   the modified data in the file.  The client should treat modified data
   with a mismatched verifier as a WRITE failure and try to recover by
   resending the WRITEs to the original data server or using another
   path to that data if the layout has not been recalled.
   Alternatively, the client can obtain a new layout or it could rewrite
   the data directly to the metadata server.  If nfl_util &
   NFL4_UFLG_COMMIT_THRU_MDS is FALSE, sending a COMMIT to the metadata
   server might have no effect.  If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS
   is FALSE, a COMMIT sent to the metadata server should be used only to
   commit data that was written to the metadata server.  See
   Section 12.7.6 for recovery options.

13.8.  The Layout Iomode

   The layout iomode need not be used by the metadata server when
   servicing NFSv4.1 file-based layouts, although in some circumstances
   it may be useful.  For example, if the server implementation supports
   reading from read-only replicas or mirrors, it would be useful for
   the server to return a layout enabling the client to do so.  As such,
   the client SHOULD set the iomode based on its intent to read or write
   the data.  The client may default to an iomode of LAYOUTIOMODE4_RW.

Top      Up      ToC       Page 329 
   The iomode need not be checked by the data servers when clients
   perform I/O.  However, the data servers SHOULD still validate that
   the client holds a valid layout and return an error if the client
   does not.

13.9.  Metadata and Data Server State Coordination

13.9.1.  Global Stateid Requirements

   When the client sends I/O to a data server, the stateid used MUST NOT
   be a layout stateid as returned by LAYOUTGET or sent by
   CB_LAYOUTRECALL.  Permitted stateids are based on one of the
   following: an OPEN stateid (the stateid field of data type OPEN4resok
   as returned by OPEN), a delegation stateid (the stateid field of data
   types open_read_delegation4 and open_write_delegation4 as returned by
   OPEN or WANT_DELEGATION, or as sent by CB_PUSH_DELEG), or a stateid
   returned by the LOCK or LOCKU operations.  The stateid sent to the
   data server MUST be sent with the seqid set to zero, indicating the
   most current version of that stateid, rather than indicating a
   specific non-zero seqid value.  In no case is the use of special
   stateid values allowed.

   The stateid used for I/O MUST have the same effect and be subject to
   the same validation on a data server as it would if the I/O was being
   performed on the metadata server itself in the absence of pNFS.  This
   has the implication that stateids are globally valid on both the
   metadata and data servers.  This requires the metadata server to
   propagate changes in LOCK and OPEN state to the data servers, so that
   the data servers can validate I/O accesses.  This is discussed
   further in Section 13.9.2.  Depending on when stateids are
   propagated, the existence of a valid stateid on the data server may
   act as proof of a valid layout.

   Clients performing I/O operations need to select an appropriate
   stateid based on the locks (including opens and delegations) held by
   the client and the various types of state-owners sending the I/O
   requests.  The rules for doing so when referencing data servers are
   somewhat different from those discussed in Section 8.2.5, which apply
   when accessing metadata servers.

   The following rules, applied in order of decreasing priority, govern
   the selection of the appropriate stateid:

   o  If the client holds a delegation for the file in question, the
      delegation stateid should be used.

Top      Up      ToC       Page 330 
   o  Otherwise, there must be an OPEN stateid for the current open-
      owner, and that OPEN stateid for the open file in question is
      used, unless mandatory locking prevents that.  See below.

   o  If the data server had previously responded with NFS4ERR_LOCKED to
      use of the OPEN stateid, then the client should use the byte-range
      lock stateid whenever one exists for that open file with the
      current lock-owner.

   o  Special stateids should never be used.  If they are used, the data
      server MUST reject the I/O with an NFS4ERR_BAD_STATEID error.

13.9.2.  Data Server State Propagation

   Since the metadata server, which handles byte-range lock and open-
   mode state changes as well as ACLs, might not be co-located with the
   data servers where I/O accesses are validated, the server
   implementation MUST take care of propagating changes of this state to
   the data servers.  Once the propagation to the data servers is
   complete, the full effect of those changes MUST be in effect at the
   data servers.  However, some state changes need not be propagated
   immediately, although all changes SHOULD be propagated promptly.
   These state propagations have an impact on the design of the control
   protocol, even though the control protocol is outside of the scope of
   this specification.  Immediate propagation refers to the synchronous
   propagation of state from the metadata server to the data server(s);
   the propagation must be complete before returning to the client.

13.9.2.1.  Lock State Propagation

   If the pNFS server supports mandatory byte-range locking, any
   mandatory byte-range locks on a file MUST be made effective at the
   data servers before the request that establishes them returns to the
   caller.  The effect MUST be the same as if the mandatory byte-range
   lock state were synchronously propagated to the data servers, even
   though the details of the control protocol may avoid actual transfer
   of the state under certain circumstances.

   On the other hand, since advisory byte-range lock state is not used
   for checking I/O accesses at the data servers, there is no semantic
   reason for propagating advisory byte-range lock state to the data
   servers.  Since updates to advisory locks neither confer nor remove
   privileges, these changes need not be propagated immediately, and may
   not need to be propagated promptly.  The updates to advisory locks
   need only be propagated when the data server needs to resolve a
   question about a stateid.  In fact, if byte-range locking is not
   mandatory (i.e., is advisory) the clients are advised to avoid using

Top      Up      ToC       Page 331 
   the byte-range lock-based stateids for I/O.  The stateids returned by
   OPEN are sufficient and eliminate overhead for this kind of state
   propagation.

   If a client gets back an NFS4ERR_LOCKED error from a data server,
   this is an indication that mandatory byte-range locking is in force.
   The client recovers from this by getting a byte-range lock that
   covers the affected range and re-sends the I/O with the stateid of
   the byte-range lock.

13.9.2.2.  Open and Deny Mode Validation

   Open and deny mode validation MUST be performed against the open and
   deny mode(s) held by the data servers.  When access is reduced or a
   deny mode made more restrictive (because of CLOSE or OPEN_DOWNGRADE),
   the data server MUST prevent any I/Os that would be denied if
   performed on the metadata server.  When access is expanded, the data
   server MUST make sure that no requests are subsequently rejected
   because of open or deny issues that no longer apply, given the
   previous relaxation.

13.9.2.3.  File Attributes

   Since the SETATTR operation has the ability to modify state that is
   visible on both the metadata and data servers (e.g., the size), care
   must be taken to ensure that the resultant state across the set of
   data servers is consistent, especially when truncating or growing the
   file.

   As described earlier, the LAYOUTCOMMIT operation is used to ensure
   that the metadata is synchronized with changes made to the data
   servers.  For the NFSv4.1-based data storage protocol, it is
   necessary to re-synchronize state such as the size attribute, and the
   setting of mtime/change/atime.  See Section 12.5.4 for a full
   description of the semantics regarding LAYOUTCOMMIT and attribute
   synchronization.  It should be noted that by using an NFSv4.1-based
   layout type, it is possible to synchronize this state before
   LAYOUTCOMMIT occurs.  For example, the control protocol can be used
   to query the attributes present on the data servers.

   Any changes to file attributes that control authorization or access
   as reflected by ACCESS calls or READs and WRITEs on the metadata
   server, MUST be propagated to the data servers for enforcement on
   READ and WRITE I/O calls.  If the changes made on the metadata server
   result in more restrictive access permissions for any user, those
   changes MUST be propagated to the data servers synchronously.

Top      Up      ToC       Page 332 
   The OPEN operation (Section 18.16.4) does not impose any requirement
   that I/O operations on an open file have the same credentials as the
   OPEN itself (unless EXCHGID4_FLAG_BIND_PRINC_STATEID is set when
   EXCHANGE_ID creates the client ID), and so it requires the server's
   READ and WRITE operations to perform appropriate access checking.
   Changes to ACLs also require new access checking by READ and WRITE on
   the server.  The propagation of access-right changes due to changes
   in ACLs may be asynchronous only if the server implementation is able
   to determine that the updated ACL is not more restrictive for any
   user specified in the old ACL.  Due to the relative infrequency of
   ACL updates, it is suggested that all changes be propagated
   synchronously.

13.10.  Data Server Component File Size

   A potential problem exists when a component data file on a particular
   data server has grown past EOF; the problem exists for both dense and
   sparse layouts.  Imagine the following scenario: a client creates a
   new file (size == 0) and writes to byte 131072; the client then seeks
   to the beginning of the file and reads byte 100.  The client should
   receive zeroes back as a result of the READ.  However, if the
   striping pattern directs the client to send the READ to a data server
   other than the one that received the client's original WRITE, the
   data server servicing the READ may believe that the file's size is
   still 0 bytes.  In that event, the data server's READ response will
   contain zero bytes and an indication of EOF.  The data server can
   only return zeroes if it knows that the file's size has been
   extended.  This would require the immediate propagation of the file's
   size to all data servers, which is potentially very costly.
   Therefore, the client that has initiated the extension of the file's
   size MUST be prepared to deal with these EOF conditions.  When the
   offset in the arguments to READ is less than the client's view of the
   file size, if the READ response indicates EOF and/or contains fewer
   bytes than requested, the client will interpret such a response as a
   hole in the file, and the NFS client will substitute zeroes for the
   data.

   The NFSv4.1 protocol only provides close-to-open file data cache
   semantics; meaning that when the file is closed, all modified data is
   written to the server.  When a subsequent OPEN of the file is done,
   the change attribute is inspected for a difference from a cached
   value for the change attribute.  For the case above, this means that
   a LAYOUTCOMMIT will be done at close (along with the data WRITEs) and
   will update the file's size and change attribute.  Access from
   another client after that point will result in the appropriate size
   being returned.

Top      Up      ToC       Page 333 
13.11.  Layout Revocation and Fencing

   As described in Section 12.7, the layout-type-specific storage
   protocol is responsible for handling the effects of I/Os that started
   before lease expiration and extend through lease expiration.  The
   LAYOUT4_NFSV4_1_FILES layout type can prevent all I/Os to data
   servers from being executed after lease expiration (this prevention
   is called "fencing"), without relying on a precise client lease timer
   and without requiring data servers to maintain lease timers.  The
   LAYOUT4_NFSV4_1_FILES pNFS server has the flexibility to revoke
   individual layouts, and thus fence I/O on a per-file basis.

   In addition to lease expiration, the reasons a layout can be revoked
   include: client fails to respond to a CB_LAYOUTRECALL, the metadata
   server restarts, or administrative intervention.  Regardless of the
   reason, once a client's layout has been revoked, the pNFS server MUST
   prevent the client from sending I/O for the affected file from and to
   all data servers; in other words, it MUST fence the client from the
   affected file on the data servers.

   Fencing works as follows.  As described in Section 13.1, in COMPOUND
   procedure requests to the data server, the data filehandle provided
   by the PUTFH operation and the stateid in the READ or WRITE operation
   are used to ensure that the client has a valid layout for the I/O
   being performed; if it does not, the I/O is rejected with
   NFS4ERR_PNFS_NO_LAYOUT.  The server can simply check the stateid and,
   additionally, make the data filehandle stale if the layout specified
   a data filehandle that is different from the metadata server's
   filehandle for the file (see the nfl_fh_list description in
   Section 13.3).

   Before the metadata server takes any action to revoke layout state
   given out by a previous instance, it must make sure that all layout
   state from that previous instance are invalidated at the data
   servers.  This has the following implications.

   o  The metadata server must not restripe a file until it has
      contacted all of the data servers to invalidate the layouts from
      the previous instance.

   o  The metadata server must not give out mandatory locks that
      conflict with layouts from the previous instance without either
      doing a specific layout invalidation (as it would have to do
      anyway) or doing a global data server invalidation.

Top      Up      ToC       Page 334 
13.12.  Security Considerations for the File Layout Type

   The NFSv4.1 file layout type MUST adhere to the security
   considerations outlined in Section 12.9.  NFSv4.1 data servers MUST
   make all of the required access checks on each READ or WRITE I/O as
   determined by the NFSv4.1 protocol.  If the metadata server would
   deny a READ or WRITE operation on a file due to its ACL, mode
   attribute, open access mode, open deny mode, mandatory byte-range
   lock state, or any other attributes and state, the data server MUST
   also deny the READ or WRITE operation.  This impacts the control
   protocol and the propagation of state from the metadata server to the
   data servers; see Section 13.9.2 for more details.

   The methods for authentication, integrity, and privacy for data
   servers based on the LAYOUT4_NFSV4_1_FILES layout type are the same
   as those used by metadata servers.  Metadata and data servers use ONC
   RPC security flavors to authenticate, and SECINFO and SECINFO_NO_NAME
   to negotiate the security mechanism and services to be used.  Thus,
   when using the LAYOUT4_NFSV4_1_FILES layout type, the impact on the
   RPC-based security model due to pNFS (as alluded to in Sections 1.7.1
   and 1.7.2.2) is zero.

   For a given file object, a metadata server MAY require different
   security parameters (secinfo4 value) than the data server.  For a
   given file object with multiple data servers, the secinfo4 value
   SHOULD be the same across all data servers.  If the secinfo4 values
   across a metadata server and its data servers differ for a specific
   file, the mapping of the principal to the server's internal user
   identifier MUST be the same in order for the access-control checks
   based on ACL, mode, open and deny mode, and mandatory locking to be
   consistent across on the pNFS server.

   If an NFSv4.1 implementation supports pNFS and supports NFSv4.1 file
   layouts, then the implementation MUST support the SECINFO_NO_NAME
   operation on both the metadata and data servers.



(page 334 continued on part 12)

Next RFC Part