tech-invite   World Map     

IETF     RFCs     Groups     SIP     ABNFs    |    3GPP     Specs     Gloss.     Arch.     IMS     UICC    |    Misc.    |    search     info

RFC 7145

 
 
 

Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification

Part 2 of 5, p. 20 to 43
Prev RFC Part       Next RFC Part

 


prevText      Top      Up      ToC       Page 20 
3.  Upper-Layer Interface Requirements

   This section discusses the upper-layer interface requirements in the
   form of an abstract model of the required interactions between the
   iSCSI layer and the iSER layer.  The abstract model used here is
   derived from the architectural model described in [DA].  [DA] also
   provides a functional overview of the interactions between the iSCSI
   layer and the Datamover layer as intended by the Datamover
   Architecture.

   The interface requirements are specified by Operational Primitives.
   An Operational Primitive is an abstract functional interface
   procedure between the iSCSI layer and the iSER layer that requests
   one layer to perform a specific action on behalf of the other layer
   or notifies the other layer of some event.  Whenever an Operational
   Primitive in invoked, the Connection_Handle qualifier is used to
   identify a particular iSCSI connection.  For some Operational
   Primitives, a Data_Descriptor is used to identify the iSCSI/SCSI data
   buffer associated with the requested or completed operation.

   The abstract model and the Operational Primitives defined in this
   section facilitate the description of the iSER protocol.  In the rest
   of the iSER specification, the compliance statements related to the
   use of these Operational Primitives are only for the purpose of the

Top      Up      ToC       Page 21 
   required interactions between the iSCSI layer and the iSER layer.
   Note that the compliance statements related to the Operational
   Primitives in the rest of this specification only mandate functional
   equivalence on implementations, but do not put any requirements on
   the implementation specifics of the interface between the iSCSI layer
   and the iSER layer.

   Each Operational Primitive is invoked with a set of qualifiers which
   specify the information context for performing the specific action
   being requested of the Operational Primitive.  While the qualifiers
   are required, the method of realizing the qualifiers (e.g., by
   passing synchronously with invocation, or by retrieving from task
   context, or by retrieving from shared memory, etc.) is implementation
   dependent.

3.1.  Operational Primitives offered by iSER

   The iSER protocol layer MUST support the following Operational
   Primitives to be used by the iSCSI protocol layer.

3.1.1.  Send_Control

      Input qualifiers:  Connection_Handle, BHS and AHS (if any) of the
      iSCSI PDU, PDU-specific qualifiers

      Return results:  Not specified

   This is used by the iSCSI layers at the initiator and the target to
   request the outbound transfer of an iSCSI control-type PDU (see
   Section 7.2).  Qualifiers that only apply for a particular control-
   type PDU are known as PDU-specific qualifiers, e.g.,
   ImmediateDataSize for a SCSI Write command.  For details on PDU-
   specific qualifiers, see Section 7.3.  The iSCSI layer can only
   invoke the Send_Control Operational Primitive when the connection is
   in iSER-assisted mode.

3.1.2.  Put_Data

      Input qualifiers:  Connection_Handle, content of a SCSI Data-In
      PDU header, Data_Descriptor, Notify_Enable

      Return results:  Not specified

   This is used by the iSCSI layer at the target to request the outbound
   transfer of data for a SCSI Data-In PDU from the buffer identified by
   the Data_Descriptor qualifier.  The iSCSI layer can only invoke the
   Put_Data Operational Primitive when the connection is in iSER-
   assisted mode.

Top      Up      ToC       Page 22 
   The Notify_Enable qualifier is used to indicate to the iSER layer
   whether or not it should generate an eventual local completion
   notification to the iSCSI layer.  See Section 3.2.2 on
   Data_Completion_Notify for details.

3.1.3.  Get_Data

      Input qualifiers:  Connection_Handle, content of an R2T PDU,
      Data_Descriptor, Notify_Enable

      Return results:  Not specified

   This is used by the iSCSI layer at the target to request the inbound
   transfer of solicited data requested by an R2T PDU into the buffer
   identified by the Data_Descriptor qualifier.  The iSCSI layer can
   only invoke the Get_Data Operational Primitive when the connection is
   in iSER-assisted mode.

   The Notify_Enable qualifier is used to indicate to the iSER layer
   whether or not it should generate the eventual local completion
   notification to the iSCSI layer.  See Section 3.2.2 on
   Data_Completion_Notify for details.

3.1.4.  Allocate_Connection_Resources

      Input qualifiers:  Connection_Handle, Resource_Descriptor
      (optional)

      Return results:  Status

   This is used by the iSCSI layers at the initiator and the target to
   request the allocation of all connection resources necessary to
   support RCaP for an operational iSCSI/iSER connection.  The iSCSI
   layer may optionally specify the implementation-specific resource
   requirements for the iSCSI connection using the Resource_Descriptor
   qualifier.

   A return result of Status=success means the invocation succeeded, and
   a return result of Status=failure means that the invocation failed.
   If the invocation is for a Connection_Handle for which an earlier
   invocation succeeded, the request will be ignored by the iSER layer
   and the result of Status=success will be returned.  Only one
   Allocate_Connection_Resources Operational Primitive invocation can be
   outstanding for a given Connection_Handle at any time.

Top      Up      ToC       Page 23 
3.1.5.  Deallocate_Connection_Resources

      Input qualifiers:  Connection_Handle

      Return results:  Not specified

   This is used by the iSCSI layers at the initiator and the target to
   request the deallocation of all connection resources that were
   allocated earlier as a result of a successful invocation of the
   Allocate_Connection_Resources Operational Primitive.

3.1.6.  Enable_Datamover

      Input qualifiers:  Connection_Handle,
      Transport_Connection_Descriptor, Final Login_Response_PDU
      (optional)

      Return results:  Not specified

   This is used by the iSCSI layers at the initiator and the target to
   request that iSER-assisted mode be used for the connection.  The
   Transport_Connection_Descriptor qualifier is used to identify the
   specific connection associated with the Connection_Handle.  The iSCSI
   layer can only invoke the Enable_Datamover Operational Primitive when
   there was a corresponding prior resource allocation.

   The Final_Login_Response_PDU input qualifier is applicable only for a
   target and contains the final Login Response PDU that concludes the
   iSCSI Login Phase.

3.1.7.  Connection_Terminate

      Input qualifiers:  Connection_Handle

      Return results:  Not specified

   This is used by the iSCSI layers at the initiator and the target to
   request that a specified iSCSI/iSER connection be terminated and all
   associated connection and task resources be freed.  When this
   Operational Primitive invocation returns to the iSCSI layer, the
   iSCSI layer may assume full ownership of all iSCSI-level resources,
   e.g., I/O Buffers, associated with the connection.

Top      Up      ToC       Page 24 
3.1.8.  Notice_Key_Values

      Input qualifiers:  Connection_Handle, number of keys, list of Key-
      Value pairs

      Return results:  Not specified

   This is used by the iSCSI layers at the initiator and the target to
   request the iSER layer to take note of the specified Key-Value pairs
   that were negotiated by the iSCSI peers for the connection.

3.1.9.  Deallocate_Task_Resources

      Input qualifiers:  Connection_Handle, ITT

      Return results:  Not specified

   This is used by the iSCSI layers at the initiator and the target to
   request the deallocation of all RCaP-specific resources allocated by
   the iSER layer for the task identified by the ITT qualifier.  The
   iSER layer may require a certain number of RCaP-specific resources
   associated with the ITT for each new iSCSI task.  In the normal
   course of execution, these task-level resources in the iSER layer are
   assumed to be transparently allocated on each task initiation and
   deallocated on the conclusion of each task as appropriate.  In
   exception scenarios where the task does not conclude with a SCSI
   Response PDU, the iSER layer needs to be notified of the individual
   task terminations to aid its task-level resource management.  This
   Operational Primitive is used for this purpose and is not needed when
   a SCSI Response PDU normally concludes a task.  Note that RCaP-
   specific task resources are deallocated by the iSER layer when a SCSI
   Response PDU normally concludes a task, even if the SCSI status was
   not success.

3.2.  Operational Primitives Used by iSER

   The iSER layer MUST use the following Operational Primitives offered
   by the iSCSI protocol layer when the connection is in iSER-assisted
   mode.

Top      Up      ToC       Page 25 
3.2.1.  Control_Notify

      Input qualifiers:  Connection_Handle, an iSCSI control-type PDU

      Return results:  Not specified

   This is used by the iSER layers at the initiator and the target to
   notify the iSCSI layer of the availability of an inbound iSCSI
   control-type PDU.  A PDU is described as "available" to the iSCSI
   layer when the iSER layer notifies the iSCSI layer of the reception
   of that inbound PDU, along with an implementation-specific indication
   as to where the received PDU is.

3.2.2.  Data_Completion_Notify

      Input qualifiers:  Connection_Handle, ITT, SN

      Return results:  Not specified

   This is used by the iSER layer to notify the iSCSI layer of the
   completion of the outbound data transfer that was requested by the
   iSCSI layer only if the invocation of the Put_Data Operational
   Primitive (see Section 3.1.2) was qualified with Notify_Enable set.
   SN refers to the DataSN associated with the SCSI Data-In PDU.

   This is used by the iSER layer to notify the iSCSI layer of the
   completion of the inbound data transfer that was requested by the
   iSCSI layer only if the invocation of the Get_Data Operational
   Primitive (see Section 3.1.3) was qualified with Notify_Enable set.
   SN refers to the R2TSN associated with the R2T PDU.

3.2.3.  Data_ACK_Notify

      Input qualifier:  Connection_Handle, ITT, DataSN

      Return results:  Not specified

   This is used by the iSER layer at the target to notify the iSCSI
   layer of the arrival of the data acknowledgement (as defined in
   [iSCSI]) requested earlier by the iSCSI layer for the outbound data
   transfer via an invocation of the Put_Data Operational Primitive
   where the A-bit in the SCSI Data-In PDU is set to one.  See Section
   7.3.5.  DataSN refers to the expected DataSN of the next SCSI Data-In
   PDU that immediately follows the SCSI Data-In PDU with the A-bit set
   to which this notification corresponds, with semantics as defined in
   [iSCSI].

Top      Up      ToC       Page 26 
3.2.4.  Connection_Terminate_Notify

      Input qualifiers:  Connection_Handle

      Return results:  Not specified

   This is used by the iSER layers at the initiator and the target to
   notify the iSCSI layer of the unsolicited termination or failure of
   an iSCSI/iSER connection.  The iSER layer MUST deallocate the
   connection and task resources associated with the terminated
   connection before the invocation of this Operational Primitive.  Note
   that the Connection_Terminate_Notify Operational Primitive is not
   invoked when the termination of the connection was earlier requested
   by the local iSCSI layer.

3.3.  iSCSI Protocol Usage Requirements

   To operate in iSER-assisted mode, the iSCSI layers at both the
   initiator and the target MUST negotiate the RDMAExtensions key (see
   Section 6.3) to "Yes" on the leading connection.  If the
   RDMAExtensions key is not negotiated to "Yes", then iSER-assisted
   mode MUST NOT be used.  If the RDMAExtensons key is negotiated to
   "Yes", but the invocation of the Allocate_Connection_Resources
   Operational Primitive to the iSER layer fails, the iSCSI layer MUST
   fail the iSCSI Login process or terminate the connection as
   appropriate.  See Section 10.1.3.1 for details.

   If the RDMAExtensions key is negotiated to "Yes", the iSCSI layer
   MUST satisfy the following protocol usage requirements from the iSER
   protocol:

   1.  The iSCSI layer at the initiator MUST set ExpDataSN to zero in
       Task Management Function Requests for Task Allegiance
       Reassignment for read/bidirectional commands, so as to cause the
       target to send all unacknowledged read data.

   2.  The iSCSI layer at the target MUST always return the SCSI status
       in a separate SCSI Response PDU for read commands, i.e., there
       MUST NOT be a "phase collapse" in concluding a SCSI Read Command.

   3.  The iSCSI layers at both the initiator and the target MUST
       support the keys as defined in Section 6 on Login/Text
       Operational Keys.  If used as specified, these keys MUST NOT be
       answered with NotUnderstood, and the semantics as defined MUST be
       followed for each iSER-assisted connection.

   4.  The iSCSI layer at the initiator MUST NOT issue SNACKs for PDUs.

Top      Up      ToC       Page 27 
4.  Lower-Layer Interface Requirements

4.1.  Interactions with the RCaP Layer

   The iSER protocol layer is layered on top of an RCaP layer (see
   Figure 1) and the following are the key features that are assumed to
   be supported by any RCaP layer:

   *  The RCaP layer supports all basic RDMA operations, including the
      RDMA Write Operation, RDMA Read Operation, and Send Operation.

   *  The RCaP layer provides reliable, in-order message delivery and
      direct data placement.

   *  When the iSER layer initiates an RDMA Read Operation following an
      RDMA Write Operation on one RCaP Stream, the RDMA Read Response
      Message processing on the remote node will be started only after
      the preceding RDMA Write Message payload is placed in the memory
      of the remote node.

   *  The RCaP layer encapsulates a single iSER Message into a single
      RCaP Message on the Data Source side.  The RCaP layer decapsulates
      the iSER Message before delivering it to the iSER layer on the
      Data Sink side.

   *  For an RCaP layer that supports the Send with Invalidate Message
      (e.g., iWARP), when the iSER layer provides the STag to be
      remotely invalidated to the RCaP layer for a Send with Invalidate
      Message, the RCaP layer uses this STag as the STag to be
      invalidated in the Send with Invalidate Message.

   *  The RCaP layer uses the STag and Tagged Offset provided by the
      iSER layer for the RDMA Write and RDMA Read Request Messages.

   *  When the RCaP layer delivers the content of an RDMA Send Message
      to the iSER layer, the RCaP layer provides the length of the RDMA
      Send Message.  This ensures that the iSER layer does not have to
      carry a length field in the iSER header.

   *  When the RCaP layer delivers the Send Message to the iSER layer,
      it notifies the iSER layer with the mechanism provided on that
      interface.

   *  For an RCaP layer that supports the Send with Invalidate Message
      (e.g., iWARP), when the RCaP layer delivers a Send with Invalidate
      Message to the iSER layer, it passes the value of the STag that
      was invalidated.

Top      Up      ToC       Page 28 
   *  The RCaP layer propagates all status and error indications to the
      iSER layer.

   *  For a transport layer that operates in byte stream mode such as
      TCP, the RCaP implementation supports the enabling of the RDMA
      mode after connection establishment and the exchange of Login
      parameters in byte stream mode.  For a transport layer that
      provides message delivery capability such as [IB], the RCaP
      implementation supports the direct use of the messaging capability
      by the iSCSI layer for the Login Phase after connection
      establishment and before enabling iSER-assisted mode.  (In the
      specific example of InfiniBand [IB], the iSCSI layer uses IB
      messages to transfer iSCSI PDUs for the Login Phase after
      connection establishment and before enabling iSER-assisted mode.)

   *  Whenever the iSER layer terminates the RCaP Stream, the RCaP layer
      terminates the associated connection.

4.2.  Interactions with the Transport Layer

   After the iSER connection is established, the RCaP layer and the
   underlying transport layer are responsible for maintaining the
   connection and reporting to the iSER layer any connection failures.

5.  Connection Setup and Termination

5.1.  iSCSI/iSER Connection Setup

   During connection setup, the iSCSI layer at the initiator is
   responsible for establishing a connection with the target.  After the
   connection is established, the iSCSI layers at the initiator and the
   target enter the Login Phase using the same rules as outlined in
   [iSCSI].  The connection transitions into the iSCSI Full Feature
   Phase in iSER-assisted mode following a successful login negotiation
   between the initiator and the target in which iSER-assisted mode is
   negotiated and the connection resources necessary to support RCaP
   have been allocated at both the initiator and the target.  The same
   connection MUST be used for both the iSCSI Login Phase and the
   subsequent iSER-assisted Full Feature Phase.

   For a transport layer that operates in byte stream mode such as TCP,
   the RCaP implementation supports the enabling of the RDMA mode after
   connection establishment and the exchange of Login parameters in byte
   stream mode.  For a transport layer that provides message delivery
   capability such as [IB], the RCaP implementation supports the use of
   the messaging capability by the iSCSI layer directly for the Login
   Phase after connection establishment before enabling iSER-assisted
   mode.

Top      Up      ToC       Page 29 
   iSER-assisted mode MUST NOT be enabled unless it is negotiated on the
   leading connection during the LoginOperationalNegotiation stage of
   the iSCSI Login Phase.  iSER-assisted mode is negotiated using the
   RDMAExtensions=<boolean-value> key.  Both the initiator and the
   target MUST exchange the RDMAExtensions key with the value set to
   "Yes" to enable iSER-assisted mode.  If both the initiator and the
   target fail to negotiate the RDMAExtensions key set to "Yes", then
   the connection MUST continue with the login semantics as defined in
   [iSCSI].  If the RDMAExtensions key is not negotiated to Yes, then
   for some RCaP implementation (such as [IB]), the existing connection
   may need to be torn down and a new connection may need to be
   established in TCP-capable mode.  (For InfiniBand, this will require
   a connection like [IPoIB].)

   iSER-assisted mode is defined for a Normal session only, and the
   RDMAExtensions key MUST NOT be negotiated for a Discovery session.
   Discovery sessions are always conducted using the transport layer as
   described in [iSCSI].

   An iSER-enabled node is not required to initiate the RDMAExtensions
   key exchange if its preference is for the Traditional iSCSI mode.
   The RDMAExtensions key, if offered, MUST be sent in the first
   available Login Response or Login Request PDU in the
   LoginOperationalNegotiation stage.  This is due to the fact that the
   value of some Login parameters might depend on whether or not iSER-
   assisted mode is enabled.

   iSER-assisted mode is a session-wide attribute.  If both the
   initiator and the target negotiated RDMAExtensions="Yes" on the
   leading connection of a session, then all subsequent connections of
   the same session MUST enable iSER-assisted mode without having to
   exchange RDMAExtensions keys during the iSCSI Login Phase.
   Conversely, if both the initiator and the target failed to negotiate
   RDMAExtensions to "Yes" on the leading connection of a session, then
   the RDMAExtensions key MUST NOT be negotiated further on any
   additional subsequent connection of the session.

   When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest
   and the DataDigest keys MUST be negotiated to "None" on all
   iSCSI/iSER connections participating in that iSCSI session.  This is
   because, for an iSCSI/iSER connection, RCaP is responsible for
   providing error detection that is at least as good as a 32-bit CRC
   for all iSER Messages.  Furthermore, all SCSI Read data are sent
   using RDMA Write Messages instead of the SCSI Data-In PDUs, and all
   solicited SCSI Write data are sent using RDMA Read Response Messages
   instead of the SCSI Data-Out PDUs.  HeaderDigest and DataDigest that
   apply to iSCSI PDUs would not be appropriate for RDMA Read and RDMA
   Write operations used with iSER.

Top      Up      ToC       Page 30 
5.1.1.  Initiator Behavior

   If the outcome of the iSCSI negotiation is to enable iSER-assisted
   mode, then on the initiator side, prior to sending the Login Request
   with the T (Transit) bit set to one and the NSG (Next Stage) field
   set to FullFeaturePhase, the iSCSI layer SHOULD request the iSER
   layer to allocate the connection resources necessary to support RCaP
   by invoking the Allocate_Connection_Resources Operational Primitive.
   The connection resources required are defined by the implementation
   and are outside the scope of this specification.  The iSCSI layer may
   invoke the Notice_Key_Values Operational Primitive before invoking
   the Allocate_Connection_Resources Operational Primitive to request
   the iSER layer to take note of the negotiated values of the iSCSI
   keys for the connection.  The specific keys to be passed in as input
   qualifiers are implementation dependent.  These may include, but are
   not limited to, MaxOutstandingR2T and ErrorRecoveryLevel.

   Among the connection resources allocated at the initiator is the
   Inbound RDMA Read Queue Depth (IRD).  As described in Section 9.5.1,
   R2Ts are transformed by the target into RDMA Read operations.  IRD
   limits the maximum number of simultaneously incoming outstanding RDMA
   Read Requests per an RCaP Stream from the target to the initiator.
   The required value of IRD is outside the scope of the iSER
   specification.  The iSER layer at the initiator MUST set IRD to 1 or
   higher if R2Ts are to be used in the connection.  However, the iSER
   layer at the initiator MAY set IRD to zero based on implementation
   configuration; setting IRD to zero indicates that no R2Ts will be
   used on that connection.  Initially, the iSER-IRD value at the
   initiator SHOULD be set to the IRD value at the initiator and MUST
   NOT be more than the IRD value.

   On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be
   set to zero since the iSER layer at the initiator does not issue RDMA
   Read Requests to the target.

   Failure to allocate the requested connection resources locally
   results in a login failure, and its handling is described in Section
   10.1.3.1.

   The iSER layer MUST return a success status to the iSCSI layer in
   response to the Allocate_Connection_Resources Operational Primitive.

Top      Up      ToC       Page 31 
   After the target returns the Login Response with the T bit set to one
   and the NSG field set to FullFeaturePhase, and a Status-Class of 0x00
   (Success), the iSCSI layer MUST invoke the Enable_Datamover
   Operational Primitive with the following qualifiers.  (See Section
   10.1.4.6 for the case when the Status-Class is not Success.)

      a. Connection_Handle that identifies the iSCSI connection.

      b. Transport_Connection_Descriptor that identifies the specific
         transport connection associated with the Connection_Handle.

   The iSER layer MUST send the iSER Hello Message as the first iSER
   Message only if iSERHelloRequired is negotiated to "Yes".  See
   Section 5.1.3 on iSER Hello Exchange.

   If the iSCSI layer on the initiator side allocates the connection
   resources to support RCaP only after it receives the final Login
   Response PDU from the target, then it may not be able to handle the
   number of unexpected iSCSI control-type PDUs (as declared by the
   MaxOutstandingUnexpectedPDUs key from the initiator) that can be sent
   by the target before the buffer resources are allocated at the
   initiator side.  In this case, the iSERHelloRequired key SHOULD be
   negotiated to "Yes" so that the initiator can allocate the connection
   resources before sending the iSER Hello Message.  See Section 5.1.3
   for more details.

5.1.2.  Target Behavior

   If the outcome of the iSCSI negotiation is to enable iSER-assisted
   mode, then on the target side, prior to sending the Login Response
   with the T (Transit) bit set to one and the NSG (Next Stage) field
   set to FullFeaturePhase, the iSCSI layer MUST request the iSER layer
   to allocate the resources necessary to support RCaP by invoking the
   Allocate_Connection_Resources Operational Primitive.  The connection
   resources required are defined by implementation and are outside the
   scope of this specification.  Optionally, the iSCSI layer may invoke
   the Notice_Key_Values Operational Primitive before invoking the
   Allocate_Connection_Resources Operational Primitive to request the
   iSER layer to take note of the negotiated values of the iSCSI keys
   for the connection.  The specific keys to be passed in as input
   qualifiers are implementation dependent.  These may include, but not
   limited to, MaxOutstandingR2T and ErrorRecoveryLevel.

   Premature allocation of RCaP connection resources can expose an iSER
   target to a resource exhaustion attack on those resources via
   multiple iSER connections that progress only to the point at which
   the implementation allocates the RCaP connection resources.  The
   countermeasure for this attack is initiator authentication; the iSCSI

Top      Up      ToC       Page 32 
   layer MUST NOT request the iSER layer to allocate the connection
   resources necessary to support RCaP until the iSCSI layer is
   sufficiently far along in the iSCSI Login Phase that it is reasonably
   certain that the peer side is not an attacker.  In particular, if the
   Login Phase includes a SecurityNegotiation stage, the iSCSI layer
   MUST defer the connection resource allocation (i.e., invoking the
   Allocate_Connection_Resources Operational Primitive) to the
   LoginOperationalNegotiation stage ([iSCSI]) so that the resource
   allocation occurs after the authentication phase is completed.

   Among the connection resources allocated at the target is the
   Outbound RDMA Read Queue Depth (ORD).  As described in Section 9.5.1,
   R2Ts are transformed by the target into RDMA Read operations.  The
   ORD limits the maximum number of simultaneously outstanding RDMA Read
   Requests per RCaP Stream from the target to the initiator.
   Initially, the iSER-ORD value at the target SHOULD be set to the ORD
   value at the target.

   On the other hand, the IRD at the target MAY be set to zero since the
   iSER layer at the target does not expect RDMA Read Requests to be
   issued by the initiator.

   Failure to allocate the requested connection resources locally
   results in a login failure, and its handling is described in Section
   10.1.3.1.

   If the iSER layer at the target is successful in allocating the
   connection resources necessary to support RCaP, the following events
   MUST occur in the specified sequence:

   1. The iSER layer MUST return a success status to the iSCSI layer in
      response to the Allocate_Connection_Resources Operational
      Primitive.

   2. The iSCSI layer MUST invoke the Enable_Datamover Operational
      Primitive with the following qualifiers:

      a. Connection_Handle that identifies the iSCSI connection.

      b. Transport_Connection_Descriptor that identifies the specific
         transport connection associated with the Connection_Handle.

      c. The final transport-layer (e.g., TCP) message containing the
         Login Response with the T bit set to one and the NSG field set
         to FullFeaturePhase.

Top      Up      ToC       Page 33 
   3. The iSER layer MUST send the final Login Response PDU in the
      native transport mode to conclude the iSCSI Login Phase.  If the
      underlying transport is TCP, then the iSER layer MUST send the
      final Login Response PDU in byte stream mode.

   4. After receiving the iSER Hello Message from the initiator, the
      iSER layer MUST respond with the iSER HelloReply Message to be
      sent as the first iSER Message if iSERHelloRequired is negotiated
      to "Yes".  If the iSER layer receives an iSER Hello Message when
      iSERHelloRequired is negotiated to "No", then this MUST be treated
      as an iSER protocol error.  See Section 5.1.3 on iSER Hello
      Exchange for more details.

   Note: In the above sequence, the operations as described in items 3
   and 4 MUST be performed atomically for iWARP connections.  Failure to
   do this may result in race conditions.

5.1.3.  iSER Hello Exchange

   If iSERHelloRequired is negotiated to "Yes", the first iSER Message
   sent by the iSER layer at the initiator to the target MUST be the
   iSER Hello Message.  The iSER Hello Message is used by the iSER layer
   at the initiator to declare iSER parameters to the target.  See
   Section 9.3 on iSER Header Format for iSER Hello Message.
   Conversely, if iSERHelloRequired is negotiated to "No", then the iSER
   layer at the initiator MUST NOT send an iSER Hello Message.

   In response to the iSER Hello Message, the iSER layer at the target
   MUST return the iSER HelloReply Message as the first iSER Message
   sent by the target if iSERHelloRequired is negotiated to "Yes".  The
   iSER HelloReply Message is used by the iSER layer at the target to
   declare iSER parameters to the initiator.  See Section 9.4 on iSER
   Header Format for iSER HelloReply Message.  If the iSER layer
   receives an iSER Hello Message when iSERHelloRequired is negotiated
   to "No", then this MUST be treated as an iSER protocol error.  See
   Section 10.1.3.4 on iSER Protocol Errors on for more details.

   In the iSER Hello Message, the iSER layer at the initiator declares
   the iSER-IRD value to the target.

   Upon receiving the iSER Hello Message, the iSER layer at the target
   MUST set the iSER-ORD value to the minimum of the iSER-ORD value at
   the target and the iSER-IRD value declared by the initiator.  In
   order to free up the unused resources, the iSER layer at the target
   MAY adjust (lower) its ORD value to match the iSER-ORD value if the
   iSER-ORD value is smaller than the ORD value at the target.

Top      Up      ToC       Page 34 
   In the iSER HelloReply Message, the iSER layer at the target declares
   the iSER-ORD value to the initiator.

   Upon receiving the iSER HelloReply Message, the iSER layer at the
   initiator MAY adjust (lower) its IRD value to match the iSER-ORD
   value in order to free up the unused resources, if the iSER-ORD value
   declared by the target is smaller than the iSER-IRD value declared by
   the initiator.

   It is an iSER-level negotiation failure if the iSER parameters
   declared in the iSER Hello Message by the initiator are unacceptable
   to the target.  This includes the following:

   *  The initiator-declared iSER-IRD value is greater than 0, and the
      target-declared iSER-ORD value is 0.

   *  The initiator-supported and the target-supported iSER protocol
      versions do not overlap.

   See Section 10.1.3.2 on the handling of the error situation.

   An initiator that conforms to [RFC5046] allocates connection
   resources before sending the Login Request with the T (Transit) bit
   set to one and the NSG (Next Stage) field set to FullFeaturePhase.
   (For brevity, this is referred to as "early" connection allocation.)
   The current iSER specification relaxes this requirement to allow an
   initiator to allocate connection resources after it receives the
   final Login Response PDU from the target.  (For brevity, this is
   referred to as "late" connection allocation.)  An initiator that
   employs "late" connection allocation may encounter problems (e.g.,
   RCaP connection closure) with a target that sends unexpected iSCSI
   PDUs immediately upon transitioning to Full Feature Phase, as allowed
   by the negotiated value of the MaxOutstandingUnexpectedPDUs key.  The
   only way to prevent this situation in full generality is to use iSER
   Hello Messages, as they enable the initiator to allocate its
   connection resources before sending its iSER Hello Message.  The
   iSERHelloRequired key is used by the initiator to determine if it is
   dealing with a target that supports the iSER Hello exchanges.
   Fortunately, known iSER target implementations do not take full
   advantage of the number of allowed unexpected PDUs immediately upon
   transitioning into Full Feature Phase, thus enabling an initiator
   workaround that involves a smaller quantity of connection resources
   prior to Full Feature Phase, as explained further below.

   In the following summary, where "late" connection allocation is
   practiced, an initiator that follows [RFC5046] is referred to as an
   "old" initiator; otherwise, it is referred to as a "new" initiator.
   Similarly, a target that does not support the iSERHelloRequired key

Top      Up      ToC       Page 35 
   (and responds with "NotUnderstood" when negotiating the
   iSERHelloRequired key) is referred to as an "old" target; otherwise,
   it is referred to as a "new" target.  Note that an "old" target can
   still support the iSER Hello exchanges, but this fact is not known by
   the initiator.  A "new" target can also respond with "No" when
   negotiating the iSERHelloRequired key.  In this case, its behavior
   with respect to "late" connection allocation is similar to an "old"
   target.

   A "new" initiator will work fine with a "new" target.

   For an "old" initiator and an "old" target, the failure by the
   initiator to handle the number of unexpected iSCSI control-type PDUs
   that are sent by the target before the buffer resources are allocated
   at the initiator can result in the failure of the iSER session caused
   by closure of the underlying RCaP connection.  For the "old" target,
   there is a known implementation that sends one unexpected iSCSI
   control-type PDU after sending the final Login Response and then
   waits awhile before sending the next one.  This tends to alleviate
   somewhat the buffer allocation problem at the initiator.

   For a "new" initiator and an "old" target, the failure by the
   initiator to handle the number of unexpected iSCSI control-type PDUs
   that are sent by the target before the buffer resources are allocated
   at the initiator can result in the failure of the iSER session caused
   by closure of the underlying RCaP connection.  A "new" initiator MAY
   choose to terminate the connection; otherwise, it SHOULD do one of
   the following:

   1. Allocate the connection resources before sending the final Login
      Request PDU.

   2. Allocate one or more buffers for receiving unexpected control-type
      PDUs from the target before sending the final Login Request PDU.
      This reduces the possibility of the unexpected control-type PDUs
      causing the RCaP connection to close before the connection
      resources have been allocated.

Top      Up      ToC       Page 36 
   For an "old" initiator and a "new" target, if the iSERHelloRequired
   key is not negotiated, a "new" target MUST still respond with the
   iSER HelloReply Message when it receives the iSER Hello Message.  If
   the iSERHelloRequired key is negotiated to "No" or "NotUnderstood", a
   "new" target MAY choose to terminate the connection; otherwise, it
   SHOULD delay sending any unexpected control-type PDUs until one of
   the following events has occurred:

   1. A PDU is received from the initiator after it sends the final
      Login Response PDU.

   2. A system-configurable timeout period (say, one second) has
      expired.

5.2.  iSCSI/iSER Connection Termination

5.2.1.  Normal Connection Termination at the Initiator

   The iSCSI layer at the initiator terminates an iSCSI/iSER connection
   normally by invoking the Send_Control Operational Primitive qualified
   with the Logout Request PDU.  The iSER layer at the initiator MUST
   use a Send Message to send the Logout Request PDU to the target.  The
   SendSE Message should be used if supported by the RCaP layer (e.g.,
   iWARP).  After the iSER layer at the initiator receives the Send
   Message containing the Logout Response PDU from the target, it MUST
   notify the iSCSI layer by invoking the Control_Notify Operational
   Primitive qualified with the Logout Response PDU.

   After the iSCSI logout process is complete, the iSCSI layer at the
   target is responsible for closing the iSCSI/iSER connection as
   described in Section 5.2.2.  After the RCaP layer at the initiator
   reports that the connection has been closed, the iSER layer at the
   initiator MUST deallocate all connection and task resources (if any)
   associated with the connection, and invalidate the Local Mappings (if
   any) before notifying the iSCSI layer by invoking the
   Connection_Terminate_Notify Operational Primitive.

5.2.2.  Normal Connection Termination at the Target

   Upon receiving the Send Message containing the Logout Request PDU,
   the iSER layer at the target MUST notify the iSCSI layer at the
   target by invoking the Control_Notify Operational Primitive qualified
   with the Logout Request PDU.  The iSCSI layer completes the logout
   process by invoking the Send_Control Operational Primitive qualified
   with the Logout Response PDU.  The iSER layer at the target MUST use
   a Send Message to send the Logout Response PDU to the initiator.  The
   SendSE Message should be used if supported by the RCaP layer (e.g.,
   iWARP).  After the iSCSI logout process is complete, the iSCSI layer

Top      Up      ToC       Page 37 
   at the target MUST request the iSER layer at the target to terminate
   the RCaP Stream by invoking the Connection_Terminate Operational
   Primitive.

   As part of the termination process, the RCaP layer MUST close the
   connection.  When the RCaP layer notifies the iSER layer after the
   RCaP Stream and the associated connection are terminated, the iSER
   layer MUST deallocate all connection and task resources (if any)
   associated with the connection, and invalidate the Local and Remote
   Mappings (if any).

5.2.3.  Termination without Logout Request/Response PDUs

5.2.3.1.  Connection Termination Initiated by the iSCSI layer

   The Connection_Terminate Operational Primitive MAY be invoked by the
   iSCSI layer to request the iSER layer to terminate the RCaP Stream
   without having previously exchanged the Logout Request and Logout
   Response PDUs between the two iSCSI/iSER nodes.  As part of the
   termination process, the RCaP layer will close the connection.  When
   the RCaP layer notifies the iSER layer after the RCaP Stream and the
   associated connection are terminated, the iSER layer MUST perform the
   following actions.

   If the Connection_Terminate Operational Primitive is invoked by the
   iSCSI layer at the target, then the iSER layer at the target MUST
   deallocate all connection and task resources (if any) associated with
   the connection, and invalidate the Local and Remote Mappings (if
   any).

   If the Connection_Terminate Operational Primitive is invoked by the
   iSCSI layer at the initiator, then the iSER layer at the initiator
   MUST deallocate all connection and task resources (if any) associated
   with the connection, and invalidate the Local Mappings (if any).

5.2.3.2.  Connection Termination Notification to the iSCSI layer

   If the iSCSI/iSER connection is terminated without the invocation of
   Connection_Terminate from the iSCSI layer, the iSER layer MUST notify
   the iSCSI layer that the iSCSI/iSER connection has been terminated by
   invoking the Connection_Terminate_Notify Operational Primitive.

   Prior to invoking Connection_Terminate_Notify, the iSER layer at the
   target MUST deallocate all connection and task resources (if any)
   associated with the connection, and invalidate the Local and Remote
   Mappings (if any).

Top      Up      ToC       Page 38 
   Prior to invoking Connection_Terminate_Notify, the iSER layer at the
   initiator MUST deallocate all connection and task resources (if any)
   associated with the connection, and invalidate the Local Mappings (if
   any).

   If the remote iSCSI/iSER node initiated the closing of the connection
   (e.g., by sending a TCP FIN or TCP RST), the iSER layer MUST notify
   the iSCSI layer after the RCaP layer reports that the connection is
   closed by invoking the Connection_Terminate_Notify Operational
   Primitive.

   Another example of a connection termination without a preceding
   logout is when the iSCSI layer at the initiator does an implicit
   logout (connection reinstatement).

6.  Login/Text Operational Keys

   Certain iSCSI login/text operational keys have restricted usage in
   iSER, and additional keys are used to support the iSER protocol
   functionality.  All other keys defined in [iSCSI] and not discussed
   in this section may be used on iSCSI/iSER connections with the same
   semantics.

6.1.  HeaderDigest and DataDigest

   Irrelevant when: RDMAExtensions=Yes

   Negotiations resulting in RDMAExtensions=Yes for a session imply
   HeaderDigest=None and DataDigest=None for all connections in that
   session and override the settings, whether default or configured.

6.2.  MaxRecvDataSegmentLength

   For an iSCSI connection belonging to a session in which
   RDMAExtensions=Yes was negotiated on the leading connection of the
   session, MaxRecvDataSegmentLength need not be declared in the Login
   Phase, and MUST be ignored if it is declared.  Instead,
   InitiatorRecvDataSegmentLength (as described in Section 6.5) and
   TargetRecvDataSegmentLength (as described in Section 6.4) keys are
   negotiated.  The values of the local and remote
   MaxRecvDataSegmentLength are derived from the
   InitiatorRecvDataSegmentLength and TargetRecvDataSegmentLength keys.

   In the Full Feature Phase, the initiator MUST consider the value of
   its local MaxRecvDataSegmentLength (that it would have declared to
   the target) as having the value of InitiatorRecvDataSegmentLength,
   and the value of the remote MaxRecvDataSegmentLength (that would have
   been declared by the target) as having the value of

Top      Up      ToC       Page 39 
   TargetRecvDataSegmentLength.  Similarly, the target MUST consider the
   value of its local MaxRecvDataSegmentLength (that it would have
   declared to the initiator) as having the value of
   TargetRecvDataSegmentLength, and the value of the remote
   MaxRecvDataSegmentLength (that would have been declared by the
   initiator) as having the value of InitiatorRecvDataSegmentLength.

   Note that RFC 3720 requires that when a target receives a NOP-Out
   request with a valid Initiator Task Tag, it responds with a NOP-In
   with the same Initiator Task Tag that was provided in the NOP-Out
   request.  Furthermore, it returns the first MaxRecvDataSegmentLength
   bytes of the initiator-provided Ping Data.  Since there is no
   MaxRecvDataSegmentLength common to the initiator and the target in
   iSER, the length of the data sent with the NOP-Out request MUST NOT
   exceed InitiatorMaxRecvDataSegmentLength.

   The MaxRecvDataSegmentLength key is applicable only for iSCSI
   control-type PDUs.

6.3.  RDMAExtensions

   Use: LO (leading only)

   Senders: Initiator and Target

   Scope: SW (session-wide)

   RDMAExtensions=<boolean-value>

   Irrelevant when: SessionType=Discovery

   Default is No

   Result function is AND

   This key is used by the initiator and the target to negotiate the
   support for iSER-assisted mode.  To enable the use of iSER-assisted
   mode, both the initiator and the target MUST exchange
   RDMAExtensions=Yes.  iSER-assisted mode MUST NOT be used if either
   the initiator or the target offers RDMAExtensions=No.

   An iSER-enabled node is not required to initiate the RDMAExtensions
   key exchange if it prefers to operate in the Traditional iSCSI mode.
   However, if the RDMAExtensions key is to be negotiated, an initiator
   MUST offer the key in the first Login Request PDU in the
   LoginOperationalNegotiation stage of the leading connection, and a
   target MUST offer the key in the first Login Response PDU with which
   it is allowed to do so (i.e., the first Login Response PDU issued

Top      Up      ToC       Page 40 
   after the first Login Request PDU with the C bit set to zero) in the
   LoginOperationalNegotiation stage of the leading connection.  In
   response to the offered key=value pair of RDMAExtensions=yes, an
   initiator MUST respond in the next Login Request PDU with which it is
   allowed to do so, and a target MUST respond in the next Login
   Response PDU with which it is allowed to do so.

   Negotiating the RDMAExtensions key first enables a node to negotiate
   the optimal value for other keys.  Certain iSCSI keys such as
   MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T,
   ImmediateData, etc., may be negotiated differently depending on
   whether the connection is in Traditional iSCSI mode or iSER-assisted
   mode.

6.4.  TargetRecvDataSegmentLength

   Use: IO (Initialize only)

   Senders: Initiator and Target

   Scope: CO (connection-only)

   Irrelevant when: RDMAExtensions=No

   TargetRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>

   Default is 8192 bytes

   Result function is minimum

   This key is relevant only for the iSCSI connection of an iSCSI
   session if RDMAExtensions=Yes was negotiated on the leading
   connection of the session.  It is used by the initiator and the
   target to negotiate the maximum size of the data segment that an
   initiator may send to the target in an iSCSI control-type PDU in the
   Full Feature Phase.  For SCSI Command PDUs and SCSI Data-Out PDUs
   containing non-immediate unsolicited data to be sent by the
   initiator, the initiator MUST send all non-Final PDUs with a data
   segment size of exactly TargetRecvDataSegmentLength whenever the PDUs
   constitute a data sequence whose size is larger than
   TargetRecvDataSegmentLength.

Top      Up      ToC       Page 41 
6.5.  InitiatorRecvDataSegmentLength

   Use: IO (Initialize only)

   Senders: Initiator and Target

   Scope: CO (connection-only)

   Irrelevant when: RDMAExtensions=No

   InitiatorRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>

   Default is 8192 bytes

   Result function is minimum

   This key is relevant only for the iSCSI connection of an iSCSI
   session if RDMAExtensions=Yes was negotiated on the leading
   connection of the session.  It is used by the initiator and the
   target to negotiate the maximum size of the data segment that a
   target may send to the initiator in an iSCSI control-type PDU in the
   Full Feature Phase.

6.6.  OFMarker and IFMarker

   Irrelevant when: RDMAExtensions=Yes

   Negotiations resulting in RDMAExtensions=Yes for a session imply
   OFMarker=No and IFMarker=No for all connections in that session and
   override the settings, whether default or configured.

6.7.  MaxOutstandingUnexpectedPDUs

   Use: LO (leading only), Declarative

   Senders: Initiator and Target

   Scope: SW (session-wide)

   Irrelevant when: RDMAExtensions=No

   MaxOutstandingUnexpectedPDUs=
      <numerical-value-from-2-to-(2**32-1) | 0>

   Default is 0

Top      Up      ToC       Page 42 
   This key is used by the initiator and the target to declare the
   maximum number of outstanding "unexpected" iSCSI control-type PDUs
   that it can receive in the Full Feature Phase.  It is intended to
   allow the receiving side to determine the amount of buffer resources
   needed beyond the normal flow control mechanism available in iSCSI.
   An initiator or target should select a value such that it would not
   impose an unnecessary constraint on the iSCSI layer under normal
   circumstances.  The value of 0 is defined to indicate that the
   declarer has no limit on the maximum number of outstanding
   "unexpected" iSCSI control-type PDUs that it can receive.  See
   Sections 8.1.1 and 8.1.2 for the usage of this key.  Note that iSER
   Hello and HelloReply Messages are not iSCSI control-type PDUs and are
   not affected by this key.

   For interoperability with implementations based on [RFC5046], this
   key SHOULD be negotiated because the default value of 0 in [RFC5046]
   is problematic for most implementations as it does not impose a bound
   on resources consumable by unexpected PDUs.

6.8.  MaxAHSLength

   Use: LO (leading only), Declarative

   Senders: Initiator and Target

   Scope: SW (session-wide)

   Irrelevant when: RDMAExtensions=No

   MaxAHSLength=<numerical-value-from-2-to-(2**32-1) | 0>

   Default is 256

   This key is used by the initiator and target to declare the maximum
   size of AHS in an iSCSI control-type PDU that it can receive in the
   Full Feature Phase.  It is intended to allow the receiving side to
   determine the amount of resources needed for receive buffering.  An
   initiator or target should select a value such that it would not
   impose an unnecessary constraint on the iSCSI layer under normal
   circumstances.  The value of 0 is defined to indicate that the
   declarer has no limit on the maximum size of AHS in iSCSI control-
   type PDUs that it can receive.

   For interoperability with implementations based on [RFC5046], an
   initiator or target MAY terminate the connection if it anticipates
   MaxAHSLength to be greater than 256 and the key is not understood by
   its peer.

Top      Up      ToC       Page 43 
6.9.  TaggedBufferForSolicitedDataOnly

   Use: LO (leading only), Declarative

   Senders: Initiator

   Scope: SW (session-wide)

   RDMAExtensions=<boolean-value>

   Irrelevant when: RDMAExtensions=No

   Default is No

   This key is used by the initiator to declare to the target the usage
   of the Write Base Offset in the iSER header of an iSCSI control-type
   PDU.  When set to No, the Base Offset is associated with an I/O
   buffer that contains all the write data, including both unsolicited
   and solicited data.  When set to Yes, the Base Offset is associated
   with an I/O buffer that only contains solicited data.

6.10.  iSERHelloRequired

   Use: LO (leading only), Declarative

   Senders: Initiator

   Scope: SW (session-wide)

   RDMAExtensions=<boolean-value>

   Irrelevant when: RDMAExtensions=No

   Default is No

   This key is relevant only for the iSCSI connection of an iSCSI
   session if RDMAExtensions=Yes was negotiated on the leading
   connection of the session.  It is used by the initiator to declare to
   the target whether the iSER Hello Exchange is required.  When set to
   Yes, the iSER layers MUST perform the iSER Hello Exchange as
   described in Section 5.1.3.  When set to No, the iSER layers MUST NOT
   perform the iSER Hello Exchange.


Next RFC Part