Tech-invite3GPPspaceIETFspace
959493929190898887868584838281807978777675747372717069686766656463626160595857565554535251504948474645444342414039383736353433323130292827262524232221201918171615141312111009080706050403020100
in Index   Prev   Next

RFC 5046

Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)

Pages: 85
Obsoleted by:  7145
Updated by:  7146
Part 2 of 4 – Pages 19 to 35
First   Prev   Next

ToP   noToC   RFC5046 - Page 19   prevText

3. Upper Layer Interface Requirements

This section discusses the upper layer interface requirements in the form of an abstract model of the required interactions between the iSCSI layer and the iSER layer. The abstract model used here is derived from the architectural model described in [DA]. [DA] also provides a functional overview of the interactions between the iSCSI layer and the Datamover layer as intended by the Datamover Architecture. The interface requirements are specified by Operational Primitives. An Operational Primitive is an abstract functional interface procedure between the iSCSI layer and the iSER layer that requests one layer to perform a specific action on behalf of the other layer or notifies the other layer of some event. Whenever an Operational Primitive in invoked, the Connection_Handle qualifier is used to identify a particular iSCSI connection. For some Operational
ToP   noToC   RFC5046 - Page 20
   Primitives, a Data_Descriptor is used to identify the iSCSI/SCSI data
   buffer associated with the requested or completed operation.

   The abstract model and the Operational Primitives defined in this
   section facilitate the description of the iSER protocol.  In the rest
   of the iSER specification, the compliance statements related to the
   use of these Operational Primitives are only for the purpose of the
   required interactions between the iSCSI layer and the iSER layer.
   Note that the compliance statements related to the Operational
   Primitives in the rest of this specification only mandate functional
   equivalence on implementations, but do not put any requirements on
   the implementation specifics of the interface between the iSCSI layer
   and the iSER layer.

   Each Operational Primitive is invoked with a set of qualifiers that
   specify the information context for performing the specific action
   being requested of the Operational Primitive.  While the qualifiers
   are required, the method of realizing the qualifiers (e.g., by
   passing synchronously with invocation, or by retrieving from task
   context, or by retrieving from shared memory, etc.) is implementation
   dependent.

3.1. Operational Primitives Offered by iSER

The iSER protocol layer MUST support the following Operational Primitives to be used by the iSCSI protocol layer.

3.1.1. Send_Control

Input qualifiers: Connection_Handle, BHS and AHS (if any) of the iSCSI PDU, PDU-specific qualifiers Return results: Not specified This is used by the iSCSI layers at the initiator and the target to request the outbound transfer of an iSCSI control-type PDU (see Section 7.2). Qualifiers that only apply for a particular control- type PDU are known as PDU-specific qualifiers, e.g., ImmediateDataSize for a SCSI write command. For details on PDU- specific qualifiers, see Section 7.3. The iSCSI layer can only invoke the Send_Control Operational Primitive when the connection is in iSER-assisted mode.

3.1.2. Put_Data

Input qualifiers: Connection_Handle, content of a SCSI Data-in PDU header, Data_Descriptor, Notify_Enable
ToP   noToC   RFC5046 - Page 21
      Return results:  Not specified

   This is used by the iSCSI layer at the target to request the outbound
   transfer of data for a SCSI Data-in PDU from the buffer identified by
   the Data_Descriptor qualifier.  The iSCSI layer can only invoke the
   Put_Data Operational Primitive when the connection is in iSER-
   assisted mode.

   The Notify_Enable qualifier is used to indicate to the iSER layer
   whether or not it should generate an eventual local completion
   notification to the iSCSI layer.  See Section 3.2.2 on
   Data_Completion_Notify for details.

3.1.3. Get_Data

Input qualifiers: Connection_Handle, content of an R2T PDU, Data_Descriptor, Notify_Enable Return results: Not specified This is used by the iSCSI layer at the target to request the inbound transfer of solicited data requested by an R2T PDU into the buffer identified by the Data_Descriptor qualifier. The iSCSI layer can only invoke the Get_Data Operational Primitive when the connection is in iSER-assisted mode. The Notify_Enable qualifier is used to indicate to the iSER layer whether or not it should generate the eventual local completion notification to the iSCSI layer. See Section 3.2.2 on Data_Completion_Notify for details.

3.1.4. Allocate_Connection_Resources

Input qualifiers: Connection_Handle, Resource_Descriptor (optional) Return results: Status This is used by the iSCSI layers at the initiator and the target to request the allocation of all connection resources necessary to support RCaP for an operational iSCSI/iSER connection. The iSCSI layer may optionally specify the implementation-specific resource requirements for the iSCSI connection using the Resource_Descriptor qualifier. A return result of Status=success means that the invocation succeeded, and a return result of Status=failure means that the invocation failed. If the invocation is for a Connection_Handle for
ToP   noToC   RFC5046 - Page 22
   which an earlier invocation succeeded, the request will be ignored by
   the iSER layer and the result of Status=success will be returned.
   Only one Allocate_Connection_Resources Operational Primitive
   invocation can be outstanding for a given Connection_Handle at any
   time.

3.1.5. Deallocate_Connection_Resources

Input qualifiers: Connection_Handle Return results: Not specified This is used by the iSCSI layers at the initiator and the target to request the deallocation of all connection resources that were allocated earlier as a result of a successful invocation of the Allocate_Connection_Resources Operational Primitive.

3.1.6. Enable_Datamover

Input qualifiers: Connection_Handle, Transport_Connection_Descriptor, Final Login_Response_PDU (optional) Return results: Not specified This is used by the iSCSI layers at the initiator and the target to request that a specified iSCSI connection be transitioned to iSER- assisted mode. The Transport_Connection_Descriptor qualifier is used to identify the specific connection associated with the Connection_Handle. The iSCSI layer can only invoke the Enable_Datamover Operational Primitive when there is a corresponding prior resource allocation. The Final_Login_Response_PDU input qualifier is applicable only for a target, and contains the final Login Response PDU that concludes the iSCSI Login Phase. If the underlying transport is TCP, the final Login Response PDU must be sent as a byte stream as expected by the iSCSI layer at the initiator. When this qualifier is used, the iSER layer at the target MUST transmit this final Login Response PDU before transitioning to iSER-assisted mode.

3.1.7. Connection_Terminate

Input qualifiers: Connection_Handle Return results: Not specified
ToP   noToC   RFC5046 - Page 23
   This is used by the iSCSI layers at the initiator and the target to
   request that a specified iSCSI/iSER connection be terminated and all
   associated connection and task resources be freed.  When this
   Operational Primitive invocation returns to the iSCSI layer, the
   iSCSI layer may assume full ownership of all iSCSI-level resources,
   e.g., I/O Buffers, associated with the connection.

3.1.8. Notice_Key_Values

Input qualifiers: Connection_Handle, number of keys, list of Key-Value pairs Return results: Not specified This is used by the iSCSI layers at the initiator and the target to request that the iSER layer take note of the specified Key-Value pairs that were negotiated by the iSCSI peers for the connection.

3.1.9. Deallocate_Task_Resources

Input qualifiers: Connection_Handle, ITT Return results: Not specified This is used by the iSCSI layers at the initiator and the target to request the deallocation of all RCaP-specific resources allocated by the iSER layer for the task identified by the ITT qualifier. The iSER layer may require a certain number of RCaP-specific resources associated with the ITT for each new iSCSI task. In the normal course of execution, these task-level resources in the iSER layer are assumed to be transparently allocated on each task initiation and deallocated on the conclusion of each task as appropriate. In exception scenarios where the task does not conclude with a SCSI Response PDU, the iSER layer needs to be notified of the individual task terminations to aid its task-level resource management. This Operational Primitive is used for this purpose, and is not needed when a SCSI Response PDU normally concludes a task. Note that RCaP- specific task resources are deallocated by the iSER layer when a SCSI Response PDU normally concludes a task, even if the SCSI status was not success.

3.2. Operational Primitives Used by iSER

The iSER layer MUST use the following Operational Primitives offered by the iSCSI protocol layer when the connection is in iSER-assisted mode.
ToP   noToC   RFC5046 - Page 24

3.2.1. Control_Notify

Input qualifiers: Connection_Handle, an iSCSI control-type PDU Return results: Not specified This is used by the iSER layers at the initiator and the target to notify the iSCSI layer of the availability of an inbound iSCSI control-type PDU. A PDU is described as "available" to the iSCSI layer when the iSER layer notifies the iSCSI layer of the reception of that inbound PDU, along with an implementation-specific indication as to where the received PDU is.

3.2.2. Data_Completion_Notify

Input qualifiers: Connection_Handle, ITT, SN Return results: Not specified This is used by the iSER layer to notify the iSCSI layer of the completion of outbound data transfer that was requested by the iSCSI layer only if the invocation of the Put_Data Operational Primitive (see Section 3.1.2) was qualified with Notify_Enable set. SN refers to the DataSN associated with the SCSI Data-in PDU. This is used by the iSER layer to notify the iSCSI layer of the completion of inbound data transfer that was requested by the iSCSI layer only if the invocation of the Get_Data Operational Primitive (see Section 3.1.3) was qualified with Notify_Enable set. SN refers to the R2TSN associated with the R2T PDU.

3.2.3. Data_ACK_Notify

Input qualifier: Connection_Handle, ITT, DataSN Return results: Not specified This is used by the iSER layer at the target to notify the iSCSI layer of the arrival of the data acknowledgement (as defined in [RFC3720]) requested earlier by the iSCSI layer for the outbound data transfer via an invocation of the Put_Data Operational Primitive where the A-bit in the SCSI Data-in PDU is set to 1. See Section 7.3.5. DataSN refers to the expected DataSN of the next SCSI Data-in PDU, which immediately follows the SCSI Data-in PDU with the A-bit set to which this notification corresponds, with semantics as defined in [RFC3720].
ToP   noToC   RFC5046 - Page 25

3.2.4. Connection_Terminate_Notify

Input qualifiers: Connection_Handle Return results: Not specified This is used by the iSER layers at the initiator and the target to notify the iSCSI layer of the unsolicited termination or failure of an iSCSI/iSER connection. The iSER layer MUST deallocate the connection and task resources associated with the terminated connection before the invocation of this Operational Primitive. Note that the Connection_Terminate_Notify Operational Primitive is not invoked when the termination of the connection is earlier requested by the local iSCSI layer.

3.3. iSCSI Protocol Usage Requirements

To operate in an iSER-assisted mode, the iSCSI layers at both the initiator and the target MUST negotiate the RDMAExtensions key (see Section 6.3) to "Yes" on the leading connection. If the RDMAExtensions key is not negotiated to "Yes", then iSER-assisted mode MUST NOT be used. If the RDMAExtensions key is negotiated to "Yes" but the invocation of the Allocate_Connection_Resources Operational Primitive to the iSER layer fails, the iSCSI layer MUST fail the iSCSI Login process or terminate the connection as appropriate. See Section 10.1.3.1 for details. If the RDMAExtensions key is negotiated to "Yes", the iSCSI layer MUST satisfy the following protocol usage requirements from the iSER protocol: 1. The iSCSI layer at the initiator MUST set ExpDataSN to 0 in Task Management Function Requests for Task Allegiance Reassignment for read/bidirectional commands, so as to cause the target to send all unacknowledged read data. 2. The iSCSI layer at the target MUST always return the SCSI status in a separate SCSI Response PDU for read commands, i.e., there MUST NOT be a "phase collapse" in concluding a SCSI read command. 3. The iSCSI layers at both the initiator and the target MUST support the keys as defined in Section 6 on Login/Text Operational Keys. If used as specified, these keys MUST NOT be answered with NotUnderstood, and the semantics as defined MUST be followed for each iSER-assisted connection. 4. The iSCSI layer at the initiator MUST NOT issue SNACKs for PDUs.
ToP   noToC   RFC5046 - Page 26

4. Lower Layer Interface Requirements

4.1. Interactions with the RCaP Layer

The iSER protocol layer is layered on top of an RCaP layer (see Figure 1) and the following are the key features that are assumed to be supported by any RCaP layer: * The RCaP layer supports all basic RDMA operations, including RDMA Write Operation, RDMA Read Operation, Send Operation, Send with Invalidate Operation, Send with Solicited Event Operation, Send with Solicited Event and Invalidate Operation, and Terminate Operation. * The RCaP layer provides reliable, in-order message delivery and direct data placement. * When the iSER layer initiates an RDMA Read Operation following an RDMA Write Operation on one RCaP Stream, the RDMA Read Response Message processing on the remote node will be started only after the preceding RDMA Write Message payload is placed in the memory of the remote node. * The RCaP layer encapsulates a single iSER Message into a single RCaP Message on the Data Source side. The RCaP layer decapsulates the iSER Message before delivering it to the iSER layer on the Data Sink side. * When the iSER layer provides the STag to be remotely invalidated to the RCaP layer for a SendInvSE Message, the RCaP layer uses this STag as the STag to be invalidated in the SendInvSE Message. * The RCaP layer uses the STag and Tagged Offset provided by the iSER layer for the RDMA Write and RDMA Read Request Messages. * When the RCaP layer delivers the content of an RDMA Send Message Type to the iSER layer, the RCaP layer provides the length of the RDMA Send message. This ensures that the iSER layer does not have to carry a length field in the iSER header. * When the RCaP layer delivers the SendSE or SendInvSE Message to the iSER layer, it notifies the iSER layer with the mechanism provided on that interface. * When the RCaP layer delivers a SendInvSE Message to the iSER layer, it passes the value of the STag that was invalidated.
ToP   noToC   RFC5046 - Page 27
   *  The RCaP layer propagates all status and error indications to the
      iSER layer.

   *  For a transport layer that operates in byte stream mode such as
      TCP, the RCaP implementation supports the enabling of the RDMA
      mode after connection establishment and the exchange of Login
      parameters in byte stream mode.  For a transport layer that
      provides message delivery capability such as [IB], the RCaP
      implementation supports the use of the messaging capability by the
      iSCSI layer directly for the Login Phase after connection
      establishment before enabling iSER-assisted mode.

   *  Whenever the iSER layer terminates the RCaP Stream, the RCaP layer
      terminates the associated connection.

4.2. Interactions with the Transport Layer

The iSER layer does not directly setup the transport layer connection (e.g., TCP, or [IB]). During connection setup, the iSCSI layer is responsible for setting up the connection. If the login is successful, the iSCSI layer invokes the Enable_Datamover Operational Primitive to request the iSER layer to transition to the iSER- assisted mode for that iSCSI connection. See Section 5.1 on iSCSI/iSER connection setup. After transitioning to iSER-assisted mode, the RCaP layer and the underlying transport layer are responsible for maintaining the connection and reporting to the iSER layer any connection failures.

5. Connection Setup and Termination

5.1. iSCSI/iSER Connection Setup

During connection setup, the iSCSI layer at the initiator is responsible for establishing a connection with the target. After the connection is established, the iSCSI layers at the initiator and the target enter the Login Phase using the same rules as outlined in [RFC3720]. Transition to iSER-assisted mode occurs when the connection transitions into the iSCSI Full Feature Phase following a successful login negotiation between the initiator and the target in which iSER-assisted mode is negotiated and the connection resources necessary to support RCaP have been allocated at both the initiator and the target. The same connection MUST be used for both the iSCSI Login Phase and the subsequent iSER-assisted Full Feature Phase. iSER-assisted mode MUST be enabled only if it is negotiated on the leading connection during the LoginOperationalNegotiation stage of the iSCSI Login Phase. iSER-assisted mode is negotiated using the RDMAExtensions=<boolean-value> key. Both the initiator and the
ToP   noToC   RFC5046 - Page 28
   target MUST exchange the RDMAExtensions key with the value set to
   "Yes" to enable iSER-assisted mode.  If both the initiator and the
   target fail to negotiate the RDMAExtensions key set to "Yes", then
   the connection MUST continue with the login semantics as defined in
   [RFC3720].  If the RDMAExtensions key is not negotiated to Yes, then
   for some RCaP implementation (such as [IB]), the connection may need
   to be re-established in TCP capable mode.  (For InfiniBand this will
   require an [IPoIB] type connection.)

   iSER-assisted mode is defined for a Normal session only and the
   RDMAExtensions key MUST NOT be negotiated for a Discovery session.
   Discovery sessions are always conducted using the transport layer as
   described in [RFC3720].

   An iSER enabled node is not required to initiate the RDMAExtensions
   key exchange if its preference is for the Traditional iSCSI mode.
   The RDMAExtensions key, if offered, MUST be sent in the first
   available Login Response or Login Request PDU in the
   LoginOperationalNegotiation stage.  This is due to the fact that the
   value of some login parameters might depend on whether iSER-assisted
   mode is enabled.

   iSER-assisted mode is a session-wide attribute.  If both the
   initiator and the target negotiate RDMAExtensions="Yes" on the
   leading connection of a session, then all subsequent connections of
   the same session MUST enable iSER-assisted mode without having to
   exchange an RDMAExtensions key during the iSCSI Login Phase.

   Conversely, if both the initiator and the target fail to negotiate
   RDMAExtensions to "Yes" on the leading connection of a session, then
   the RDMAExtensions key MUST NOT be negotiated further on any
   additional subsequent connection of the session.

   When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest
   and the DataDigest keys MUST be negotiated to "None" on all
   iSCSI/iSER connections participating in that iSCSI session.  This is
   because, for an iSCSI/iSER connection, RCaP is responsible for
   providing error detection that is at least as good as a 32-bit CRC
   for all iSER Messages.  Furthermore, all SCSI Read data are sent
   using RDMA Write Messages instead of the SCSI Data-in PDUs, and all
   solicited SCSI write data are sent using RDMA Read Response Messages
   instead of the SCSI Data-out PDUs.  HeaderDigest and DataDigest that
   apply to iSCSI PDUs, would not be appropriate for RDMA Read and RDMA
   Write operations used with iSER.
ToP   noToC   RFC5046 - Page 29

5.1.1. Initiator Behavior

If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, then on the initiator side, prior to sending the Login Request with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI layer MUST request that the iSER layer allocate the connection resources necessary to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive. The connection resources required are defined by implementation and are outside the scope of this specification. The iSCSI layer may invoke the Notice_Key_Values Operational Primitive before invoking the Allocate_Connection_Resources Operational Primitive to request that the iSER layer take note of the negotiated values of the iSCSI keys for the connection. The specific keys to be passed as input qualifiers are implementation dependent. These may include, but are not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. To minimize the potential for a denial-of service attack, the iSCSI layer MUST NOT request that the iSER layer allocate the connection resources necessary to support RCaP until the iSCSI layer is sufficiently far along in the iSCSI Login Phase that it is reasonably certain that the peer side is not an attacker. In particular, if the Login Phase includes a SecurityNegotiation stage, the iSCSI layer MUST defer the connection resource allocation (i.e., invoking the Allocate_Connection_Resources Operational Primitive) to the LoginOperationalNegotiation stage [RFC3720] so that the resource allocation occurs after the authentication phase is completed. Among the connection resources allocated at the initiator is the Inbound RDMA Read Queue Depth (IRD). As described in Section 9.5.1, R2Ts are transformed by the target into RDMA Read operations. IRD limits the maximum number of simultaneously incoming outstanding RDMA Read Requests per an RCaP Stream from the target to the initiator. The required value of IRD is outside the scope of the iSER specification. The iSER layer at the initiator MUST set IRD to 1 or higher if R2Ts are to be used in the connection. However, the iSER layer at the initiator MAY set IRD to 0 based on implementation configuration, which indicates that no R2Ts will be used on that connection. Initially, the iSER-IRD value at the initiator SHOULD be set to the IRD value at the initiator and MUST NOT be more than the IRD value. On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be set to 0, since the iSER layer at the initiator does not issue RDMA Read Requests to the target.
ToP   noToC   RFC5046 - Page 30
   Failure to allocate the requested connection resources locally
   results in a login failure and its handling is described in Section
   10.1.3.1.

   If the iSER layer at the initiator is successful in allocating the
   connection resources necessary to support RCaP, the following events
   MUST occur in the specified sequence:

   1.  The iSER layer MUST return a success status to the iSCSI layer in
       response to the Allocate_Connection_Resources Operational
       Primitive.

   2.  After the target returns the Login Response with the T bit set to
       1 and the NSG field set to FullFeaturePhase, and a status class
       of 0 (Success), the iSCSI layer MUST request that the iSER layer
       transition to iSER-assisted mode by invoking the Enable_Datamover
       Operational Primitive with the following qualifiers.  (See
       Section 10.1.4.6 for the case when the status class is not
       Success.):

       a.  Connection_Handle that identifies the iSCSI connection.

       b.  Transport_Connection_Descriptor that identifies the specific
           transport connection associated with the Connection_Handle.

   3.  If necessary, the iSER layer should enable RCaP and transition
       the connection to iSER-assisted mode.  When the RCaP is iWARP,
       then this step MUST be done.  Not all RCaPs may need it depending
       on the RCaP Stream start-up state.

   4.  The iSER layer MUST send the iSER Hello Message as the first iSER
       Message.  See Section 5.1.3 on iSER Hello Exchange.

5.1.2. Target Behavior

If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, then on the target side, prior to sending the Login Response with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI layer MUST request that the iSER layer allocate the resources necessary to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive. The connection resources required are defined by implementation and are outside the scope of this specification. Optionally, the iSCSI layer may invoke the Notice_Key_Values Operational Primitive before invoking the Allocate_Connection_Resources Operational Primitive to request that the iSER layer take note of the negotiated values of the iSCSI keys for the connection. The specific keys to be passed as input
ToP   noToC   RFC5046 - Page 31
   qualifiers are implementation dependent.  These may include, but are
   not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc.

   To minimize the potential for a denial-of-service attack, the iSCSI
   layer MUST NOT request that the iSER layer allocate the connection
   resources necessary to support RCaP until the iSCSI layer is
   sufficiently far along in the iSCSI Login Phase that it is reasonably
   certain that the peer side is not an attacker.  In particular, if the
   Login Phase includes a SecurityNegotiation stage, the iSCSI layer
   MUST defer the connection resource allocation (i.e., invoking the
   Allocate_Connection_Resources Operational Primitive) to the
   LoginOperationalNegotiation stage [RFC3720] so that the resource
   allocation occurs after the authentication phase is completed.

   Among the connection resources allocated at the target is the
   Outbound RDMA Read Queue Depth (ORD).  As described in Section 9.5.1,
   R2Ts are transformed by the target into RDMA Read operations.  The
   ORD limits the maximum number of simultaneously outstanding RDMA Read
   Requests per RCaP Stream from the target to the initiator.
   Initially, the iSER-ORD value at the target SHOULD be set to the ORD
   value at the target.

   On the other hand, the IRD at the target MAY be set to 0 since the
   iSER layer at the target does not expect RDMA Read Requests to be
   issued by the initiator.

   Failure to allocate the requested connection resources locally
   results in a login failure and its handling is described in Section
   10.1.3.1.

   If the iSER layer at the target is successful in allocating the
   connection resources necessary to support RCaP, the following events
   MUST occur in the specified sequence:

   1.  The iSER layer MUST return a success status to the iSCSI layer in
       response to the Allocate_Connection_Resources Operational
       Primitive.

   2.  The iSCSI layer MUST request that the iSER layer transition to
       iSER-assisted mode by invoking the Enable_Datamover Operational
       Primitive with the following qualifiers:

       a.  Connection_Handle that identifies the iSCSI connection.

       b.  Transport_Connection_Descriptor that identifies the specific
           transport connection associated with the Connection_Handle.
ToP   noToC   RFC5046 - Page 32
       c.  The final transport layer (e.g., TCP) message containing the
           Login Response with the T bit set to 1 and the NSG field set
           to FullFeaturePhase.

   3.  The iSER layer MUST send the final Login Response PDU in the
       native transport mode to conclude the iSCSI Login Phase.  If the
       underlying transport is TCP, then the iSER layer MUST send the
       final Login Response PDU in byte stream mode.

   4.  After sending the final Login Response PDU, the iSER layer should
       enable RCaP if necessary and transition the connection to iSER-
       assisted mode.  When the RCaP is iWARP, then this step MUST be
       done.  Not all RCaPs may need it depending on the RCaP Stream
       start-up state.

   5.  After receiving the iSER Hello Message from the initiator, the
       iSER layer MUST respond with the iSER HelloReply Message to be
       sent as the first iSER Message.  See Section 5.1.3 on iSER Hello
       Exchange for more details.

   Note: In the above sequence, the operations as described in bullets 3
   and 4 MUST be performed atomically for iWARP connections.  Failure to
   do this may result in race conditions.

5.1.3. iSER Hello Exchange

After the connection transitions into iSER-assisted mode, the first iSER Message sent by the iSER layer at the initiator to the target MUST be the iSER Hello Message. The iSER Hello Message is used by the iSER layer at the initiator to declare iSER parameters to the target. See Section 9.3 on iSER Header Format for the iSER Hello Message. In response to the iSER Hello Message, the iSER layer at the target MUST return the iSER HelloReply Message as the first iSER Message sent by the target. The iSER HelloReply Message is used by the iSER layer at the target to declare iSER parameters to the initiator. See Section 9.4 on iSER Header Format for the iSER HelloReply Message. In the iSER Hello Message, the iSER layer at the initiator declares the iSER-IRD value to the target. Upon receiving the iSER Hello Message, the iSER layer at the target MUST set the iSER-ORD value to the minimum of the iSER-ORD value at the target and the iSER-IRD value declared by the initiator. The iSER layer at the target MAY adjust (lower) its ORD value to match the iSER-ORD value if the iSER-ORD value is smaller than the ORD value at the target in order to free up the unused resources.
ToP   noToC   RFC5046 - Page 33
   In the iSER HelloReply Message, the iSER layer at the target declares
   the iSER-ORD value to the initiator.

   Upon receiving the iSER HelloReply Message, the iSER layer at the
   initiator MAY adjust (lower) its IRD value to match the iSER-ORD
   value in order to free up the unused resources, if the iSER-ORD value
   declared by the target is smaller than the iSER-IRD value declared by
   the initiator.

   It is an iSER level negotiation failure if the iSER parameters
   declared in the iSER Hello Message by the initiator are unacceptable
   to the target.  This includes the following:

   *  The initiator-declared iSER-IRD value is greater than 0 and the
      target-declared iSER-ORD value is 0.

   *  The initiator-supported and the target-supported iSER protocol
      versions do not overlap.

   See Section 10.1.3.2 for the handling of the error situation.

5.2. iSCSI/iSER Connection Termination

5.2.1. Normal Connection Termination at the Initiator

The iSCSI layer at the initiator terminates an iSCSI/iSER connection normally by invoking the Send_Control Operational Primitive qualified with the Logout Request PDU. The iSER layer at the initiator MUST use a SendSE Message to send the Logout Request PDU to the target. After the iSER layer at the initiator receives the SendSE Message containing the Logout Response PDU from the target, it MUST notify the iSCSI layer by invoking the Control_Notify Operational Primitive qualified with the Logout Response PDU. After the iSCSI logout process is complete, the iSCSI layer at the target is responsible for closing the iSCSI/iSER connection as described in Section 5.2.2. After the RCaP layer at the initiator reports that the connection has been closed, the iSER layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local Mapping(s) (if any) that associate the ITT(s) used on that connection to the local STag(s) before notifying the iSCSI layer by invoking the Connection_Terminate_Notify Operational Primitive.
ToP   noToC   RFC5046 - Page 34

5.2.2. Normal Connection Termination at the Target

Upon receiving the SendSE Message containing the Logout Request PDU, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the Logout Request PDU. The iSCSI layer completes the logout process by invoking the Send_Control Operational Primitive qualified with the Logout Response PDU. The iSER layer at the target MUST use a SendSE Message to send the Logout Response PDU to the initiator. After the iSCSI logout process is complete, the iSCSI layer at the target MUST request that the iSER layer at the target terminate the RCaP Stream by invoking the Connection_Terminate Operational Primitive. As part of the termination process, the RCaP layer MUST close the connection. When the RCaP layer notifies the iSER layer after the RCaP Stream and the associated connection are terminated, the iSER layer MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mapping(s) (if any) that associate the ITT(s) used on that connection to the local STag(s) and the Advertised STag(s) respectively.

5.2.3. Termination without Logout Request/Response PDUs

5.2.3.1. Connection Termination Initiated by the iSCSI Layer
The Connection_Terminate Operational Primitive MAY be invoked by the iSCSI layer to request that the iSER layer terminate the RCaP Stream without having previously exchanged the Logout Request and Logout Response PDUs between the two iSCSI/iSER nodes. As part of the termination process, the RCaP layer will close the connection. When the RCaP layer notifies the iSER layer after the RCaP Stream and the associated connection are terminated, the iSER layer MUST perform the following actions. If the Connection_Terminate Operational Primitive is invoked by the iSCSI layer at the target, then the iSER layer at the target MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mappings (if any) that associate the ITT(s) used on the connection to the local STag(s) and the Advertised STag(s), respectively. If the Connection_Terminate Operational Primitive is invoked by the iSCSI layer at the initiator, then the iSER layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local Mapping(s) (if any) that associate the ITT(s) used on the connection to the local STag(s).
ToP   noToC   RFC5046 - Page 35
5.2.3.2. Connection Termination Notification to the iSCSI Layer
If the iSCSI/iSER connection is terminated without the invocation of Connection_Terminate from the iSCSI layer, the iSER layer MUST notify the iSCSI layer that the iSCSI/iSER connection has been terminated by invoking the Connection_Terminate_Notify Operational Primitive. Prior to invoking Connection_Terminate_Notify, the iSER layer at the target MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mappings (if any) that associate the ITT(s) used on the connection to the local STag(s) and the Advertised STag(s), respectively. Prior to invoking Connection_Terminate_Notify, the iSER layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local Mappings (if any) that associate the ITT(s) used on the connection to the local STag(s). If the remote iSCSI/iSER node initiated the closing of the connection (e.g., by sending a TCP FIN or TCP RST), the iSER layer MUST notify the iSCSI layer after the RCaP layer reports that the connection is closed by invoking the Connection_Terminate_Notify Operational Primitive. Another example of a connection termination without a preceding logout is when the iSCSI layer at the initiator does an implicit logout (connection reinstatement).


(page 35 continued on part 3)

Next Section