tech-invite   World Map     

IETF     RFCs     Groups     SIP     ABNFs    |    3GPP     Specs     Glossaries     Architecture     IMS     UICC    |    search     info

RFC 5046


Pages: 85
Top     in Index     Prev     Next
 

Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)

Part 1 of 4, p. 1 to 19
None       Next RFC Part

Obsoleted by:    7145
Updated by:    7146


Top       ToC       Page 1 
Network Working Group                                              M. Ko
Request for Comments: 5046                               IBM Corporation
Category: Standards Track                                 M. Chadalapaka
                                                 Hewlett-Packard Company
                                                              J. Hufferd
                                                           Brocade, Inc.
                                                                U. Elzur
                                                                 H. Shah
                                                               P. Thaler
                                                    Broadcom Corporation
                                                            October 2007


      Internet Small Computer System Interface (iSCSI) Extensions
                 for Remote Direct Memory Access (RDMA)

Status of This Memo

   This document specifies an Internet standards track protocol for the
   Internet community, and requests discussion and suggestions for
   improvements.  Please refer to the current edition of the "Internet
   Official Protocol Standards" (STD 1) for the standardization state
   and status of this protocol.  Distribution of this memo is unlimited.

Abstract

   Internet Small Computer System Interface (iSCSI) Extensions for
   Remote Direct Memory Access (RDMA) provides the RDMA data transfer
   capability to iSCSI by layering iSCSI on top of an RDMA-Capable
   Protocol, such as the iWARP protocol suite.  An RDMA-Capable Protocol
   provides RDMA Read and Write services, which enable data to be
   transferred directly into SCSI I/O Buffers without intermediate data
   copies.  This document describes the extensions to the iSCSI protocol
   to support RDMA services as provided by an RDMA-Capable Protocol,
   such as the iWARP protocol suite.

Top       Page 2 
Table of Contents

   1. Introduction ....................................................5
      1.1. Motivation .................................................5
      1.2. Architectural Goals ........................................6
      1.3. Protocol Overview ..........................................7
      1.4. RDMA Services and iSER .....................................8
           1.4.1. STag ................................................8
           1.4.2. Send ................................................9
           1.4.3. RDMA Write ..........................................9
           1.4.4. RDMA Read ...........................................9
      1.5. SCSI Read Overview ........................................10
      1.6. SCSI Write Overview .......................................10
      1.7. iSCSI/iSER Layering .......................................10
   2. Definitions and Acronyms .......................................11
      2.1. Definitions ...............................................11
      2.2. Acronyms ..................................................17
      2.3. Conventions ...............................................19
   3. Upper Layer Interface Requirements .............................19
      3.1. Operational Primitives Offered by iSER ....................20
           3.1.1. Send_Control .......................................20
           3.1.2. Put_Data ...........................................20
           3.1.3. Get_Data ...........................................21
           3.1.4. Allocate_Connection_Resources ......................21
           3.1.5. Deallocate_Connection_Resources ....................22
           3.1.6. Enable_Datamover ...................................22
           3.1.7. Connection_Terminate ...............................22
           3.1.8. Notice_Key_Values ..................................23
           3.1.9. Deallocate_Task_Resources ..........................23
      3.2. Operational Primitives Used by iSER .......................23
           3.2.1. Control_Notify .....................................24
           3.2.2. Data_Completion_Notify .............................24
           3.2.3. Data_ACK_Notify ....................................24
           3.2.4. Connection_Terminate_Notify ........................25
      3.3. iSCSI Protocol Usage Requirements .........................25
   4. Lower Layer Interface Requirements .............................26
      4.1. Interactions with the RCaP Layer ..........................26
      4.2. Interactions with the Transport Layer .....................27
   5. Connection Setup and Termination ...............................27
      5.1. iSCSI/iSER Connection Setup ...............................27
           5.1.1. Initiator Behavior .................................29
           5.1.2. Target Behavior ....................................30
           5.1.3. iSER Hello Exchange ................................32
      5.2. iSCSI/iSER Connection Termination .........................33
           5.2.1. Normal Connection Termination at the Initiator .....33
           5.2.2. Normal Connection Termination at the Target ........34
           5.2.3. Termination without Logout Request/Response PDUs ...34

Top      ToC       Page 3 
   6. Login/Text Operational Keys ....................................35
      6.1. HeaderDigest and DataDigest ...............................35
      6.2. MaxRecvDataSegmentLength ..................................36
      6.3. RDMAExtensions ............................................36
      6.4. TargetRecvDataSegmentLength ...............................37
      6.5. InitiatorRecvDataSegmentLength ............................38
      6.6. OFMarker and IFMarker .....................................38
      6.7. MaxOutstandingUnexpectedPDUs ..............................38
   7. iSCSI PDU Considerations .......................................39
      7.1. iSCSI Data-Type PDU .......................................39
      7.2. iSCSI Control-Type PDU ....................................40
      7.3. iSCSI PDUs ................................................40
           7.3.1. SCSI Command .......................................40
           7.3.2. SCSI Response ......................................42
           7.3.3. Task Management Function Request/Response ..........44
           7.3.4. SCSI Data-Out ......................................45
           7.3.5. SCSI Data-In .......................................46
           7.3.6. Ready to Transfer (R2T) ............................48
           7.3.7. Asynchronous Message ...............................50
           7.3.8. Text Request and Text Response .....................50
           7.3.9. Login Request and Login Response ...................50
           7.3.10. Logout Request and Logout Response ................51
           7.3.11. SNACK Request .....................................51
           7.3.12. Reject ............................................51
           7.3.13. NOP-Out and NOP-In ................................51
   8. Flow Control and STag Management ...............................52
      8.1. Flow Control for RDMA Send Message Types ..................52
           8.1.1. Flow Control for Control-Type PDUs from the
                  Initiator ..........................................52
           8.1.2. Flow Control for Control-Type PDUs from the
                  Target .............................................55
      8.2. Flow Control for RDMA Read Resources ......................56
      8.3. STag Management ...........................................56
           8.3.1. Allocation of STags ................................57
           8.3.2. Invalidation of STags ..............................57
   9. iSER Control and Data Transfer .................................58
      9.1. iSER Header Format ........................................58
      9.2. iSER Header Format for the iSCSI Control-Type PDU .........59
      9.3. iSER Header Format for the iSER Hello Message .............60
      9.4. iSER Header Format for the iSER HelloReply Message ........61
      9.5. SCSI Data Transfer Operations .............................62
           9.5.1. SCSI Write Operation ...............................62
           9.5.2. SCSI Read Operation ................................63
           9.5.3. Bidirectional Operation ............................64
   10. iSER Error Handling and Recovery ..............................64
      10.1. Error Handling ...........................................64
           10.1.1. Errors in the Transport Layer .....................64
           10.1.2. Errors in the RCaP Layer ..........................65

Top      ToC       Page 4 
           10.1.3. Errors in the iSER Layer ..........................66
           10.1.4. Errors in the iSCSI Layer .........................67
      10.2. Error Recovery ...........................................69
           10.2.1. PDU Recovery ......................................69
           10.2.2. Connection Recovery ...............................70
   11. Security Considerations .......................................71
   12. References ....................................................71
      12.1. Normative References .....................................71
      12.2. Informative References ...................................72
   Appendix A. iWARP Message Format for iSER .........................73
      A.1. iWARP Message Format for iSER Hello Message ...............73
      A.2. iWARP Message Format for iSER HelloReply Message ..........74
      A.3. iWARP Message Format for SCSI Read Command PDU ............75
      A.4. iWARP Message Format for SCSI Read Data ...................76
      A.5. iWARP Message Format for SCSI Write Command PDU ...........77
      A.6. iWARP Message Format for RDMA Read Request ................78
      A.7. iWARP Message Format for Solicited SCSI Write Data ........79
      A.8. iWARP Message Format for SCSI Response PDU ................80
   Appendix B. Architectural Discussion of iSER over InfiniBand ......81
      B.1. The Host Side of the iSCSI and iSER Connections
           in InfiniBand .............................................81
      B.2. The Storage Side of the iSCSI and iSER Mixed
           Network Environment .......................................82
      B.3. Discovery Processes for an InfiniBand Host ................82
      B.4. IBTA Connection Specifications ............................83
   Acknowledgments ...................................................83

Table of Figures

   Figure 1. Example of iSCSI/iSER Layering in Full Feature Phase ....11
   Figure 2. iSER Header Format ......................................58
   Figure 3. iSER Header Format for iSCSI Control-Type PDU ...........59
   Figure 4. iSER Header Format for iSER Hello Message ...............60
   Figure 5. iSER Header Format for iSER HelloReply Message ..........61
   Figure 6. SendSE Message containing an iSER Hello Message .........72
   Figure 7. SendSE Message containing an iSER HelloReply Message ....74
   Figure 8. SendSE Message containing a SCSI Read Command PDU .......75
   Figure 9. RDMA Write Message containing SCSI Read Data ............76
   Figure 10. SendSE Message containing a SCSI Write Command PDU .....77
   Figure 11. RDMA Read Request Message ..............................78
   Figure 12. RDMA Read Response Message containing SCSI Write Data ..79
   Figure 13. SendInvSE Message containing SCSI Response PDU .........80
   Figure 14. iSCSI and iSER on IB ...................................81
   Figure 15. Storage Controller with TCP, iWARP, and IB Connections .82

Top      ToC       Page 5 
1.  Introduction

1.1.  Motivation

   The iSCSI protocol [RFC3720] is a mapping of the SCSI Architecture
   Model (see [SAM2]) over the TCP protocol.  SCSI commands are carried
   by iSCSI requests, and SCSI responses and status are carried by iSCSI
   responses.  Other iSCSI protocol exchanges and SCSI data are also
   transported in iSCSI Protocol Data Units (PDUs).

   Out-of-order TCP segments in the Traditional iSCSI model have to be
   stored and reassembled before the iSCSI protocol layer within an end
   node can place the data in the iSCSI buffers.  This reassembly is
   required because not every TCP segment is likely to contain an iSCSI
   header to enable its placement, and TCP itself does not have a
   built-in mechanism for signaling Upper Level Protocol (ULP) message
   boundaries to aid placement of out-of-order segments.  This TCP
   reassembly at high network speeds is quite counter-productive for the
   following reasons: wasted memory bandwidth in data copying, the need
   for reassembly memory, wasted CPU cycles in data copying, and the
   general store-and-forward latency from an application perspective.
   TCP reassembly was recognized as a serious issue in [RFC3720], and
   the notion of a "sync and steering layer" was introduced that is
   optional to implement and use.  One specific sync and steering
   mechanism, called "markers", was defined in [RFC3720], which provides
   an application-level way of framing iSCSI Protocol Data Units (PDUs)
   within the TCP data stream even when the TCP segments are not yet
   reassembled to be in-order.

   With these defined techniques in [RFC3720], a Network Interface
   Controller customized for iSCSI (SNIC) could offload the TCP/IP
   processing and support direct data placement, but most iSCSI
   implementations do not support iSCSI "markers", making SNIC marker-
   based direct data placement unusable in practice.

   The iWARP protocol stack provides direct data placement functionality
   that is usable in practice.  In addition, there is interest in using
   iSCSI with other Remote Direct Memory Access (RDMA) protocol stacks
   that support direct data placement, such as the one provided by
   InfiniBand.  The generic term RDMA-Capable Protocol (RCaP) is used to
   refer to the RDMA functionality provided by such protocol stacks.

   With the availability of RDMA-Capable Controllers within a host
   system, which does not have SNICs, it is appropriate for iSCSI to be
   able to exploit the direct data placement function of the RDMA-
   Capable Controller like other applications.

Top      ToC       Page 6 
   iSCSI Extensions for RDMA (iSER) is designed precisely to take
   advantage of generic RDMA technologies -- iSER's goal is to permit
   iSCSI to employ direct data placement and RDMA capabilities using a
   generic RDMA-Capable Controller.  In summary, the iSCSI/iSER protocol
   stack is designed to enable scaling to high speeds by relying on a
   generic data placement process and RDMA technologies and products,
   which enable direct data placement of both in-order and out-of-order
   data.

   This document describes iSER as a protocol extension to iSCSI, both
   for convenience of description and because it is true in a very
   strict protocol sense.  However, note that iSER is in reality
   extending the connectivity of the iSCSI protocol defined in
   [RFC3720], and the name iSER reflects this reality.

   When the iSCSI protocol as defined in [RFC3720] (i.e., without the
   iSER enhancements) is intended in the rest of the document, the term
   "Traditional iSCSI" is used to make the intention clear.

1.2.  Architectural Goals

   This section summarizes the architectural goals that guided the
   design of iSER.

   1. Provide an RDMA data transfer model for iSCSI that enables direct
      in-order or out-of-order data placement of SCSI data into pre-
      allocated SCSI buffers while maintaining in-order data delivery.

   2. Not require any major changes to the SCSI Architecture Model
      [SAM2] and SCSI command set standards.

   3. Utilize existing iSCSI infrastructure (sometimes referred to as
      "iSCSI ecosystem") including but not limited to MIB,
      bootstrapping, negotiation, naming and discovery, and security.

   4. Require a session to operate in the Traditional iSCSI data
      transfer mode if iSER is not supported by either the initiator or
      the target (i.e., not require iSCSI Full Feature Phase
      interoperability between an end node operating in Traditional
      iSCSI mode, and an end node operating in iSER-assisted mode).

   5. Allow initiator and target implementations to utilize generic
      RDMA-Capable Controllers such as RDMA-enabled Network Interface
      Controllers (RNICs), or to implement iSCSI and iSER in software
      (not require iSCSI- or iSER-specific assists in the RCaP
      implementation or RDMA-Capable Controller).

Top      ToC       Page 7 
   6. Require full and only generic RCaP functionality at both the
      initiator and the target.

   7. Implement a lightweight Datamover protocol for iSCSI with minimal
      state maintenance.

1.3.  Protocol Overview

   Consistent with the architectural goals stated in Section 2.2, the
   iSER protocol does not require changes in the iSCSI ecosystem or any
   related SCSI specifications.  The iSER protocol defines the mapping
   of iSCSI PDUs to RCaP Messages in such a way that it is entirely
   feasible to realize iSCSI/iSER implementations that are based on
   generic RDMA-Capable Controllers.  The iSER protocol layer requires
   minimal state maintenance to assist an iSCSI Full Feature Phase
   connection, besides being oblivious to the notion of an iSCSI
   session.  The crucial protocol aspects of iSER may be summarized
   thus:

   1. iSER-assisted mode is negotiated during the iSCSI login for each
      session, and an entire iSCSI session can only operate in one mode
      (i.e., a connection in a session cannot operate in iSER-assisted
      mode if a different connection of the same session is already in
      Full Feature Phase in the Traditional iSCSI mode).

   2. Once in iSER-assisted mode, all iSCSI interactions on that
      connection use RCaP Messages.

   3. A Send Message Type is used for carrying an iSCSI control-type PDU
      preceded by an iSER header.  See Section 7.2 for more details on
      iSCSI control-type PDUs.

   4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages are
      used for carrying control and all data information associated with
      the iSCSI data-type PDUs.  See Section 7.1 for more details on
      iSCSI data-type PDUs.

   5. Target drives all data transfer (with the exception of iSCSI
      unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA
      Read Requests and RDMA Writes, respectively.

   6. RCaP is responsible for ensuring data integrity.  (For example,
      iWARP includes a CRC-enhanced framing layer called Marker PDU
      Aligned Framing for TCP (MPA) on top of TCP; and for InfiniBand,
      the CRCs are included in the Reliable Connection mode).  For this
      reason, iSCSI header and data digests are negotiated to "None" for
      iSCSI/iSER sessions.

Top      ToC       Page 8 
   7. The iSCSI error recovery hierarchy defined in [RFC3720] is fully
      supported by iSER.  (However, see Section 7.3.11 on the handling
      of SNACK Request PDUs.)

   8. iSER requires no changes to iSCSI authentication, security, and
      text mode negotiation mechanisms.

   Note that Traditional iSCSI implementations may have to be adapted to
   employ iSER.  It is expected that the adaptation when required is
   likely to be centered around the upper layer interface requirements
   of iSER (Section 3).

1.4.  RDMA Services and iSER

   iSER is designed to work with software and/or hardware protocol
   stacks providing the protocol services defined in RCaP documents such
   as [RDMAP], [IB], etc.  The following subsections describe the key
   protocol elements of RCaP services that iSER relies on.

1.4.1.  STag

   A Steering Tag (STag) is the identifier of an I/O Buffer unique to an
   RDMA-Capable Controller that the iSER layer Advertises to the remote
   iSCSI/iSER node in order to complete a SCSI I/O.

   In iSER, Advertisement is the act of informing the target by the
   initiator that an I/O Buffer is available at the initiator for RDMA
   Read or RDMA Write access by the target.  The initiator Advertises
   the I/O Buffer by including the STag in the header of an iSER Message
   containing the SCSI Command PDU to the target.  The base Tagged
   Offset is not explicitly specified, but the target must always assume
   it as zero.  The buffer length is as specified in the SCSI Command
   PDU.

   The iSER layer at the initiator Advertises the STag for the I/O
   Buffer of each SCSI I/O to the iSER layer at the target in the iSER
   header of the Send with Solicited Event (SendSE) Message containing
   the SCSI Command PDU, unless the I/O can be completely satisfied by
   unsolicited data alone.

   The iSER layer at the target provides the STag for the I/O Buffer
   that is the Data Sink of an RDMA Read Operation (Section 2.4.4) to
   the RCaP layer on the initiator node -- i.e., this is completely
   transparent to the iSER layer at the initiator.

   The iSER protocol is defined so that the Advertised STag is
   automatically invalidated upon a normal completion of the associated
   task.  This automatic invalidation is realized via the Send with

Top      ToC       Page 9 
   Solicited Event and Invalidate (SendInvSE) Message carrying the SCSI
   Response PDU.  There are two exceptions to this automatic
   invalidation -- bidirectional commands, and abnormal completion of a
   command.  The iSER layer at the initiator is required to explicitly
   invalidate the STag in these cases, in addition to sanity checking
   the automatic invalidation even when that does happen.

1.4.2.  Send

   Send is the RDMA Operation that is not addressed to an Advertised
   buffer by the sending side, and thus uses Untagged buffers on the
   receiving side.

   The iSER layer at the initiator uses the Send Operation to transmit
   any iSCSI control-type PDU to the target.  As an example, the
   initiator uses Send Operations to transfer iSER Messages containing
   SCSI Command PDUs to the iSER layer at the target.

   An iSER layer at the target uses the Send Operation to transmit any
   iSCSI control-type PDU to the initiator.  As an example, the target
   uses Send Operations to transfer iSER Messages containing SCSI
   Response PDUs to the iSER layer at the initiator.

1.4.3.  RDMA Write

   RDMA Write is the RDMA Operation that is used to place data into an
   Advertised buffer on the receiving side.  The sending side addresses
   the Message using an STag and a Tagged Offset that are valid on the
   Data Sink.

   The iSER layer at the target uses the RDMA Write Operation to
   transfer the contents of a local I/O Buffer to an Advertised I/O
   Buffer at the initiator.  The iSER layer at the target uses the RDMA
   Write to transfer whole or part of the data required to complete a
   SCSI read command.

   The iSER layer at the initiator does not employ RDMA Writes.

1.4.4.  RDMA Read

   RDMA Read is the RDMA Operation that is used to retrieve data from an
   Advertised buffer on a remote node.  The sending side of the RDMA
   Read Request addresses the Message using an STag and a Tagged Offset
   that are valid on the Data Source in addition to providing a valid
   local STag and Tagged Offset that identify the Data Sink.

   The iSER layer at the target uses the RDMA Read Operation to transfer
   the contents of an Advertised I/O Buffer at the initiator to a local

Top      ToC       Page 10 
   I/O Buffer at the target.  The iSER layer at the target uses the RDMA
   Read to fetch whole or part of the data required to complete a SCSI
   write command.

   The iSER layer at the initiator does not employ RDMA Reads.

1.5.  SCSI Read Overview

   The iSER layer at the initiator receives the SCSI Command PDU from
   the iSCSI layer.  The iSER layer at the initiator generates an STag
   for the I/O Buffer of the SCSI Read and Advertises the buffer by
   including the STag as part of the iSER header for the PDU.  The iSER
   Message is transferred to the target using a SendSE Message.

   The iSER layer at the target uses one or more RDMA Writes to transfer
   the data required to complete the SCSI Read.

   The iSER layer at the target uses a SendInvSE Message to transfer the
   SCSI Response PDU back to the iSER layer at the initiator.  The iSER
   layer at the initiator notifies the iSCSI layer of the availability
   of the SCSI Response PDU.

1.6.  SCSI Write Overview

   The iSER layer at the initiator receives the SCSI Command PDU from
   the iSCSI layer.  If solicited data transfer is involved, the iSER
   layer at the initiator generates an STag for the I/O Buffer of the
   SCSI Write and Advertises the buffer by including the STag as part of
   the iSER header for the PDU.  The iSER Message is transferred to the
   target using a SendSE Message.

   The iSER layer at the initiator may optionally send one or more non-
   immediate unsolicited data PDUs to the target using Send Message
   Types.

   If solicited data transfer is involved, the iSER layer at the target
   uses one or more RDMA Reads to transfer the data required to complete
   the SCSI Write.

   The iSER layer at the target uses a SendInvSE Message to transfer the
   SCSI Response PDU back to the iSER layer at the initiator.  The iSER
   layer at the initiator notifies the iSCSI layer of the availability
   of the SCSI Response PDU.

1.7.  iSCSI/iSER Layering

   iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer
   and the RCaP layer.  Note that the RCaP layer may be composed of one

Top      ToC       Page 11 
   or more distinct protocol layers depending on the specifics of the
   RCaP.  Figure 1 shows an example of the relationship between SCSI,
   iSCSI, iSER, and the different RCaP layers.  For TCP, the RCaP is
   iWARP.  For InfiniBand, the RCaP is the Reliable Connected Transport
   Service.  Note that the iSCSI layer as described here supports the
   RDMA Extensions as used in iSER.

                 +-------------------------------------+
                 |              SCSI                   |
                 +-------------------------------------+
                 |              iSCSI                  |
      DI ------> +-------------------------------------+
                 |              iSER                   |
                 +---------+--------------+------------+
                 |  RDMAP  |              |            |
                 +---------+  InfiniBand  |            |
                 |   DDP   |   Reliable   |   Other    |
                 +---------+  Connected   |   RDMA-    |
                 |   MPA   |  Transport   |  Capable   |
                 +---------+   Service    |  Protocol  |
                 |   TCP   |              |            |
                 +---------+--------------+------------+
                 |         |  InfiniBand  |   Other    |
                 |    IP   |   Network    |  Network   |
                 |         |    Layer     |   Layer    |
                 +---------+--------------+------------+

   Figure 1.  Example of iSCSI/iSER Layering in Full Feature Phase

2.  Definitions and Acronyms

2.1.  Definitions

   Advertisement (Advertised, Advertise, Advertisements, Advertises) -
      The act of informing a remote iSER layer that a local node's
      buffer is available to it.  A Node makes a buffer available for
      incoming RDMA Read Request Message or incoming RDMA Write Message
      access by informing the remote iSER layer of the Tagged Buffer
      identifiers (STag, TO, and buffer length).  Note that this
      Advertisement of Tagged Buffer information is the responsibility
      of the iSER layer on either end and is not defined by the RDMA-
      Capable Protocol.  A typical method would be for the iSER layer to
      embed the Tagged Buffer's STag, TO, and buffer length in a Send
      Message destined for the remote iSER layer.

   Completion (Completed, Complete, Completes) - Completion is defined
      as the process by the RDMA-Capable Protocol layer to inform the

Top      ToC       Page 12 
      iSER layer, that a particular RDMA Operation has performed all
      functions specified for the RDMA Operation.

   Connection - A connection is a logical circuit between the initiator
      and the target, e.g., a TCP connection.  Communication between the
      initiator and the target occurs over one or more connections.  The
      connections carry control messages, SCSI commands, parameters, and
      data within iSCSI Protocol Data Units (iSCSI PDUs).

   Connection Handle - An information element that identifies the
      particular iSCSI connection and is unique for a given iSCSI-iSER
      pair.  Every invocation of an Operational Primitive is qualified
      with the Connection Handle.

   Data Sink - The peer receiving a data payload.  Note that the Data
      Sink can be required to both send and receive RCaP Messages to
      transfer a data payload.

   Data Source - The peer sending a data payload.  Note that the Data
      Source can be required to both send and receive RCaP Messages to
      transfer a data payload.

   Datamover Interface (DI) - The interface between the iSCSI layer and
      the Datamover layer as described in [DA].

   Datamover Layer - A layer that is directly below the iSCSI layer and
      above the underlying transport layers.  This layer exposes and
      uses a set of transport independent Operational Primitives for the
      communication between the iSCSI layer and itself.  The Datamover
      layer, operating in conjunction with the transport layers, moves
      the control and data information on the iSCSI connection.  In this
      specification, the iSER layer is the Datamover layer.

   Datamover Protocol - A Datamover protocol is the wire-protocol that
      is defined to realize the Datamover layer functionality.  In this
      specification, the iSER protocol is the Datamover protocol.

   Event - An indication provided by the RDMA-Capable Protocol layer to
      the iSER layer to indicate a Completion or other condition
      requiring immediate attention.

   Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming
      outstanding RDMA Read Requests that the RDMA-Capable Controller
      can handle on a particular RCaP Stream at the Data Source.  For
      some RDMA-Capable Protocol layers, the term "IRD" may be known by
      a different name.  For example, for InfiniBand, the equivalent for
      IRD is the Responder Resources.

Top      ToC       Page 13 
   Invalidate STag - A mechanism used to prevent the Remote Peer from
      reusing a previous explicitly Advertised STag, until the iSER
      layer at the local node makes it available through a subsequent
      explicit Advertisement.

   I/O Buffer - A buffer that is used in a SCSI Read or Write operation
      so SCSI data may be sent from or received into that buffer.

   iSCSI - The iSCSI protocol as defined in [RFC3720] is a mapping of
      the SCSI Architecture Model of SAM-2 over TCP.

   iSCSI control-type PDU - Any iSCSI PDU that is not an iSCSI data-
      type PDU and also not a SCSI Data-out PDU carrying solicited data
      is defined as an iSCSI control-type PDU.  Specifically, it is to
      be noted that SCSI Data-out PDUs for unsolicited data are defined
      as iSCSI control-type PDUs.

   iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI
      PDU that causes data transfer, transparent to the remote iSCSI
      layer, to take place between the peer iSCSI nodes on a Full
      Feature Phase iSCSI connection.  An iSCSI data-type PDU, when
      requested for transmission by the sender iSCSI layer, results in
      the associated data transfer without the participation of the
      remote iSCSI layer, i.e. the PDU itself is not delivered as-is to
      the remote iSCSI layer.  The following iSCSI PDUs constitute the
      set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU.

   iSCSI Layer - A layer in the protocol stack implementation within an
      end node that implements the iSCSI protocol and interfaces with
      the iSER layer via the Datamover Interface.

   iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI layer at the
      initiator and the iSCSI layer at the target divide their
      communications into messages.  The term "iSCSI protocol data unit"
      (iSCSI PDU) is used for these messages.

   iSCSI/iSER Connection - An iSER-assisted iSCSI connection.

   iSCSI/iSER Session - An iSER-assisted iSCSI session.

   iSCSI-iSER Pair - The iSCSI layer and the underlying iSER layer.

   iSER - iSCSI Extensions for RDMA, the protocol defined in this
      document.

   iSER-assisted - A term generally used to describe the operation of
      iSCSI when the iSER functionality is also enabled below the iSCSI
      layer for the specific iSCSI/iSER connection in question.

Top      ToC       Page 14 
   iSER-IRD - This variable represents the maximum number of incoming
      outstanding RDMA Read Requests that the iSER layer at the
      initiator declares on a particular RCaP Stream.

   iSER-ORD - This variable represents the maximum number of outstanding
      RDMA Read Requests that the iSER layer can initiate on a
      particular RCaP Stream.  This variable is maintained only by the
      iSER layer at the target.

   iSER Layer - The layer that implements the iSCSI Extensions for RDMA
      (iSER) protocol.

   iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and
      [MPA] when layered above [TCP].  [RDMAP] and [DDP] may be layered
      above SCTP or other transport protocols.

   Local Mapping - A task state record maintained by the iSER layer that
      associates the Initiator Task Tag to the local STag(s).  The
      specifics of the record structure are implementation dependent.

   Local Peer - The implementation of the RDMA-Capable Protocol on the
      local end of the connection.  Used to refer to the local entity
      when describing protocol exchanges or other interactions between
      two Nodes.

   Node - A computing device attached to one or more links of a network.
      A Node in this context does not refer to a specific application or
      protocol instantiation running on the computer.  A Node may
      consist of one or more RDMA-Capable Controllers installed in a
      host computer.

   Operational Primitive - An Operational Primitive is an abstract
      functional interface procedure that requests that another layer
      perform a specific action on the requestor's behalf or notifies
      the other layer of some event.  The Datamover Interface between an
      iSCSI layer and a Datamover layer within an iSCSI end node uses a
      set of Operational Primitives to define the functional interface
      between the two layers.  Note that not every invocation of an
      Operational Primitive may elicit a response from the requested
      layer.  A full discussion of the Operational Primitive types and
      request-response semantics available to iSCSI and iSER can be
      found in [DA].

   Outbound RDMA Read Queue Depth (ORD) - The maximum number of
      outstanding RDMA Read Requests that the RDMA-Capable Controller
      can initiate on a particular RCaP Stream at the Data Sink.  For

Top      ToC       Page 15 
      some RDMA-Capable Protocol layer, the term "ORD" may be known by a
      different name.  For example, for InfiniBand, the equivalent for
      ORD is the Initiator Depth.

   Phase-Collapse - Refers to the optimization in iSCSI where the SCSI
      status is transferred along with the final SCSI Data-in PDU from a
      target.  See Section 3.2 in [RFC3720].

   RCaP Message - One or more packets of the network layer comprising a
      single RDMA Operation or a part of an RDMA Read Operation of the
      RDMA-Capable Protocol.  For iWARP, an RCaP Message is known as an
      RDMAP Message.

   RCaP Stream - A single bidirectional association between the peer
      RDMA-Capable Protocol layers on two Nodes over a single
      transport-level stream.  For iWARP, an RCaP Stream is known as an
      RDMAP Stream, and the association is created when the connection
      transitions to iSER-assisted mode following a successful Login
      Phase during which iSER support is negotiated.

   RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that
      provides a reliable RDMA transport functionality, e.g., iWARP,
      InfiniBand, etc.

   RDMA-Capable Controller - A network I/O adapter or embedded
      controller with RDMA functionality.  For example, for iWARP, this
      could be an RNIC, and for InfiniBand, this could be a HCA (Host
      Channel Adapter) or TCA (Target Channel Adapter).

   RDMA-enabled Network Interface Controller (RNIC) - A network I/O
      adapter or embedded controller with iWARP functionality.

   RDMA Operation - A sequence of RCaP Messages, including control
      Messages, to transfer data from a Data Source to a Data Sink.  The
      following RDMA Operations are defined - RDMA Write Operation, RDMA
      Read Operation, Send Operation, Send with Invalidate Operation,
      Send with Solicited Event Operation, Send with Solicited Event and
      Invalidate Operation, and Terminate Operation.

   RDMA Protocol (RDMAP) - A wire protocol that supports RDMA Operations
      to transfer ULP data between a Local Peer and the Remote Peer as
      described in [RDMAP].

   RDMA Read Operation - An RDMA Operation used by the Data Sink to
      transfer the contents of a Data Source buffer from the Remote Peer
      to a Data Sink buffer at the Local Peer.  An RDMA Read operation
      consists of a single RDMA Read Request Message and a single RDMA
      Read Response Message.

Top      ToC       Page 16 
   RDMA Read Request - An RCaP Message used by the Data Sink to request
      that the Data Source transfer the contents of a buffer.  The RDMA
      Read Request Message describes both the Data Source and the Data
      Sink buffers.

   RDMA Read Response - An RCaP Message used by the Data Source to
      transfer the contents of a buffer to the Data Sink, in response to
      an RDMA Read Request.  The RDMA Read Response Message only
      describes the Data Sink buffer.

   RDMA Write Operation - An RDMA Operation used by the Data Source to
      transfer the contents of a Data Source buffer from the Local Peer
      to a Data Sink buffer at the Remote Peer.  The RDMA Write Message
      only describes the Data Sink buffer.

   Remote Direct Memory Access (RDMA) - A method of accessing memory on
      a remote system in which the local system specifies the remote
      location of the data to be transferred.  Employing an RDMA-
      Capable Controller in the remote system allows the access to take
      place without interrupting the processing of the CPU(s) on the
      system.

   Remote Mapping - A task state record maintained by the iSER layer
      that associates the Initiator Task Tag to the Advertised STag(s).
      The specifics of the record structure are implementation
      dependent.

   Remote Peer - The implementation of the RDMA-Capable Protocol on the
      opposite end of the connection.  Used to refer to the remote
      entity when describing protocol exchanges or other interactions
      between two Nodes.

   SCSI Layer - This layer builds/receives SCSI CDBs (Command Descriptor
      Blocks) and sends/receives them with the remaining command execute
      [SAM2] parameters to/from the iSCSI layer.

   Send - An RDMA Operation that transfers the contents of a Buffer from
      the Local Peer to a Buffer at the Remote Peer.

   Send Message Type - A Send Message, Send with Invalidate Message,
      Send with Solicited Event Message, or Send with Solicited Event
      and Invalidate Message.

   SendInvSE Message - A Send with Solicited Event and Invalidate
      Message.

   SendSE Message - A Send with Solicited Event Message.

Top      ToC       Page 17 
   Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for an
      R2T PDU.  The semantics for both types of sequence numbers are as
      defined in [RFC3720].

   Session, iSCSI Session - The group of connections that link an
      initiator SCSI port with a target SCSI port form an iSCSI session
      (equivalent to a SCSI I-T nexus).  Connections can be added to and
      removed from a session even while the I-T nexus is intact.  Across
      all connections within a session, an initiator sees one and the
      same target.

   Solicited Event (SE) - A facility by which an RDMA Operation sender
      may cause an Event to be generated at the recipient, if the
      recipient is configured to generate such an Event, when a Send
      with Solicited Event or Send with Solicited Event and Invalidate
      Message is received.

   Steering Tag (STag) - An identifier of a Tagged Buffer on a Node
      (Local or Remote) as defined in [RDMAP] and [DDP].  For other
      RDMA-Capable Protocols, the Steering Tag may be known by different
      names but will be herein referred to as STags.  For example, for
      InfiniBand, a Remote STag is known as an R-Key, and a local STag
      is known as an L-Key, and both will be considered STags.

   Tagged Buffer - A buffer that is explicitly Advertised to the iSER
      layer at the remote node through the exchange of an STag, Tagged
      Offset, and length.

   Tagged Offset (TO) - The offset within a Tagged Buffer.

   Traditional iSCSI - Refers to the iSCSI protocol as defined in
      [RFC3720] (i.e. without the iSER enhancements).

   Untagged Buffer - A buffer that is not explicitly Advertised to the
      iSER layer at the remode node.

2.2.  Acronyms

      Acronym        Definition
      --------------------------------------------------------------

      AHS            Additional Header Segment

      BHS            Basic Header Segment

      CO             Connection Only

      CRC            Cyclic Redundancy Check

Top      ToC       Page 18 
      DDP            Direct Data Placement Protocol

      DI             Datamover Interface

      HCA            Host Channel Adapter

      IANA           Internet Assigned Numbers Authority

      IB             InfiniBand

      IETF           Internet Engineering Task Force

      I/O            Input - Output

      IO             Initialize Only

      IP             Internet Protocol

      IPoIB          IP over InfiniBand

      IPsec          Internet Protocol Security

      iSER           iSCSI Extensions for RDMA

      ITT            Initiator Task Tag

      LO             Leading Only

      MPA            Marker PDU Aligned Framing for TCP

      NOP            No Operation

      NSG            Next Stage (during the iSCSI Login Phase)

      OS             Operating System

      PDU            Protocol Data Unit

      R2T            Ready To Transfer

      R2TSN          Ready To Transfer Sequence Number

      RDMA           Remote Direct Memory Access

      RDMAP          Remote Direct Memory Access Protocol

      RFC            Request For Comments

Top      ToC       Page 19 
      RNIC           RDMA-enabled Network Interface Controller

      SAM2           SCSI Architecture Model - 2

      SCSI           Small Computer Systems Interface

      SNACK          Selective Negative Acknowledgment - also
                     Sequence Number Acknowledgement for data

      STag           Steering Tag

      SW             Session Wide

      TCA            Target Channel Adapter

      TCP            Transmission Control Protocol

      TMF            Task Management Function

      TTT            Target Transfer Tag

      TO             Tagged Offset

      ULP            Upper Level Protocol

2.3.  Conventions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].



(page 19 continued on part 2)

Next RFC Part