tech-invite   World Map     

IETF     RFCs     Groups     SIP     ABNFs    |    3GPP     Specs     Glossaries     Architecture     IMS     UICC    |    search     info

RFC 7143

 
 
 

Internet Small Computer System Interface (iSCSI) Protocol (Consolidated)

Part 5 of 10, p. 92 to 131
Prev RFC Part       Next RFC Part

 


prevText      Top      Up      ToC       Page 92 
7.  iSCSI Error Handling and Recovery

7.1.  Overview

7.1.1.  Background

   The following two considerations prompted the design of much of the
   error recovery functionality in iSCSI:

      - An iSCSI PDU may fail the digest check and be dropped, despite
        being received by the TCP layer.  The iSCSI layer must
        optionally be allowed to recover such dropped PDUs.

      - A TCP connection may fail at any time during the data transfer.
        All the active tasks must optionally be allowed to be continued
        on a different TCP connection within the same session.

   Implementations have considerable flexibility in deciding what degree
   of error recovery to support, when to use it, and by which mechanisms
   to achieve the required behavior.  Only the externally visible
   actions of the error recovery mechanisms must be standardized to
   ensure interoperability.

   This section describes a general model for recovery in support of
   interoperability.  See Appendix D for further details on how the
   described model may be implemented.  Compliant implementations do not
   have to match the implementation details of this model as presented,
   but the external behavior of such implementations must correspond to
   the externally observable characteristics of the presented model.

7.1.2.  Goals

   The major design goals of the iSCSI error recovery scheme are as
   follows:

      - Allow iSCSI implementations to meet different requirements by
        defining a collection of error recovery mechanisms from which
        implementations may choose.

      - Ensure interoperability between any two implementations
        supporting different sets of error recovery capabilities.

      - Define the error recovery mechanisms to ensure command ordering
        even in the face of errors, for initiators that demand ordering.

      - Do not make additions in the fast path, but allow moderate
        complexity in the error recovery path.

Top      Up      ToC       Page 93 
      - Prevent both the initiator and target from attempting to recover
        the same set of PDUs at the same time.  For example, there must
        be a clear "error recovery functionality distribution" between
        the initiator and target.

7.1.3.  Protocol Features and State Expectations

   The initiator mechanisms defined in connection with error recovery
   are:

      a) NOP-Out to probe sequence numbers of the target (Section 11.18)

      b) Command retry (Section 7.2.1)

      c) Recovery R2T support (Section 7.8)

      d) Requesting retransmission of status/data/R2T using the SNACK
         facility (Section 11.16)

      e) Acknowledging the receipt of the data (Section 11.16)

      f) Reassigning the connection allegiance of a task to a different
         TCP connection (Section 7.2.2)

      g) Terminating the entire iSCSI session to start afresh
         (Section 7.1.4.4)

   The target mechanisms defined in connection with error recovery are:

      a) NOP-In to probe sequence numbers of the initiator
         (Section 11.19)

      b) Requesting retransmission of data using the recovery R2T
         feature (Section 7.8)

      c) SNACK support (Section 11.16)

      d) Requesting that parts of read data be acknowledged
         (Section 11.7.2)

      e) Allegiance reassignment support (Section 7.2.2)

      f) Terminating the entire iSCSI session to force the initiator to
         start over (Section 7.1.4.4)

   For any outstanding SCSI command, it is assumed that iSCSI, in
   conjunction with SCSI at the initiator, is able to keep enough
   information to be able to rebuild the command PDU and that outgoing

Top      Up      ToC       Page 94 
   data is available (in host memory) for retransmission while the
   command is outstanding.  It is also assumed that at the target,
   incoming data (read data) MAY be kept for recovery, or it can be
   reread from a device server.

   It is further assumed that a target will keep the "status and sense"
   for a command it has executed if it supports status retransmission.

   A target that agrees to support data retransmission is expected to be
   prepared to retransmit the outgoing data (i.e., Data-In) on request
   until either the status for the completed command is acknowledged or
   the data in question has been separately acknowledged.

7.1.4.  Recovery Classes

   iSCSI enables the following classes of recovery (in the order of
   increasing scope of affected iSCSI tasks):

      - within a command (i.e., without requiring command restart)

      - within a connection (i.e., without requiring the connection to
        be rebuilt, but perhaps requiring command restart)

      - connection recovery (i.e., perhaps requiring connections to be
        rebuilt and commands to be reissued)

      - session recovery

   The recovery scenarios detailed in the rest of this section are
   representative rather than exclusive.  In every case, they detail the
   lowest recovery class that MAY be attempted.  The implementer is left
   to decide under which circumstances to escalate to the next recovery
   class and/or what recovery classes to implement.  Both the iSCSI
   target and initiator MAY escalate the error handling to an error
   recovery class, which impacts a larger number of iSCSI tasks in any
   of the cases identified in the following discussion.

   In all classes, the implementer has the choice of deferring errors to
   the SCSI initiator (with an appropriate response code), in which case
   the task, if any, has to be removed from the target and all the side
   effects, such as ACA, must be considered.

   The use of within-connection and within-command recovery classes MUST
   NOT be attempted before the connection is in the Full Feature Phase.

Top      Up      ToC       Page 95 
   In the detailed description of the recovery classes, the mandating
   terms (MUST, SHOULD, MAY, etc.) indicate normative actions to be
   executed if the recovery class is supported (see Section 7.1.5 for
   the related negotiation semantics) and used.

7.1.4.1.  Recovery Within-command

   At the target, the following cases lend themselves to within-command
   recovery:

      Lost data PDU - realized through one of the following:

      a) Data digest error - dealt with as specified in Section 7.8,
         using the option of a recovery R2T

      b) Sequence reception timeout (no data or partial-data-and-no-
         F-bit) - considered an implicit sequence error and dealt with
         as specified in Section 7.9, using the option of a recovery R2T

      c) Header digest error, which manifests as a sequence reception
         timeout or a sequence error - dealt with as specified in
         Section 7.9, using the option of a recovery R2T

   At the initiator, the following cases lend themselves to within-
   command recovery:

      Lost data PDU or lost R2T - realized through one of the following:

      a) Data digest error - dealt with as specified in Section 7.8,
         using the option of a SNACK

      b) Sequence reception timeout (no status) or response reception
         timeout - dealt with as specified in Section 7.9, using the
         option of a SNACK

      c) Header digest error, which manifests as a sequence reception
         timeout or a sequence error - dealt with as specified in
         Section 7.9, using the option of a SNACK

   To avoid a race with the target, which may already have a recovery
   R2T or a termination response on its way, an initiator SHOULD NOT
   originate a SNACK for an R2T based on its internal timeouts (if any).
   Recovery in this case is better left to the target.

   The timeout values used by the initiator and target are outside the
   scope of this document.  A sequence reception timeout is generally a
   large enough value to allow the data sequence transfer to be
   complete.

Top      Up      ToC       Page 96 
7.1.4.2.  Recovery Within-connection

   At the initiator, the following cases lend themselves to within-
   connection recovery:

      a) Requests not acknowledged for a long time.  Requests are
         acknowledged explicitly through the ExpCmdSN or implicitly by
         receiving data and/or status.  The initiator MAY retry
         non-acknowledged commands as specified in Section 7.2.

      b) Lost iSCSI numbered response.  It is recognized by either
         identifying a data digest error on a Response PDU or a Data-In
         PDU carrying the status, or receiving a Response PDU with a
         higher StatSN than expected.  In the first case, digest error
         handling is done as specified in Section 7.8, using the option
         of a SNACK.  In the second case, sequence error handling is
         done as specified in Section 7.9, using the option of a SNACK.

   At the target, the following cases lend themselves to within-
   connection recovery:

      - Status/Response not acknowledged for a long time.  The target
        MAY issue a NOP-In (with a valid Target Transfer Tag or
        otherwise) that carries the next status sequence number it is
        going to use in the StatSN field.  This helps the initiator
        detect any missing StatSN(s) and issue a SNACK for the status.

   The timeout values used by the initiator and the target are outside
   the scope of this document.

7.1.4.3.  Connection Recovery

   At an iSCSI initiator, the following cases lend themselves to
   connection recovery:

      a) TCP connection failure: The initiator MUST close the
         connection.  It then MUST either implicitly or explicitly log
         out the failed connection with the reason code "remove the
         connection for recovery" and reassign connection allegiance for
         all commands still in progress associated with the failed
         connection on one or more connections (some or all of which MAY
         be newly established connections) using the "TASK REASSIGN"
         task management function (see Section 11.5.1).  For an
         initiator, a command is in progress as long as it has not
         received a response or a Data-In PDU including status.

Top      Up      ToC       Page 97 
         Note: The logout function is mandatory.  However, a new
         connection establishment is only mandatory if the failed
         connection was the last or only connection in the session.

      b) Receiving an Asynchronous Message that indicates that one or
         all connections in a session have been dropped.  The initiator
         MUST handle it as a TCP connection failure for the
         connection(s) referred to in the message.

   At an iSCSI target, the following cases lend themselves to connection
   recovery:

      - TCP connection failure: The target MUST close the connection
        and, if more than one connection is available, the target SHOULD
        send an Asynchronous Message that indicates that it has dropped
        the connection.  Then, the target will wait for the initiator to
        continue recovery.

7.1.4.4.  Session Recovery

   Session recovery should be performed when all other recovery attempts
   have failed.  Very simple initiators and targets MAY perform session
   recovery on all iSCSI errors and rely on recovery on the SCSI layer
   and above.

   Session recovery implies the closing of all TCP connections,
   internally aborting all executing and queued tasks for the given
   initiator at the target, terminating all outstanding SCSI commands
   with an appropriate SCSI service response at the initiator, and
   restarting a session on a new set of connection(s) (TCP connection
   establishment and login on all new connections).

   For possible clearing effects of session recovery on SCSI and iSCSI
   objects, refer to Appendix E.

7.1.5.  Error Recovery Hierarchy

   The error recovery classes described so far are organized into a
   hierarchy for ease in understanding and to limit the complexity of
   the implementation.  With a few well-defined recovery levels,
   interoperability is easier to achieve.  The attributes of this
   hierarchy are as follows:

      a) Each level is a superset of the capabilities of the previous
         level.  For example, Level 1 support implies supporting all
         capabilities of Level 0 and more.

Top      Up      ToC       Page 98 
      b) As a corollary, supporting a higher error recovery level means
         increased sophistication and possibly an increase in resource
         requirements.

      c) Supporting error recovery level "n" is advertised and
         negotiated by each iSCSI entity by exchanging the text key
         "ErrorRecoveryLevel=n".  The lower of the two exchanged values
         is the operational ErrorRecoveryLevel for the session.

   The following diagram represents the error recovery hierarchy.

                            +
                           / \
                          / 2 \      <-- Connection recovery
                         +-----+
                        /   1   \    <-- Digest failure recovery
                       +---------+
                      /     0     \  <-- Session failure recovery
                     +-------------+

   The following table lists the error recovery (ER) capabilities
   expected from the implementations that support each error recovery
   level.

    +-------------------+--------------------------------------------+
    |ErrorRecoveryLevel | Associated Error Recovery Capabilities     |
    +-------------------+--------------------------------------------+
    |        0          | Session recovery class                     |
    |                   | (Session Recovery)                         |
    +-------------------+--------------------------------------------+
    |        1          | Digest failure recovery (see Note below)   |
    |                   | plus the capabilities of ER Level 0        |
    +-------------------+--------------------------------------------+
    |        2          | Connection recovery class                  |
    |                   | (Connection Recovery)                      |
    |                   | plus the capabilities of ER Level 1        |
    +-------------------+--------------------------------------------+

   Note: Digest failure recovery is comprised of two recovery classes:
   the Within-connection recovery class (recovery within-connection) and
   the Within-command recovery class (recovery within-command).

   When a defined value of ErrorRecoveryLevel is proposed by an
   originator in a text negotiation, the originator MUST support the
   functionality defined for the proposed value and, additionally,
   functionality corresponding to any defined value numerically less
   than the proposed value.  When a defined value of ErrorRecoveryLevel

Top      Up      ToC       Page 99 
   is returned by a responder in a text negotiation, the responder MUST
   support the functionality corresponding to the ErrorRecoveryLevel it
   is accepting.

   When either party attempts to use error recovery functionality beyond
   what is negotiated, the recovery attempts MAY fail, unless an
   a priori agreement outside the scope of this document exists between
   the two parties to provide such support.

   Implementations MUST support error recovery level "0", while the rest
   are OPTIONAL to implement.  In implementation terms, the above
   striation means that the following incremental sophistication with
   each level is required:

    +-------------------+--------------------------------------------+
    | Level Transition  | Incremental Requirement                    |
    +-------------------+--------------------------------------------+
    |        0->1       | PDU retransmissions on the same connection |
    +-------------------+--------------------------------------------+
    |        1->2       | Retransmission across connections and      |
    |                   | allegiance reassignment                    |
    +-------------------+--------------------------------------------+

7.2.  Retry and Reassign in Recovery

   This section summarizes two important and somewhat related iSCSI
   protocol features used in error recovery.

7.2.1.  Usage of Retry

   By resending the same iSCSI Command PDU ("retry") in the absence of a
   command acknowledgment (by way of an ExpCmdSN update) or a response,
   an initiator attempts to "plug" (what it thinks are) the
   discontinuities in CmdSN ordering on the target end.  Discarded
   command PDUs, due to digest errors, may have created these
   discontinuities.

   Retry MUST NOT be used for reasons other than plugging command
   sequence gaps and, in particular, cannot be used for requesting PDU
   retransmissions from a target.  Any such PDU retransmission requests
   for a currently allegiant command in progress may be made using the
   SNACK mechanism described in Section 11.16, although the usage of
   SNACK is OPTIONAL.

Top      Up      ToC       Page 100 
   If initiators, as part of plugging command sequence gaps as described
   above, inadvertently issue retries for allegiant commands already in
   progress (i.e., targets did not see the discontinuities in CmdSN
   ordering), the duplicate commands are silently ignored by targets as
   specified in Section 4.2.2.1.

   When an iSCSI command is retried, the command PDU MUST carry the
   original Initiator Task Tag and the original operational attributes
   (e.g., flags, function names, LUN, CDB, etc.) as well as the original
   CmdSN.  The command being retried MUST be sent on the same connection
   as the original command, unless the original connection was already
   successfully logged out.

7.2.2.  Allegiance Reassignment

   By issuing a "TASK REASSIGN" task management request
   (Section 11.5.1), the initiator signals its intent to continue an
   already active command (but with no current connection allegiance) as
   part of connection recovery.  This means that a new connection
   allegiance is requested for the command, which seeks to associate it
   to the connection on which the task management request is being
   issued.  Before the allegiance reassignment is attempted for a task,
   an implicit or explicit Logout with the reason code "remove the
   connection for recovery" (see Section 11.14.1) MUST be successfully
   completed for the previous connection to which the task was
   allegiant.

   In reassigning connection allegiance for a command, the target SHOULD
   continue the command from its current state.  For example, when
   reassigning read commands, the target SHOULD take advantage of the
   ExpDataSN field provided by the Task Management Function Request
   (which must be set to 0 if there was no data transfer) and bring the
   read command to completion by sending the remaining data and sending
   (or resending) the status.  The ExpDataSN acknowledges all data sent
   up to, but not including, the Data-In PDU and/or R2T with the DataSN
   (or R2TSN) equal to the ExpDataSN.  However, targets may choose to
   send/receive all unacknowledged data or all of the data on a
   reassignment of connection allegiance if unable to recover or
   maintain accurate state.  Initiators MUST NOT subsequently request
   data retransmission through Data SNACK for PDUs numbered less than
   the ExpDataSN (i.e., prior to the acknowledged sequence number).  For
   all types of commands, a reassignment request implies that the task
   is still considered in progress by the initiator, and the target must
   conclude the task appropriately if the target returns the "Function
   complete" response to the reassignment request.  This might possibly
   involve retransmission of data/R2T/status PDUs as necessary but MUST
   involve the (re)transmission of the status PDU.

Top      Up      ToC       Page 101 
   It is OPTIONAL for targets to support the allegiance reassignment.
   This capability is negotiated via the ErrorRecoveryLevel text key
   during the login time.  When a target does not support allegiance
   reassignment, it MUST respond with a task management response code of
   "Task allegiance reassignment not supported".  If allegiance
   reassignment is supported by the target but the task is still
   allegiant to a different connection, or a successful recovery Logout
   of the previously allegiant connection was not performed, the target
   MUST respond with a task management response code of "Task still
   allegiant".

   If allegiance reassignment is supported by the target, the task
   management response to the reassignment request MUST be issued before
   the reassignment becomes effective.

   If a SCSI command that involves data input is reassigned, any SNACK
   Tag it holds for a final response from the original connection is
   deleted, and the default value of 0 MUST be used instead.

7.3.  Usage of Reject PDU in Recovery

   Targets MUST NOT implicitly terminate an active task by sending a
   Reject PDU for any PDU exchanged during the life of the task.  If the
   target decides to terminate the task, a Response PDU (SCSI, Text,
   Task, etc.) must be returned by the target to conclude the task.  If
   the task had never been active before the Reject (i.e., the Reject is
   on the command PDU), targets should not send any further responses
   because the command itself is being discarded.

   The above rule means that the initiator can eventually expect a
   response on receiving Rejects, if the received Reject is for a PDU
   other than the command PDU itself.  The non-command Rejects only have
   diagnostic value in logging the errors, and they can be used for
   retransmission decisions by the initiators.

   The CmdSN of the rejected command PDU (if it is a non-immediate
   command) MUST NOT be considered received by the target (i.e., a
   command sequence gap must be assumed for the CmdSN), even though the
   CmdSN of the rejected command PDU may be reliably ascertained.  Upon
   receiving the Reject, the initiator MUST plug the CmdSN gap in order
   to continue to use the session.  The gap may be plugged by either
   transmitting a command PDU with the same CmdSN or aborting the task
   (see Section 7.11 for information regarding how an abort may plug a
   CmdSN gap).

Top      Up      ToC       Page 102 
   When a data PDU is rejected and its DataSN can be ascertained, a
   target MUST advance the ExpDataSN for the current data burst if a
   recovery R2T is being generated.  The target MAY advance its
   ExpDataSN if it does not attempt to recover the lost data PDU.

7.4.  Error Recovery Considerations for Discovery Sessions

7.4.1.  ErrorRecoveryLevel for Discovery Sessions

   The negotiation of the key ErrorRecoveryLevel is not required for
   Discovery sessions -- i.e., for sessions that negotiated
   "SessionType=Discovery" -- because the default value of 0 is
   necessary and sufficient for Discovery sessions.  It is, however,
   possible that some legacy iSCSI implementations might attempt to
   negotiate the ErrorRecoveryLevel key on Discovery sessions.  When
   such a negotiation attempt is made by the remote side, a compliant
   iSCSI implementation MUST propose a value of 0 (zero) in response.
   The operational ErrorRecoveryLevel for Discovery sessions thus MUST
   be 0.  This naturally follows from the functionality constraints that
   Section 4.3 imposes on Discovery sessions.

7.4.2.  Reinstatement Semantics for Discovery Sessions

   Discovery sessions are intended to be relatively short-lived.
   Initiators are not expected to establish multiple Discovery sessions
   to the same iSCSI Network Portal.  An initiator may use the same
   iSCSI Initiator Name and ISID when establishing different unique
   sessions with different targets and/or different portal groups.  This
   behavior is discussed in Section 10.1.1 and is, in fact, encouraged
   as conservative reuse of ISIDs.

   The ISID RULE in Section 4.4.3 states that there must not be more
   than one session with a matching 4-tuple: <InitiatorName, ISID,
   TargetName, TargetPortalGroupTag>.  While the spirit of the ISID RULE
   applies to Discovery sessions the same as it does for Normal
   sessions, note that some Discovery sessions differ from the Normal
   sessions in two important aspects:

      a) Because Appendix C allows a Discovery session to be established
         without specifying a TargetName key in the Login Request PDU
         (let us call such a session an "Unnamed" Discovery session),
         there is no target node context to enforce the ISID RULE.

      b) Portal groups are defined only in the context of a target node.
         When the TargetName key is NULL-valued (i.e., not specified),
         the TargetPortalGroupTag thus cannot be ascertained to enforce
         the ISID RULE.

Top      Up      ToC       Page 103 
   The following two sections describe Unnamed Discovery sessions and
   Named Discovery sessions, respectively.

7.4.2.1.  Unnamed Discovery Sessions

   For Unnamed Discovery sessions, neither the TargetName nor the
   TargetPortalGroupTag is available to the targets in order to enforce
   the ISID RULE.  Therefore, the following rule applies.

   UNNAMED ISID RULE: Targets MUST enforce the uniqueness of the
   following 4-tuple for Unnamed Discovery sessions: <InitiatorName,
   ISID, NULL, TargetAddress>.  The following semantics are implied by
   this uniqueness requirement.

   Targets SHOULD allow concurrent establishment of one Discovery
   session with each of its Network Portals by the same initiator port
   with a given iSCSI Node Name and an ISID.  Each of the concurrent
   Discovery sessions, if established by the same initiator port to
   other Network Portals, MUST be treated as independent sessions --
   i.e., one session MUST NOT reinstate the other.

   A new Unnamed Discovery session that has a matching <InitiatorName,
   ISID, NULL, TargetAddress> to an existing Discovery session MUST
   reinstate the existing Unnamed Discovery session.  Note thus that
   only an Unnamed Discovery session may reinstate another Unnamed
   Discovery session.

7.4.2.2.  Named Discovery Sessions

   For Named Discovery sessions, the TargetName key is specified by the
   initiator, and thus the target can unambiguously ascertain the
   TargetPortalGroupTag as well.  Since all the four elements of the
   4-tuple are known, the ISID RULE MUST be enforced by targets with no
   changes from Section 4.4.3 semantics.  A new session with a matching
   <InitiatorName, ISID, TargetName, TargetPortalGroupTag> thus will
   reinstate an existing session.  Note in this case that any new iSCSI
   session (Discovery or Normal) with the matching 4-tuple may reinstate
   an existing Named Discovery iSCSI session.

7.4.3.  Target PDUs during Discovery

   Targets SHOULD NOT send any responses other than a Text Response and
   Logout Response on a Discovery session, once in the Full Feature
   Phase.

   Implementation Note: A target may simply drop the connection in a
   Discovery session when it would have requested a Logout via an Async
   Message on Normal sessions.

Top      Up      ToC       Page 104 
7.5.  Connection Timeout Management

   iSCSI defines two session-global timeout values (in seconds) --
   Time2Wait and Time2Retain -- that are applicable when an iSCSI Full
   Feature Phase connection is taken out of service either intentionally
   or by an exception.  Time2Wait is the initial "respite time" before
   attempting an explicit/implicit Logout for the CID in question or
   task reassignment for the affected tasks (if any).  Time2Retain is
   the maximum time after the initial respite interval that the task
   and/or connection state(s) is/are guaranteed to be maintained on the
   target to cater to a possible recovery attempt.  Recovery attempts
   for the connection and/or task(s) SHOULD NOT be made before
   Time2Wait seconds but MUST be completed within Time2Retain seconds
   after that initial Time2Wait waiting period.

7.5.1.  Timeouts on Transport Exception Events

   A transport connection shutdown or a transport reset without any
   preceding iSCSI protocol interactions informing the endpoints of the
   fact causes a Full Feature Phase iSCSI connection to be abruptly
   terminated.  The timeout values to be used in this case are the
   negotiated values of DefaultTime2Wait (Section 13.15) and
   DefaultTime2Retain (Section 13.16) text keys for the session.

7.5.2.  Timeouts on Planned Decommissioning

   Any planned decommissioning of a Full Feature Phase iSCSI connection
   is preceded by either a Logout Response PDU or an Async Message PDU.
   The Time2Wait and Time2Retain field values (Section 11.15) in a
   Logout Response PDU, and the Parameter2 and Parameter3 fields of an
   Async Message (AsyncEvent types "drop the connection" or "drop all
   the connections"; see Section 11.9.1), specify the timeout values to
   be used in each of these cases.

   These timeout values are only applicable for the affected connection
   and the tasks active on that connection.  These timeout values have
   no bearing on initiator timers (if any) that are already running on
   connections or tasks associated with that session.

7.6.  Implicit Termination of Tasks

   A target implicitly terminates the active tasks due to iSCSI protocol
   dynamics in the following cases:

      a) When a connection is implicitly or explicitly logged out with
         the reason code "close the connection" and there are active
         tasks allegiant to that connection.

Top      Up      ToC       Page 105 
      b) When a connection fails and eventually the connection state
         times out (state transition M1 in Section 8.2.2), and there are
         active tasks allegiant to that connection.

      c) When a successful Logout with the reason code "remove the
         connection for recovery" is performed while there are active
         tasks allegiant to that connection, and those tasks eventually
         time out after the Time2Wait and Time2Retain periods without
         allegiance reassignment.

      d) When a connection is implicitly or explicitly logged out with
         the reason code "close the session" and there are active tasks
         in that session.

   If the tasks terminated in cases a), b), c), and d) above are SCSI
   tasks, they must be internally terminated as if with CHECK CONDITION
   status.  This status is only meaningful for appropriately handling
   the internal SCSI state and SCSI side effects with respect to
   ordering, because this status is never communicated back as a
   terminating status to the initiator.  However, additional actions may
   have to be taken at the SCSI level, depending on the SCSI context as
   defined by the SCSI standards (e.g., queued commands and ACA; UA for
   the next command on the I_T nexus in cases a), b), and c); etc. --
   see [SAM2] and [SPC3]).

7.7.  Format Errors

   The following two explicit violations of PDU layout rules are format
   errors:

      a) Illegal contents of any PDU header field except the Opcode
         (legal values are specified in Section 11).

      b) Inconsistent field contents (consistent field contents are
         specified in Section 11).

   Format errors indicate a major implementation flaw in one of the
   parties.

   When a target or an initiator receives an iSCSI PDU with a format
   error, it MUST immediately terminate all transport connections in the
   session with either a connection close or a connection reset, and
   escalate the format error to session recovery (see Section 7.1.4.4).

   All initiator-detected PDU construction errors MUST be considered as
   format errors.  Some examples of such errors are:

      - NOP-In with a valid TTT but an invalid LUN

Top      Up      ToC       Page 106 
      - NOP-In with a valid ITT (i.e., a NOP-In response) and also a
        valid TTT

      - SCSI Response PDU with Status=CHECK CONDITION, but
        DataSegmentLength = 0

7.8.  Digest Errors

   The discussion below regarding the legal choices in handling digest
   errors excludes session recovery as an explicit option, but either
   party detecting a digest error may choose to escalate the error to
   session recovery.

   When a target or an initiator receives any iSCSI PDU with a header
   digest error, it MUST either discard the header and all data up to
   the beginning of a later PDU or close the connection.  Because the
   digest error indicates that the length field of the header may have
   been corrupted, the location of the beginning of a later PDU needs to
   be reliably ascertained by other means, such as the operation of a
   Sync and Steering layer.

   When a target receives any iSCSI PDU with a payload digest error, it
   MUST answer with a Reject PDU with a reason code of Data-Digest-Error
   and discard the PDU.

   - If the discarded PDU is a solicited or unsolicited iSCSI data PDU
     (for immediate data in a command PDU, the non-data PDU rule below
     applies), the target MUST do one of the following:

     a) Request retransmission with a recovery R2T.

     b) Terminate the task with a SCSI Response PDU with a CHECK
        CONDITION Status and an iSCSI Condition of "Protocol Service CRC
        error" (Section 11.4.7.2).  If the target chooses to implement
        this option, it MUST wait to receive all the data (signaled by a
        data PDU with the Final bit set for all outstanding R2Ts) before
        sending the SCSI Response PDU.  A task management command (such
        as an ABORT TASK) from the initiator during this wait may also
        conclude the task.

   - No further action is necessary for targets if the discarded PDU is
     a non-data PDU.  In the case of immediate data being present on a
     discarded command, the immediate data is implicitly recovered when
     the task is retried (see Section 7.2.1), followed by the entire
     data transfer for the task.

   When an initiator receives any iSCSI PDU with a payload digest error,
   it MUST discard the PDU.

Top      Up      ToC       Page 107 
      - If the discarded PDU is an iSCSI data PDU, the initiator MUST do
        one of the following:

        a) Request the desired data PDU through SNACK.  In response to
           the SNACK, the target MUST either resend the data PDU or
           reject the SNACK with a Reject PDU with a reason code of
           "SNACK reject", in which case:

           a.1) If the status has not already been sent for the command,
                the target MUST terminate the command with a CHECK
                CONDITION Status and an iSCSI Condition of "SNACK
                rejected" (Section 11.4.7.2).

           a.2) If the status was already sent, no further action is
                necessary for the target.  The initiator in this case
                MUST wait for the status to be received and then discard
                it, so as to internally signal the completion with CHECK
                CONDITION Status and an iSCSI Condition of "Protocol
                Service CRC error" (Section 11.4.7.2).

        b) Abort the task and terminate the command with an error.

      - If the discarded PDU is a response PDU or an unsolicited PDU
        (e.g., Async, Reject), the initiator MUST do one of the
        following:

        a) Request PDU retransmission with a status of SNACK.

        b) Log out the connection for recovery, and continue the tasks
           on a different connection instance as described in
           Section 7.2.

        c) Log out to close the connection (abort all the commands
           associated with the connection).

      Note that an unsolicited PDU carries the next StatSN value on an
      iSCSI connection, thereby advancing the StatSN.  When an initiator
      discards one of these PDUs due to a payload digest error, the
      entire PDU, including the header, MUST be discarded.
      Consequently, the initiator MUST treat the exception like a loss
      of any other solicited response PDU.

7.9.  Sequence Errors

   When an initiator receives an iSCSI R2T/data PDU with an out-of-order
   R2TSN/DataSN or a SCSI Response PDU with an ExpDataSN that implies
   missing data PDU(s), it means that the initiator must have detected a
   header or payload digest error on one or more earlier R2T/data PDUs.

Top      Up      ToC       Page 108 
   The initiator MUST address these implied digest errors as described
   in Section 7.8.  When a target receives a data PDU with an out-of-
   order DataSN, it means that the target must have hit a header or
   payload digest error on at least one of the earlier data PDUs.  The
   target MUST address these implied digest errors as described in
   Section 7.8.

   When an initiator receives an iSCSI status PDU with an out-of-order
   StatSN that implies missing responses, it MUST address the one or
   more missing status PDUs as described in Section 7.8.  As a side
   effect of receiving the missing responses, the initiator may discover
   missing data PDUs.  If the initiator wants to recover the missing
   data for a command, it MUST NOT acknowledge the received responses
   that start from the StatSN of the relevant command until it has
   completed receiving all the data PDUs of the command.

   When an initiator receives duplicate R2TSNs (due to proactive
   retransmission of R2Ts by the target) or duplicate DataSNs (due to
   proactive SNACKs by the initiator), it MUST discard the duplicates.

7.10.  Message Error Checking

   In iSCSI implementations to date, there has been some uncertainty
   regarding the extent to which incoming messages have to be checked
   for protocol errors, beyond what is strictly required for processing
   the inbound message.  This section addresses this question.

   Unless this document requires it, an iSCSI implementation is not
   required to do an exhaustive protocol conformance check on an
   incoming iSCSI PDU.  The iSCSI implementation in particular is not
   required to double-check the remote iSCSI implementation's
   conformance to protocol requirements.

7.11.  SCSI Timeouts

   An iSCSI initiator MAY attempt to plug a command sequence gap on the
   target end (in the absence of an acknowledgment of the command by way
   of the ExpCmdSN) before the ULP timeout by retrying the
   unacknowledged command, as described in Section 7.2.

   On a ULP timeout for a command (that carried a CmdSN of n), if the
   iSCSI initiator intends to continue the session it MUST abort the
   command by using either an appropriate Task Management Function
   Request for the specific command or a "close the connection" logout.

Top      Up      ToC       Page 109 
   When using an ABORT TASK, if the ExpCmdSN is still less than (n + 1),
   the target may see the abort request while missing the original
   command itself, due to one of the following reasons:

      - The original command was dropped due to digest error.

      - The connection on which the original command was sent was
        successfully logged out.  On logout, the unacknowledged commands
        issued on the connection being logged out are discarded.

   If the abort request is received and the original command is missing,
   targets MUST consider the original command with that RefCmdSN as
   received and issue a task management response with the response code
   "Function complete".  This response concludes the task on both ends.
   If the abort request is received and the target can determine (based
   on the Referenced Task Tag) that the command was received and
   executed, and also that the response was sent prior to the abort,
   then the target MUST respond with the response code "Task Does Not
   Exist".

7.12.  Negotiation Failures

   Text Request and Response sequences, when used to set/negotiate
   operational parameters, constitute the negotiation/parameter setting.
   A negotiation failure is considered to be one or more of the
   following:

      - For a negotiated key, none of the choices are acceptable to one
        of the sides in the negotiation.

      - For a declarative key, the declared value is not acceptable to
        the other side in the negotiation.

      - The Text Request timed out and possibly terminated.

      - The Text Request was answered with a Reject PDU.

   The following two rules should be used to address negotiation
   failures:

      a) During login, any failure in negotiation MUST be considered a
         login process failure; the Login Phase, along with the
         connection, MUST be terminated.  If the target detects the
         failure, it must terminate the login with the appropriate Login
         response code.

Top      Up      ToC       Page 110 
      b) A failure in negotiation during the Full Feature Phase will
         terminate the entire negotiation sequence, which may consist of
         a series of Text Requests that use the same Initiator Task Tag.
         The operational parameters of the session or the connection
         MUST continue to be the values agreed upon during an earlier
         successful negotiation (i.e., any partial results of this
         unsuccessful negotiation MUST NOT take effect and MUST be
         discarded).

7.13.  Protocol Errors

   Mapping framed messages over a "streaming" connection such as TCP
   makes the proposed mechanisms vulnerable to simple software framing
   errors.  On the other hand, the introduction of framing mechanisms to
   limit the effects of these errors may be onerous on performance for
   simple implementations.  Command sequence numbers and the mechanisms
   for dropping and reestablishing connections (discussed earlier in
   Section 7 and its subsections) help handle this type of mapping
   errors.

   All violations of iSCSI PDU exchange sequences specified in this
   document are also protocol errors.  This category of errors can only
   be addressed by fixing the implementations; iSCSI defines Reject and
   response codes to enable this.

7.14.  Connection Failures

   iSCSI can keep a session in operation if it is able to keep/establish
   at least one TCP connection between the initiator and the target in a
   timely fashion.  Targets and/or initiators may recognize a failing
   connection by either transport-level means (TCP), a gap in the
   command sequence number, a response stream that is not filled for a
   long time, or a failing iSCSI NOP (acting as a ping).  The latter MAY
   be used periodically to increase the speed and likelihood of
   detecting connection failures.  As an example for transport-level
   means, initiators and targets MAY also use the keep-alive option (see
   [RFC1122]) on the TCP connection to enable early link failure
   detection on otherwise idle links.

Top      Up      ToC       Page 111 
   On connection failure, the initiator and target MUST do one of the
   following:

      a) Attempt connection recovery within the session (Connection
         Recovery).

      b) Log out the connection with the reason code "close the
         connection" (Section 11.14.5), reissue missing commands, and
         implicitly terminate all active commands.  This option requires
         support for the Within-connection recovery class (recovery
         within-connection).

      c) Perform session recovery (Session Recovery).

   Either side may choose to escalate to session recovery (via the
   initiator dropping all the connections or via an Async Message that
   announces the similar intent from a target), and the other side MUST
   give it precedence.  On a connection failure, a target MUST terminate
   and/or discard all of the active immediate commands, regardless of
   which of the above options is used (i.e., immediate commands are not
   recoverable across connection failures).

7.15.  Session Errors

   If all of the connections of a session fail and cannot be
   reestablished in a short time, or if initiators detect protocol
   errors repeatedly, an initiator may choose to terminate a session and
   establish a new session.

   In this case, the initiator takes the following actions:

      - Resets or closes all the transport connections.

      - Terminates all outstanding requests with an appropriate response
        before initiating a new session.  If the same I_T nexus is
        intended to be reestablished, the initiator MUST employ session
        reinstatement (see Section 6.3.5).

   When the session timeout (the connection state timeout for the last
   failed connection) happens on the target, it takes the following
   actions:

      - Resets or closes the TCP connections (closes the session).

      - Terminates all active tasks that were allegiant to the
        connection(s) that constituted the session.

Top      Up      ToC       Page 112 
   A target MUST also be prepared to handle a session reinstatement
   request from the initiator that may be addressing session errors.

8.  State Transitions

   iSCSI connections and iSCSI sessions go through several well-defined
   states from the time they are created to the time they are cleared.

   The connection state transitions are described in two separate but
   dependent sets of state diagrams for ease in understanding.  The
   first set of diagrams, "standard connection state diagrams",
   describes the connection state transitions when the iSCSI connection
   is not waiting for, or undergoing, a cleanup by way of an explicit or
   implicit logout.  The second set, "connection cleanup state diagram",
   describes the connection state transitions while performing the iSCSI
   connection cleanup.  While the first set has two diagrams -- one each
   for initiator and target -- the second set has a single diagram
   applicable to both initiators and targets.

   The "session state diagram" describes the state transitions an iSCSI
   session would go through during its lifetime, and it depends on the
   states of possibly multiple iSCSI connections that participate in the
   session.

   States and transitions are described in text, tables, and diagrams.
   The diagrams are used for illustration.  The text and the tables are
   the governing specification.

8.1.  Standard Connection State Diagrams

8.1.1.  State Descriptions for Initiators and Targets

   State descriptions for the standard connection state diagram are as
   follows:

   S1: FREE

       - initiator: State on instantiation, or after successful
         connection closure.

       - target: State on instantiation, or after successful
         connection closure.

Top      Up      ToC       Page 113 
   S2: XPT_WAIT

       - initiator: Waiting for a response to its transport
         connection establishment request.

       - target: Illegal.

   S3: XPT_UP

       - initiator: Illegal.

       - target: Waiting for the login process to commence.

   S4: IN_LOGIN

       - initiator: Waiting for the login process to conclude,
         possibly involving several PDU exchanges.

       - target: Waiting for the login process to conclude,
         possibly involving several PDU exchanges.

   S5: LOGGED_IN

       - initiator: In the Full Feature Phase, waiting for all
         internal, iSCSI, and transport events.

       - target: In the Full Feature Phase, waiting for all internal,
         iSCSI, and transport events.

   S6: IN_LOGOUT

       - initiator: Waiting for a Logout Response.

       - target: Waiting for an internal event signaling completion
         of logout processing.

   S7: LOGOUT_REQUESTED

       - initiator: Waiting for an internal event signaling
         readiness to proceed with Logout.

       - target: Waiting for the Logout process to start after
         having requested a Logout via an Async Message.

Top      Up      ToC       Page 114 
   S8: CLEANUP_WAIT

       - initiator: Waiting for the context and/or resources to
         initiate the cleanup processing for this CSM.

       - target: Waiting for the cleanup process to start for this CSM.

8.1.2.  State Transition Descriptions for Initiators and Targets

   T1:

       - initiator: Transport connect request was made (e.g., TCP SYN
         sent).

       - target: Illegal.

   T2:

       - initiator: Transport connection request timed out, a
         transport reset was received, or an internal event of
         receiving a Logout Response (success) on another connection
         for a "close the session" Logout Request was received.

       - target: Illegal.

   T3:

       - initiator: Illegal.

       - target: Received a valid transport connection request that
         establishes the transport connection.

   T4:

       - initiator: Transport connection established, thus
         prompting the initiator to start the iSCSI Login.

       - target: Initial iSCSI Login Request was received.

   T5:

       - initiator: The final iSCSI Login Response with a Status-Class
         of zero was received.

       - target: The final iSCSI Login Request to conclude the
         Login Phase was received, thus prompting the target to send
         the final iSCSI Login Response with a Status-Class of zero.

Top      Up      ToC       Page 115 
   T6:

       - initiator: Illegal.

       - target: Timed out waiting for an iSCSI Login, transport
         disconnect indication was received, transport reset was
         received, or an internal event indicating a transport
         timeout was received.  In all these cases, the connection is
         to be closed.

   T7:

       - initiator: One of the following events caused the transition:

         a) The final iSCSI Login Response was received with a
            non-zero Status-Class.

         b) Login timed out.

         c) A transport disconnect indication was received.

         d) A transport reset was received.

         e) An internal event indicating a transport timeout was
            received.

         f) An internal event of receiving a Logout Response
            (success) on another connection for a "close the
            session" Logout Request was received.

       In all these cases, the transport connection is closed.

       - target: One of the following events caused the transition:

         a) The final iSCSI Login Request to conclude the Login
            Phase was received, prompting the target to send the
            final iSCSI Login Response with a non-zero Status-Class.

         b) Login timed out.

         c) A transport disconnect indication was received.

         d) A transport reset was received.

         e) An internal event indicating a transport timeout was
            received.

Top      Up      ToC       Page 116 
         f) On another connection, a "close the session" Logout Request
            was received.

       In all these cases, the connection is to be closed.

   T8:

       - initiator: An internal event of receiving a Logout
         Response (success) on another connection for a "close the
         session" Logout Request was received, thus closing this
         connection and requiring no further cleanup.

       - target: An internal event of sending a Logout Response
         (success) on another connection for a "close the session"
         Logout Request was received, or an internal event of a
         successful connection/session reinstatement was received,
         thus prompting the target to close this connection cleanly.

   T9, T10:

       - initiator: An internal event that indicates the readiness
         to start the Logout process was received, thus prompting an
         iSCSI Logout to be sent by the initiator.

       - target: An iSCSI Logout Request was received.

   T11, T12:

       - initiator: An Async PDU with AsyncEvent "Request Logout"
         was received.

       - target: An internal event that requires the decommissioning
         of the connection was received, thus causing an Async PDU with
         an AsyncEvent "Request Logout" to be sent.

   T13:

       - initiator: An iSCSI Logout Response (success) was received,
         or an internal event of receiving a Logout Response (success)
         on another connection for a "close the session" Logout Request
         was received.

       - target: An internal event was received that indicates
         successful processing of the Logout, which prompts an iSCSI
         Logout Response (success) to be sent; an internal event of
         sending a Logout Response (success) on another connection
         for a "close the session" Logout Request was received; or

Top      Up      ToC       Page 117 
         an internal event of a successful connection/session
         reinstatement was received.  In all these cases, the
         transport connection is closed.

   T14:

       - initiator: An Async PDU with AsyncEvent "Request Logout"
         was received again.

       - target: Illegal.

   T15, T16:

       - initiator: One or more of the following events caused this
         transition:

         a) An internal event that indicates a transport connection
            timeout was received, thus prompting a transport reset
            or transport connection closure.

         b) A transport reset was received.

         c) A transport disconnect indication was received.

         d) An Async PDU with AsyncEvent "Drop connection" (for this
            CID) was received.

         e) An Async PDU with AsyncEvent "Drop all connections" was
            received.

       - target: One or more of the following events caused this
         transition:

         a) Internal event that indicates that a transport connection
            timeout was received, thus prompting a transport reset
            or transport connection closure.

         b) An internal event of a failed connection/session
            reinstatement was received.

         c) A transport reset was received.

         d) A transport disconnect indication was received.

         e) An internal emergency cleanup event was received, which
            prompts an Async PDU with AsyncEvent "Drop connection" (for
            this CID), or event "Drop all connections".

Top      Up      ToC       Page 118 
   T17:

       - initiator: One or more of the following events caused this
         transition:

         a) A Logout Response (failure, i.e., a non-zero status)
            was received, or Logout timed out.

         b) Any of the events specified for T15 and T16 occurred.

       - target: One or more of the following events caused this
         transition:

         a) An internal event that indicates a failure of the
            Logout processing was received, which prompts a
            Logout Response (failure, i.e., a non-zero status)
            to be sent.

         b) Any of the events specified for T15 and T16 occurred.

   T18:

       - initiator: An internal event of receiving a Logout
         Response (success) on another connection for a "close the
         session" Logout Request was received.

       - target: An internal event of sending a Logout Response
         (success) on another connection for a "close the session"
         Logout Request was received, or an internal event of a
         successful connection/session reinstatement was received.
         In both these cases, the connection is closed.

   The CLEANUP_WAIT state (S8) implies that there are possible iSCSI
   tasks that have not reached conclusion and are still considered
   busy.

8.1.3.  Standard Connection State Diagram for an Initiator

   Symbolic names for states:

      S1: FREE

      S2: XPT_WAIT

      S4: IN_LOGIN

      S5: LOGGED_IN

Top      Up      ToC       Page 119 
      S6: IN_LOGOUT

      S7: LOGOUT_REQUESTED

      S8: CLEANUP_WAIT

   States S5, S6, and S7 constitute the Full Feature Phase operation of
   the connection.

   The state diagram is as follows:

                        -------<-------------+
            +--------->/ S1    \<----+       |
         T13|       +->\       /<-+   \      |
            |      /    ---+---    \   \     |
            |     /        |     T2 \   |    |
            |  T8 |        |T1       |  |    |
            |     |        |        /   |T7  |
            |     |        |       /    |    |
            |     |        |      /     |    |
            |     |        V     /     /     |
            |     |     ------- /     /      |
            |     |    / S2    \     /       |
            |     |    \       /    /        |
            |     |     ---+---    /         |
            |     |        |T4    /          |
            |     |        V     /           | T18
            |     |     ------- /            |
            |     |    / S4    \             |
            |     |    \       /             |
            |     |     ---+---              |         T15
            |     |        |T5      +--------+---------+
            |     |        |       /T16+-----+------+  |
            |     |        |      /   -+-----+--+   |  |
            |     |        |     /   /  S7   \  |T12|  |
            |     |        |    / +->\       /<-+   V  V
            |     |        |   / /    -+-----       -------
            |     |        |  / /T11   |T10        /  S8   \
            |     |        V / /       V  +----+   \       /
            |     |      ---+-+-      ----+--  |    -------
            |     |     / S5    \T9  / S6    \<+      ^
            |     +-----\       /--->\       / T14    |
            |            -------      --+---+---------+T17
            +---------------------------+

Top      Up      ToC       Page 120 
   The following state transition table represents the above diagram.
   Each row represents the starting state for a given transition, which,
   after taking a transition marked in a table cell, would end in the
   state represented by the column of the cell.  For example, from
   state S1, the connection takes the T1 transition to arrive at
   state S2.  The fields marked "-" correspond to undefined transitions.

      +----+---+---+---+---+----+---+
      |S1  |S2 |S4 |S5 |S6 |S7  |S8 |
   ---+----+---+---+---+---+----+---+
    S1| -  |T1 | - | - | - | -  | - |
   ---+----+---+---+---+---+----+---+
    S2|T2  |-  |T4 | - | - | -  | - |
   ---+----+---+---+---+---+----+---+
    S4|T7  |-  |-  |T5 | - | -  | - |
   ---+----+---+---+---+---+----+---+
    S5|T8  |-  |-  | - |T9 |T11 |T15|
   ---+----+---+---+---+---+----+---+
    S6|T13 |-  |-  | - |T14|-   |T17|
   ---+----+---+---+---+---+----+---+
    S7|T18 |-  |-  | - |T10|T12 |T16|
   ---+----+---+---+---+---+----+---+
    S8| -  |-  |-  | - | - | -  | - |
   ---+----+---+---+---+---+----+---+

8.1.4.  Standard Connection State Diagram for a Target

   Symbolic names for states:

      S1: FREE

      S3: XPT_UP

      S4: IN_LOGIN

      S5: LOGGED_IN

      S6: IN_LOGOUT

      S7: LOGOUT_REQUESTED

      S8: CLEANUP_WAIT

   States S5, S6, and S7 constitute the Full Feature Phase operation of
   the connection.

Top      Up      ToC       Page 121 
   The state diagram is as follows:

                           -------<-------------+
               +--------->/ S1    \<----+       |
            T13|       +->\       /<-+   \      |
               |      /    ---+---    \   \     |
               |     /        |     T6 \   |    |
               |  T8 |        |T3       |  |    |
               |     |        |        /   |T7  |
               |     |        |       /    |    |
               |     |        |      /     |    |
               |     |        V     /     /     |
               |     |     ------- /     /      |
               |     |    / S3    \     /       |
               |     |    \       /    /        | T18
               |     |     ---+---    /         |
               |     |        |T4    /          |
               |     |        V     /           |
               |     |     ------- /            |
               |     |    / S4    \             |
               |     |    \       /             |
               |     |     ---+---         T15  |
               |     |        |T5      +--------+---------+
               |     |        |       /T16+-----+------+  |
               |     |        |      /  -+-----+---+   |  |
               |     |        |     /   /  S7   \  |T12|  |
               |     |        |    / +->\       /<-+   V  V
               |     |        |   / /    -+-----       -------
               |     |        |  / /T11   |T10        /  S8   \
               |     |        V / /       V           \       /
               |     |      ---+-+-      -------       -------
               |     |     / S5    \T9  / S6    \        ^
               |     +-----\       /--->\       /        |
               |            -------      --+---+---------+T17
               +---------------------------+

Top      Up      ToC       Page 122 
   The following state transition table represents the above diagram and
   follows the conventions described for the initiator diagram.

      +----+---+---+---+---+----+---+
      |S1  |S3 |S4 |S5 |S6 |S7  |S8 |
   ---+----+---+---+---+---+----+---+
    S1| -  |T3 | - | - | - | -  | - |
   ---+----+---+---+---+---+----+---+
    S3|T6  |-  |T4 | - | - | -  | - |
   ---+----+---+---+---+---+----+---+
    S4|T7  |-  |-  |T5 | - | -  | - |
   ---+----+---+---+---+---+----+---+
    S5|T8  |-  |-  | - |T9 |T11 |T15|
   ---+----+---+---+---+---+----+---+
    S6|T13 |-  |-  | - |-  |-   |T17|
   ---+----+---+---+---+---+----+---+
    S7|T18 |-  |-  | - |T10|T12 |T16|
   ---+----+---+---+---+---+----+---+
    S8| -  |-  |-  | - | - | -  | - |
   ---+----+---+---+---+---+----+---+

8.2.  Connection Cleanup State Diagram for Initiators and Targets

   Symbolic names for states:

      R1: CLEANUP_WAIT (same as S8)

      R2: IN_CLEANUP

      R3: FREE (same as S1)

   Whenever a connection state machine in cleanup (let's call it CSM-C)
   enters the CLEANUP_WAIT state (S8), it must go through the state
   transitions described in the connection cleanup state diagram, using
   either a) a separate Full Feature Phase connection (let's call it
   CSM-E, for explicit) in the LOGGED_IN state in the same session or
   b) a new transport connection (let's call it CSM-I, for implicit) in
   the FREE state that is to be added to the same session.  In the CSM-E
   case, an explicit logout for the CID that corresponds to CSM-C (as
   either a connection or session logout) needs to be performed to
   complete the cleanup.  In the CSM-I case, an implicit logout for the
   CID that corresponds to CSM-C needs to be performed by way of
   connection reinstatement (Section 6.3.4) for that CID.  In either
   case, the protocol exchanges on CSM-E or CSM-I determine the state
   transitions for CSM-C.  Therefore, this cleanup state diagram is only
   applicable to the instance of the connection in cleanup (i.e.,
   CSM-C).  In the case of an implicit logout, for example, CSM-C

Top      Up      ToC       Page 123 
   reaches FREE (R3) at the time CSM-I reaches LOGGED_IN.  In the case
   of an explicit logout, CSM-C reaches FREE (R3) when CSM-E receives a
   successful Logout Response while continuing to be in the LOGGED_IN
   state.

   An initiator must initiate an explicit or implicit connection logout
   for a connection in the CLEANUP_WAIT state, if the initiator intends
   to continue using the associated iSCSI session.

   The following state diagram applies to both initiators and targets.
   (M1, M2, M3, and M4 are defined in Section 8.2.2.)

                           ---------
                          / R1      \
                      +---\         /<-+
                     /     ----+----    \
                    /          |         \ M3
                 M1 |          |M2        |
                    |          |         /
                    |          |        /
                    |          |       /
                    |          V      /
                    |       ---------/
                    |      / R2      \
                    |      \         /
                    |       ---------
                    |          |
                    |          |M4
                    |          |
                    |          |
                    |          |
                    |          V
                    |       --------
                    |      / R3     \
                    +----->\        /
                            --------

Top      Up      ToC       Page 124 
   The following state transition table represents the above diagram and
   follows the same conventions as in earlier sections.

        +----+----+----+
        |R1  |R2  |R3  |
   -----+----+----+----+
    R1  | -  |M2  |M1  |
   -----+----+----+----+
    R2  |M3  | -  |M4  |
   -----+----+----+----+
    R3  | -  | -  | -  |
   -----+----+----+----+

8.2.1.  State Descriptions for Initiators and Targets

   R1: CLEANUP_WAIT (same as S8)

       - initiator: Waiting for the internal event to initiate the
         cleanup processing for CSM-C.

       - target: Waiting for the cleanup process to start for CSM-C.

   R2: IN_CLEANUP

       - initiator: Waiting for the connection cleanup process to
         conclude for CSM-C.

       - target: Waiting for the connection cleanup process to conclude
         for CSM-C.

   R3: FREE (same as S1)

       - initiator: End state for CSM-C.

       - target: End state for CSM-C.

8.2.2.  State Transition Descriptions for Initiators and Targets

   M1: One or more of the following events was received:

       - initiator:

         * An internal event that indicates connection state timeout.

         * An internal event of receiving a successful Logout Response
           on a different connection for a "close the session" Logout.

Top      Up      ToC       Page 125 
       - target:

         * An internal event that indicates connection state timeout.

         * An internal event of sending a Logout Response (success) on a
           different connection for a "close the session" Logout
           Request.

   M2: An implicit/explicit logout process was initiated by the
       initiator.

       - In CSM-I usage:

         * initiator: An internal event requesting the connection (or
           session) reinstatement was received, thus prompting a
           connection (or session) reinstatement Login to be sent,
           transitioning CSM-I to state IN_LOGIN.

         * target: A connection/session reinstatement Login was received
           while in state XPT_UP.

       - In CSM-E usage:

         * initiator: An internal event was received that indicates that
           an explicit logout was sent for this CID in state LOGGED_IN.

         * target: An explicit logout was received for this CID in state
           LOGGED_IN.

   M3: Logout failure was detected.

       - In CSM-I usage:

         * initiator: CSM-I failed to reach LOGGED_IN and arrived into
           FREE instead.

         * target: CSM-I failed to reach LOGGED_IN and arrived into FREE
           instead.

       - In CSM-E usage:

         * initiator: either CSM-E moved out of LOGGED_IN, or Logout
           timed out and/or aborted, or Logout Response (failure) was
           received.

Top      Up      ToC       Page 126 
         * target: either CSM-E moved out of LOGGED_IN, Logout timed out
           and/or aborted, or an internal event that indicates that a
           failed Logout processing was received.  A Logout Response
           (failure) was sent in the last case.

   M4: Successful implicit/explicit logout was performed.

       - In CSM-I usage:

         * initiator: CSM-I reached state LOGGED_IN, or an internal
           event of receiving a Logout Response (success) on another
           connection for a "close the session" Logout Request was
           received.

         * target: CSM-I reached state LOGGED_IN, or an internal event
           of sending a Logout Response (success) on a different
           connection for a "close the session" Logout Request was
           received.

       - In CSM-E usage:

         * initiator: CSM-E stayed in LOGGED_IN and received a Logout
           Response (success), or an internal event of receiving a
           Logout Response (success) on another connection for a "close
           the session" Logout Request was received.

         * target: CSM-E stayed in LOGGED_IN and an internal event
           indicating a successful Logout processing was received, or an
           internal event of sending a Logout Response (success) on a
           different connection for a "close the session" Logout Request
           was received.

8.3.  Session State Diagrams

8.3.1.  Session State Diagram for an Initiator

   Symbolic names for states:

      Q1: FREE

      Q3: LOGGED_IN

      Q4: FAILED

   State Q3 represents the Full Feature Phase operation of the session.

Top      Up      ToC       Page 127 
   The state diagram is as follows.  (N1, N3, N4, N5, and N6 are defined
   in Section 8.3.4.)

                                   ---------
                                  / Q1      \
                      +---------->\         /<-+
                     /             ----+----   |
                    /                  |       |N3
                N6  |                  |N1     |
                    |                  |       |
                    |       N4         |       |
                    | +------------+   |      /
                    | |            |   |     /
                    | |            |   |    /
                    | |            V   V   /
                  --+-+---         -------+-
                 / Q4     \ N5    / Q3      \
                 \        /<------\         /
                  --------         ---------

   The state transition table is as follows:

        +---+---+---+
        |Q1 |Q3 |Q4 |
   -----+---+---+---+
    Q1  | - |N1 | - |
   -----+---+---+---+
    Q3  |N3 | - |N5 |
   -----+---+---+---+
    Q4  |N6 |N4 | - |
   -----+---+---+---+

8.3.2.  Session State Diagram for a Target

   Symbolic names for states:

      Q1: FREE

      Q2: ACTIVE

      Q3: LOGGED_IN

      Q4: FAILED

      Q5: IN_CONTINUE

   State Q3 represents the Full Feature Phase operation of the session.

Top      Up      ToC       Page 128 
   The state diagram is as follows:

                                           ---------
                     +------------------->/ Q1      \
                    /     +-------------->\         /<-+
                    |     |                ---+-----   |
                    |     |                 ^ |        |N3
                 N6 |     |N11            N9| V N1     |
                    |     |                 +--------  |
                    |     |                / Q2      \ |
                    |     |                \         / |
                    |  ---+-----            +--+-----  |
                    | / Q5      \              |       |
                    | \         / N10          |       |
                    |  -+-+----+-----------+   | N2   /
                    |   ^ |                |   |     /
                    | N7| |N8              |   |    /
                    |   | |                |   V   /
                  --+---+-V                V------+-
                 / Q4      \ N5           / Q3      \
                 \         /<-------------\         /
                  ---------                ---------

   The state transition table is as follows:

        +----+----+----+----+----+
        |Q1  |Q2  |Q3  |Q4  |Q5  |
   -----+----+----+----+----+----+
    Q1  | -  |N1  | -  | -  | -  |
   -----+----+----+----+----+----+
    Q2  |N9  | -  |N2  | -  | -  |
   -----+----+----+----+----+----+
    Q3  |N3  | -  | -  |N5  | -  |
   -----+----+----+----+----+----+
    Q4  |N6  | -  | -  | -  |N7  |
   -----+----+----+----+----+----+
    Q5  |N11 | -  |N10 |N8  | -  |
   -----+----+----+----+----+----+

Top      Up      ToC       Page 129 
8.3.3.  State Descriptions for Initiators and Targets

   Q1: FREE

       - initiator: State on instantiation or after cleanup.

       - target: State on instantiation or after cleanup.

   Q2: ACTIVE

       - initiator: Illegal.

       - target: The first iSCSI connection in the session transitioned
         to IN_LOGIN, waiting for it to complete the login process.

   Q3: LOGGED_IN

       - initiator: Waiting for all session events.

       - target: Waiting for all session events.

   Q4: FAILED

       - initiator: Waiting for session recovery or session
         continuation.

       - target: Waiting for session recovery or session continuation.

   Q5: IN_CONTINUE

       - initiator: Illegal.

       - target: Waiting for session continuation attempt to reach a
         conclusion.

8.3.4.  State Transition Descriptions for Initiators and Targets

   N1:

       - initiator: At least one transport connection reached the
         LOGGED_IN state.

       - target: The first iSCSI connection in the session had reached
         the IN_LOGIN state.

Top      Up      ToC       Page 130 
   N2:

       - initiator: Illegal.

       - target: At least one iSCSI connection reached the LOGGED_IN
         state.

   N3:

       - initiator: Graceful closing of the session via session closure
         (Section 6.3.6).

       - target: Graceful closing of the session via session closure
         (Section 6.3.6) or a successful session reinstatement cleanly
         closed the session.

   N4:

       - initiator: A session continuation attempt succeeded.

       - target: Illegal.

   N5:
       - initiator: Session failure (Section 6.3.6) occurred.

       - target: Session failure (Section 6.3.6) occurred.

   N6:

       - initiator: Session state timeout occurred, or a session
         reinstatement cleared this session instance.  This results in
         the freeing of all associated resources, and the session state
         is discarded.

       - target: Session state timeout occurred, or a session
         reinstatement cleared this session instance.  This results in
         the freeing of all associated resources, and the session state
         is discarded.

   N7:

       - initiator: Illegal.

       - target: A session continuation attempt was initiated.

Top      Up      ToC       Page 131 
   N8:

       - initiator: Illegal.

       - target: The last session continuation attempt failed.

   N9:

       - initiator: Illegal.

       - target: Login attempt on the leading connection failed.

   N10:

       - initiator: Illegal.

       - target: A session continuation attempt succeeded.

   N11:

       - initiator: Illegal.

       - target: A successful session reinstatement cleanly closed the
         session.

9.  Security Considerations

   Historically, native storage systems have not had to consider
   security, because their environments offered minimal security risks.
   That is, these environments consisted of storage devices either
   directly attached to hosts or connected via a Storage Area Network
   (SAN) distinctly separate from the communications network.  The use
   of storage protocols, such as SCSI, over IP networks requires that
   security concerns be addressed.  iSCSI implementations must provide
   means of protection against active attacks (e.g., pretending to be
   another identity; message insertion, deletion, modification, and
   replaying) and passive attacks (e.g., eavesdropping, gaining
   advantage by analyzing the data sent over the line).

   Although technically possible, iSCSI SHOULD NOT be configured without
   security, specifically in-band authentication; see Section 9.2.
   iSCSI configured without security should be confined to closed
   environments that have very limited and well-controlled security
   risks.  [RFC3723] specifies the mechanisms that must be used in order
   to mitigate risks fully described in that document.

   The following section describes the security mechanisms provided by
   an iSCSI implementation.


Next RFC Part