tech-invite   World Map     

IETF     RFCs     Groups     SIP     ABNFs    |    3GPP     Specs     Glossaries     Architecture     IMS     UICC    |    search     info

RFC 5040

 
 
 

A Remote Direct Memory Access Protocol Specification

Part 3 of 3, p. 46 to 66
Prev RFC Part

 


prevText      Top      Up      ToC       Page 46 
8.  Security Considerations

   This section references the resources that discuss protocol- specific
   security considerations and implications of using RDMAP with existing
   security services.  A detailed analysis of the security issues around
   implementation and use of the RDMAP can be found in [RDMASEC].

   [RDMASEC] introduces the RDMA reference model and discusses how the
   resources of this model are vulnerable to attacks and the types of
   attack these vulnerabilities are subject to.  It also details the
   levels of Trust available in this peer-to-peer model and how this
   defines the nature of resource sharing.

   The IPsec requirements for RDDP are based on the version of IPsec
   specified in RFC 2401 [RFC2401] and related RFCs, as profiled by RFC
   3723 [RFC3723], despite the existence of a newer version of IPsec
   specified in RFC 4301 [RFC4301] and related RFCs [RFC4303],
   [RFC4306], [RFC4835].  One of the important early applications of the
   RDDP protocols is their use with iSCSI [iSER]; RDDP's IPsec
   requirements follow those of IPsec in order to facilitate that usage
   by allowing a common profile of IPsec to be used with iSCSI and the
   RDDP protocols.  In the future, RFC 3723 may be updated to the newer
   version of IPsec, and the IPsec security requirements of any such
   update should apply uniformly to iSCSI and the RDDP protocols.

8.1.  Summary of RDMAP-Specific Security Requirements

   [RDMASEC] defines the security requirements for the implementation of
   the components of the RDMA reference model, namely the RDMA enabled
   NIC (RNIC) and the Privileged Resource Manager.  An RDMAP
   implementation conforming to this specification MUST conform to these
   requirements.

Top      Up      ToC       Page 47 
8.1.1.  RDMAP (RNIC) Requirements

   RDMAP provides several countermeasures for all types of attacks as
   introduced in [RDMASEC].  In the following, this specification lists
   all security requirements that MUST be implemented by the RNIC.  A
   more detailed discussion of RNIC security requirements can be found
   in Section 5 of [RDMASEC].

   1.  An RNIC MUST ensure that a specific Stream in a specific
       Protection Domain cannot access an STag in a different Protection
       Domain.

   2.  An RNIC MUST ensure that if an STag is limited in scope to a
       single Stream, no other Stream can use the STag.

   3.  An RNIC MUST ensure that a Remote Peer is not able to access
       memory outside of the buffer specified when the STag was enabled
       for remote access.

   4.  An RNIC MUST provide a mechanism for the ULP to establish and
       revoke the association of a ULP Buffer to an STag and TO range.

   5.  An RNIC MUST provide a mechanism for the ULP to establish and
       revoke read, write, or read and write access to the ULP Buffer
       referenced by an STag.

   6.  An RNIC MUST ensure that the network interface can no longer
       modify an Advertised Buffer after the ULP revokes remote access
       rights for an STag.

   7.  An RNIC MUST ensure that a Remote Peer is not able to invalidate
       an STag enabled for remote access, if the STag is shared on
       multiple streams.

   8.  An RNIC MUST choose the value of STags in a way difficult to
       predict.  It is RECOMMENDED to sparsely populate them over the
       full available range.

   9.  An RNIC MUST NOT enable sharing a Completion Queue (CQ) across
       ULPs that do not share partial mutual trust.

   10. An RNIC MUST ensure that if a CQ overflows, any Streams that do
       not use the CQ MUST remain unaffected.

   11. An RNIC implementation SHOULD provide a mechanism to cap the
       number of outstanding RDMA Read Requests.

Top      Up      ToC       Page 48 
   12. An RNIC MUST NOT enable firmware to be loaded on the RNIC
       directly from an untrusted Local Peer or Remote Peer, unless the
       Peer is properly authenticated*, and the update is done via a
       secure protocol, such as IPsec.

       * by a mechanism outside the scope of this specification.  The
         mechanism presumably entails authenticating that the remote ULP
         has the right to perform the update.

8.1.2.  Privileged Resource Manager Requirements

   With RDMAP, all reservations of local resources are initiated from
   local ULPs.  To protect from local attacks including unfair resource
   distribution and gaining unauthorized access to RNIC resources, a
   Privileged Resource Manager (PRM) must be implemented, which manages
   all local resource allocation.  Note that the PRM must not be
   provided as an independent component, and its functionality can also
   be implemented as part of the privileged ULP or as part of the RNIC
   itself.

   A PRM implementation must meet the following security requirements (a
   more detailed discussion of PRM security requirements can be found in
   Section 5 of [RDMASEC]):

   1.  All Non-Privileged ULP interactions with the RNIC Engine that
       could affect other ULPs MUST be done using the Resource Manager
       as a proxy.

   2.  All ULP resource allocation requests for scarce resources MUST
       also be done using a Privileged Resource Manager.

   3.  The Privileged Resource Manager MUST NOT assume that different
       ULPs share Partial Mutual Trust unless there is a mechanism to
       ensure that the ULPs do indeed share partial mutual trust.

   4.  If Non-Privileged ULPs are supported, the Privileged Resource
       Manager MUST verify that the Non-Privileged ULP has the right to
       access a specific Data Buffer before allowing an STag for which
       the ULP has access rights to be associated with a specific Data
       Buffer.

   5.  The Privileged Resource Manager MUST control the allocation of CQ
       entries.

   6.  The Privileged Resource Manager SHOULD prevent a Local Peer from
       allocating more than its fair share of resources.

Top      Up      ToC       Page 49 
   7.  RDMA Read Request Queue resource consumption MUST be controlled
       by the Privileged Resource Manager such that RDMAP/DDP Streams
       that do not share Partial Mutual Trust do not share RDMA Read
       Request Queue resources.

   8.  If an RNIC provides the ability to share receive buffers across
       multiple Streams, the combination of the RNIC and the Privileged
       Resource Manager MUST be able to detect if the Remote Peer is
       attempting to consume more than its fair share of resources so
       that the Local Peer can apply countermeasures to detect and
       prevent the attack.

8.2.  Security Services for RDMAP

   RDMAP is using IP-based network services to control, read, and write
   data buffers over the network.  Therefore, all exchanged control and
   data packets are vulnerable to spoofing, tampering, and information
   disclosure attacks.

   RDMAP Streams that are subject to impersonation attacks or Stream
   hijacking attacks can be authenticated, have their integrity
   protected, and be protected from replay attacks.  Furthermore,
   confidentiality protection can be used to protect from eavesdropping.

8.2.1.  Available Security Services

   The IPsec protocol suite [RFC2401] defines strong countermeasures to
   protect an IP stream from those attacks.  Several levels of
   protection can guarantee session confidentiality, per-packet source
   authentication, per-packet integrity, and correct packet sequencing.

   RDMAP security may also profit from SSL or TLS security services
   provided for TCP-based ULPs [RFC4346].  Used underneath RDMAP, these
   security services also provide for stream authentication, data
   integrity, and confidentiality.  As discussed in [RDMASEC],
   limitations on the maximum packet length to be carried over the
   network and potentially inefficient out-of-order packet processing at
   the data sink make SSL and TLS less appropriate for RDMAP than IPsec.

   If SSL is layered on top of RDMAP, SSL does not protect the RDMAP
   headers.  Thus, a man-in-the-middle attack can still occur by
   modifying the RDMAP header to incorrectly place the data into the
   wrong buffer, thus effectively corrupting the data stream.

   By remaining independent of ULP and LLP security protocols, RDMAP
   will benefit from continuing improvements at those layers.  Users are
   provided flexibility to adapt to their specific security requirements
   and the ability to adapt to future security challenges.  Given this,

Top      Up      ToC       Page 50 
   the vulnerabilities of RDMAP to active third-party interference are
   no greater than any other protocol running over an LLP such as TCP or
   SCTP.

8.2.2.  Requirements for IPsec Services for RDMAP

   Because IPsec is designed to secure arbitrary IP packet streams,
   including streams where packets are lost, RDMAP can run on top of
   IPsec without any change.  IPsec packets are processed (e.g.,
   integrity checked and possibly decrypted) in the order they are
   received, and an RDMAP Data Sink will process the decrypted RDMA
   Messages contained in these packets in the same manner as RDMA
   Messages contained in unsecured IP packets.

   The IP Storage working group has defined the normative IPsec
   requirements for IP Storage [RFC3723].  Portions of this
   specification are applicable to the RDMAP.  In particular, a
   compliant implementation of IPsec services for RDMAP MUST meet the
   requirements as outlined in Section 2.3 of [RFC3723].  Without
   replicating the detailed discussion in [RFC3723], this includes the
   following requirements:

   1.  The implementation MUST support IPsec ESP [RFC2406], as well as
       the replay protection mechanisms of IPsec.  When ESP is utilized,
       per-packet data origin authentication, integrity, and replay
       protection MUST be used.

   2.  It MUST support ESP in tunnel mode and MAY implement ESP in
       transport mode.

   3.  It MUST support IKE [RFC2409] for peer authentication,
       negotiation of security associations, and key management, using
       the IPsec DOI [RFC2407].

   4.  It MUST NOT interpret the receipt of a IKE Phase 2 delete message
       as a reason for tearing down the RDMAP stream.  Since IPsec
       acceleration hardware may only be able to handle a limited number
       of active IKE Phase 2 SAs, idle SAs may be dynamically brought
       down, and a new SA be brought up again, if activity resumes.

   5.  It MUST support peer authentication using a pre-shared key, and
       MAY support certificate-based peer authentication using digital
       signatures.  Peer authentication using the public key encryption
       methods [RFC2409] SHOULD NOT be used.

Top      Up      ToC       Page 51 
   6.  It MUST support IKE Main Mode and SHOULD support Aggressive Mode.
       IKE Main Mode with pre-shared key authentication SHOULD NOT be
       used when either of the peers uses a dynamically assigned IP
       address.

   7.  When digital signatures are used to achieve authentication,
       either IKE Main Mode or IKE Aggressive Mode MAY be used.  In
       these cases, an IKE negotiator SHOULD use IKE Certificate Request
       Payload(s) to specify the certificate authority (or authorities)
       that are trusted in accordance with its local policy.  IKE
       negotiators SHOULD check the pertinent Certificate Revocation
       List (CRL) before accepting a PKI certificate for use in IKE's
       authentication procedures.

   8.  Access to locally stored secret information (pre-shared or
       private key for digital signing) must be suitably restricted,
       since compromise of the secret information nullifies the security
       properties of the IKE/IPsec protocols.

   9.  It MUST follow the guidelines of Section 2.3.4 of [RFC3723] on
       the setting of IKE parameters to achieve a high level of
       interoperability without requiring extensive configuration.

   Furthermore, implementation and deployment of the IPsec services for
   RDDP should follow the Security Considerations outlined in Section 5
   of [RFC3723].

9.  IANA Considerations

   This document requests no direct action from IANA.  The following
   consideration is listed here as commentary.

   If RDMAP was enabled a priori for a ULP by connecting to a well-known
   port, this well-known port would be registered for the RDMAP with
   IANA.  The registration of the well-known port will be the
   responsibility of the ULP specification.

Top      Up      ToC       Page 52 
10.  References

10.1.  Normative References

   [DDP]     Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
             Data Placement over Reliable Transports", RFC 5041, October
             2007.

   [iSER]    Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H.,
             and P. Thaler, "Internet Small Computer System Interface
             (iSCSI) Extensions for Remote Direct Memory Access (RDMA)"
             RFC 5046, October 2007.

   [MPA]     Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
             Carrier, "Marker PDU Aligned Framing for TCP
             Specification", RFC 5044, October 2007.

   [RDMASEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement
             Protocol (DDP) / Remote Direct Memory Access Protocol
             (RDMAP) Security", RFC 5042, October 2007.

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2406] Kent, S. and R. Atkinson, "IP Encapsulating Security
             Payload (ESP)", RFC 2406, November 1998.

   [RFC2407] Piper, D., "The Internet IP Security Domain of
             Interpretation of ISAKMP", RFC 2407, November 1998.

   [RFC2409] Harkins, D. and D. Carrel, "The Internet Key Exchange
             (IKE)", RFC 2409, November 1998.

   [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
             Travostino, "Securing Block Storage Protocols over IP", RFC
             3723, April 2004.

   [RFC2401] Kent, S. and R. Atkinson, "Security Architecture for the
             Internet Protocol", RFC 2401, November 1998.

   [SCTP]    Stewart, R., Ed., "Stream Control Transmission Protocol",
             RFC 4960, September 2007.

   [TCP]     Postel, J., "Transmission Control Protocol", STD 7, RFC
             793, September 1981.

Top      Up      ToC       Page 53 
10.2.  Informative References

   [RFC4301] Kent, S. and K. Seo, "Security Architecture for the
             Internet Protocol", RFC 4301, December 2005.

   [RFC4303] Kent, S., "IP Encapsulating Security Payload (ESP)", RFC
             4303, December 2005.

   [RFC4306] Kaufman, C., "Internet Key Exchange (IKEv2) Protocol", RFC
             4306, December 2005.

   [RFC4346] Dierks, T. and E. Rescorla, "The TLS Protocol Version 1.1",
             RFC 4346, April 2006.

   [RFC4835] Manral, V., "Cryptographic Algorithm Implementation
             Requirements for Encapsulating Security Payload (ESP) and
             Authentication Header (AH)", RFC 4835, April 2007.

Top      Up      ToC       Page 54 
Appendix A.  DDP Segment Formats for RDMA Messages

   This appendix is for information only and is NOT part of the
   standard. It simply depicts the DDP Segment format for the various
   RDMA Messages.

A.1.  DDP Segment for RDMA Write

   The following figure depicts an RDMA Write, DDP Segment:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |   DDP Control | RDMA Control  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Data Sink STag                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                   Data Sink Tagged Offset                     |
   +                                                               +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                   RDMA Write ULP Payload                      |
   //                                                             //
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

            Figure 11:  RDMA Write, DDP Segment Format

Top      Up      ToC       Page 55 
A.2.  DDP Segment for RDMA Read Request

   The following figure depicts an RDMA Read Request, DDP Segment:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |  DDP Control  | RDMA Control  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Reserved (Not Used)                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |              DDP (RDMA Read Request) Queue Number             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        DDP (RDMA Read Request) Message Sequence Number        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |             DDP (RDMA Read Request) Message Offset            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     Data Sink STag (SinkSTag)                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   +                  Data Sink Tagged Offset (SinkTO)             +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  RDMA Read Message Size (RDMARDSZ)            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     Data Source STag (SrcSTag)                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   +                 Data Source Tagged Offset (SrcTO)             +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

            Figure 12: RDMA Read Request, DDP Segment format

Top      Up      ToC       Page 56 
A.3.  DDP Segment for RDMA Read Response

   The following figure depicts an RDMA Read Response, DDP Segment:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |  DDP Control  | RDMA Control  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Data Sink STag                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                   Data Sink Tagged Offset                     |
   +                                                               +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                RDMA Read Response ULP Payload                 |
   //                                                             //
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

           Figure 13: RDMA Read Response, DDP Segment Format

A.4.  DDP Segment for Send and Send with Solicited Event

   The following figure depicts a Send and Send with Solicited
   Request, DDP Segment:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |  DDP Control  | RDMA Control  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Reserved (Not Used)                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       (Send) Queue Number                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                 (Send) Message Sequence Number                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      (Send) Message Offset                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Send ULP Payload                        |
   //                                                             //
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Figure 14: Send and Send with Solicited Event, DDP Segment Format

Top      Up      ToC       Page 57 
A.5.  DDP Segment for Send with Invalidate and Send with SE and
      Invalidate

   The following figure depicts a Send with Invalidate and Send with
   Solicited and Invalidate Request, DDP Segment:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |   DDP Control | RDMA Control  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Invalidate STag                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       (Send) Queue Number                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                 (Send) Message Sequence Number                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      (Send) Message Offset                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Send ULP Payload                        |
   //                                                             //
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Figure 15: Send with Invalidate and Send with SE and Invalidate,
                            DDP Segment Format

Top      Up      ToC       Page 58 
A.6.  DDP Segment for Terminate

   The following figure depicts a Terminate, DDP Segment:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   |   DDP Control | RDMA Control  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Reserved (Not Used)                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                   DDP (Terminate) Queue Number                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |             DDP (Terminate) Message Sequence Number           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  DDP (Terminate) Message Offset               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Terminate Control             |      Reserved           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  DDP Segment Length (if any)  |                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
   |                                                               |
   +                                                               +
   |                 Terminated DDP Header (if any)                |
   +                                                               +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   //                                                             //
   |                 Terminated RDMA Header (if any)               |
   +                                                               +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

               Figure 16: Terminate, DDP Segment Format

Top      Up      ToC       Page 59 
Appendix B.  Ordering and Completion Table

   The following table summarizes the ordering relationships that are
   defined in Section 5.5, "Ordering and Completions", from the
   standpoint of the local peer issuing the two Operations.  Note that
   in the table that follows, Send includes Send, Send with Invalidate,
   Send with Solicited Event, and Send with Solicited Event and
   Invalidate.

   ------+-------+----------------+----------------+----------------
   First | Later | Placement      | Placement      | Ordering
    Op   | Op    | guarantee at   | guarantee at   | guarantee at
         |       | Remote Peer    | Local Peer     | Remote Peer
         |       |                |                |
   ------+-------+----------------+----------------+----------------
   Send  | Send  | No placement   | Not applicable | Completed in
         |       | guarantee. If  |                | order.
         |       | guarantee is   |                |
         |       | necessary, see |                |
         |       | footnote 1.    |                |
   ------+-------+----------------+----------------+----------------
   Send  | RDMA  | No placement   | Not applicable | Not applicable
         | Write | guarantee. If  |                |
         |       | guarantee is   |                |
         |       | necessary, see |                |
         |       | footnote 1.    |                |
   ------+-------+----------------+----------------+----------------
   Send  | RDMA  | No placement   | RDMA Read      | RDMA Read
         | Read  | guarantee      | Response       | Response
         |       | between Send   | Payload will   | Message will
         |       | Payload and    | not be placed  | not be
         |       | RDMA Read      | at the local   | generated until
         |       | Request Header | peer until the | Send has been
         |       |                | Send Payload is| Completed
         |       |                | placed at the  |
         |       |                | Remote Peer    |
   ------+-------+----------------+----------------+----------------
   RDMA  | Send  | No placement   | Not applicable | Not applicable
   Write |       | guarantee. If  |                |
         |       | guarantee is   |                |
         |       | necessary, see |                |
         |       | footnote 1.    |                |

Top      Up      ToC       Page 60 
   ------+-------+----------------+----------------+----------------
   RDMA  | RDMA  | No placement   | Not applicable | Not applicable
   Write | Write | guarantee. If  |                |
         |       | guarantee is   |                |
         |       | necessary, see |                |
         |       | footnote 1.    |                |
   ------+-------+----------------+----------------+----------------
   RDMA  | RDMA  | No placement   | RDMA Read      | Not applicable
   Write | Read  | guarantee      | Response       |
         |       | between RDMA   | Payload will   |
         |       | Write Payload  | not be placed  |
         |       | and RDMA Read  | at the local   |
         |       | Request Header | peer until the |
         |       |                | RDMA Write     |
         |       |                | Payload is     |
         |       |                | placed at the  |
         |       |                | Remote Peer    |
   ------+-------+----------------+----------------+----------------
   RDMA  | Send  | No placement   | Send Payload   | Not applicable
   Read  |       | guarantee      | may be placed  |
         |       | between RDMA   | at the remote  |
         |       | Read Request   | peer before the|
         |       | Header and Send| RDMA Read      |
         |       | payload        | Response is    |
         |       |                | generated.     |
         |       |                | If guarantee is|
         |       |                | necessary, see |
         |       |                | footnote 2.    |
   ------+-------+----------------+----------------+----------------
   RDMA  | RDMA  | No placement   | RDMA Write     | Not applicable
   Read  | Write | guarantee      | Payload may be |
         |       | between RDMA   | placed at the  |
         |       | Read Request   | Remote Peer    |
         |       | Header and RDMA| before the RDMA|
         |       | Write payload  | Read Response  |
         |       |                | is generated.  |
         |       |                | If guarantee is|
         |       |                | necessary, see |
         |       |                | footnote 2.    |

Top      Up      ToC       Page 61 
   ------+-------+----------------+----------------+----------------
   RDMA  | RDMA  | No placement   | No placement   | Second RDMA
   Read  | Read  | guarantee of   | guarantee of   | Read Response
         |       | the two RDMA   | the two RDMA   | will not be
         |       | Read Request   | Read Response  | generated until
         |       | Headers        | Payloads.      | first RDMA Read
         |       | Additionally,  |                | Response is
         |       | there is no    |                | generated.
         |       | guarantee that |                |
         |       | the Tagged     |                |
         |       | Buffers        |                |
         |       | referenced in  |                |
         |       | the RDMA Read  |                |
         |       | will be read in|                |
         |       | order          |                |

                    Figure 17: Operation Ordering

   Footnote 1:  If the guarantee is necessary, a ULP may insert an RDMA
   Read operation and wait for it to complete to act as a Fence.

   Footnote 2:  If the guarantee is necessary, a ULP may wait for the
   RDMA Read operation to complete before performing the Send.

Appendix C.  Contributors

   Dwight Barron
   Hewlett-Packard Company
   20555 SH 249
   Houston, TX  77070-2698 USA
   Phone:  281-514-2769
   EMail:  dwight.barron@hp.com


   Caitlin Bestler
   Broadcom Corporation
   16215 Alton Parkway
   Irvine, CA  92619-7013 USA
   Phone:  949-926-6383
   EMail:  caitlinb@broadcom.com


   John Carrier
   Cray, Inc.
   411 First Avenue S, Suite 600
   Seattle, WA  98104-2860 USA
   Phone: 206-701-2090
   EMail: carrier@cray.com

Top      Up      ToC       Page 62 
   Ted Compton
   EMC Corporation
   Research Triangle Park, NC  27709 USA
   Phone: 919-248-6075
   EMail: compton_ted@emc.com


   Uri Elzur
   Broadcom Corporation
   16215 Alton Parkway
   Irvine, California  92619-7013 USA
   Phone: +1 (949) 585-6432
   EMail: Uri@Broadcom.com


   Hari Ghadia
   Gen10 Technology, Inc.
   1501 W Shady Grove Road
   Grand Prairie, TX 75050
   Phone: (972) 301 3630
   EMail: hghadia@gen10technology.com


   Howard C. Herbert
   Intel Corporation
   MS CH7-404
   5000 West Chandler Blvd.
   Chandler, Arizona  85226
   Phone: 480-554-3116
   EMail: howard.c.herbert@intel.com


   Mike Ko
   IBM
   650 Harry Rd.
   San Jose, CA  95120
   Phone: (408) 927-2085
   EMail: mako@us.ibm.com


   Mike Krause
   Hewlett-Packard Company
   43LN
   19410 Homestead Road
   Cupertino, CA  95014 USA
   Phone: 408-447-3191
   EMail: krause@cup.hp.com

Top      Up      ToC       Page 63 
   Dave Minturn
   Intel Corporation
   MS JF1-210
   5200 North East Elam Young Parkway
   Hillsboro, Oregon  97124
   Phone: 503-712-4106
   EMail: dave.b.minturn@intel.com


   Mike Penna
   Broadcom Corporation
   16215 Alton Parkway
   Irvine, California  92619-7013 USA
   Phone: +1 (949) 926-7149
   EMail: MPenna@Broadcom.com


   Jim Pinkerton
   Microsoft, Inc.
   One Microsoft Way
   Redmond, WA  98052 USA
   EMail:  jpink@microsoft.com


   Hemal Shah
   Broadcom Corporation
   5300 California Avenue
   Irvine, CA 92617 USA
   Phone: +1 (949) 926-6941
   EMail: hemal@broadcom.com


   Allyn Romanow
   Cisco Systems
   170 W Tasman Drive
   San Jose, CA  95134 USA
   Phone: +1 408 525 8836
   EMail: allyn@cisco.com


   Tom Talpey
   Network Appliance
   1601 Trapelo Road #16
   Waltham, MA  02451 USA
   Phone: +1 (781) 768-5329
   EMail: thomas.talpey@netapp.com

Top      Up      ToC       Page 64 
   Patricia Thaler
   Broadcom Corporation
   16215 Alton Parkway
   Irvine, CA  92619-7013 USA
   Phone: +1-916-570-2707
   EMail: pthaler@broadcom.com


   Jim Wendt
   Hewlett-Packard Company
   8000 Foothills Boulevard MS 5668
   Roseville, CA  95747-5668 USA
   Phone: +1 916 785 5198
   EMail: jim_wendt@hp.com


   Madeline Vega
   IBM
   11400 Burnet Rd. Bld.45-2L-007
   Austin, TX  78758 USA
   Phone: 512-838-7739
   EMail: mvega1@us.ibm.com


   Claudia Salzberg
   IBM
   11501 Burnet Rd. Bld.902-5B-014
   Austin, TX  78758 USA
   Phone: 512-838-5156
   EMail: salzberg@us.ibm.com

Top      Up      ToC       Page 65 
Authors' Addresses

   Renato J. Recio
   IBM Corp.
   11501 Burnett Road
   Austin, TX  78758 USA
   Phone: 512-838-3685
   EMail: recio@us.ibm.com


   Bernard Metzler
   IBM Research GmbH
   Zurich Research Laboratory
   Saeumerstrasse 4
   CH-8803 Rueschlikon, Switzerland
   Phone: +41 44 724 8605
   EMail: bmt@zurich.ibm.com


   Paul R. Culley
   Hewlett-Packard Company
   20555 SH 249
   Houston, TX  77070-2698 USA
   Phone: 281-514-5543
   EMail: paul.culley@hp.com


   Jeff Hilland
   Hewlett-Packard Company
   20555 SH 249
   Houston, TX  77070-2698 USA
   Phone: 281-514-9489
   EMail: jeff.hilland@hp.com


   Dave Garcia
   24100 Hutchinson Rd.
   Los Gatos, CA 95033 USA
   Phone: +1 (831) 247-4464
   Email: Dave.Garcia@StanfordAlumni.org

Top      Up      ToC       Page 66 
Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
   OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   ietf-ipr@ietf.org.