Tech-invite3GPPspecsGlossariesIETFRFCsGroupsSIPABNFsWorld Map

RFC 5042


Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security

Part 3 of 3, p. 38 to 52
Prev RFC Part


prevText      Top      Up      ToC       Page 38 
7.  Attacks from Local Peers

   This section describes local attacks that are possible against the
   RDMA system defined in Figure 1 - RDMA Security Model and the RNIC
   Engine resources defined in Section 2.2.

Top      Up      ToC       Page 39 
7.1.  Local ULP Attacking a Shared CQ

   DOS attacks against a Shared Completion Queue (CQ - see Section
   2.2.6, Completion Queues) can be caused by either the local ULP or
   the Remote Peer if either attempts to cause more completions than its
   fair share of the number of entries; thus, potentially starving
   another unrelated ULP such that no Completion Queue entries are

   A Completion Queue entry can potentially be maliciously consumed by a
   completion from the Send Queue or a completion from the Receive
   Queue.  In the former, the attacker is the local ULP.  In the latter,
   the attacker is the Remote Peer.

   A form of attack can occur where the local ULPs can consume resources
   on the CQ.  A local ULP that is slow to free resources on the CQ by
   not reaping the completion status quickly enough could stall all
   other local ULPs attempting to use that CQ.

   For these reasons, an RNIC MUST NOT enable sharing a CQ across ULPs
   that do not share Partial Mutual Trust.

7.2.  Local Peer Attacking the RDMA Read Request Queue

   If RDMA Read Request Queue resources are pooled across multiple
   Streams, one attack is if the local ULP attempts to unfairly allocate
   RDMA Read Request Queue resources for its Streams.  For example, a
   local ULP attempts to allocate all available resources on a specific
   RDMA Read Request Queue for its Streams, thereby denying the resource
   to ULPs sharing the RDMA Read Request Queue.  The same type of
   argument applies even if the RDMA Read Request is not shared, but a
   local ULP attempts to allocate all the RNIC's resources when the
   queue is created.

   Thus, access to interfaces that allocate RDMA Read Request Queue
   entries MUST be restricted to a trusted Local Peer, such as a
   Privileged Resource Manager.  The Privileged Resource Manager SHOULD
   prevent a local ULP from allocating more than its fair share of

7.3.  Local ULP Attacking the PTT and STag Mapping

   If a Non-Privileged ULP is able to directly manipulate the RNIC Page
   Translation Tables (which translate from an STag to a host address),
   it is possible that the Non-Privileged ULP could point the Page
   Translation Table at an unrelated Stream's or ULP's buffers and,
   thereby, be able to gain access to information of the unrelated

Top      Up      ToC       Page 40 
   As discussed in Section 2, Architectural Model, introduction of a
   Privileged Resource Manager to arbitrate the mapping requests is an
   effective countermeasure.  This enables the Privileged Resource
   Manager to ensure that a local ULP can only initialize the Page
   Translation Table (PTT) to point to its own buffers.

   Thus, if Non-Privileged ULPs are supported, the Privileged Resource
   Manager MUST verify that the Non-Privileged ULP has the right to
   access a specific Data Buffer before allowing an STag for which the
   ULP has access rights to be associated with a specific Data Buffer.
   This can be done when the Page Translation Table is initialized to
   access the Data Buffer or when the STag is initialized to point to a
   group of Page Translation Table entries, or both.

8.  Security considerations

   Please see Sections 5, Attacks That Can be Mitigated with End-to-End
   Security; Section 6, Attacks from Remote Peers; and Section 7,
   Attacks from Local Peers, for a detailed analysis of attacks and
   normative countermeasures to mitigate the attacks.

   Additionally, the appendices provide a summary of the security
   requirements for specific audiences.  Appendix A, ULP Issues for RDDP
   Client/Server Protocols, provides a summary of implementation issues
   and requirements for applications that implement a traditional
   client/server style of interaction.  It provides additional insight
   and applicability of the normative text in Sections 5, 6, and 7.
   Appendix B, Summary of RNIC and ULP Implementation Requirements,
   provides a convenient summary of normative requirements for

9.  IANA Considerations

   IANA considerations are not addressed by this document.  Any IANA
   considerations resulting from the use of DDP or RDMA must be
   addressed in the relevant standards.

10.  References

10.1.  Normative References

   [DDP]         Shah, H., Pinkerton, J., Recio, R., and P. Culley,
                 "Direct Data Placement over Reliable Transports", RFC
                 5041, October 2007.

   [RDMAP]       Recio, R., Culley, P., Garcia, D., and J. Hilland, "A
                 Remote Direct Memory Access Protocol Specification",
                 RFC 5040, October 2007.

Top      Up      ToC       Page 41 
   [RFC2401]     Kent, S. and R. Atkinson, "Security Architecture for
                 the Internet Protocol", RFC 2401, November 1998.

   [RFC2402]     Kent, S. and R. Atkinson, "IP Authentication Header",
                 RFC 2402, November 1998.

   [RFC2406]     Kent, S. and R. Atkinson, "IP Encapsulating Security
                 Payload (ESP)", RFC 2406, November 1998.

   [RFC2409]     Harkins, D. and D. Carrel, "The Internet Key Exchange
                 (IKE)", RFC 2409, November 1998.

   [RFC3723]     Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
                 Travostino, "Securing Block Storage Protocols over IP",
                 RFC 3723, April 2004.

   [RFC4960]     Stewart, R., Ed., "Stream Control Transmission
                 Protocol", RFC 4960, September 2007.

   [RFC793]      Postel, J., "Transmission Control Protocol", STD 7, RFC
                 793, September 1981.

10.2.  Informative References

   [RFC4301]     Kent, S. and K. Seo, "Security Architecture for the
                 Internet Protocol", RFC 4301, December 2005.

   [RFC4346]     Dierks, T. and E. Rescorla, "The Transport Layer
                 Security (TLS) Protocol Version 1.1", RFC 4346, April

   [RFC4949]     Shirey, R., "Internet Security Glossary, Version 2",
                 RFC 4949, August 2007.

                 Bestler, C. and L. Coene, "Applicability of Remote
                 Direct Memory Access Protocol (RDMA) and Direct Data
                 Placement (DDP)", RFC 5045, October 2007.

                 Williams, N., "On the Use of Channel Bindings to Secure
                 Channels", Work in Progress, July 2004.

   [VERBS-RDMAC] "RDMA Protocol Verbs Specification", RDMA Consortium
                 standard, April 2003, <

Top      Up      ToC       Page 42 
                 "RDMA enabled NIC (RNIC) Verbs Overview", slide
                 presentation by Renato Recio, April 2003,

   [RFC3552]     Rescorla, E. and B. Korver, "Guidelines for Writing RFC
                 Text on Security Considerations", BCP 72, RFC 3552,
                 July 2003.

   [INFINIBAND]  "InfiniBand Architecture Specification Volume 1",
                 release 1.2, InfiniBand Trade Association standard,
                 <>.  Verbs are
                 documented in chapter 11.

   [DTLS]        Rescorla, E. and N. Modadugu, "Datagram Transport Layer
                 Security", RFC 4347, April 2006.

   [iSCSI]       Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M.,
                 and E. Zeidner, "Internet Small Computer Systems
                 Interface (iSCSI)", RFC 3720, April 2004.

   [iSER]        Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah,
                 H., and P. Thaler, "Internet Small Computer System
                 Interface (iSCSI) Extensions for Remote Direct Memory
                 Access (RDMA)", RFC 5046, October 2007.

   [NFSv4]       Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
                 Beame, C., Eisler, M., and D. Noveck, "Network File
                 System (NFS) version 4 Protocol", RFC 3530, April 2003.

   [NFSv4.1]     Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
                 "NFSv4 Minor Version 1", Work in Progress, September

Top      Up      ToC       Page 43 
Appendix A. ULP Issues for RDDP Client/Server Protocols

   This section is a normative appendix to the document that is focused
   on client/server ULP implementation requirements to ensure a secure
   server implementation.

   The prior sections outlined specific attacks and their
   countermeasures.  This section summarizes the attacks and
   countermeasures that have been defined in the prior section, which
   are applicable to creation of a secure ULP (e.g., application)
   server.  A ULP server is defined as a ULP that must be able to
   communicate with many clients that do not necessarily have a trust
   relationship with each other, and to ensure that each client cannot
   attack another client through server interactions.  Further, the
   server may wish to use multiple Streams to communicate with a
   specific client, and those Streams may share mutual trust.  Note that
   this section assumes a compliant RNIC and Privileged Resource Manager
   implementation - thus, it focuses specifically on ULP server (e.g.,
   application) implementation issues.

   All of the prior section's details on attacks and countermeasures
   apply to the server; thus, requirements that are repeated in this
   section use non-normative "must", "should", and "may".  In some
   cases, normative SHOULD statements for the ULP from the main body of
   this document are made MUST statements for the ULP server because the
   operating conditions can be refined to make the motives for a SHOULD
   inapplicable.  If a prior SHOULD is changed to a MUST in this
   section, it is explicitly noted and it uses uppercase normative

   The following list summarizes the relevant attacks that clients can
   mount on the shared server by re-stating the previous normative
   statements to be client/server specific.  Note that each
   client/server ULP may employ explicit RDMA Operations (RDMA Read,
   RDMA Write) in differing fashions.  Therefore, where appropriate,
   "Local ULP", "Local Peer", and "Remote Peer" are used in place of
   "server" or "client", in order to retain full generality of each

   *   Spoofing

     *   Sections 5.1.1 to 5.1.3.  For protection against many forms of
         spoofing attacks, enable IPsec.

     *   Section 6.1.1, Using an STag on a Different Stream.  To ensure
         that one client cannot access another client's data via use of
         the other client's STag, the server ULP must either scope an
         STag to a single Stream or use a unique Protection Domain per

Top      Up      ToC       Page 44 
         client.  If a single client has multiple Streams that share
         Partial Mutual Trust, then the STag can be shared between the
         associated Streams by using a single Protection Domain among
         the associated Streams (see Section 5.4.4, ULPs That Provide
         Security, for additional issues).  To prevent unintended
         sharing of STags within the associated Streams, a server ULP
         should use STags in such a fashion that it is difficult to
         predict the next allocated STag number.

   *   Tampering

     *   6.2.2 Modifying a Buffer after Indication.  Before the local
         ULP operates on a buffer that was written by the Remote Peer
         using an RDMA Write or RDMA Read, the local ULP MUST ensure the
         buffer can no longer be modified by invalidating the STag for
         remote access (note that this is stronger than the SHOULD in
         Section 6.2.2).  This can be done either by explicitly revoking
         remote access rights for the STag when the Remote Peer
         indicates the operation has completed, or by checking to make
         sure the Remote Peer Invalidated the STag through the RDMAP
         Invalidate capability.  If the Remote Peer did not invalidate
         the STag, the local ULP then explicitly revokes the STag remote
         access rights.

   *   Information Disclosure

     *   6.3.2, Using RDMA Read to Access Stale Data.  In a general
         purpose server environment, there is no compelling rationale
         not to require a buffer to be initialized before remote read is
         enabled (and an enormous downside of unintentionally sharing
         data). Thus, a local ULP MUST (this is stronger than the SHOULD
         in Section 6.3.2) ensure that no stale data is contained in a
         buffer before remote read access rights are granted to a Remote
         Peer (this can be done by zeroing the contents of the memory,
         for example).

     *   6.3.3, Accessing a Buffer after the Transfer.  This mitigation
         is already covered by Section 6.2.2 (above).

     *   6.3.4, Accessing Unintended Data with a Valid STag.  The ULP
         must set the base and bounds of the buffer when the STag is
         initialized to expose only the data to be retrieved.

     *   6.3.5, RDMA Read into an RDMA Write Buffer.  If a peer only
         intends a buffer to be exposed for remote write access, it must
         set the access rights to the buffer to only enable remote write

Top      Up      ToC       Page 45 
     *   6.3.6, Using Multiple STags That Alias to the Same Buffer.  The
         requirement in Section 6.1.1 (above) mitigates this attack.  A
         server buffer is exposed to only one client at a time to ensure
         that no information disclosure or information tampering occurs
         between peers.

     *   5.3, Network-Based Eavesdropping.  Confidentiality services
         should be enabled by the ULP if this threat is a concern.

   *   Denial of Service

     *, Multiple Streams Sharing Receive Buffers.  ULP memory
         footprint size can be important for some server ULPs.  If a
         server ULP is expecting significant network traffic from
         multiple clients, using a receive buffer queue per Stream where
         there is a large number of Streams can consume substantial
         amounts of memory.  Thus, a receive queue that can be shared by
         multiple Streams is attractive.

         However, because of the attacks outlined in this section,
         sharing a single receive queue between multiple clients must
         only be done if a mechanism is in place to ensure that one
         client cannot consume receive buffers in excess of its limits,
         as defined by each ULP.  For multiple Streams within a single
         client ULP (which presumably shared Partial Mutual Trust), this
         added overhead may be avoided.

     *   7.1 Local ULP Attacking a Shared CQ.  The normative RNIC
         mitigations require that the RNIC not enable sharing of a CQ if
         the local ULPs do not share Partial Mutual Trust.  Thus, while
         the ULP is not allowed to enable this feature in an unsafe
         mode, if the two local ULPs share Partial Mutual Trust, they
         must behave in the following manner:

         1) The sizing of the completion queue is based on the size of
         the receive queue and send queues, as documented in,
         Remote or Local Peer Attacking a Shared CQ.

         2) The local ULP ensures that CQ entries are reaped frequently
         enough to adhere to Section's rules.

     *, Remote or Local Peer Attacking a Shared CQ.  There are
         two mitigations specified in this section - one requires a
         worst-case size of the CQ, and can be implemented entirely
         within the Privileged Resource Manager.  The second approach
         requires cooperation with the local ULP server (not to post too
         many buffers), and enables a smaller CQ to be used.

Top      Up      ToC       Page 46 
         In some server environments, partial trust of the server ULP
         (but not the clients) is acceptable; thus, the smaller CQ fully
         mitigates the remote attacker.  In other environments, the
         local server ULP could also contain untrusted elements that can
         attack the local machine (or have bugs).  In those
         environments, the worst-case size of the CQ must be used.

     *, Attacking the RDMA Read Request Queue.  The section
         requires a server's Privileged Resource Manager not to allow
         sharing of RDMA Read Request Queues across multiple Streams
         that do not share Partial Mutual Trust for a ULP that performs
         RDMA Read operations to server buffers.  However, because the
         server ULP knows which of its Streams best share Partial Mutual
         Trust, this requirement can be reflected back to the ULP.  The
         ULP (i.e., server) requirement, in this case, is that it MUST
         NOT allow RDMA Read Request Queues to be shared between ULPs
         that do not have Partial Mutual Trust.

     *   6.4.5, Remote Invalidate an STag Shared on Multiple Streams.
         This mitigation is already covered by Section 6.2.2 (above).

Appendix B. Summary of RNIC and ULP Implementation Requirements

   This appendix is informative.

   Below is a summary of implementation requirements for the RNIC:

   *   3 Trust and Resource Sharing

   *   5.4.5 Requirements for IPsec Encapsulation of DDP

   *   6.1.1 Using an STag on a Different Stream

   *   6.2.1 Buffer Overrun - RDMA Write or Read Response

   *   6.2.2 Modifying a Buffer after Indication

   *   6.4.1 RNIC Resource Consumption

   * Multiple Streams Sharing Receive Buffers

   * Remote or Local Peer Attacking a Shared CQ

   * Attacking the RDMA Read Request Queue

   *   6.4.6 Remote Peer Attacking an Unshared CQ

   *   6.5 Elevation of Privilege 39

Top      Up      ToC       Page 47 
   *   7.1 Local ULP Attacking a Shared CQ

   *   7.3 Local ULP Attacking the PTT and STag Mapping

   Below is a summary of implementation requirements for the ULP above
   the RNIC:

   *   5.3 Information Disclosure - Network-Based Eavesdropping

   *   6.1.1 Using an STag on a Different Stream

   *   6.2.2 Modifying a Buffer after Indication

   *   6.3.2 Using RDMA Read to Access Stale Data

   *   6.3.3 Accessing a Buffer after the Transfer

   *   6.3.4 Accessing Unintended Data with a Valid STag

   *   6.3.5 RDMA Read into an RDMA Write Buffer

   *   6.3.6 Using Multiple STags That Alias to the Same Buffer

   *   6.4.5 Remote Invalidate an STag Shared on Multiple Streams

Appendix C. Partial Trust Taxonomy

   This appendix is informative.

   Partial Trust is defined as when one party is willing to assume that
   another party will refrain from a specific attack or set of attacks,
   the parties are said to be in a state of Partial Trust.  Note that
   the partially trusted peer may attempt a different set of attacks.
   This may be appropriate for many ULPs where any adverse effects of
   the betrayal is easily confined and does not place other clients or
   ULPs at risk.

   The Trust Models described in this section have three primary
   distinguishing characteristics.  The Trust Model refers to a local
   ULP and Remote Peer, which are intended to be the local and remote
   ULP instances communicating via RDMA/DDP.

Top      Up      ToC       Page 48 
   *   Local Resource Sharing (yes/no) - When local resources are
       shared, they are shared across a grouping of RDMAP/DDP Streams.
       If local resources are not shared, the resources are dedicated on
       a per Stream basis.  Resources are defined in Section 2.2,
       Resources.  The advantage of not sharing resources between
       Streams is that it reduces the types of attacks that are
       possible.  The disadvantage is that ULPs might run out of

   *   Local Partial Trust (yes/no) - Local Partial Trust is determined
       based on whether the local grouping of RDMAP/DDP Streams (which
       typically equates to one ULP or group of ULPs) mutually trust
       each other not to perform a specific set of attacks.

   *   Remote Partial Trust (yes/no) - The Remote Partial Trust level is
       determined based on whether the local ULP of a specific RDMAP/DDP
       Stream partially trusts the Remote Peer of the Stream (see the
       definition of Partial Trust in Section 1, Introduction).

   Not all the combinations of the trust characteristics are expected to
   be used by ULPs.  This document specifically analyzes five ULP Trust
   Models that are expected to be in common use.  The Trust Models are
   as follows:

   *   NS-NT - Non-Shared Local Resources, no Local Trust, no Remote
       Trust; typically, a server ULP that wants to run in the safest
       mode possible.  All attack mitigations are in place to ensure
       robust operation.

   *   NS-RT - Non-Shared Local Resources, no Local Trust, Remote
       Partial Trust; typically, a peer-to-peer ULP that has, by some
       method outside of the scope of this document, authenticated the
       Remote Peer.  Note that unless some form of key based
       authentication is used on a per RDMA/DDP Stream basis, it may not
       be possible for man-in-the-middle attacks to occur.

   *   S-NT - Shared Local Resources, no Local Trust, no Remote Trust;
       typically, a server ULP that runs in an untrusted environment
       where the amount of resources required is either too large or too
       dynamic to dedicate for each RDMAP/DDP Stream.

   *   S-LT - Shared Local Resources, Local Partial Trust, no Remote
       Trust; typically, a ULP that provides a session layer and uses
       multiple Streams, to provides additional throughput or fail-over
       capabilities.  All the Streams within the local ULP partially
       trust each other, but do not trust the Remote Peer.  This Trust
       Model may be appropriate for embedded environments.

Top      Up      ToC       Page 49 
   *   S-T - Shared Local Resources, Local Partial Trust, Remote Partial
       Trust; typically, a distributed application, such as a
       distributed database application or High Performance Computer
       (HPC) application, which is intended to run on a cluster.  Due to
       extreme resource and performance requirements, the application
       typically authenticates with all of its peers and then runs in a
       highly trusted environment.  The application peers are all in a
       single application fault domain and depend on one another to be
       well-behaved when accessing data structures.  If a trusted Remote
       Peer has an implementation defect that results in poor behavior,
       the entire application could be corrupted.

   Models NS-NT and S-NT, above, are typical for Internet networking -
   neither the local ULP nor the Remote Peer is trusted.  Sometimes,
   optimizations can be done that enable sharing of Page Translation
   Tables across multiple local ULPs; thus, Model S-LT can be
   advantageous.  Model S-T is typically used when resource scaling
   across a large parallel ULP makes it infeasible to use any other
   model.  Resource scaling issues can either be due to performance
   around scaling or because there simply are not enough resources.
   Model NS-RT is probably the least likely model to be used, but is
   presented for completeness.


   Sara Bitan
   Microsoft Corporation

   Allyn Romanow
   Cisco Systems
   170 W Tasman Drive
   San Jose, CA 95134 USA
   Phone: +1 (408) 525-8836

   Catherine Meadows
   Naval Research Laboratory
   Code 5543
   Washington, DC 20375 USA

Top      Up      ToC       Page 50 
   Patricia Thaler
   Agilent Technologies, Inc.
   1101 Creekside Ridge Drive, #100
   Roseville, CA 95678 USA
   Phone: +1 (916) 788-5662

   James Livingston
   NEC Solutions (America), Inc.
   7525 166th Ave. N.E., Suite D210
   Redmond, WA 98052-7811 USA
   Phone: +1 (425) 897-2033

   John Carrier
   Cray Inc.
   411 First Avenue S, Suite 600
   Seattle, WA 98104-2860
   Phone: 206-701-2090

   Caitlin Bestler
   49 Discovery
   Irvine, CA 92618

   Bernard Aboba
   Microsoft Corporation
   One Microsoft Way USA
   Redmond, WA 98052
   Phone: +1 (425) 706-6606

Top      Up      ToC       Page 51 
Authors' Addresses

   James Pinkerton
   Microsoft Corporation
   One Microsoft Way
   Redmond, WA 98052 USA
   Phone: +1 (425) 705-5442

   Ellen Deleganes
   P.O. Box 9245
   Brooks, OR 97305
   Phone: (503) 642-3950

Top      Up      ToC       Page 52 
Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at