7.1. Local ULP Attacking a Shared CQ
DOS attacks against a Shared Completion Queue (CQ - see Section
2.2.6, Completion Queues) can be caused by either the local ULP or
the Remote Peer if either attempts to cause more completions than its
fair share of the number of entries; thus, potentially starving
another unrelated ULP such that no Completion Queue entries are
A Completion Queue entry can potentially be maliciously consumed by a
completion from the Send Queue or a completion from the Receive
Queue. In the former, the attacker is the local ULP. In the latter,
the attacker is the Remote Peer.
A form of attack can occur where the local ULPs can consume resources
on the CQ. A local ULP that is slow to free resources on the CQ by
not reaping the completion status quickly enough could stall all
other local ULPs attempting to use that CQ.
For these reasons, an RNIC MUST NOT enable sharing a CQ across ULPs
that do not share Partial Mutual Trust.
7.2. Local Peer Attacking the RDMA Read Request Queue
If RDMA Read Request Queue resources are pooled across multiple
Streams, one attack is if the local ULP attempts to unfairly allocate
RDMA Read Request Queue resources for its Streams. For example, a
local ULP attempts to allocate all available resources on a specific
RDMA Read Request Queue for its Streams, thereby denying the resource
to ULPs sharing the RDMA Read Request Queue. The same type of
argument applies even if the RDMA Read Request is not shared, but a
local ULP attempts to allocate all the RNIC's resources when the
queue is created.
Thus, access to interfaces that allocate RDMA Read Request Queue
entries MUST be restricted to a trusted Local Peer, such as a
Privileged Resource Manager. The Privileged Resource Manager SHOULD
prevent a local ULP from allocating more than its fair share of
7.3. Local ULP Attacking the PTT and STag Mapping
If a Non-Privileged ULP is able to directly manipulate the RNIC Page
Translation Tables (which translate from an STag to a host address),
it is possible that the Non-Privileged ULP could point the Page
Translation Table at an unrelated Stream's or ULP's buffers and,
thereby, be able to gain access to information of the unrelated
As discussed in Section 2, Architectural Model, introduction of a
Privileged Resource Manager to arbitrate the mapping requests is an
effective countermeasure. This enables the Privileged Resource
Manager to ensure that a local ULP can only initialize the Page
Translation Table (PTT) to point to its own buffers.
Thus, if Non-Privileged ULPs are supported, the Privileged Resource
Manager MUST verify that the Non-Privileged ULP has the right to
access a specific Data Buffer before allowing an STag for which the
ULP has access rights to be associated with a specific Data Buffer.
This can be done when the Page Translation Table is initialized to
access the Data Buffer or when the STag is initialized to point to a
group of Page Translation Table entries, or both.
8. Security considerations
Please see Sections 5, Attacks That Can be Mitigated with End-to-End
Security; Section 6, Attacks from Remote Peers; and Section 7,
Attacks from Local Peers, for a detailed analysis of attacks and
normative countermeasures to mitigate the attacks.
Additionally, the appendices provide a summary of the security
requirements for specific audiences. Appendix A, ULP Issues for RDDP
Client/Server Protocols, provides a summary of implementation issues
and requirements for applications that implement a traditional
client/server style of interaction. It provides additional insight
and applicability of the normative text in Sections 5, 6, and 7.
Appendix B, Summary of RNIC and ULP Implementation Requirements,
provides a convenient summary of normative requirements for
9. IANA Considerations
IANA considerations are not addressed by this document. Any IANA
considerations resulting from the use of DDP or RDMA must be
addressed in the relevant standards.
10.1. Normative References
[DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley,
"Direct Data Placement over Reliable Transports", RFC
5041, October 2007.
[RDMAP] Recio, R., Culley, P., Garcia, D., and J. Hilland, "A
Remote Direct Memory Access Protocol Specification",
RFC 5040, October 2007.
"RDMA enabled NIC (RNIC) Verbs Overview", slide
presentation by Renato Recio, April 2003,
[RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC
Text on Security Considerations", BCP 72, RFC 3552,
[INFINIBAND] "InfiniBand Architecture Specification Volume 1",
release 1.2, InfiniBand Trade Association standard,
<http://www.infinibandta.org/specs>. Verbs are
documented in chapter 11.
[DTLS] Rescorla, E. and N. Modadugu, "Datagram Transport Layer
Security", RFC 4347, April 2006.
[iSCSI] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M.,
and E. Zeidner, "Internet Small Computer Systems
Interface (iSCSI)", RFC 3720, April 2004.
[iSER] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah,
H., and P. Thaler, "Internet Small Computer System
Interface (iSCSI) Extensions for Remote Direct Memory
Access (RDMA)", RFC 5046, October 2007.
[NFSv4] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Beame, C., Eisler, M., and D. Noveck, "Network File
System (NFS) version 4 Protocol", RFC 3530, April 2003.
[NFSv4.1] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"NFSv4 Minor Version 1", Work in Progress, September
Appendix A. ULP Issues for RDDP Client/Server Protocols
This section is a normative appendix to the document that is focused
on client/server ULP implementation requirements to ensure a secure
The prior sections outlined specific attacks and their
countermeasures. This section summarizes the attacks and
countermeasures that have been defined in the prior section, which
are applicable to creation of a secure ULP (e.g., application)
server. A ULP server is defined as a ULP that must be able to
communicate with many clients that do not necessarily have a trust
relationship with each other, and to ensure that each client cannot
attack another client through server interactions. Further, the
server may wish to use multiple Streams to communicate with a
specific client, and those Streams may share mutual trust. Note that
this section assumes a compliant RNIC and Privileged Resource Manager
implementation - thus, it focuses specifically on ULP server (e.g.,
application) implementation issues.
All of the prior section's details on attacks and countermeasures
apply to the server; thus, requirements that are repeated in this
section use non-normative "must", "should", and "may". In some
cases, normative SHOULD statements for the ULP from the main body of
this document are made MUST statements for the ULP server because the
operating conditions can be refined to make the motives for a SHOULD
inapplicable. If a prior SHOULD is changed to a MUST in this
section, it is explicitly noted and it uses uppercase normative
The following list summarizes the relevant attacks that clients can
mount on the shared server by re-stating the previous normative
statements to be client/server specific. Note that each
client/server ULP may employ explicit RDMA Operations (RDMA Read,
RDMA Write) in differing fashions. Therefore, where appropriate,
"Local ULP", "Local Peer", and "Remote Peer" are used in place of
"server" or "client", in order to retain full generality of each
* Sections 5.1.1 to 5.1.3. For protection against many forms of
spoofing attacks, enable IPsec.
* Section 6.1.1, Using an STag on a Different Stream. To ensure
that one client cannot access another client's data via use of
the other client's STag, the server ULP must either scope an
STag to a single Stream or use a unique Protection Domain per
client. If a single client has multiple Streams that share
Partial Mutual Trust, then the STag can be shared between the
associated Streams by using a single Protection Domain among
the associated Streams (see Section 5.4.4, ULPs That Provide
Security, for additional issues). To prevent unintended
sharing of STags within the associated Streams, a server ULP
should use STags in such a fashion that it is difficult to
predict the next allocated STag number.
* 6.2.2 Modifying a Buffer after Indication. Before the local
ULP operates on a buffer that was written by the Remote Peer
using an RDMA Write or RDMA Read, the local ULP MUST ensure the
buffer can no longer be modified by invalidating the STag for
remote access (note that this is stronger than the SHOULD in
Section 6.2.2). This can be done either by explicitly revoking
remote access rights for the STag when the Remote Peer
indicates the operation has completed, or by checking to make
sure the Remote Peer Invalidated the STag through the RDMAP
Invalidate capability. If the Remote Peer did not invalidate
the STag, the local ULP then explicitly revokes the STag remote
* Information Disclosure
* 6.3.2, Using RDMA Read to Access Stale Data. In a general
purpose server environment, there is no compelling rationale
not to require a buffer to be initialized before remote read is
enabled (and an enormous downside of unintentionally sharing
data). Thus, a local ULP MUST (this is stronger than the SHOULD
in Section 6.3.2) ensure that no stale data is contained in a
buffer before remote read access rights are granted to a Remote
Peer (this can be done by zeroing the contents of the memory,
* 6.3.3, Accessing a Buffer after the Transfer. This mitigation
is already covered by Section 6.2.2 (above).
* 6.3.4, Accessing Unintended Data with a Valid STag. The ULP
must set the base and bounds of the buffer when the STag is
initialized to expose only the data to be retrieved.
* 6.3.5, RDMA Read into an RDMA Write Buffer. If a peer only
intends a buffer to be exposed for remote write access, it must
set the access rights to the buffer to only enable remote write
* 6.3.6, Using Multiple STags That Alias to the Same Buffer. The
requirement in Section 6.1.1 (above) mitigates this attack. A
server buffer is exposed to only one client at a time to ensure
that no information disclosure or information tampering occurs
* 5.3, Network-Based Eavesdropping. Confidentiality services
should be enabled by the ULP if this threat is a concern.
* Denial of Service
* 220.127.116.11, Multiple Streams Sharing Receive Buffers. ULP memory
footprint size can be important for some server ULPs. If a
server ULP is expecting significant network traffic from
multiple clients, using a receive buffer queue per Stream where
there is a large number of Streams can consume substantial
amounts of memory. Thus, a receive queue that can be shared by
multiple Streams is attractive.
However, because of the attacks outlined in this section,
sharing a single receive queue between multiple clients must
only be done if a mechanism is in place to ensure that one
client cannot consume receive buffers in excess of its limits,
as defined by each ULP. For multiple Streams within a single
client ULP (which presumably shared Partial Mutual Trust), this
added overhead may be avoided.
* 7.1 Local ULP Attacking a Shared CQ. The normative RNIC
mitigations require that the RNIC not enable sharing of a CQ if
the local ULPs do not share Partial Mutual Trust. Thus, while
the ULP is not allowed to enable this feature in an unsafe
mode, if the two local ULPs share Partial Mutual Trust, they
must behave in the following manner:
1) The sizing of the completion queue is based on the size of
the receive queue and send queues, as documented in 18.104.22.168,
Remote or Local Peer Attacking a Shared CQ.
2) The local ULP ensures that CQ entries are reaped frequently
enough to adhere to Section 22.214.171.124's rules.
* 126.96.36.199, Remote or Local Peer Attacking a Shared CQ. There are
two mitigations specified in this section - one requires a
worst-case size of the CQ, and can be implemented entirely
within the Privileged Resource Manager. The second approach
requires cooperation with the local ULP server (not to post too
many buffers), and enables a smaller CQ to be used.
In some server environments, partial trust of the server ULP
(but not the clients) is acceptable; thus, the smaller CQ fully
mitigates the remote attacker. In other environments, the
local server ULP could also contain untrusted elements that can
attack the local machine (or have bugs). In those
environments, the worst-case size of the CQ must be used.
* 188.8.131.52, Attacking the RDMA Read Request Queue. The section
requires a server's Privileged Resource Manager not to allow
sharing of RDMA Read Request Queues across multiple Streams
that do not share Partial Mutual Trust for a ULP that performs
RDMA Read operations to server buffers. However, because the
server ULP knows which of its Streams best share Partial Mutual
Trust, this requirement can be reflected back to the ULP. The
ULP (i.e., server) requirement, in this case, is that it MUST
NOT allow RDMA Read Request Queues to be shared between ULPs
that do not have Partial Mutual Trust.
* 6.4.5, Remote Invalidate an STag Shared on Multiple Streams.
This mitigation is already covered by Section 6.2.2 (above).
Appendix B. Summary of RNIC and ULP Implementation Requirements
This appendix is informative.
Below is a summary of implementation requirements for the RNIC:
* 3 Trust and Resource Sharing
* 5.4.5 Requirements for IPsec Encapsulation of DDP
* 6.1.1 Using an STag on a Different Stream
* 6.2.1 Buffer Overrun - RDMA Write or Read Response
* 6.2.2 Modifying a Buffer after Indication
* 6.4.1 RNIC Resource Consumption
* 184.108.40.206 Multiple Streams Sharing Receive Buffers
* 220.127.116.11 Remote or Local Peer Attacking a Shared CQ
* 18.104.22.168 Attacking the RDMA Read Request Queue
* 6.4.6 Remote Peer Attacking an Unshared CQ
* 6.5 Elevation of Privilege 39
* 7.1 Local ULP Attacking a Shared CQ
* 7.3 Local ULP Attacking the PTT and STag Mapping
Below is a summary of implementation requirements for the ULP above
* 5.3 Information Disclosure - Network-Based Eavesdropping
* 6.1.1 Using an STag on a Different Stream
* 6.2.2 Modifying a Buffer after Indication
* 6.3.2 Using RDMA Read to Access Stale Data
* 6.3.3 Accessing a Buffer after the Transfer
* 6.3.4 Accessing Unintended Data with a Valid STag
* 6.3.5 RDMA Read into an RDMA Write Buffer
* 6.3.6 Using Multiple STags That Alias to the Same Buffer
* 6.4.5 Remote Invalidate an STag Shared on Multiple Streams
Appendix C. Partial Trust Taxonomy
This appendix is informative.
Partial Trust is defined as when one party is willing to assume that
another party will refrain from a specific attack or set of attacks,
the parties are said to be in a state of Partial Trust. Note that
the partially trusted peer may attempt a different set of attacks.
This may be appropriate for many ULPs where any adverse effects of
the betrayal is easily confined and does not place other clients or
ULPs at risk.
The Trust Models described in this section have three primary
distinguishing characteristics. The Trust Model refers to a local
ULP and Remote Peer, which are intended to be the local and remote
ULP instances communicating via RDMA/DDP.
* Local Resource Sharing (yes/no) - When local resources are
shared, they are shared across a grouping of RDMAP/DDP Streams.
If local resources are not shared, the resources are dedicated on
a per Stream basis. Resources are defined in Section 2.2,
Resources. The advantage of not sharing resources between
Streams is that it reduces the types of attacks that are
possible. The disadvantage is that ULPs might run out of
* Local Partial Trust (yes/no) - Local Partial Trust is determined
based on whether the local grouping of RDMAP/DDP Streams (which
typically equates to one ULP or group of ULPs) mutually trust
each other not to perform a specific set of attacks.
* Remote Partial Trust (yes/no) - The Remote Partial Trust level is
determined based on whether the local ULP of a specific RDMAP/DDP
Stream partially trusts the Remote Peer of the Stream (see the
definition of Partial Trust in Section 1, Introduction).
Not all the combinations of the trust characteristics are expected to
be used by ULPs. This document specifically analyzes five ULP Trust
Models that are expected to be in common use. The Trust Models are
* NS-NT - Non-Shared Local Resources, no Local Trust, no Remote
Trust; typically, a server ULP that wants to run in the safest
mode possible. All attack mitigations are in place to ensure
* NS-RT - Non-Shared Local Resources, no Local Trust, Remote
Partial Trust; typically, a peer-to-peer ULP that has, by some
method outside of the scope of this document, authenticated the
Remote Peer. Note that unless some form of key based
authentication is used on a per RDMA/DDP Stream basis, it may not
be possible for man-in-the-middle attacks to occur.
* S-NT - Shared Local Resources, no Local Trust, no Remote Trust;
typically, a server ULP that runs in an untrusted environment
where the amount of resources required is either too large or too
dynamic to dedicate for each RDMAP/DDP Stream.
* S-LT - Shared Local Resources, Local Partial Trust, no Remote
Trust; typically, a ULP that provides a session layer and uses
multiple Streams, to provides additional throughput or fail-over
capabilities. All the Streams within the local ULP partially
trust each other, but do not trust the Remote Peer. This Trust
Model may be appropriate for embedded environments.
* S-T - Shared Local Resources, Local Partial Trust, Remote Partial
Trust; typically, a distributed application, such as a
distributed database application or High Performance Computer
(HPC) application, which is intended to run on a cluster. Due to
extreme resource and performance requirements, the application
typically authenticates with all of its peers and then runs in a
highly trusted environment. The application peers are all in a
single application fault domain and depend on one another to be
well-behaved when accessing data structures. If a trusted Remote
Peer has an implementation defect that results in poor behavior,
the entire application could be corrupted.
Models NS-NT and S-NT, above, are typical for Internet networking -
neither the local ULP nor the Remote Peer is trusted. Sometimes,
optimizations can be done that enable sharing of Page Translation
Tables across multiple local ULPs; thus, Model S-LT can be
advantageous. Model S-T is typically used when resource scaling
across a large parallel ULP makes it infeasible to use any other
model. Resource scaling issues can either be due to performance
around scaling or because there simply are not enough resources.
Model NS-RT is probably the least likely model to be used, but is
presented for completeness.
170 W Tasman Drive
San Jose, CA 95134 USA
Phone: +1 (408) 525-8836
Naval Research Laboratory
Washington, DC 20375 USA
Agilent Technologies, Inc.
1101 Creekside Ridge Drive, #100
Roseville, CA 95678 USA
Phone: +1 (916) 788-5662
NEC Solutions (America), Inc.
7525 166th Ave. N.E., Suite D210
Redmond, WA 98052-7811 USA
Phone: +1 (425) 897-2033
411 First Avenue S, Suite 600
Seattle, WA 98104-2860
Irvine, CA 92618
One Microsoft Way USA
Redmond, WA 98052
Phone: +1 (425) 706-6606
One Microsoft Way
Redmond, WA 98052 USA
Phone: +1 (425) 705-5442
P.O. Box 9245
Brooks, OR 97305
Phone: (503) 642-3950
Full Copyright Statement
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at