RFC 5042
Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security
7. Attacks from Local Peers
This section describes local attacks that are possible against the RDMA system defined in Figure 1 - RDMA Security Model and the RNIC Engine resources defined in Section 2.2.
7.1. Local ULP Attacking a Shared CQ
DOS attacks against a Shared Completion Queue (CQ - see Section 2.2.6, Completion Queues) can be caused by either the local ULP or the Remote Peer if either attempts to cause more completions than its fair share of the number of entries; thus, potentially starving another unrelated ULP such that no Completion Queue entries are available. A Completion Queue entry can potentially be maliciously consumed by a completion from the Send Queue or a completion from the Receive Queue. In the former, the attacker is the local ULP. In the latter, the attacker is the Remote Peer. A form of attack can occur where the local ULPs can consume resources on the CQ. A local ULP that is slow to free resources on the CQ by not reaping the completion status quickly enough could stall all other local ULPs attempting to use that CQ. For these reasons, an RNIC MUST NOT enable sharing a CQ across ULPs that do not share Partial Mutual Trust.7.2. Local Peer Attacking the RDMA Read Request Queue
If RDMA Read Request Queue resources are pooled across multiple Streams, one attack is if the local ULP attempts to unfairly allocate RDMA Read Request Queue resources for its Streams. For example, a local ULP attempts to allocate all available resources on a specific RDMA Read Request Queue for its Streams, thereby denying the resource to ULPs sharing the RDMA Read Request Queue. The same type of argument applies even if the RDMA Read Request is not shared, but a local ULP attempts to allocate all the RNIC's resources when the queue is created. Thus, access to interfaces that allocate RDMA Read Request Queue entries MUST be restricted to a trusted Local Peer, such as a Privileged Resource Manager. The Privileged Resource Manager SHOULD prevent a local ULP from allocating more than its fair share of resources.7.3. Local ULP Attacking the PTT and STag Mapping
If a Non-Privileged ULP is able to directly manipulate the RNIC Page Translation Tables (which translate from an STag to a host address), it is possible that the Non-Privileged ULP could point the Page Translation Table at an unrelated Stream's or ULP's buffers and, thereby, be able to gain access to information of the unrelated Stream/ULP.
As discussed in Section 2, Architectural Model, introduction of a Privileged Resource Manager to arbitrate the mapping requests is an effective countermeasure. This enables the Privileged Resource Manager to ensure that a local ULP can only initialize the Page Translation Table (PTT) to point to its own buffers. Thus, if Non-Privileged ULPs are supported, the Privileged Resource Manager MUST verify that the Non-Privileged ULP has the right to access a specific Data Buffer before allowing an STag for which the ULP has access rights to be associated with a specific Data Buffer. This can be done when the Page Translation Table is initialized to access the Data Buffer or when the STag is initialized to point to a group of Page Translation Table entries, or both.8. Security considerations
Please see Sections 5, Attacks That Can be Mitigated with End-to-End Security; Section 6, Attacks from Remote Peers; and Section 7, Attacks from Local Peers, for a detailed analysis of attacks and normative countermeasures to mitigate the attacks. Additionally, the appendices provide a summary of the security requirements for specific audiences. Appendix A, ULP Issues for RDDP Client/Server Protocols, provides a summary of implementation issues and requirements for applications that implement a traditional client/server style of interaction. It provides additional insight and applicability of the normative text in Sections 5, 6, and 7. Appendix B, Summary of RNIC and ULP Implementation Requirements, provides a convenient summary of normative requirements for implementers.9. IANA Considerations
IANA considerations are not addressed by this document. Any IANA considerations resulting from the use of DDP or RDMA must be addressed in the relevant standards.10. References
10.1. Normative References
[DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, October 2007. [RDMAP] Recio, R., Culley, P., Garcia, D., and J. Hilland, "A Remote Direct Memory Access Protocol Specification", RFC 5040, October 2007.
[RFC2401] Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998. [RFC2402] Kent, S. and R. Atkinson, "IP Authentication Header", RFC 2402, November 1998. [RFC2406] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload (ESP)", RFC 2406, November 1998. [RFC2409] Harkins, D. and D. Carrel, "The Internet Key Exchange (IKE)", RFC 2409, November 1998. [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F. Travostino, "Securing Block Storage Protocols over IP", RFC 3723, April 2004. [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", RFC 4960, September 2007. [RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.10.2. Informative References
[RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005. [RFC4346] Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.1", RFC 4346, April 2006. [RFC4949] Shirey, R., "Internet Security Glossary, Version 2", RFC 4949, August 2007. [APPLICABILITY] Bestler, C. and L. Coene, "Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Data Placement (DDP)", RFC 5045, October 2007. [NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to Secure Channels", Work in Progress, July 2004. [VERBS-RDMAC] "RDMA Protocol Verbs Specification", RDMA Consortium standard, April 2003, <http://www.rdmaconsortium.org/ home/draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>.
[VERBS-RDMAC-Overview] "RDMA enabled NIC (RNIC) Verbs Overview", slide presentation by Renato Recio, April 2003, <http://www.rdmaconsortium.org/home/ RNIC_Verbs_Overview2.pdf>. [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC Text on Security Considerations", BCP 72, RFC 3552, July 2003. [INFINIBAND] "InfiniBand Architecture Specification Volume 1", release 1.2, InfiniBand Trade Association standard, <http://www.infinibandta.org/specs>. Verbs are documented in chapter 11. [DTLS] Rescorla, E. and N. Modadugu, "Datagram Transport Layer Security", RFC 4347, April 2006. [iSCSI] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004. [iSER] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H., and P. Thaler, "Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)", RFC 5046, October 2007. [NFSv4] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003. [NFSv4.1] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "NFSv4 Minor Version 1", Work in Progress, September 2007.
Appendix A. ULP Issues for RDDP Client/Server Protocols
This section is a normative appendix to the document that is focused on client/server ULP implementation requirements to ensure a secure server implementation. The prior sections outlined specific attacks and their countermeasures. This section summarizes the attacks and countermeasures that have been defined in the prior section, which are applicable to creation of a secure ULP (e.g., application) server. A ULP server is defined as a ULP that must be able to communicate with many clients that do not necessarily have a trust relationship with each other, and to ensure that each client cannot attack another client through server interactions. Further, the server may wish to use multiple Streams to communicate with a specific client, and those Streams may share mutual trust. Note that this section assumes a compliant RNIC and Privileged Resource Manager implementation - thus, it focuses specifically on ULP server (e.g., application) implementation issues. All of the prior section's details on attacks and countermeasures apply to the server; thus, requirements that are repeated in this section use non-normative "must", "should", and "may". In some cases, normative SHOULD statements for the ULP from the main body of this document are made MUST statements for the ULP server because the operating conditions can be refined to make the motives for a SHOULD inapplicable. If a prior SHOULD is changed to a MUST in this section, it is explicitly noted and it uses uppercase normative statements. The following list summarizes the relevant attacks that clients can mount on the shared server by re-stating the previous normative statements to be client/server specific. Note that each client/server ULP may employ explicit RDMA Operations (RDMA Read, RDMA Write) in differing fashions. Therefore, where appropriate, "Local ULP", "Local Peer", and "Remote Peer" are used in place of "server" or "client", in order to retain full generality of each requirement. * Spoofing * Sections 5.1.1 to 5.1.3. For protection against many forms of spoofing attacks, enable IPsec. * Section 6.1.1, Using an STag on a Different Stream. To ensure that one client cannot access another client's data via use of the other client's STag, the server ULP must either scope an STag to a single Stream or use a unique Protection Domain per
client. If a single client has multiple Streams that share
Partial Mutual Trust, then the STag can be shared between the
associated Streams by using a single Protection Domain among
the associated Streams (see Section 5.4.4, ULPs That Provide
Security, for additional issues). To prevent unintended
sharing of STags within the associated Streams, a server ULP
should use STags in such a fashion that it is difficult to
predict the next allocated STag number.
* Tampering
* 6.2.2 Modifying a Buffer after Indication. Before the local
ULP operates on a buffer that was written by the Remote Peer
using an RDMA Write or RDMA Read, the local ULP MUST ensure the
buffer can no longer be modified by invalidating the STag for
remote access (note that this is stronger than the SHOULD in
Section 6.2.2). This can be done either by explicitly revoking
remote access rights for the STag when the Remote Peer
indicates the operation has completed, or by checking to make
sure the Remote Peer Invalidated the STag through the RDMAP
Invalidate capability. If the Remote Peer did not invalidate
the STag, the local ULP then explicitly revokes the STag remote
access rights.
* Information Disclosure
* 6.3.2, Using RDMA Read to Access Stale Data. In a general
purpose server environment, there is no compelling rationale
not to require a buffer to be initialized before remote read is
enabled (and an enormous downside of unintentionally sharing
data). Thus, a local ULP MUST (this is stronger than the SHOULD
in Section 6.3.2) ensure that no stale data is contained in a
buffer before remote read access rights are granted to a Remote
Peer (this can be done by zeroing the contents of the memory,
for example).
* 6.3.3, Accessing a Buffer after the Transfer. This mitigation
is already covered by Section 6.2.2 (above).
* 6.3.4, Accessing Unintended Data with a Valid STag. The ULP
must set the base and bounds of the buffer when the STag is
initialized to expose only the data to be retrieved.
* 6.3.5, RDMA Read into an RDMA Write Buffer. If a peer only
intends a buffer to be exposed for remote write access, it must
set the access rights to the buffer to only enable remote write
access.
* 6.3.6, Using Multiple STags That Alias to the Same Buffer. The
requirement in Section 6.1.1 (above) mitigates this attack. A
server buffer is exposed to only one client at a time to ensure
that no information disclosure or information tampering occurs
between peers.
* 5.3, Network-Based Eavesdropping. Confidentiality services
should be enabled by the ULP if this threat is a concern.
* Denial of Service
* 6.4.3.1, Multiple Streams Sharing Receive Buffers. ULP memory
footprint size can be important for some server ULPs. If a
server ULP is expecting significant network traffic from
multiple clients, using a receive buffer queue per Stream where
there is a large number of Streams can consume substantial
amounts of memory. Thus, a receive queue that can be shared by
multiple Streams is attractive.
However, because of the attacks outlined in this section,
sharing a single receive queue between multiple clients must
only be done if a mechanism is in place to ensure that one
client cannot consume receive buffers in excess of its limits,
as defined by each ULP. For multiple Streams within a single
client ULP (which presumably shared Partial Mutual Trust), this
added overhead may be avoided.
* 7.1 Local ULP Attacking a Shared CQ. The normative RNIC
mitigations require that the RNIC not enable sharing of a CQ if
the local ULPs do not share Partial Mutual Trust. Thus, while
the ULP is not allowed to enable this feature in an unsafe
mode, if the two local ULPs share Partial Mutual Trust, they
must behave in the following manner:
1) The sizing of the completion queue is based on the size of
the receive queue and send queues, as documented in 6.4.3.2,
Remote or Local Peer Attacking a Shared CQ.
2) The local ULP ensures that CQ entries are reaped frequently
enough to adhere to Section 6.4.3.2's rules.
* 6.4.3.2, Remote or Local Peer Attacking a Shared CQ. There are
two mitigations specified in this section - one requires a
worst-case size of the CQ, and can be implemented entirely
within the Privileged Resource Manager. The second approach
requires cooperation with the local ULP server (not to post too
many buffers), and enables a smaller CQ to be used.
In some server environments, partial trust of the server ULP
(but not the clients) is acceptable; thus, the smaller CQ fully
mitigates the remote attacker. In other environments, the
local server ULP could also contain untrusted elements that can
attack the local machine (or have bugs). In those
environments, the worst-case size of the CQ must be used.
* 6.4.3.3, Attacking the RDMA Read Request Queue. The section
requires a server's Privileged Resource Manager not to allow
sharing of RDMA Read Request Queues across multiple Streams
that do not share Partial Mutual Trust for a ULP that performs
RDMA Read operations to server buffers. However, because the
server ULP knows which of its Streams best share Partial Mutual
Trust, this requirement can be reflected back to the ULP. The
ULP (i.e., server) requirement, in this case, is that it MUST
NOT allow RDMA Read Request Queues to be shared between ULPs
that do not have Partial Mutual Trust.
* 6.4.5, Remote Invalidate an STag Shared on Multiple Streams.
This mitigation is already covered by Section 6.2.2 (above).
Appendix B. Summary of RNIC and ULP Implementation Requirements
This appendix is informative.
Below is a summary of implementation requirements for the RNIC:
* 3 Trust and Resource Sharing
* 5.4.5 Requirements for IPsec Encapsulation of DDP
* 6.1.1 Using an STag on a Different Stream
* 6.2.1 Buffer Overrun - RDMA Write or Read Response
* 6.2.2 Modifying a Buffer after Indication
* 6.4.1 RNIC Resource Consumption
* 6.4.3.1 Multiple Streams Sharing Receive Buffers
* 6.4.3.2 Remote or Local Peer Attacking a Shared CQ
* 6.4.3.3 Attacking the RDMA Read Request Queue
* 6.4.6 Remote Peer Attacking an Unshared CQ
* 6.5 Elevation of Privilege 39
* 7.1 Local ULP Attacking a Shared CQ * 7.3 Local ULP Attacking the PTT and STag Mapping Below is a summary of implementation requirements for the ULP above the RNIC: * 5.3 Information Disclosure - Network-Based Eavesdropping * 6.1.1 Using an STag on a Different Stream * 6.2.2 Modifying a Buffer after Indication * 6.3.2 Using RDMA Read to Access Stale Data * 6.3.3 Accessing a Buffer after the Transfer * 6.3.4 Accessing Unintended Data with a Valid STag * 6.3.5 RDMA Read into an RDMA Write Buffer * 6.3.6 Using Multiple STags That Alias to the Same Buffer * 6.4.5 Remote Invalidate an STag Shared on Multiple StreamsAppendix C. Partial Trust Taxonomy
This appendix is informative. Partial Trust is defined as when one party is willing to assume that another party will refrain from a specific attack or set of attacks, the parties are said to be in a state of Partial Trust. Note that the partially trusted peer may attempt a different set of attacks. This may be appropriate for many ULPs where any adverse effects of the betrayal is easily confined and does not place other clients or ULPs at risk. The Trust Models described in this section have three primary distinguishing characteristics. The Trust Model refers to a local ULP and Remote Peer, which are intended to be the local and remote ULP instances communicating via RDMA/DDP.
* Local Resource Sharing (yes/no) - When local resources are
shared, they are shared across a grouping of RDMAP/DDP Streams.
If local resources are not shared, the resources are dedicated on
a per Stream basis. Resources are defined in Section 2.2,
Resources. The advantage of not sharing resources between
Streams is that it reduces the types of attacks that are
possible. The disadvantage is that ULPs might run out of
resources.
* Local Partial Trust (yes/no) - Local Partial Trust is determined
based on whether the local grouping of RDMAP/DDP Streams (which
typically equates to one ULP or group of ULPs) mutually trust
each other not to perform a specific set of attacks.
* Remote Partial Trust (yes/no) - The Remote Partial Trust level is
determined based on whether the local ULP of a specific RDMAP/DDP
Stream partially trusts the Remote Peer of the Stream (see the
definition of Partial Trust in Section 1, Introduction).
Not all the combinations of the trust characteristics are expected to
be used by ULPs. This document specifically analyzes five ULP Trust
Models that are expected to be in common use. The Trust Models are
as follows:
* NS-NT - Non-Shared Local Resources, no Local Trust, no Remote
Trust; typically, a server ULP that wants to run in the safest
mode possible. All attack mitigations are in place to ensure
robust operation.
* NS-RT - Non-Shared Local Resources, no Local Trust, Remote
Partial Trust; typically, a peer-to-peer ULP that has, by some
method outside of the scope of this document, authenticated the
Remote Peer. Note that unless some form of key based
authentication is used on a per RDMA/DDP Stream basis, it may not
be possible for man-in-the-middle attacks to occur.
* S-NT - Shared Local Resources, no Local Trust, no Remote Trust;
typically, a server ULP that runs in an untrusted environment
where the amount of resources required is either too large or too
dynamic to dedicate for each RDMAP/DDP Stream.
* S-LT - Shared Local Resources, Local Partial Trust, no Remote
Trust; typically, a ULP that provides a session layer and uses
multiple Streams, to provides additional throughput or fail-over
capabilities. All the Streams within the local ULP partially
trust each other, but do not trust the Remote Peer. This Trust
Model may be appropriate for embedded environments.
* S-T - Shared Local Resources, Local Partial Trust, Remote Partial
Trust; typically, a distributed application, such as a
distributed database application or High Performance Computer
(HPC) application, which is intended to run on a cluster. Due to
extreme resource and performance requirements, the application
typically authenticates with all of its peers and then runs in a
highly trusted environment. The application peers are all in a
single application fault domain and depend on one another to be
well-behaved when accessing data structures. If a trusted Remote
Peer has an implementation defect that results in poor behavior,
the entire application could be corrupted.
Models NS-NT and S-NT, above, are typical for Internet networking -
neither the local ULP nor the Remote Peer is trusted. Sometimes,
optimizations can be done that enable sharing of Page Translation
Tables across multiple local ULPs; thus, Model S-LT can be
advantageous. Model S-T is typically used when resource scaling
across a large parallel ULP makes it infeasible to use any other
model. Resource scaling issues can either be due to performance
around scaling or because there simply are not enough resources.
Model NS-RT is probably the least likely model to be used, but is
presented for completeness.
Acknowledgments
Sara Bitan
Microsoft Corporation
EMail: sarab@microsoft.com
Allyn Romanow
Cisco Systems
170 W Tasman Drive
San Jose, CA 95134 USA
Phone: +1 (408) 525-8836
EMail: allyn@cisco.com
Catherine Meadows
Naval Research Laboratory
Code 5543
Washington, DC 20375 USA
EMail: meadows@itd.nrl.navy.mil
Patricia Thaler Agilent Technologies, Inc. 1101 Creekside Ridge Drive, #100 M/S-RG10 Roseville, CA 95678 USA Phone: +1 (916) 788-5662 EMail: pat_thaler@agilent.com James Livingston NEC Solutions (America), Inc. 7525 166th Ave. N.E., Suite D210 Redmond, WA 98052-7811 USA Phone: +1 (425) 897-2033 EMail: james.livingston@necsam.com John Carrier Cray Inc. 411 First Avenue S, Suite 600 Seattle, WA 98104-2860 Phone: 206-701-2090 EMail: carrier@cray.com Caitlin Bestler Broadcom 49 Discovery Irvine, CA 92618 EMail: cait@asomi.com Bernard Aboba Microsoft Corporation One Microsoft Way USA Redmond, WA 98052 Phone: +1 (425) 706-6606 EMail: bernarda@windows.microsoft.com
Authors' Addresses
James Pinkerton Microsoft Corporation One Microsoft Way Redmond, WA 98052 USA Phone: +1 (425) 705-5442 EMail: jpink@windows.microsoft.com Ellen Deleganes Self P.O. Box 9245 Brooks, OR 97305 Phone: (503) 642-3950 EMail: deleganes@yahoo.com
Full Copyright Statement Copyright (C) The IETF Trust (2007). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.