Internet Engineering Task Force (IETF) A. Morton Request for Comments: 8172 AT&T Labs Category: Informational July 2017 ISSN: 2070-1721 Considerations for Benchmarking Virtual Network Functions and Their Infrastructure
AbstractThe Benchmarking Methodology Working Group has traditionally conducted laboratory characterization of dedicated physical implementations of internetworking functions. This memo investigates additional considerations when network functions are virtualized and performed in general-purpose hardware. Status of This Memo This document is not an Internet Standards Track specification; it is published for informational purposes. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc8172. Copyright Notice Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Considerations for Hardware and Testing . . . . . . . . . . . 4 3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4 3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5 3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6 3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7 4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 8 4.1. Comparison with Physical Network Functions . . . . . . . 8 4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8 4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 9 4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 10 4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12 5. Security Considerations . . . . . . . . . . . . . . . . . . . 12 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 7.1. Normative References . . . . . . . . . . . . . . . . . . 13 7.2. Informative References . . . . . . . . . . . . . . . . . 14 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 15 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15 RFC1242] and [RFC2544] are the cornerstones of the work. A set of service provider and vendor development goals has emerged: reduce costs while increasing flexibility of network devices and drastically reduce deployment time. Network Function Virtualization (NFV) has the promise to achieve these goals and therefore has garnered much attention. It now seems certain that some network functions will be virtualized following the success of cloud computing and virtual desktops supported by sufficient network path capacity, performance, and widespread deployment; many of the same techniques will help achieve NFV. In the context of Virtual Network Functions (VNFs), the supporting Infrastructure requires general-purpose computing systems, storage systems, networking systems, virtualization support systems (such as hypervisors), and management systems for the virtual and physical resources. There will be many potential suppliers of Infrastructure
systems and significant flexibility in configuring the systems for best performance. There are also many potential suppliers of VNFs, adding to the combinations possible in this environment. The separation of hardware and software suppliers has a profound implication on benchmarking activities: much more of the internal configuration of the black-box Device Under Test (DUT) must now be specified and reported with the results, to foster both repeatability and comparison testing at a later time. Consider the following user story as further background and motivation: I'm designing and building my NFV Infrastructure platform. The first steps were easy because I had a small number of categories of VNFs to support and the VNF vendor gave hardware recommendations that I followed. Now I need to deploy more VNFs from new vendors, and there are different hardware recommendations. How well will the new VNFs perform on my existing hardware? Which among several new VNFs in a given category are most efficient in terms of capacity they deliver? And, when I operate multiple categories of VNFs (and PNFs) *concurrently* on a hardware platform such that they share resources, what are the new performance limits, and what are the software design choices I can make to optimize my chosen hardware platform? Conversely, what hardware platform upgrades should I pursue to increase the capacity of these concurrently operating VNFs? See <http://www.etsi.org/technologies-clusters/technologies/nfv> for more background; the white papers there may be a useful starting place. The "NFV Performance & Portability Best Practices" document [NFV.PER001] is particularly relevant to BMWG. There are also documents available among the Approved ETSI NFV Specifications [Approved_ETSI_NFV], including documents describing Infrastructure performance aspects and service quality metrics, and drafts in the ETSI NFV Open Area [Draft_ETSI_NFV], which may also have relevance to benchmarking. RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
BareMetal] or other isolation environments such as Linux containers. An essential consideration is benchmarking physical and Virtual Network Functions in the same way when possible, thereby allowing direct comparison. Benchmarking combinations of physical and virtual devices and functions in a System Under Test (SUT) is another topic of keen interest. A clearly related goal is investigating benchmarks for the capacity of a general-purpose platform to host a plurality of VNF instances. Existing networking technology benchmarks will also be considered for adaptation to NFV and closely associated technologies. A non-goal is any overlap with traditional computer benchmark development and their specific metrics (e.g., SPECmark suites such as SPEC CPU). A continued non-goal is any form of architecture development related to NFV and associated technologies in BMWG, consistent with all chartered work since BMWG began in 1989.
2. Storage systems with large capacity, high speed, and high reliability 3. Network interface ports specially designed for efficient service of many virtual Network Interface Cards (NICs) 4. High-capacity Ethernet switches The components above are subjects for development of specialized benchmarks that focus on the special demands of network function deployment. Labs conducting comparisons of different VNFs may be able to use the same hardware platform over many studies, until the steady march of innovations overtakes their capabilities (as happens with the lab's traffic generation and testing devices today).
and finally, the VNF itself, with items such as: o specific function being implemented in VNF o reserved resources for each function (e.g., CPU pinning and Non- Uniform Memory Access (NUMA) node assignment) o number of VNFs (or sub-VNF components, each with its own VM) in the service function chain (see Section 1.1 of [RFC7498] for a definition of service function chain) o number of physical interfaces and links transited in the service function chain In the physical device benchmarking context, most of the corresponding Infrastructure configuration choices were determined by the vendor. Although the platform itself is now one of the configuration variables, it is important to maintain emphasis on the networking benchmarks and capture the platform variables as input factors.
operational condition. In other words, benchmarking should include scenarios with production life-cycle management of VMs and their VNFs and network connectivity in progress, including VNF scaling up/down operations, as well as static configurations. 5. All physical things can fail, and benchmarking efforts can also examine recovery aided by the virtual architecture with different approaches to resiliency. 6. The sheer number of test conditions and configuration combinations encourage increased efficiency, including automated testing arrangements, combination sub-sampling through an understanding of inter-relationships, and machine-readable test results.
Section 4.4. However, external observations remain essential as the basis for benchmarks. Internal observations with fixed specification and interpretation may be provided in parallel (as auxiliary metrics), to assist the development of operations procedures when the technology is deployed, for example. Internal metrics and measurements from open-source implementations may be the only direct source of performance results in a desired dimension, but corroborating external observations are still required to assure the integrity of measurement discipline was maintained for all reported results. A related aspect of benchmark development is where the scope includes multiple approaches to a common function under the same benchmark. For example, there are many ways to arrange for activation of a network path between interface points, and the activation times can be compared if the start-to-stop activation interval has a generic
and unambiguous definition. Thus, generic benchmark definitions are preferred over technology/protocol-specific definitions where possible. RFC6049]. Examples include mean one-way delay in Section 4.1 of [RFC6049], Packet Delay Variation (PDV) in [RFC5481], and Packet Reordering [RFC4737] [RFC4689].
X3.102]) provides a way to organize benchmarks such that there is a clear indication of coverage for the intersection of life-cycle stages and performance criteria. |----------------------------------------------------------| | | | | | | | SPEED | ACCURACY | RELIABILITY | | | | | | |----------------------------------------------------------| | | | | | | Activation | | | | | | | | | |----------------------------------------------------------| | | | | | | Operation | | | | | | | | | |----------------------------------------------------------| | | | | | | De-activation | | | | | | | | | |----------------------------------------------------------| For example, the "Time to deploy VNFs" benchmark described above would be placed in the intersection of Activation and Speed, making it clear that there are other potential performance criteria to benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups" in a set of 100 attempts. This example emphasizes that the Activation and De-activation life-cycle stages are key areas for NFV and related Infrastructure and encourages expansion beyond traditional benchmarks for normal operation. Thus, reviewing the benchmark coverage using this table (sometimes called the 3x3 Matrix) can be a worthwhile exercise in BMWG. In one of the first applications of the 3x3 Matrix in BMWG [SDN-BENCHMARK], we discovered that metrics on measured size, capacity, or scale do not easily match one of the three columns above. Following discussion, this was resolved in two ways: o Add a column, Scale, for use when categorizing and assessing the coverage of benchmarks (without measured results). An example of this use is found in [OPNFV-BENCHMARK] (and a variation may be found in [SDN-BENCHMARK]). This is the 3x4 Matrix.
o If using the matrix to report results in an organized way, keep size, capacity, and scale metrics separate from the 3x3 Matrix and incorporate them in the report with other qualifications of the results. Note that the resource utilization (e.g., CPU) metrics do not fit in the matrix. They are not benchmarks, and omitting them confirms their status as auxiliary metrics. Resource assignments are configuration parameters, and these are reported separately. This approach encourages use of the 3x3 Matrix to organize reports of results, where the capacity at which the various metrics were measured could be included in the title of the matrix (and results for multiple capacities would result in separate 3x3 Matrices, if there were sufficient measurements/results to organize in that way). For example, results for each VM and VNF could appear in the 3x3 Matrix, organized to illustrate resource occupation (CPU Cores) in a particular physical computing system, as shown below. VNF#1 .-----------. |__|__|__|__| Core 1 |__|__|__|__| |__|__|__|__| | | | | | '-----------' VNF#2 .-----------. |__|__|__|__| Cores 2-5 |__|__|__|__| |__|__|__|__| | | | | | '-----------' VNF#3 VNF#4 VNF#5 .-----------. .-----------. .-----------. |__|__|__|__| |__|__|__|__| |__|__|__|__| Core 6 |__|__|__|__| |__|__|__|__| |__|__|__|__| |__|__|__|__| |__|__|__|__| |__|__|__|__| | | | | | | | | | | | | | | | '-----------' '-----------' '-----------' VNF#6 .-----------. |__|__|__|__| Core 7 |__|__|__|__| |__|__|__|__| | | | | | '-----------'
The combination of tables above could be built incrementally, beginning with VNF#1 and one Core, then adding VNFs according to their supporting Core assignments. X-Y plots of critical benchmarks would also provide insight to the effect of increased hardware utilization. All VNFs might be of the same type, or to match a production environment, there could be VNFs of multiple types and categories. In this figure, VNFs #3-#5 are assumed to require small CPU resources, while VNF#2 requires four Cores to perform its function. IPMI2.0] has been implemented by many manufacturers and supports measurement of instantaneous energy consumption. To assess the instantaneous energy consumption of virtual resources, it may be possible to estimate the value using an overall metric based on utilization readings, according to [NFVIaas-FRAMEWORK].
[NFV.PER001] ETSI, "Network Function Virtualization: Performance & Portability Best Practices", ETSI GS NFV-PER 001, V1.1.2, December 2014. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <http://www.rfc-editor.org/info/rfc2119>. [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, <http://www.rfc-editor.org/info/rfc2544>. [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, "Terminology for Benchmarking Network-layer Traffic Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689, October 2006, <http://www.rfc-editor.org/info/rfc4689>. [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, S., and J. Perser, "Packet Reordering Metrics", RFC 4737, DOI 10.17487/RFC4737, November 2006, <http://www.rfc-editor.org/info/rfc4737>. [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for Service Function Chaining", RFC 7498, DOI 10.17487/RFC7498, April 2015, <http://www.rfc-editor.org/info/rfc7498>. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <http://www.rfc-editor.org/info/rfc8174>.
[Approved_ETSI_NFV] ETSI, Network Functions Virtualisation Technical Committee, "ETSI NFV", <http://www.etsi.org/standards-search>. [BareMetal] Popek, G. and R. Goldberg, "Formal requirements for virtualizable third generation architectures", Communications of the ACM, Volume 17, Issue 7, Pages 412-421, DOI 10.1145/361011.361073, July 1974. [Draft_ETSI_NFV] ETSI, "Network Functions Virtualisation: Specifications", <http://www.etsi.org/technologies-clusters/technologies/ nfv>. [IPMI2.0] Intel Corporation, Hewlett-Packard Company, NEC Corporation, and Dell Inc., "Intelligent Platform Management Interface Specification v2.0 (with latest errata)", April 2015, <http://www.intel.com/content/dam/www/public/us/en/ documents/specification-updates/ipmi-intelligent-platform- mgt-interface-spec-2nd-gen-v2-0-spec-update.pdf>. [NFVIaas-FRAMEWORK] Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D., Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra, "NFVIaaS Architectural Framework for Policy Based Resource Placement and Scheduling", Work in Progress, draft-krishnan-nfvrg-policy-based-rm-nfviaas-06, March 2016. [OPNFV-BENCHMARK] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking Virtual Switches in OPNFV", Work in Progress, draft-ietf-bmwg-vswitch-opnfv-04, June 2017. [RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, <http://www.rfc-editor.org/info/rfc1242>. [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, March 2009, <http://www.rfc-editor.org/info/rfc5481>.
[RFC6049] Morton, A. and E. Stephan, "Spatial Composition of Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, <http://www.rfc-editor.org/info/rfc6049>. [SDN-BENCHMARK] Vengainathan, B., Basil, A., Tassinari, M., Manral, V., and S. Banks, "Terminology for Benchmarking SDN Controller Performance", Work in Progress, draft-ietf-bmwg-sdn- controller-benchmark-term-04, June 2017. [X3.102] ANSI, "Information Systems - Data Communication Systems and Services - User-Oriented Performance Parameters Communications Framework", ANSI X3.102, 1983.