Network Working Group S. Bradner Request for Comments: 2544 Harvard University Obsoletes: 1944 J. McQuaid Category: Informational NetScout Systems March 1999 Benchmarking Methodology for Network Interconnect Devices Status of this Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Copyright Notice Copyright (C) The Internet Society (1999). All Rights Reserved. IESG Note This document is a republication of RFC 1944 correcting the values for the IP addresses which were assigned to be used as the default addresses for networking test equipment. (See section C.2.2 ). This RFC replaces and obsoletes RFC 1944.
AbstractThis document discusses and defines a number of tests that may be used to describe the performance characteristics of a network interconnecting device. In addition to defining the tests this document also describes specific formats for reporting the results of the tests. Appendix A lists the tests and conditions that we believe should be included for specific cases and gives additional information about testing practices. Appendix B is a reference listing of maximum frame rates to be used with specific frame sizes on various media and Appendix C gives some examples of frame formats to be used in testing.
This document defines a specific set of tests that vendors can use to measure and report the performance characteristics of network devices. The results of these tests will provide the user comparable data from different vendors with which to evaluate these devices. A previous document, "Benchmarking Terminology for Network Interconnect Devices" (RFC 1242), defined many of the terms that are used in this document. The terminology document should be consulted before attempting to make use of this document. Appendix A lists some of the tests and conditions that we believe should be included for specific cases.
Figure 1) Since the tester both sends the test traffic and receives it back, after the traffic has been forwarded but the DUT, the tester can easily determine if all of the transmitted packets were received and verify that the correct packets were received. The same functionality can be obtained with separate transmitting and receiving devices (see Figure 2) but unless they are remotely controlled by some computer in a way that simulates the single tester, the labor required to accurately perform some of the tests (particularly the throughput test) can be prohibitive.
+------------+ | | +------------| tester |<-------------+ | | | | | +------------+ | | | | +------------+ | | | | | +----------->| DUT |--------------+ | | +------------+ Figure 1 +--------+ +------------+ +----------+ | | | | | | | sender |-------->| DUT |--------->| receiver | | | | | | | +--------+ +------------+ +----------+ Figure 2 Figure 1 would be used. Two identical DUTs are used in the other test set up. (see Figure 3) In many cases this set up may more accurately simulate the real world. For example, connecting two LANs together with a WAN link or high speed backbone. This set up would not be as good at simulating a system where clients on a Ethernet LAN were interacting with a server on an FDDI backbone. +-----------+ | | +---------------------| tester |<---------------------+ | | | | | +-----------+ | | | | +----------+ +----------+ | | | | | | | +------->| DUT 1 |-------------->| DUT 2 |---------+ | | | | +----------+ +----------+ Figure 3
Appendix A). It is expected that all of the tests will be run without changing the configuration or setup of the DUT in any way other than that required to do the specific test. For example, it is not acceptable to change the size of frame handling buffers between tests of frame handling rates or to disable all but one transport protocol when testing the throughput of that protocol. It is necessary to modify the configuration when starting a test to determine the effect of filters on throughput, but the only change MUST be to enable the specific filter. The DUT set up SHOULD include the normally recommended routing update intervals and keep alive frequency. The specific version of the software and the exact DUT configuration, including what functions are disabled, used during the tests MUST be included as part of the report of the results. Appendix C: Test Frame Formats. These exact frame formats SHOULD be used in the tests described in this document for this protocol/media combination and that these frames will be used as a template for testing other protocol/media combinations. The specific formats that are used to define the test frames for a particular test series MUST be included in the report of the results.
In theory it would be ideal to distribute the frame sizes in a way that would evenly distribute the theoretical frame rates. These recommendations incorporate this theory but specify frame sizes which are easy to understand and remember. In addition, many of the same frame sizes are specified on each of the media types to allow for easy performance comparisons. Note: The inclusion of an unrealistically small frame size on some of the media types (i.e. with little or no space for data) is to help characterize the per-frame processing overhead of the DUT.
Routing update frames SHOULD be sent at the rate specified in Appendix C for the specific routing protocol being used in the test. Two routing update frames are defined in Appendix C for the TCP/IP over Ethernet example. The routing frames are designed to change the routing to a number of networks that are not involved in the forwarding of the test data. The first frame sets the routing table state to "A", the second one changes the state to "B". The frames MUST be alternated during the trial. The test SHOULD verify that the routing update was processed by the DUT.
Consider the following 6 port DUT: -------------- ---------| in A out X|-------- ---------| in B out Y|-------- ---------| in C out Z|-------- -------------- The addressing of the data streams for each of the inputs SHOULD be: stream sent to input A: packet to out X, packet to out Y, packet to out Z stream sent to input B: packet to out X, packet to out Y, packet to out Z stream sent to input C packet to out X, packet to out Y, packet to out Z Note that these streams each follow the same sequence so that 3 packets will arrive at output X at the same time, then 3 packets at Y, then 3 packets at Z. This procedure ensures that, as in the real world, the DUT will have to deal with multiple packets addressed to the same output at the same time.
This model is useful when the test input and output are homogenous (e.g., 64-byte IP, 802.3 frames into the DUT; 64 byte IP, 802.3 frames out), or the method of test can distinguish between dissimilar input/output. (E.g., 1518 byte IP, 802.3 frames in; 576 byte, fragmented IP, X.25 frames out.) By extending the single DUT test model, reasonable benchmarks regarding multiple DUTs or heterogeneous environments may be collected. In this extension, the single DUT is replaced by a system of interconnected network DUTs. This test methodology would support the benchmarking of a variety of device/media/service/protocol combinations. For example, a configuration for a LAN-to-WAN-to-LAN test might be: (1) 802.3-> DUT 1 -> X.25 @ 64kbps -> DUT 2 -> 802.3 Or a mixed LAN configuration might be: (2) 802.3 -> DUT 1 -> FDDI -> DUT 2 -> FDDI -> DUT 3 -> 802.3 In both examples 1 and 2, end-to-end benchmarks of each system could be empirically ascertained. Other behavior may be characterized through the use of intermediate devices. In example 2, the configuration may be used to give an indication of the FDDI to FDDI capability exhibited by DUT 2. Because multiple DUTs are treated as a single system, there are limitations to this methodology. For instance, this methodology may yield an aggregate benchmark for a tested system. That benchmark alone, however, may not necessarily reflect asymmetries in behavior between the DUTs, latencies introduce by other apparatus (e.g., CSUs/DSUs, switches), etc. Further, care must be used when comparing benchmarks of different systems by ensuring that the DUTs' features/configuration of the tested systems have the appropriate common denominators to allow comparison.
The maximum frame rate that should be used when testing WAN connections SHOULD be greater than the listed theoretical maximum rate for the frame size on that speed connection. The higher rate for WAN tests is to compensate for the fact that some vendors employ various forms of header compression. A list of maximum frame rates for LAN connections is included in Appendix B.
b) Send the "learning frames" to the "output" port and wait 2 seconds to be sure that the learning has settled. Bridge learning frames are frames with source addresses that are the same as the destination addresses used by the test frames. Learning frames for other protocols are used to prime the address resolution tables in the DUT. The formats of the learning frame that should be used are shown in the Test Frame Formats document. c) Run the test trial. d) Wait for two seconds for any residual frames to be received. e) Wait for at least five seconds for the DUT to restabilize. RFC 1242. Procedure: Send a specific number of frames at a specific rate through the DUT and then count the frames that are transmitted by the DUT. If the count of offered frames is equal to the count of received frames, the fewer frames are received than were transmitted, the rate of the offered stream is reduced and the test is rerun.
The throughput is the fastest rate at which the count of test frames transmitted by the DUT is equal to the number of test frames sent to it by the test equipment. Reporting format: The results of the throughput test SHOULD be reported in the form of a graph. If it is, the x coordinate SHOULD be the frame size, the y coordinate SHOULD be the frame rate. There SHOULD be at least two lines on the graph. There SHOULD be one line showing the theoretical frame rate for the media at the various frame sizes. The second line SHOULD be the plot of the test results. Additional lines MAY be used on the graph to report the results for each type of data stream tested. Text accompanying the graph SHOULD indicate the protocol, data stream format, and type of media used in the tests. We assume that if a single value is desired for advertising purposes the vendor will select the rate for the minimum frame size for the media. If this is done then the figure MUST be expressed in frames per second. The rate MAY also be expressed in bits (or bytes) per second if the vendor so desires. The statement of performance MUST include a/ the measured maximum frame rate, b/ the size of the frame used, c/ the theoretical limit of the media for that frame size, and d/ the type of protocol used in the test. Even if a single value is used as part of the advertising copy, the full table of results SHOULD be included in the product data sheet. RFC 1242. Procedure: First determine the throughput for DUT at each of the listed frame sizes. Send a stream of frames at a particular frame size through the DUT at the determined throughput rate to a specific destination. The stream SHOULD be at least 120 seconds in duration. An identifying tag SHOULD be included in one frame after 60 seconds with the type of tag being implementation dependent. The time at which this frame is fully transmitted is recorded (timestamp A). The receiver logic in the test equipment MUST recognize the tag information in the frame stream and record the time at which the tagged frame was received (timestamp B). The latency is timestamp B minus timestamp A as per the relevant definition frm RFC 1242, namely latency as defined for store and forward devices or latency as defined for bit forwarding devices. The test MUST be repeated at least 20 times with the reported value being the average of the recorded values.
This test SHOULD be performed with the test frame addressed to the same destination as the rest of the data stream and also with each of the test frames addressed to a new destination network. Reporting format: The report MUST state which definition of latency (from RFC 1242) was used for this test. The latency results SHOULD be reported in the format of a table with a row for each of the tested frame sizes. There SHOULD be columns for the frame size, the rate at which the latency test was run for that frame size, for the media types tested, and for the resultant latency values for each type of data stream tested. RFC 1242, of a DUT throughout the entire range of input data rates and frame sizes. Procedure: Send a specific number of frames at a specific rate through the DUT to be tested and count the frames that are transmitted by the DUT. The frame loss rate at each point is calculated using the following equation: ( ( input_count - output_count ) * 100 ) / input_count The first trial SHOULD be run for the frame rate that corresponds to 100% of the maximum rate for the frame size on the input media. Repeat the procedure for the rate that corresponds to 90% of the maximum rate used and then for 80% of this rate. This sequence SHOULD be continued (at reducing 10% intervals) until there are two successive trials in which no frames are lost. The maximum granularity of the trials MUST be 10% of the maximum rate, a finer granularity is encouraged. Reporting format: The results of the frame loss rate test SHOULD be plotted as a graph. If this is done then the X axis MUST be the input frame rate as a percent of the theoretical rate for the media at the specific frame size. The Y axis MUST be the percent loss at the particular input rate. The left end of the X axis and the bottom of the Y axis MUST be 0 percent; the right end of the X axis and the top of the Y axis MUST be 100 percent. Multiple lines on the graph MAY used to report the frame loss rate for different frame sizes, protocols, and types of data streams. Note: See section 18 for the maximum frame rates that SHOULD be used.
RFC 1242. Procedure: Send a burst of frames with minimum inter-frame gaps to the DUT and count the number of frames forwarded by the DUT. If the count of transmitted frames is equal to the number of frames forwarded the length of the burst is increased and the test is rerun. If the number of forwarded frames is less than the number transmitted, the length of the burst is reduced and the test is rerun. The back-to-back value is the number of frames in the longest burst that the DUT will handle without the loss of any frames. The trial length MUST be at least 2 seconds and SHOULD be repeated at least 50 times with the average of the recorded values being reported. Reporting format: The back-to-back results SHOULD be reported in the format of a table with a row for each of the tested frame sizes. There SHOULD be columns for the frame size and for the resultant average frame count for each type of data stream tested. The standard deviation for each measurement MAY also be reported.
DSAP 8 bits SSAP 8 bits Control 8 bits Vendor 24 bits Type 16 bits Data 8 x ( N - 18) bits FCS 32 bits ED 4 bits FS 12 bits
Figure 2) or the end node if adjacent network routing is being used.) It is assumed that the router will cache the MAC address of the requesting device. The ARP request should be sent 5 seconds before the test frame stream starts in each trial. Trial lengths of longer than 50 seconds may require that the router be configured for an extended ARP timeout. +--------+ +------------+ | | | phantom |------ P LAN A IN A------| DUT |------------| |------ P LAN B | | OUT A | router |------ P LAN C +--------+ +------------+ Figure 2 In the case where full routing is being used
73 00 net not node 74 00 00 00 00 0 78 00 00 00 00 0 82 00 00 00 07 metric 7 -- net 3 86 00 02 family = IP 88 00 00 0 90 xx xx xx net 3 IP address 93 00 net not node 94 00 00 00 00 0 98 00 00 00 00 0 102 00 00 00 07 metric 7 -- net 4 106 00 02 family = IP 108 00 00 0 110 xx xx xx net 4 IP address 113 00 net not node 114 00 00 00 00 0 118 00 00 00 00 0 122 00 00 00 07 metric 7 -- net 5 126 00 02 family = IP 128 00 00 0 130 00 net 5 IP address 133 00 net not node 134 00 00 00 00 0 138 00 00 00 00 0 142 00 00 00 07 metric 7 -- net 6 146 00 02 family = IP 148 00 00 0 150 xx xx xx net 6 IP address 153 00 net not node 154 00 00 00 00 0 158 00 00 00 00 0 162 00 00 00 07 metric 7
-- DATAGRAM HEADER offset data (hex) description 00 xx xx xx xx xx xx set to dest MAC address 06 xx xx xx xx xx xx set to source MAC address 12 08 00 type -- IP HEADER 14 45 IP version - 4 header length 5 4 byte units 15 00 TOS 16 00 2E total length* 18 00 00 ID 20 00 00 flags (3 bits) - 0 fragment offset-0 22 0A TTL 23 11 protocol - 17 (UDP) 24 C4 8D header checksum* 26 xx xx xx xx set to source IP address** 30 xx xx xx xx set to destination IP address** -- UDP HEADER 34 C0 20 source port 36 00 07 destination port 07 = Echo 38 00 1A UDP message length* 40 00 00 UDP checksum -- UDP DATA 42 00 01 02 03 04 05 06 07 some data*** 50 08 09 0A 0B 0C 0D 0E 0F * - change for different length frames ** - change for different logical streams *** - fill remainder of frame with incrementing octets, repeated if required by frame length Values to be used in Total Length and UDP message length fields: frame size total length UDP message length 64 00 2E 00 1A 128 00 6E 00 5A 256 00 EE 00 9A 512 01 EE 01 9A 768 02 EE 02 9A 1024 03 EE 03 9A 1280 04 EE 04 9A 1518 05 DC 05 C8
Full Copyright Statement Copyright (C) The Internet Society (1999). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.