Internet Engineering Task Force (IETF) G. Fioccola, Ed. Request for Comments: 8321 A. Capello Category: Experimental M. Cociglio ISSN: 2070-1721 L. Castaldelli Telecom Italia M. Chen L. Zheng Huawei Technologies G. Mirsky ZTE T. Mizrahi Marvell January 2018 Alternate-Marking Method for Passive and Hybrid Performance Monitoring
AbstractThis document describes a method to perform packet loss, delay, and jitter measurements on live traffic. This method is based on an Alternate-Marking (coloring) technique. A report is provided in order to explain an example and show the method applicability. This technology can be applied in various situations, as detailed in this document, and could be considered Passive or Hybrid depending on the application. Status of This Memo This document is not an Internet Standards Track specification; it is published for examination, experimental implementation, and evaluation. This document defines an Experimental Protocol for the Internet community. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8321.
Copyright Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5 2. Overview of the Method . . . . . . . . . . . . . . . . . . . 5 3. Detailed Description of the Method . . . . . . . . . . . . . 6 3.1. Packet Loss Measurement . . . . . . . . . . . . . . . . . 6 3.1.1. Coloring the Packets . . . . . . . . . . . . . . . . 11 3.1.2. Counting the Packets . . . . . . . . . . . . . . . . 12 3.1.3. Collecting Data and Calculating Packet Loss . . . . . 13 3.2. Timing Aspects . . . . . . . . . . . . . . . . . . . . . 13 3.3. One-Way Delay Measurement . . . . . . . . . . . . . . . . 15 3.3.1. Single-Marking Methodology . . . . . . . . . . . . . 15 3.3.2. Double-Marking Methodology . . . . . . . . . . . . . 17 3.4. Delay Variation Measurement . . . . . . . . . . . . . . . 18 4. Considerations . . . . . . . . . . . . . . . . . . . . . . . 18 4.1. Synchronization . . . . . . . . . . . . . . . . . . . . . 19 4.2. Data Correlation . . . . . . . . . . . . . . . . . . . . 19 4.3. Packet Reordering . . . . . . . . . . . . . . . . . . . . 20 5. Applications, Implementation, and Deployment . . . . . . . . 21 5.1. Report on the Operational Experiment . . . . . . . . . . 22 5.1.1. Metric Transparency . . . . . . . . . . . . . . . . . 24 6. Hybrid Measurement . . . . . . . . . . . . . . . . . . . . . 24 7. Compliance with Guidelines from RFC 6390 . . . . . . . . . . 25 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 9. Security Considerations . . . . . . . . . . . . . . . . . . . 27 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 28 10.1. Normative References . . . . . . . . . . . . . . . . . . 28 10.2. Informative References . . . . . . . . . . . . . . . . . 29 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 32 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32
RFC7680], delay [RFC7679], and jitter [RFC3393]. In view of this scenario, Service Providers need methodologies and tools to monitor and measure network performance with an adequate accuracy, in order to constantly control the quality of experience perceived by their customers. On the other hand, performance monitoring provides useful information for improving network management (e.g., isolation of network problems, troubleshooting, etc.). A lot of work related to Operations, Administration, and Maintenance (OAM), which also includes performance monitoring techniques, has been done by Standards Developing Organizations (SDOs): [RFC7276] provides a good overview of existing OAM mechanisms defined in the IETF, ITU-T, and IEEE. In the IETF, a lot of work has been done on fault detection and connectivity verification, while a minor effort has been thus far dedicated to performance monitoring. The IPPM WG has defined standard metrics to measure network performance; however, the methods developed in this WG mainly refer to focus on Active measurement techniques. More recently, the MPLS WG has defined mechanisms for measuring packet loss, one-way and two-way delay, and delay variation in MPLS networks [RFC6374], but their applicability to Passive measurements has some limitations, especially for pure connection-less networks. The lack of adequate tools to measure packet loss with the desired accuracy drove an effort to design a new method for the performance monitoring of live traffic, which is easy to implement and deploy. The effort led to the method described in this document: basically, it is a Passive performance monitoring technique, potentially applicable to any kind of packet-based traffic, including Ethernet, IP, and MPLS, both unicast and multicast. The method addresses primarily packet loss measurement, but it can be easily extended to one-way or two-way delay and delay variation measurements as well. The method has been explicitly designed for Passive measurements, but it can also be used with Active probes. Passive measurements are usually more easily understood by customers and provide much better accuracy, especially for packet loss measurements.
RFC 7799 [RFC7799] defines Passive and Hybrid Methods of Measurement. In particular, Passive Methods of Measurement are based solely on observations of an undisturbed and unmodified packet stream of interest; Hybrid Methods are Methods of Measurement that use a combination of Active Methods and Passive Methods. Taking into consideration these definitions, the Alternate-Marking Method could be considered Hybrid or Passive, depending on the case. In the case where the marking method is obtained by changing existing field values of the packets (e.g., the Differentiated Services Code Point (DSCP) field), the technique is Hybrid. In the case where the marking field is dedicated, reserved, and included in the protocol specification, the Alternate-Marking technique can be considered as Passive (e.g., Synonymous Flow Label as described in [SFL-FRAMEWORK] or OAM Marking Bits as described in [PM-MM-BIER]). The advantages of the method described in this document are: o easy implementation: it can be implemented by using features already available on major routing platforms, as described in Section 5.1, or by applying an optimized implementation of the method for both legacy and newest technologies; o low computational effort: the additional load on processing is negligible; o accurate packet loss measurement: single packet loss granularity is achieved with a Passive measurement; o potential applicability to any kind of packet-based or frame-based traffic: Ethernet, IP, MPLS, etc., and both unicast and multicast; o robustness: the method can tolerate out-of-order packets, and it's not based on "special" packets whose loss could have a negative impact; o flexibility: all the timestamp formats are allowed, because they are managed out of band. The format (the Network Time Protocol (NTP) [RFC5905] or the IEEE 1588 Precision Time Protocol (PTP) [IEEE-1588]) depends on the precision you want; and o no interoperability issues: the features required to experiment and test the method (as described in Section 5.1) are available on all current routing platforms. Both a centralized or distributed solution can be used to harvest data from the routers.
The method doesn't raise any specific need for protocol extension, but it could be further improved by means of some extension to existing protocols. Specifically, the use of Diffserv bits for coloring the packets could not be a viable solution in some cases: a standard method to color the packets for this specific application could be beneficial. RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
The method proposed in this document follows the second approach, but it doesn't use additional packets to virtually split the flow in blocks. Instead, it "marks" the packets so that the packets belonging to the same block will have the same color, whilst consecutive blocks will have different colors. Each change of color represents a sort of auto-synchronization signal that guarantees the consistency of measurements taken by different devices along the path (see also [IP-MULTICAST-PM] and [OPSAWG-P3M], where this technique was introduced). Figure 1 represents a very simple network and shows how the method can be used to measure packet loss on different network segments: by enabling the measurement on several interfaces along the path, it is possible to perform link monitoring, node monitoring, or end-to-end monitoring. The method is flexible enough to measure packet loss on any segment of the network and can be used to isolate the faulty element. Traffic Flow ========================================================> +------+ +------+ +------+ +------+ ---<> R1 <>-----<> R2 <>-----<> R3 <>-----<> R4 <>--- +------+ +------+ +------+ +------+ . . . . . . . . . . . . . <------> <-------> . . Node Packet Loss Link Packet Loss . . . <---------------------------------------------------> End-to-End Packet Loss Figure 1: Available Measurements
As discussed in the previous section, a simple way to create the blocks is to "color" the traffic (two colors are sufficient), so that packets belonging to different consecutive blocks will have different colors. Whenever the color changes, the previous block terminates and the new one begins. Hence, all the packets belonging to the same block will have the same color and packets of different consecutive blocks will have different colors. The number of packets in each block depends on the criterion used to create the blocks: o if the color is switched after a fixed number of packets, then each block will contain the same number of packets (except for any losses); and o if the color is switched according to a fixed timer, then the number of packets may be different in each block depending on the packet rate. The following figure shows how a flow looks like when it is split in traffic blocks with colored packets. A: packet with A coloring B: packet with B coloring | | | | | | | Traffic Flow | | -------------------------------------------------------------------> BBBBBBB AAAAAAAAAAA BBBBBBBBBBB AAAAAAAAAAA BBBBBBBBBBB AAAAAAA -------------------------------------------------------------------> ... | Block 5 | Block 4 | Block 3 | Block 2 | Block 1 | | | | | Figure 2: Traffic Coloring Figure 3 shows how the method can be used to measure link packet loss between two adjacent nodes. Referring to the figure, let's assume we want to monitor the packet loss on the link between two routers: router R1 and router R2. According to the method, the traffic is colored alternatively with two different colors: A and B. Whenever the color changes, the transition generates a sort of square-wave signal, as depicted in the following figure.
Color A ----------+ +-----------+ +---------- | | | | Color B +-----------+ +-----------+ Block n ... Block 3 Block 2 Block 1 <---------> <---------> <---------> <---------> <---------> Traffic Flow ===========================================================> Color ...AAAAAAAAAAA BBBBBBBBBBB AAAAAAAAAAA BBBBBBBBBBB AAAAAAA... ===========================================================> Figure 3: Computation of Link Packet Loss Traffic coloring can be done by R1 itself if the traffic is not already colored. R1 needs two counters, C(A)R1 and C(B)R1, on its egress interface: C(A)R1 counts the packets with color A and C(B)R1 counts those with color B. As long as traffic is colored as A, only counter C(A)R1 will be incremented, while C(B)R1 is not incremented; conversely, when the traffic is colored as B, only C(B)R1 is incremented. C(A)R1 and C(B)R1 can be used as reference values to determine the packet loss from R1 to any other measurement point down the path. Router R2, similarly, will need two counters on its ingress interface, C(A)R2 and C(B)R2, to count the packets received on that interface and colored with A and B, respectively. When an A block ends, it is possible to compare C(A)R1 and C(A)R2 and calculate the packet loss within the block; similarly, when the successive B block terminates, it is possible to compare C(B)R1 with C(B)R2, and so on, for every successive block. Likewise, by using two counters on the R2 egress interface, it is possible to count the packets sent out of the R2 interface and use them as reference values to calculate the packet loss from R2 to any measurement point down R2. Using a fixed timer for color switching offers better control over the method: the (time) length of the blocks can be chosen large enough to simplify the collection and the comparison of measures taken by different network devices. It's preferable to read the value of the counters not immediately after the color switch: some packets could arrive out of order and increment the counter associated with the previous block (color), so it is worth waiting for some time. A safe choice is to wait L/2 time units (where L is the duration for each block) after the color switch, to read the still counter of the previous color, so the possibility of reading a running counter instead of a still one is minimized. The drawback is that the longer the duration of the block, the less frequent the measurement can be taken.
The following table shows how the counters can be used to calculate the packet loss between R1 and R2. The first column lists the sequence of traffic blocks, while the other columns contain the counters of A-colored packets and B-colored packets for R1 and R2. In this example, we assume that the values of the counters are reset to zero whenever a block ends and its associated counter has been read: with this assumption, the table shows only relative values, which is the exact number of packets of each color within each block. If the values of the counters were not reset, the table would contain cumulative values, but the relative values could be determined simply by the difference from the value of the previous block of the same color. The color is switched on the basis of a fixed timer (not shown in the table), so the number of packets in each block is different. +-------+--------+--------+--------+--------+------+ | Block | C(A)R1 | C(B)R1 | C(A)R2 | C(B)R2 | Loss | +-------+--------+--------+--------+--------+------+ | 1 | 375 | 0 | 375 | 0 | 0 | | 2 | 0 | 388 | 0 | 388 | 0 | | 3 | 382 | 0 | 381 | 0 | 1 | | 4 | 0 | 377 | 0 | 374 | 3 | | ... | ... | ... | ... | ... | ... | | 2n | 0 | 387 | 0 | 387 | 0 | | 2n+1 | 379 | 0 | 377 | 0 | 2 | +-------+--------+--------+--------+--------+------+ Table 1: Evaluation of Counters for Packet Loss Measurements During an A block (blocks 1, 3, and 2n+1), all the packets are A-colored; therefore, the C(A) counters are incremented to the number seen on the interface, while C(B) counters are zero. Conversely, during a B block (blocks 2, 4, and 2n), all the packets are B-colored: C(A) counters are zero, while C(B) counters are incremented. When a block ends (because of color switching), the relative counters stop incrementing; it is possible to read them, compare the values measured on routers R1 and R2, and calculate the packet loss within that block. For example, looking at the table above, during the first block (A-colored), C(A)R1 and C(A)R2 have the same value (375), which corresponds to the exact number of packets of the first block (no loss). Also, during the second block (B-colored), R1 and R2 counters have the same value (388), which corresponds to the number of packets of the second block (no loss). During the third and fourth blocks,
R1 and R2 counters are different, meaning that some packets have been lost: in the example, one single packet (382-381) was lost during block three, and three packets (377-374) were lost during block four. The method applied to R1 and R2 can be extended to any other router and applied to more complex networks, as far as the measurement is enabled on the path followed by the traffic flow(s) being observed. It's worth mentioning two different strategies that can be used when implementing the method: o flow-based: the flow-based strategy is used when only a limited number of traffic flows need to be monitored. According to this strategy, only a subset of the flows is colored. Counters for packet loss measurements can be instantiated for each single flow, or for the set as a whole, depending on the desired granularity. A relevant problem with this approach is the necessity to know in advance the path followed by flows that are subject to measurement. Path rerouting and traffic load-balancing increase the issue complexity, especially for unicast traffic. The problem is easier to solve for multicast traffic, where load-balancing is seldom used and static joins are frequently used to force traffic forwarding and replication. o link-based: measurements are performed on all the traffic on a link-by-link basis. The link could be a physical link or a logical link. Counters could be instantiated for the traffic as a whole or for each traffic class (in case it is desired to monitor each class separately), but in the second case, a couple of counters are needed for each class. As mentioned, the flow-based measurement requires the identification of the flow to be monitored and the discovery of the path followed by the selected flow. It is possible to monitor a single flow or multiple flows grouped together, but in this case, measurement is consistent only if all the flows in the group follow the same path. Moreover, if a measurement is performed by grouping many flows, it is not possible to determine exactly which flow was affected by packet loss. In order to have measures per single flow, it is necessary to configure counters for each specific flow. Once the flow(s) to be monitored has been identified, it is necessary to configure the monitoring on the proper nodes. Configuring the monitoring means configuring the rule to intercept the traffic and configuring the counters to count the packets. To have just an end-to-end monitoring, it is sufficient to enable the monitoring on the first- and last-hop routers of the path: the mechanism is completely transparent to intermediate nodes and independent from the path followed by traffic flows. On the contrary, to monitor the flow on a
hop-by-hop basis along its whole path, it is necessary to enable the monitoring on every node from the source to the destination. In case the exact path followed by the flow is not known a priori (i.e., the flow has multiple paths to reach the destination), it is necessary to enable the monitoring system on every path: counters on interfaces traversed by the flow will report packet count, whereas counters on other interfaces will be null. Section 3.2; so every node that is designated as a measurement point along the path should be able to identify unambiguously the colored packets. Furthermore, [MULTIPOINT-ALT-MM] generalizes the coloring for multipoint-to-multipoint flow. In addition, it can be advantageous to color the flow as close as possible to the source because it allows an end-to-end measure if a measurement point is enabled on the last-hop router as well. For link-based measurements, all traffic needs to be colored when transmitted on the link. If the traffic had already been colored, then it has to be re-colored because the color must be consistent on the link. This means that each hop along the path must (re-)color the traffic; the color is not required to be consistent along different links. Traffic coloring can be implemented by setting a specific bit in the packet header and changing the value of that bit periodically. How to choose the marking field depends on the application and is out of scope here. However, some applications are reported in Section 5.
MULTIPOINT-ALT-MM] generalizes the counting for multipoint-to-multipoint flow. In case of link-based measurements, the behavior is similar except that coloring and counting operations are performed on a link-by-link basis at each endpoint of the link. Another important aspect to take into consideration is when to read the counters: in order to count the exact number of packets of a block, the routers must perform this operation when that block has ended; in other words, the counter for color A must be read when the current block has color B, in order to be sure that the value of the counter is stable. This task can be accomplished in two ways. The general approach suggests reading the counters periodically, many times during a block duration, and comparing these successive readings: when the counter stops incrementing, it means that the current block has ended, and its value can be elaborated safely. Alternatively, if the coloring operation is performed on the basis of a fixed timer, it is possible to configure the reading of the counters according to that timer: for example, reading the counter for color A every period in the middle of the subsequent block with color B is a safe choice. A sufficient margin should be considered between the end of a block and the reading of the counter, in order to take into account any out-of-order packets.
COLORING], while [IP-FLOW-REPORT] introduces new information elements of IP Flow Information Export (IPFIX) [RFC7011].
In practice, in addition to clock errors, the delay between measurement points also affects the implementation of the methodology because each packet can be delayed differently, and this can produce out of order at batch boundaries. This means that, without considering clock error, we wait L/2 after color switching to be sure to take a still counter. In summary, we need to take into account two contributions: clock error between network devices and the interval we need to wait to avoid packets being out of order because of network delay. The following figure explains both issues. ...BBBBBBBBB | AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA | BBBBBBBBB... |<======================================>| | L | ...=========>|<==================><==================>|<==========... | L/2 L/2 | |<===>| |<===>| d | | d |<==========================>| available counting interval Figure 4: Timing Aspects It is assumed that all network devices are synchronized to a common reference time with an accuracy of +/- A/2. Thus, the difference between the clock values of any two network devices is bounded by A. The guard band d is given by: d = A + D_max - D_min, where A is the clock accuracy, D_max is an upper bound on the network delay between the network devices, and D_min is a lower bound on the delay. The available counting interval is L - 2d that must be > 0. The condition that must be satisfied and is a requirement on the synchronization accuracy is: d < L/2.
Figure 2, R1 stores the timestamp TS(A1)R1 when it sends the first packet of block 1 (A-colored), the timestamp TS(B2)R1 when it sends the first packet of block 2 (B-colored), and so on for every other block. R2 performs the same operation on the receiving side, recording TS(A1)R2, TS(B2)R2, and so on. Since the timestamps refer to specific packets (the first packet of each block), we are sure that timestamps compared to compute delay refer to the same packets. By comparing TS(A1)R1 with TS(A1)R2 (and similarly TS(B2)R1 with TS(B2)R2, and so on), it is possible to measure the delay between R1 and R2. In order to have more measurements, it is possible to take and store more timestamps, referring to other packets within each block. In order to coherently compare timestamps collected on different routers, the clocks on the network nodes must be in sync. Furthermore, a measurement is valid only if no packet loss occurs and if packet misordering can be avoided; otherwise, the first packet of a block on R1 could be different from the first packet of the same block on R2 (for instance, if that packet is lost between R1 and R2 or it arrives after the next one). The following table shows how timestamps can be used to calculate the delay between R1 and R2. The first column lists the sequence of blocks, while other columns contain the timestamp referring to the first packet of each block on R1 and R2. The delay is computed as a difference between timestamps. For the sake of simplicity, all the values are expressed in milliseconds.
+-------+---------+---------+---------+---------+-------------+ | Block | TS(A)R1 | TS(B)R1 | TS(A)R2 | TS(B)R2 | Delay R1-R2 | +-------+---------+---------+---------+---------+-------------+ | 1 | 12.483 | - | 15.591 | - | 3.108 | | 2 | - | 6.263 | - | 9.288 | 3.025 | | 3 | 27.556 | - | 30.512 | - | 2.956 | | | - | 18.113 | - | 21.269 | 3.156 | | ... | ... | ... | ... | ... | ... | | 2n | 77.463 | - | 80.501 | - | 3.038 | | 2n+1 | - | 24.333 | - | 27.433 | 3.100 | +-------+---------+---------+---------+---------+-------------+ Table 2: Evaluation of Timestamps for Delay Measurements The first row shows timestamps taken on R1 and R2, respectively, and refers to the first packet of block 1 (which is A-colored). Delay can be computed as a difference between the timestamp on R2 and the timestamp on R1. Similarly, the second row shows timestamps (in milliseconds) taken on R1 and R2 and refers to the first packet of block 2 (which is B-colored). By comparing timestamps taken on different nodes in the network and referring to the same packets (identified using the alternation of colors), it is possible to measure delay on different network segments. For the sake of simplicity, in the above example, a single measurement is provided within a block, taking into account only the first packet of each block. The number of measurements can be easily increased by considering multiple packets in the block: for instance, a timestamp could be taken every N packets, thus generating multiple delay measurements. Taking this to the limit, in principle, the delay could be measured for each packet by taking and comparing the corresponding timestamps (possible but impractical from an implementation point of view).
the measurement error of a lot of packets. This method is robust to out-of-order packets and also to packet loss (only a small error is introduced). Moreover, it greatly reduces the number of timestamps (only one per block for each network device) that have to be collected by the management system. On the other hand, it only gives one measure for the duration of the block (for instance, 5 minutes), and it doesn't give the minimum, maximum, and median delay values [RFC6703]. This limitation could be overcome by reducing the duration of the block (for instance, from 5 minutes to a few seconds), which implicates a highly optimized implementation of the method.
Mean delay is calculated on all the packets of a sample and is a simple computation to be performed for a Single-Marking Method. In some cases, the mean delay measure is not sufficient to characterize the sample, and more statistics of delay extent data are needed, e.g., percentiles, variance, and median delay values. The conventional range (maximum-minimum) should be avoided for several reasons, including stability of the maximum delay due to the influence by outliers. RFC 5481 [RFC5481], Section 6.5 highlights how the 99.9th percentile of delay and delay variation is more helpful to performance planners. To overcome this drawback, the idea is to couple the mean delay measure for the entire batch with a Double-Marking Method, where a subset of batch packets is selected for extensive delay calculation by using a second marking. In this way, it is possible to perform a detailed analysis on these double- marked packets. Please note that there are classic algorithms for median and variance calculation, but they are out of the scope of this document. The comparison between the mean delay for the entire batch and the mean delay on these double-marked packets gives useful information since it is possible to understand if the Double-Marking measurements are actually representative of the delay trends. RFC3393]. The alternation of colors, for a Single-Marking Method, can be used as a time reference to measure delay variations. In case of Double Marking, the time reference is given by the second-marked packets. Considering the example depicted in Figure 2, R1 stores the timestamp TS(A)R1 whenever it sends the first packet of a block, and R2 stores the timestamp TS(B)R2 whenever it receives the first packet of a block. The inter-arrival jitter can be easily derived from one-way delay measurement, by evaluating the delay variation of consecutive samples. The concept of mean delay can also be applied to delay variation, by evaluating the average variation of the interval between consecutive packets of the flow from R1 to R2.