Independent Submission S. Floyd Request for Comments: 5690 ICIR Category: Informational A. Arcia ISSN: 2070-1721 D. Ros TELECOM Bretagne J. Iyengar Franklin & Marshall College February 2010 Adding Acknowledgement Congestion Control to TCP
AbstractThis document describes a possible congestion control mechanism for acknowledgement (ACKs) traffic in TCP. The document specifies an end-to-end acknowledgement congestion control mechanism for TCP that uses participation from both TCP hosts: the TCP data sender and the TCP data receiver. The TCP data sender detects lost or Explicit Congestion Notification (ECN)-marked ACK packets, and tells the TCP data receiver the ACK Ratio R to use to respond to the congestion on the reverse path from the data receiver to the data sender. The TCP data receiver sends roughly one ACK packet for every R data packets received. This mechanism is based on the acknowledgement congestion control in the Datagram Congestion Control Protocol's (DCCP's) Congestion Control Identifier (CCID) 2. This acknowledgement congestion control mechanism is being specified for further evaluation by the network community. Status of This Memo This document is not an Internet Standards Track specification; it is published for informational purposes. This is a contribution to the RFC Series, independently of any other RFC stream. The RFC Editor has chosen to publish this document at its discretion and makes no statement about its value for implementation or deployment. Documents approved for publication by the RFC Editor are not a candidate for any level of Internet Standard; see Section 2 of RFC 5741. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc5690.
IESG Note The content of this RFC was at one time considered by the IETF, and therefore it may resemble a current IETF work in progress or a published IETF work. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. 1. Introduction ....................................................3 2. Conventions and Terminology .....................................4 3. Overview ........................................................4 4. Acknowledgement Congestion Control ..............................6 4.1. The ACK Congestion Control Permitted Option ................6 4.2. The TCP ACK Ratio Option ...................................7 4.3. The Receiver: Implementing the ACK Ratio ...................7 4.4. The Sender: Determining Lost or Marked ACK Packets .........8 4.4.1. The Sender: Detecting Lost ACK Packets after a Congestion Event ...........................10 4.5. The Sender: Adjusting the ACK Ratio .......................10 4.5.1. Possible Addition: Decreasing the ACK Ratio after a Congestion Window Decrease .................12 4.6. The Receiver: Sending ACKs for Out-of-Order Data Segments ..................................................12 4.7. The Sender: Response to ACK Packets .......................13 4.8. Possible Addition: Receiver Bounds on the ACK Ratio .......15 5. Possible Complications .........................................15 5.1. Possible Complication: Delayed Acknowledgements ...........15 5.2. Possible Complication: Duplicate Acknowledgements .........15 5.3. Possible Complication: Two-Way Traffic ....................16 5.4. Possible Complication: Reordering of ACK Packets ..........16 5.5. Possible Complication: Abrupt Changes in the ACK Path .....17 5.6. Possible Complication: Corruption .........................17 5.7. Possible Complication: ACKs That Don't Contribute to Congestion .............................................17 5.8. Possible Complication: TCP Implementations that Skip ACK Packets ..........................................20
5.9. Possible Complication: Router or Middlebox-Based ACK Mechanisms ............................................21 5.10. Possible Complication: Data-Limited Senders ..............21 5.11. Other Issues .............................................22 6. Evaluating ACK Congestion Control ..............................22 6.1. Contention in Wireless Links or in Non-Switched Ethernet ..22 6.2. Keep-Alive and Other Special ACK Packets ..................22 7. Measurements of ACK Traffic and Congestion .....................23 8. Acknowledgement Congestion Control in DCCP's CCID 2 ............23 9. Security Considerations ........................................24 10. IANA Considerations ...........................................25 11. Conclusions ...................................................26 12. Acknowledgements ..............................................26 Appendix A. Related Work ..........................................27 A.1. ECN-Only Mechanisms .......................................28 A.2. Receiver-Only Mechanisms ..................................28 A.3. Middlebox-Based Mechanisms ................................29 Appendix B. Design Considerations .................................29 B.1. The TCP ACK Ratio Option, or an AckNow Bit in Data Packets? .............................................29 Normative References ..............................................30 Informative References ............................................30 RFC4340], [RFC4341]), which is a successor to the TCP acknowledgement congestion control mechanism proposed by Balakrishnan, et al. in [BPK97]. In this document we use the terminology of senders and receivers, with the sender sending data traffic and the receiver sending acknowledgement traffic in response. In CCID 2's acknowledgement congestion control, specified in Section 6.1 of [RFC4341], the receiver uses an ACK Ratio R reported to it by the sender, sending roughly one ACK packet for every R data packets received. The CCID 2 sender keeps the acknowledgement rate roughly TCP-friendly by monitoring the acknowledgement stream for lost and marked ACK packets and modifying the ACK Ratio accordingly. For every round-trip time (RTT) containing an ACK congestion event (that is, a lost or marked ACK packet), the sender halves the acknowledgement rate by doubling the ACK Ratio; for every RTT containing no ACK congestion event, the sender additively increases the acknowledgement rate through gradual decreases in the ACK Ratio.
The goal of this document is to explore a similar congestion control mechanism for acknowledgement traffic for TCP. The assumption is that in some environments with congestion on the reverse path, reducing the sending rate for ACK traffic traversing the congested path can help to reduce the congestion itself. For those environments where the reverse path is congested but where TCP ACK traffic does not appreciably contribute to that aggregate congestion, the goal is for TCP's ACK congestion control to have a minimal negative effect on the performance of the TCP connection. Adding acknowledgement congestion control as an option in TCP would require the following: * An agreement from the TCP hosts on the use of ACK congestion control. For the mechanism specified in this document, the TCP hosts would use a new TCP option, the ACK Congestion Control Permitted option. * A mechanism for the TCP sender to detect lost and ECN-marked pure acknowledgement packets. * A mechanism for adjusting the ACK Ratio. The TCP sender would adjust the ACK Ratio as specified in Section 6.1.2 of [RFC4341]. * A method for the TCP sender to inform the TCP receiver of a new value for the ACK Ratio. For the mechanism specified in this document, the TCP sender would use a new TCP option, the ACK Ratio option. RFC2119].
--------------------------------------------------------------- TCP Host A Router TCP Host B (data sender) (data receiver) ---------- ------ ---------- <--- SYN with AckCC Permitted. SYN/ACK with AckCC Permitted ---> . . . Data packets ---> <--- one ACK packet for every two data packets . . . Sender detects a lost ACK packet. Data packet with an ACK Ratio option of 4 ---> <--- one ACK packet for at most every four data packets . . . Sender detects a period with no lost ACK packets. Data packet with an ACK Ratio option of 3 ---> <--- one ACK packet for at most every three data packets --------------------------------------------------------------- Figure 1: Acknowledgement Congestion Control, Host B as the Connection Initiator, for a Connection without ECN Figure 1 gives an example of acknowledgement congestion control (AckCC) with TCP Host B as the connection initiator. During connection initiation, TCP host B sends an ACK Congestion Control Permitted option on its SYN or SYN/ACK packet. This allows TCP host A (now called the sender) to send instructions to TCP host B (now called the receiver) about the ACK Ratio to use in responding to data packets. Also during connection initiation, TCP host A sends an ACK Congestion Control Permitted option on its SYN or SYN/ACK packet. In combination with TCP host B's sending of an ACK Congestion Control Permitted option, and with the negotiation of ECN-Capability as specified in [RFC3168], this would allow TCP host B to send its ACK packets as ECN-Capable. The TCP receiver starts with an ACK Ratio of two, generally sending one ACK packet for every two data packets received.
The TCP sender detects a lost or ECN-marked ACK packet from the TCP receiver and sends an ACK Ratio option of four to the receiver. The TCP receiver changes to an ACK Ratio of four, sending one ACK packet for at most four data packets. The TCP sender uses Appropriate Byte Counting and rate-based pacing in responding to these ACK packets. The TCP sender detects a period with no lost ACK packets and sends an ACK Ratio option of three to the TCP receiver. The TCP receiver changes back to an ACK Ratio of three, sending one ACK packet for at most three data packets. RFC4341]), but it may also take advantage of explicit congestion information from the network, conveyed by ECN [RFC3168], if available. The ECN specification ([RFC3168], see Section 6.1.4) prohibits a TCP receiver from setting the ECT(0) or ECT(1) codepoints in IP packets carrying pure ACKs, but *only* as long as the receiver does *not* implement any form of ACK congestion control. Unlike some of the related work cited in the appendix, in this document we are proposing an end-to- end ACK congestion control mechanism that controls congestion on the reverse path (the path followed by the ACK traffic) by detecting and responding to either marked or dropped ACK packets.
If TCP end-point B receives an ACK Congestion Control Permitted option from TCP end-point A, then the TCP end-points may use ACK congestion control on the pure acknowledgements sent from A to B. If TCP end-point B receives an ACK Congestion Control Permitted option from TCP end-point A and also sent an ACK Congestion Control Permitted option to TCP end-point A, and if ECN-Capability has been negotiated, then TCP end-point B can send its pure ACK packets as ECN-Capable. TCP ACK Congestion Control Permitted Option: Kind: TBD1 +-----------+-----------+ | Kind=TBD1 | Length=2 | +-----------+-----------+ When ACK congestion control is used, the default initial ACK Ratio is two, with the receiver acknowledging at least every other data packet.
If the receiver has buffer limitations, the receiver might have to acknowledge K packets, for some K less than the current ACK Ratio R. In this case, the sender could observe from the acknowledgements that the receiver is acknowledging less than R packets. It is possible for there to be lost or marked ACK packets when there haven't yet been any lost or marked data packets. Thus, the sender could increase the ACK Ratio R even during the initial slow-start. [RFC5681] recommends that the receiver SHOULD acknowledge out-of- order data packets immediately, sending an immediate duplicate ACK when it receives a data segment above a gap in the sequence space, and sending an immediate ACK when it receives a data segment that fills in all or part of a gap in the sequence space. When ACK congestion control is being used and the ACK Ratio is at most two, the TCP receiver acknowledges each out-of-order data packet immediately. For an ACK Ratio greater than two, Section 4.6 specifies in detail the receiver's behavior for sending ACKs for out- of-order data packets.
Out-of-order data packets: If the ACK Ratio is at most two, then the TCP receiver sends a duplicate acknowledgement (DupACK) for every out-of-order data packet. In this case, the TCP sender should be able to detect lost DupACK packets by counting the number of DupACKs that arrive between the beginning of the loss event and the arrival of the first full or partial ACK, and comparing this number with the number of DupACKs that should have arrived (based on the number of packets being ACKed by the full or partial ACK). Simulations and/or experiments will be needed to determine whether, in practice, it works for the TCP sender to assess lost ACK packets during loss events, for an ACK Ratio of at most two. If the ACK Ratio is greater than two, the TCP receiver does not send a DupACK for every out-of-order data packet, as specified in Section 4.6. For simplicity, the TCP sender does not attempt to detect lost ACK packets during loss events involving forward-path data traffic. That is, as soon as the sender infers a packet loss for a forward-path data packet, it stops detection of ACK loss on the reverse path. The sender waits until a new cumulative acknowledgement is received that covers the retransmitted data, and then restarts detection of ACK loss for reverse-path traffic. Detecting lost ACK packets after changes in the ACK Ratio: In detecting lost ACK packets, the sender relies on its knowledge of the ACK Ratio used by the receiver. But when the sender makes a change in the ACK Ratio and then receives ACK packets, how does the sender know whether the receiver was using the new or the old ACK Ratio when it sent those ACK packets? As specified in the next section, the sender can make only one of two possible changes to the ACK Ratio within one round-trip time. The sender can decrease the ACK Ratio by one, from R to R-1, or the sender can double the ACK Ratio, increasing it from R to 2R. But, in detecting lost ACK packets after an increase in the ACK Ratio, the sender needs to know whether the receiver was using the old ACK Ratio R or the new ACK Ratio 2R. The sender sends ACK Ratio options only on data packets, and these data packets are acknowledged by the receiver. One possibility would be for the sender to save the sequence number of the last data packet that contained an ACK Ratio option and to remember whether that ACK Ratio option was for an increase or a decrease in the ACK Ratio. Then, if the sender receives an ACK packet acknowledging the saved sequence number, the sender knows that the receiver has begun using the new ACK Ratio.
It *might* be sufficient for the sender just to save the information of whether the last change in the ACK Ratio was an increase or a decrease, without saving the sequence number associated with the last ACK Ratio option. In this way, if the sender recently increased the ACK Ratio from R to 2R, the sender could be more cautious in detecting lost ACK packets. Another possibility would be that, after sending an ACK Ratio option, the sender waits until that data has been ACKed, with the new ACK Ratio in use by the receiver, before resuming the detection of lost ACK packets. However, we do not explore either of these approaches in more detail in this document. Section 6.1.2 of [RFC4341], as follows. The ACK Ratio always meets the following three constraints. (1) The ACK Ratio is an integer. (2) The minimum ACK sending rate: The ACK Ratio does not exceed max(2, cwnd/(K*MSS)), rounded up, for K=2. As a result, the TCP receiver generally sends at least two ACKs in response to a window of at least four full-sized segments.
(3) If the congestion window is at least as large as four full-sized segments, then the ACK Ratio is at least two. In other words, an ACK Ratio of one is only allowed when the congestion window is at most three full-sized segments. The sender changes the ACK Ratio within those constraints as follows. For each congestion window of data with lost or marked ACK packets, the ACK Ratio R is doubled; for each cwnd/(MSS*(R^2 - R)) consecutive congestion windows of data with no lost or marked ACK packets, the ACK Ratio is decreased by 1. (See Appendix A of RFC 4341 for the derivation. Note that Appendix A of RFC 4341 assumes a congestion window W in packets, while we use cwnd in bytes.) As stated in the previous section, when the ACK Ratio is greater than two, the sender does not attempt to detect lost ACK packets during loss events for forward-path traffic. For a constant congestion window, these modifications to the ACK Ratio give an ACK sending rate that is roughly TCP-friendly. Of course, cwnd usually varies over time; the dynamics will be rather complex, but roughly TCP friendly. We recommend that the sender determines when to decrease the ACK Ratio by one (i.e., by calculating the number of in-order data packets to count) right after an ACK loss event. The frequency of ACK Ratio negotiations: The sender need not keep the ACK Ratio completely up to date. For instance, it may rate-limit ACK Ratio renegotiations to once every four or five round-trip times, or to once every second or two. The sender should not attempt to change the ACK Ratio more than once per round-trip time. In particular, before sending a packet with a new value for the ACK Ratio, the sender should verify that the receiver has acknowledged a data packet containing an ACK Ratio option for the old value of the ACK Ratio. Additionally, the sender may enforce a minimum ACK Ratio of two, or it may set the ACK Ratio to one for half-connections with persistent congestion windows of 1 or 2 packets. The minimum ACK sending rate: From rule (2) above, the TCP receiver always sends at least K=2 ACKs for a window of data, even in the face of very heavy congestion on the reverse path. We would note, however, that if congestion is sufficiently heavy, all the ACK packets are dropped, and then the sender falls back on an exponentially backed-off timeout. Thus, if congestion is sufficiently heavy on the reverse path, then the sender reduces its sending rate on the forward
path, which reduces the rate on the reverse path as well. One possibility would be to use a higher minimum ACK-sending rate, adding a constant upper bound on the ACK Ratio. That is, if the ACK Ratio also had an upper bound of J, independent of cwnd, then the receiver would always send at least one ACK for every J data packets, regardless of the level of congestion on the reverse path. RFC 5681 says that "a TCP receiver SHOULD send an immediate duplicate ACK when an out-of-order segment arrives". After three duplicate ACKs are received, the TCP sender infers a packet loss and implements
fast retransmit and fast recovery, retransmitting the missing packet. When the ACK Ratio is at most two, the TCP receiver should still send an immediate duplicate ACK when an out-of-order segment arrives. In general, when the ACK Ratio is greater than two, the TCP receiver still should send an immediate duplicate ACK for each of the first three out-of-order segments that arrive in a reordering event. (We define a reordering event at the receiver as beginning when an out- of-order segment arrives, and ending when the receiver holds no more out-of-order segments.) However, when the ACK Ratio is greater than two, after the first three duplicate ACKs have been sent, the TCP receiver should perform ACK congestion control on the remaining ACKs to be sent during the current reordering event. That is, after the first three duplicate ACKs have been sent, the TCP receiver should return to sending an ACK for every R segments, instead of sending an ACK for every out-of-order segment in that reordering event. (We note that the fast recovery procedure of the TCP sender might have to be modified to take this change into account.) In addition, a receiver must not withhold an ACK for more than 500 ms. We note that in an environment with systematic reordering in the data path (e.g., every set of K data packets arrives in inverted order, for some value of K), the guideline above could result in the receiver sending an ACK for every data packet, regardless of the ACK Ratio. In such an environment with persistent reordering, the receiver may decide not to send an immediate duplicate ACK for each of the first three out-of-order segments that arrive in a reordering event. We leave the investigation of mechanisms for effective ACK congestion control in environments with systematic reordering for future work.
congestion window cwnd. As specified in RFC 5681, in slow-start the TCP sender increases cwnd by one full-sized segment for each new ACK received (in this context, a "new ACK" is an ACK that acknowledges new data). RFC 5681 also specifies that in congestion avoidance, the TCP sender increases cwnd by roughly 1/cwnd full-sized segments for each ACK received, resulting in an increase in cwnd of roughly one full-sized segment per round-trip time. In this case, the use of a large ACK Ratio would slow down the increase of the sender's congestion window. RFC 5681 notes that during congestion avoidance, it is also acceptable to count the number of bytes acknowledged by new ACKs and to increase cwnd based on the number of bytes acknowledged, rather than on the number of new ACKs received. Thus, the sender should use this form of byte counting with acknowledgement congestion control, so that the acknowledgement congestion control doesn't slow down the window increases for the data traffic sent by the sender. Because rate-based pacing should be used with acknowledgement congestion control, as recommended earlier in this section, the TCP sender may increase the congestion window by more than two MSS for each ACK. We note that for Appropriate Byte Counting (ABC) as specified in [RFC3465], during slow-start the sender is allowed to increase the congestion window by at most two MSS for each ACK. It has not yet been determined whether, with acknowledgement congestion control, the TCP sender could use ABC during slow-start. If ABC is used with acknowledgement congestion control, then when the TCP sender is in slow-start and the ACK Ratio is greater than two, the TCP sender may increase the congestion window by more that two MSS in response to a single ACK. Section 4.2 of [LL07] explores some of the issues with the use of ABC for TCP connections with a fixed ACK Ratio greater than two. Inferring lost data packets: As cited earlier, RFC 5681 infers that a packet has been lost after it receives three duplicate acknowledgements. Because ACK congestion control is only used when there is congestion on the reverse path, after a packet loss, one or more of the three duplicate ACKs sent by the receiver could be lost on the reverse path, and the receiver might wait until it has received R more out-of-order segments before sending the next duplicate ACK. All this could slow down fast recovery and fast retransmit quite a bit. The use of SACK can help reduce the potential delay in detecting a lost packet. With SACK, a TCP sender can use the information in the SACK option to detect when the receiver has
received at least three out-of-order data packets and to initiate fast retransmit and fast recovery in this case, even if the TCP sender has not yet received three duplicate ACKs. Section 4.3, RFC 5681 states that "a TCP receiver SHOULD send an immediate duplicate ACK when an out-of-order segment arrives", and that "a TCP receiver SHOULD send an immediate ACK when the incoming segment fills in all or part of a gap in the sequence space" [RFC5681]. When ACK congestion control is used, the TCP receiver instead uses the guidelines from Section 4.6 to govern the sending of duplicate ACKs. More work would be useful to evaluate the
advantages and disadvantages of this approach in terms of the potential delay in triggering fast retransmit, and to explore alternate possibilities. RFC4653].
J TCP conns. link L -> J TCP conns. data -> |---| |---| <- ACKs <-------------> | | | | <-------------> | | <-------------> | | <-------------> | | | | <-------------> K TCP conns. |---| |---| K TCP conns. ACKs -> <- link L1 <- data Figure 2. A Scenario with J Forward and K Reverse TCP Connections
To explore the relative contribution of ACK traffic on congestion, it is useful to consider a simple scenario with a congested unidirectional link L carrying data traffic from J TCP connections (the forward TCP connections) and ACK traffic from K TCP connections (the reverse TCP connections). We assume that all TCP connections have the same round-trip time R and the same data packet size S of 1500 bytes. We further assume that all of the forward TCP connections have the same data packet drop rate p and the same congestion window W, and that all of the reverse TCP connections have the same congestion window W1 and the same ACK packet drop rate p1. (The packet drop rate for data packets is defined as the fraction of arriving data packets that are dropped; similarly, the packet drop rate for ACK packets is the fraction of arriving ACK packets that are dropped.) The J TCP connections each use a bandwidth on link L of 1500*W/R bytes per second, and the K TCP connections, without ACK congestion control, each use a bandwidth on link L of 40*(W1/2)/R bytes per second. This gives a ratio of 75*(J/K)*(W/W1) for TCP data bandwidth to TCP ACK bandwidth on link L. The ratio J/K is the ratio between the number of forward and reverse TCP connections on link L, and could have a wide range of values (e.g., large for an access link from a web server, and small for an access link to a web server). For this scenario, the ratio W/W1 is largely a function of the different levels of congestion on the forward and reverse paths. To explore the possibilities, we will consider some of the range of congestion control mechanisms for the congested link. First, we consider scenarios where the limitation on the congested path is in the link bandwidth in bytes per second. Cases (1), (2), (3), (5), and (7) below represent the best scenarios for ACK congestion control, where the fraction of packet drops for TCP ACK packets roughly matches the TCP ACK packets' contribution to congestion. (In several of these cases this is, at best, a rough match because the data packets are a factor in the bandwidth and in the queue limitations, while the TCP ACK packets are only a factor in the queue limitations.) Cases (4) and (8) below represent problematic scenarios where the fraction of packet drops for TCP ACK packets is much higher than the TCP ACK packets' contribution to congestion (in terms of taking space in a congested queue, using scarce CPU cycles at the congested router, or using scarce bandwidth). Case (6) below represents scenarios where ACK congestion control would not be effective because it would not be invoked. In the scenarios in case (6), the fraction of packet drops for TCP ACK packets would be much smaller than the TCP ACK packets' contribution to congestion.
(1) The Drop-Tail queue for link L is measured in packets. In this case, the congested queue can accommodate N packets (regardless of packet size), there is a limitation of both bandwidth in bytes per second and also in queue space in packets, and large data packets and small TCP ACK packets should see similar packet drop rates. Although TCP ACK packets most likely aren't a major factor in the bandwidth limitation, they can be a significant contribution to the limitation of queue space. So, while the packet drop rate for ACK packets could be high in times of congestion, the ACK packets are contributing to that congestion somewhat by using scarce buffer space. (2) The Drop-Tail queue is measured in bytes. In this case, the congested queue can accommodate M bytes of packets, and TCP ACK packets don't make a significant contribution to either the bandwidth limitation or to the limitation in queue space. It is also the case that, in this scenario, even if there is heavy congestion, the packet drop rate for TCP ACK packets should be small (because small ACK packets can often find space on the congested queue when large data packets can't find space). In this case, ACK congestion control should not present any problems; the TCP ACK packets aren't contributing significantly to congestion and aren't experiencing significant packet drop rates. (3) The RED queue is in packet mode and is measured in packets. This is similar to case (1) above. Because the queue is measured in packets, small TCP ACK packets contribute to the limitation in queue space but not to the limitation in link bandwidth. Because the queue is in packet mode, large data packets and small TCP ACK packets should see similar packet drop rates. (4) The RED queue is in packet mode but is measured in bytes. Because the queue is measured in bytes, small TCP ACK packets don't contribute significantly to either the limitation in queue space or to the limitation in link bandwidth. Because the queue is in packet mode, large data packets and small TCP ACK packets should see similar packet drop rates. If it existed, this case would be problematic, because the TCP ACK packets would not be contributing significantly to the congestion but they would see a similar packet drop rate as the large data packets that are contributing to congestion. (5) The RED queue is in byte mode and is measured in bytes. This is similar to case (2) above. Because the queue is measured in bytes, small TCP ACK packets don't contribute significantly to either the limitation in queue space or to the limitation in link
bandwidth. At the same time, because the queue is in byte mode, small TCP ACK packets see much smaller packet drop rates than those of large data packets. (6) The RED queue is in byte mode but is measured in packets. Because the queue is measured in packets, small TCP ACK packets contribute to the limitation in queue space but not to the limitation in link bandwidth. Because the queue is in byte mode, small TCP ACK packets see much smaller packet drop rates than those of large data packets. If this case existed, TCP ACK packets would contribute somewhat to congestion but would see a much smaller packet drop rate than that of large data packets. Next, we consider scenarios where the limitation on the congested link is in CPU cycles at the router in packets per second, not in bandwidth in bytes per second. (7) The CPU load imposed by TCP ACK packets is similar to the load imposed by other packets (e.g., TCP data packets). ACK congestion control would be useful in this scenario, particularly if TCP ACK packets saw the same packet drop rates as TCP data packets. (8) The CPU load imposed by TCP ACK packets is much less than the load imposed by other packets (e.g., TCP data packets). If TCP ACK packets saw a smaller packet drop rate than TCP data packets, then the TCP ACK packet drop rate would roughly match the TCP ACK packets' contribution to congestion, and this would be good. If TCP ACK packets saw the same packet drop rate as TCP data packets, this case would be problematic, because the TCP ACK packets would not be contributing significantly to the congestion, but they would see a similar packet drop rate as the large data packets that are contributing to congestion.
One possibility for addressing this problem would be for TCP receivers using ACK congestion control to be required to send an acknowledgement for each R packets, for ACK Ratio R. In this case, if the receiver received a large burst of data packets back-to-back, the receiver would be required to send a responding burst of ACK packets, one for each set of R data packets. A second possibility for addressing this problem would be to define a TCP option or flag that the TCP receiver could use when sending an ACK packet to inform the sender that the TCP receiver `skipped' some ACK packets, so that the sender should not infer ACK loss if some previous ACK packets seem to be missing. Future work will explore the costs and benefits of these two approaches. BPK97], [WWCM99], [BA03], [KLS07]). We are not aware of the deployment of ACK filtering in the Internet, but any testing of ACK congestion control would have to look for interactions with any middlebox-based mechanisms regarding ACK packets. In particular, we would consider interactions of ACK congestion control with the possible deployment of ACK filtering on satellite links, cable modems, or the like.
congestion-controlled ACK flow and a timely ACK response to the sender for the specific case of a connection with a data-limited sender and a congested ACK path. AJ03], [KIA07]). At the same time, contention on the shared medium won't necessarily result in dropped ACK packets, and therefore wouldn't necessarily be detected by ACK congestion control. KEEP-ALIVE]. This keep-alive mechanism is not addressed in TCP specifications. However, such keep-alive packets, if used, should not interact with ACK congestion control one way or another. For ACK congestion control, the ACK Ratio is set small enough to allow the receiver to
generally send at least two ACKs for a window of data. In addition, the receiver uses a delayed ACK timer with the ACK Ratio, always sending an acknowledgement if the delayed ACK timer expires. Thus, ACK congestion control will never cause the receiver to delay indefinitely in sending an acknowledgement for a received data packet. Some TCP implementations send pure ACK packets as window probes, to solicit an ACK packet from the other end with current window information. Such ACK packets will generally be orthogonal to the ACK congestion control specified in this document. TCP receivers also can send pure ACK packets as window update packets announcing a new value for the receive window, even when the acknowledgement number and SACK options in the ACK packet are not new. The receiver may send window update packets even if the ACK congestion control mechanism would say that it is not time yet to send a pure ACK. The sender will not necessarily be able to detect the loss of a window update ACK packet. Studies]. Are there any studies that show the relative packet drop rates for TCP data and ACK traffic, for particular links or for particular TCP connections? Are there any studies of congested links that show the fraction of traffic on the congested link, or in the congested queue, that consist of TCP ACK packets? RFC4340], the congestion control mechanism for the CCID 2 profile is based on that of TCP. This section briefly discusses some of the issues that have been addressed in the acknowledgement congestion control already standardized in CCID 2 [RFC4341].
Rate-based pacing: For CCID 2, RFC 4341 says that "senders MAY use a form of rate- based pacing when sending multiple data packets liberated by a single ACK packet, rather than sending all liberated data packets in a single burst." However, rate-based pacing is not required in CCID 2. Increasing the congestion window: For CCID 2, RFC 4341 says that "when cwnd < ssthresh, meaning that the sender is in slow-start, the congestion window is increased by one packet for every two newly acknowledged data packets with ACK Vector State 0 (not ECN-marked), up to a maximum of ACK Ratio/2 packets per acknowledgement. This is a modified form of Appropriate Byte Counting [RFC3465] that is consistent with TCP's current standard (which does not include byte counting), but allows CCID 2 to increase as aggressively as TCP when CCID 2's ACK Ratio is greater than the default value of two. When cwnd >= ssthresh, the congestion window is increased by one packet for every window of data acknowledged without lost or marked packets."
control and still sending ACK packets as ECN-Capable, so ACK congestion control is not a necessary component for receivers to cheat about sending ECN-Capable ACK packets. One question would be whether there is any way for receivers to cheat about sending ECN- Capable ACK packets and not using appropriate ACK congestion control without this cheating being easily detected by the sender. What about the ability of routers or middleboxes to detect TCP receivers that cheat by inappropriately sending ACK packets as ECN- Capable? The router will only know if the receiver is authorized to send ACK packets as ECN-Capable if the router can see traffic on both the forward and reverse paths and monitored both the SYN and SYN/ACK packets (and was able to read the TCP options in the packet headers). If ACK congestion control has been negotiated, the router will only know if ACK congestion control is being used correctly by the receiver if it can monitor the ACK Ratio options sent from the sender to the receiver. If ACK congestion control is being used, the router will not necessarily be able to tell if ACK congestion control is being used correctly by the sender, because drops of ACK packets might be occurring after the ACK packets have left the router. However, if the router sees the ACK Ratio options sent from the sender, the router will be able to tell if the sender is correctly accounting for those ACK packets that are dropped or ECN-marked on the path from the receiver to the router. http://www.iana.org): Kind Length Meaning Reference ---- ------ --------------------------------- ----------- TBD1 2 ACK Congestion Control Permitted [RFCXXXX] TBD2 3 ACK Ratio [RFCXXXX] In the absence of TCP option numbers allocated by IANA, experimenters may use the TCP Option Numbers set aside for Experimentation in RFC 4727 [RFC4727]. As stressed in Section 1 of RFC 3692 [RFC3692], the TCP Option Numbers in the experimental range are intended for experimentation and testing and not for wide or general deployments; these option numbers could be in use by other experimentors for other purposes.
RFC3449] discusses TCP performance problems that arise in TCP connections over asymmetric paths. Section 3 of RFC 3449 describes in detail how congestion on the ACK path can affect overall TCP performance. RFC 3449 also outlines a number of proposed mitigations, including ACK congestion control. The experimental ACK congestion control mechanism discussed in that RFC relies on ECN, with the TCP sender detecting congestion on the ACK path from ECN- marked packets. RFC 3449 also discusses two receiver-based mechanisms, the Window Prediction Mechanism (WPM) [CLP98] and Acknowledgement based on Cwnd Estimation (ACE) [MJW00], for using a dynamic ACK Ratio. RFC 3449 also considers link- and network-layer techniques that address congestion on the upstream path. These include header compression as well as bandwidth management techniques for the upstream link, including ACK filtering and ACK reconstruction. RFC 3135 [RFC3135], "Performance Enhancing Proxies Intended to Mitigate Link-Related Degradations", surveys a range of Performance Enhancing Proxies used to improve TCP behavior, including proxies for ACK filtering and reconstruction. RFC 2760 [RFC2760], "Ongoing TCP Research Related to Satellites", discusses both ACK congestion control and ACK filtering and reconstruction, with detailed descriptions of the mechanisms proposed by Balakrishnan, et al. in [BPK97]. Landstrom, et al. in [LL07] explore a mechanism where the receiver sends only four acknowledgements per window of data, along with the sender using a form of Appropriate Byte Counting. In addition, the receiver reverts to a lower acknowledgement frequency after a timeout, to avoid unnecessary retransmit timeouts. One conclusion of the paper is that pacing at the sender introduces an additional delay and might not be necessary. A key result of the paper is that, with the use of some form of byte counting at the sender, it is possible for TCP to use a lower acknowledgement frequency than that of delayed acknowledgements.
BPK97] describe the use of ECN to detect congestion in the return path, in order to reduce the sending rate of ACKs. The use of a RED queue in the reverse path allows for marking of ACK packets. The sender echoes back ECN congestion marks to the receiver. The receiver keeps an ACK Ratio d (called the "delayed-ACK factor"), specifying the number of data segments that have to be received before the receiver sends a new ACK. The ACK Ratio d is managed using multiplicative-increase, additive-decrease; upon reception of a congestion mark, the receiver doubles the value of d (hence dividing the ACK sending rate by two). The ACK Ratio decreases linearly for each RTT in which no ECN-marked ACKs are received. Multiple congestion marks received in an RTT are treated as a single congestion event, i.e., d can be doubled at most once per RTT. The TCP timestamp option is used to keep track of the RTT values. MJW00], Tam Ming-Chit, et al. propose a receiver-based method for calculating an "appropriate" number of ACKs per congestion window (cwnd) of data, in order to alleviate congestion on the reverse path. The sender's cwnd is estimated at the receiver by counting the number of received packets per RTT (which also has to be estimated by the receiver). From this estimate, a simple algorithm is used to compute the number of ACKs to be sent per cwnd. The algorithm enforces a lower bound on the number of ACKs per cwnd, aiming at minimizing the probability of timeout at the sender due to ACK loss. Similarly, the ACK Ratio is upper-bounded so as to avoid excessive ACK delay. Blandford, et al. [BGG+07] propose an end-to-end, receiver-oriented scheme called "smartacking". The algorithm is based upon the receiver's monitoring the inter-segment arrival time for data packets and adapting the ACK sending rate in response. When the bottleneck link is underutilized, ACKs are sent frequently (up to one ACK per received segment) to promote fast growth of the congestion window. On the other hand, when the bottleneck is close to full utilization, the algorithm tries to reduce control traffic overhead and slow congestion window growth by generating ACKs at the minimum rate needed to keep the data pipe full. Reducing the number of ACKs (or, equivalently, increasing the amount of bytes acknowledged by each ACK) can increase the burstiness of the TCP sender. Hence, any mechanism as those cited above should be coupled with a burst mitigation technique, such as rate-based pacing, that paces the sending of data segments ([AB05], [ASA00], [BPK97]).
BPK97] from Balakrishnan, et al. is a router- based technique that tries to reduce the number of ACKs sent over the congested return link. With AF, an arriving ACK may replace preceding, older ACKs at the bottleneck queue. An aggressive replacement policy might guarantee that at most one ACK per connection is waiting in the queue, alleviating congestion. However, as in other proposals, care must be taken to avoid sender timeouts in case the (too few) ACKs resulting from the filtering get lost. The idea of filtering ACKs has been extended in [YMH03] to deal with SACK information. Aweya, et al. [AOM02] present a middlebox-based approach for mitigating data packet bursts and for controlling the uplink ACK congestion. The main idea is to perform pacing on ACK segments on an edge device close to the sender, so as to control the ACK arrival rate at the sender.
An additional disadvantage of the AckNow approach is that it would add complications and difficulties for the default cases of the receiver using an ACK Ratio of one or two, as is done in the absence of ACK congestion control. For these reasons, we have specified that the sender determines the ACK Ratio to use and tells the receiver, rather than the sender setting an AckNow bit in the TCP Header of selected data packets. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC3465] Allman, M., "TCP Congestion Control with Appropriate Byte Counting (ABC)", RFC 3465, February 2003. [RFC3692] Narten, T., "Assigning Experimental and Testing Numbers Considered Useful", BCP 82, RFC 3692, January 2004. [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, March 2006. [RFC4341] Floyd, S. and E. Kohler, "Profile for Datagram Congestion Control Protocol (DCCP) Congestion Control ID 2: TCP-like Congestion Control", RFC 4341, March 2006. [RFC4727] Fenner, B., "Experimental Values In IPv4, IPv6, ICMPv4, ICMPv6, UDP, and TCP Headers", RFC 4727, November 2006. [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion Control", RFC 5681, September 2009. [RFC2760] Allman, M., Ed., Dawkins, S., Glover, D., Griner, J., Tran, D., Henderson, T., Heidemann, J., Touch, J., Kruse, H., Ostermann, S., Scott, K., and J. Semke, "Ongoing TCP Research Related to Satellites", RFC 2760, February 2000. [RFC3135] Border, J., Kojo, M., Griner, J., Montenegro, G., and Z. Shelby, "Performance Enhancing Proxies Intended to Mitigate Link-Related Degradations", RFC 3135, June 2001.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, September 2001. [RFC3449] Balakrishnan, H., Padmanabhan, V., Fairhurst, G., and M. Sooriyabandara, "TCP Performance Implications of Network Path Asymmetry", BCP 69, RFC 3449, December 2002. [RFC4653] Bhandarkar, S., Reddy, A., Allman, M., and E. Blanton, "Improving the Robustness of TCP to Non-Congestion Events", RFC 4653, August 2006. [ASA00] Aggarwal, A., Savage, S., and T. Anderson, "Understanding the Performance of TCP Pacing", INFOCOM (3), pp. 1157-1165, 2000. [AB05] Allman, M., and E. Blanton, "Notes on Burst Mitigation for Transport Protocols", SIGCOMM, Computer Communications Review, 35(2):5360, 2005. [AJ03] Altman, E., and T. Jimenez, "Novel Delayed ACK Techniques for Improving TCP Performance in Multihop Wireless Networks", Proc. of the Personal Wireless Communications, 2003. [AOM02] Aweya, J., Ouellette, M., and D. Y. Montuno, "A Self- regulating TCP Acknowledgement (ack) Pacing Scheme", International Journal of Network Management, 12(3):145163, 2002. [BA03] Barakat, C., and E. Altman, "On ACK Filtering on a Slow Reverse Channel", International Journal of Satellite Communications and Networking, V.21 N.3, 2003. [BPK97] Balakrishnan, H., Padmanabhan, V., and Katz, R., "The Effects of Asymmetry on TCP Performance", Third ACM/IEEE Mobicom Conference, September 1997. [BGG+07] Blandford, D.K., Goldman, S.A., Gorinsky, S., Zhou, Y., and D.R. Dooly, "Smartacking: Improving TCP Performance from the Receiving End", Journal of Internet Engineering, 1(1), 2007. [CLP98] Calveras, A., Linares, J., and J. Paradells, "Window Prediction Mechanism for Improving TCP in Wireless Asymmetric Links". Proc. IEEE Global Communications Conference (GLOBECOM), Sydney Australia, pp. 533-538, November 1998.
[KIA07] Keceli, F., Inan, I., and E. Ayanoglu, "TCP ACK Congestion Control and Filtering for Fairness Provision in the Uplink of IEEE 802.11 Infrastructure Basic Service Set", Proc. IEEE ICC 2007, June 2007. [KEEP-ALIVE] Busatto, F., "TCP Keepalive HOWTO", May 2007, http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/index.html. [KLS07] Kim, H., Lee, H., and S. Shin, "On the Cross-Layer Impact of TCP ACK Thinning on IEEE 802.11 Wireless MAC Dynamics", IEICE Transactions on Communications, 2007. [LL07] Landstrom, S., and Larzon, L.A., "Reducing the TCP Acknowledgement Frequency", SIGCOMM, Computer Communications Review, July 2007. [MJW00] Ming-Chit, I.T., Jinsong, D., and W. Wang, "Improving TCP Performance Over Asymmetric Networks", SIGCOMM, Computer Communications Review (CCR), Vol.30, No.3, 2000. [Studies] Floyd, S., "Measurement Studies of End-to-End Congestion Control in the Internet", http://www.icir.org/floyd/ccmeasure.html. [WWCM99] Wu, H., Wu, J., Cheng, S., and J. Ma, "ACK Filtering on Bandwidth Asymmetry Networks", IEEE Communications, 1999. [YMH03] Yu, L., Minhua, Y., and Z. Huimin, "The Improvement of TCP Performance in Bandwidth Asymmetric Network", IEEE PIMRC, 1:482-486, September 2003.
http://www.ula.ve David Ros Networking, Security & Multimedia (RSM) Dpt. TELECOM Bretagne Rue de la Chataigneraie, CS 17607 35576 Cesson Sevigne Cedex France EMail: David.Ros@telecom-bretagne.eu Janardhan R. Iyengar Math and Computer Science Franklin & Marshall College P. O. Box 3003 Lancaster, PA 17604-3003 USA EMail: firstname.lastname@example.org