Network Working Group V. Paxson Request for Comments: 2525 Editor Category: Informational ACIRI / ICSI M. Allman NASA Glenn Research Center/Sterling Software S. Dawson Real-Time Computing Laboratory W. Fenner Xerox PARC J. Griner NASA Glenn Research Center I. Heavens Spider Software Ltd. K. Lahey NASA Ames Research Center/MRJ J. Semke Pittsburgh Supercomputing Center B. Volz Process Software Corporation March 1999 Known TCP Implementation Problems Status of this Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Copyright Notice Copyright (C) The Internet Society (1999). All Rights Reserved. 1. INTRODUCTION....................................................2 2. KNOWN IMPLEMENTATION PROBLEMS...................................3 2.1 No initial slow start........................................3 2.2 No slow start after retransmission timeout...................6 2.3 Uninitialized CWND...........................................9 2.4 Inconsistent retransmission.................................11 2.5 Failure to retain above-sequence data.......................13 2.6 Extra additive constant in congestion avoidance.............17 2.7 Initial RTO too low.........................................23 2.8 Failure of window deflation after loss recovery.............26 2.9 Excessively short keepalive connection timeout..............28 2.10 Failure to back off retransmission timeout..................31
2.11 Insufficient interval between keepalives....................34 2.12 Window probe deadlock.......................................36 2.13 Stretch ACK violation.......................................40 2.14 Retransmission sends multiple packets.......................43 2.15 Failure to send FIN notification promptly...................45 2.16 Failure to send a RST after Half Duplex Close...............47 2.17 Failure to RST on close with data pending...................50 2.18 Options missing from TCP MSS calculation....................54 3. SECURITY CONSIDERATIONS........................................56 4. ACKNOWLEDGEMENTS...............................................56 5. REFERENCES.....................................................57 6. AUTHORS' ADDRESSES.............................................58 7. FULL COPYRIGHT STATEMENT.......................................60
Implications Why the problem is viewed as a problem. Relevant RFCs The RFCs defining the TCP specification with which the problem conflicts. These RFCs often qualify behavior using terms such as MUST, SHOULD, MAY, and others written capitalized. See RFC 2119 for the exact interpretation of these terms. Trace file demonstrating the problem One or more ASCII trace files demonstrating the problem, if applicable. Trace file demonstrating correct behavior One or more examples of how correct behavior appears in a trace, if applicable. References References that further discuss the problem. How to detect How to test an implementation to see if it exhibits the problem. This discussion may include difficulties and subtleties associated with causing the problem to manifest itself, and with interpreting traces to detect the presence of the problem (if applicable). How to fix For known causes of the problem, how to correct the implementation. RFC 1122, 126.96.36.199, to engage in a "slow start" by initializing its congestion window, cwnd, to one packet (one segment of the maximum size). (Note that an experimental change to TCP, documented in [RFC2414], allows an initial value somewhat larger than one packet.) It subsequently increases cwnd by one packet for each ACK it receives for new data. The minimum of cwnd and the
receiver's advertised window bounds the highest sequence number the TCP can transmit. A TCP that fails to initialize and increment cwnd in this fashion exhibits "No initial slow start". Significance In congested environments, detrimental to the performance of other connections, and possibly to the connection itself. Implications A TCP failing to slow start when beginning a connection results in traffic bursts that can stress the network, leading to excessive queueing delays and packet loss. Implementations exhibiting this problem might do so because they suffer from the general problem of not including the required congestion window. These implementations will also suffer from "No slow start after retransmission timeout". There are different shades of "No initial slow start". From the perspective of stressing the network, the worst is a connection that simply always sends based on the receiver's advertised window, with no notion of a separate congestion window. Another form is described in "Uninitialized CWND" below. Relevant RFCs RFC 1122 requires use of slow start. RFC 2001 gives the specifics of slow start. Trace file demonstrating it Made using tcpdump [Jacobson89] recording at the connection responder. No losses reported by the packet filter. 10:40:42.244503 B > A: S 1168512000:1168512000(0) win 32768 <mss 1460,nop,wscale 0> (DF) [tos 0x8] 10:40:42.259908 A > B: S 3688169472:3688169472(0) ack 1168512001 win 32768 <mss 1460> 10:40:42.389992 B > A: . ack 1 win 33580 (DF) [tos 0x8] 10:40:42.664975 A > B: P 1:513(512) ack 1 win 32768 10:40:42.700185 A > B: . 513:1973(1460) ack 1 win 32768 10:40:42.718017 A > B: . 1973:3433(1460) ack 1 win 32768 10:40:42.762945 A > B: . 3433:4893(1460) ack 1 win 32768 10:40:42.811273 A > B: . 4893:6353(1460) ack 1 win 32768 10:40:42.829149 A > B: . 6353:7813(1460) ack 1 win 32768 10:40:42.853687 B > A: . ack 1973 win 33580 (DF) [tos 0x8] 10:40:42.864031 B > A: . ack 3433 win 33580 (DF) [tos 0x8]
After the third packet, the connection is established. A, the connection responder, begins transmitting to B, the connection initiator. Host A quickly sends 6 packets comprising 7812 bytes, even though the SYN exchange agreed upon an MSS of 1460 bytes (implying an initial congestion window of 1 segment corresponds to 1460 bytes), and so A should have sent at most 1460 bytes. The ACKs sent by B to A in the last two lines indicate that this trace is not a measurement error (slow start really occurring but the corresponding ACKs having been dropped by the packet filter). A second trace confirmed that the problem is repeatable. Trace file demonstrating correct behavior Made using tcpdump recording at the connection originator. No losses reported by the packet filter. 12:35:31.914050 C > D: S 1448571845:1448571845(0) win 4380 <mss 1460> 12:35:32.068819 D > C: S 1755712000:1755712000(0) ack 1448571846 win 4096 12:35:32.069341 C > D: . ack 1 win 4608 12:35:32.075213 C > D: P 1:513(512) ack 1 win 4608 12:35:32.286073 D > C: . ack 513 win 4096 12:35:32.287032 C > D: . 513:1025(512) ack 1 win 4608 12:35:32.287506 C > D: . 1025:1537(512) ack 1 win 4608 12:35:32.432712 D > C: . ack 1537 win 4096 12:35:32.433690 C > D: . 1537:2049(512) ack 1 win 4608 12:35:32.434481 C > D: . 2049:2561(512) ack 1 win 4608 12:35:32.435032 C > D: . 2561:3073(512) ack 1 win 4608 12:35:32.594526 D > C: . ack 3073 win 4096 12:35:32.595465 C > D: . 3073:3585(512) ack 1 win 4608 12:35:32.595947 C > D: . 3585:4097(512) ack 1 win 4608 12:35:32.596414 C > D: . 4097:4609(512) ack 1 win 4608 12:35:32.596888 C > D: . 4609:5121(512) ack 1 win 4608 12:35:32.733453 D > C: . ack 4097 win 4096 References This problem is documented in [Paxson97]. How to detect For implementations always manifesting this problem, it shows up immediately in a packet trace or a sequence plot, as illustrated above.
How to fix If the root problem is that the implementation lacks a notion of a congestion window, then unfortunately this requires significant work to fix. However, doing so is important, as such implementations also exhibit "No slow start after retransmission timeout". RFC 1122, 188.8.131.52, to engage in "slow start" by initializing its congestion window, cwnd, to one packet (one segment of the maximum size). It subsequently increases cwnd by one packet for each ACK it receives for new data until it reaches the "congestion avoidance" threshold, ssthresh, at which point the congestion avoidance algorithm for updating the window takes over. A TCP that fails to enter slow start upon a timeout exhibits "No slow start after retransmission timeout". Significance In congested environments, severely detrimental to the performance of other connections, and also the connection itself. Implications Entering slow start upon timeout forms one of the cornerstones of Internet congestion stability, as outlined in [Jacobson88]. If TCPs fail to do so, the network becomes at risk of suffering "congestion collapse" [RFC896]. Relevant RFCs RFC 1122 requires use of slow start after loss. RFC 2001 gives the specifics of how to implement slow start. RFC 896 describes congestion collapse. The retransmission timeout discussed here should not be confused with the separate "fast recovery" retransmission mechanism discussed in RFC 2001. Trace file demonstrating it Made using tcpdump recording at the sending TCP (A). No losses reported by the packet filter.
10:40:59.090612 B > A: . ack 357125 win 33580 (DF) [tos 0x8] 10:40:59.222025 A > B: . 357125:358585(1460) ack 1 win 32768 10:40:59.868871 A > B: . 357125:358585(1460) ack 1 win 32768 10:41:00.016641 B > A: . ack 364425 win 33580 (DF) [tos 0x8] 10:41:00.036709 A > B: . 364425:365885(1460) ack 1 win 32768 10:41:00.045231 A > B: . 365885:367345(1460) ack 1 win 32768 10:41:00.053785 A > B: . 367345:368805(1460) ack 1 win 32768 10:41:00.062426 A > B: . 368805:370265(1460) ack 1 win 32768 10:41:00.071074 A > B: . 370265:371725(1460) ack 1 win 32768 10:41:00.079794 A > B: . 371725:373185(1460) ack 1 win 32768 10:41:00.089304 A > B: . 373185:374645(1460) ack 1 win 32768 10:41:00.097738 A > B: . 374645:376105(1460) ack 1 win 32768 10:41:00.106409 A > B: . 376105:377565(1460) ack 1 win 32768 10:41:00.115024 A > B: . 377565:379025(1460) ack 1 win 32768 10:41:00.123576 A > B: . 379025:380485(1460) ack 1 win 32768 10:41:00.132016 A > B: . 380485:381945(1460) ack 1 win 32768 10:41:00.141635 A > B: . 381945:383405(1460) ack 1 win 32768 10:41:00.150094 A > B: . 383405:384865(1460) ack 1 win 32768 10:41:00.158552 A > B: . 384865:386325(1460) ack 1 win 32768 10:41:00.167053 A > B: . 386325:387785(1460) ack 1 win 32768 10:41:00.175518 A > B: . 387785:389245(1460) ack 1 win 32768 10:41:00.210835 A > B: . 389245:390705(1460) ack 1 win 32768 10:41:00.226108 A > B: . 390705:392165(1460) ack 1 win 32768 10:41:00.241524 B > A: . ack 389245 win 8760 (DF) [tos 0x8] The first packet indicates the ack point is 357125. 130 msec after receiving the ACK, A transmits the packet after the ACK point, 357125:358585. 640 msec after this transmission, it retransmits 357125:358585, in an apparent retransmission timeout. At this point, A's cwnd should be one MSS, or 1460 bytes, as A enters slow start. The trace is consistent with this possibility. B replies with an ACK of 364425, indicating that A has filled a sequence hole. At this point, A's cwnd should be 1460*2 = 2920 bytes, since in slow start receiving an ACK advances cwnd by MSS. However, A then launches 19 consecutive packets, which is inconsistent with slow start. A second trace confirmed that the problem is repeatable. Trace file demonstrating correct behavior Made using tcpdump recording at the sending TCP (C). No losses reported by the packet filter. 12:35:48.442538 C > D: P 465409:465921(512) ack 1 win 4608 12:35:48.544483 D > C: . ack 461825 win 4096 12:35:48.703496 D > C: . ack 461825 win 4096 12:35:49.044613 C > D: . 461825:462337(512) ack 1 win 4608
12:35:49.192282 D > C: . ack 465921 win 2048 12:35:49.192538 D > C: . ack 465921 win 4096 12:35:49.193392 C > D: P 465921:466433(512) ack 1 win 4608 12:35:49.194726 C > D: P 466433:466945(512) ack 1 win 4608 12:35:49.350665 D > C: . ack 466945 win 4096 12:35:49.351694 C > D: . 466945:467457(512) ack 1 win 4608 12:35:49.352168 C > D: . 467457:467969(512) ack 1 win 4608 12:35:49.352643 C > D: . 467969:468481(512) ack 1 win 4608 12:35:49.506000 D > C: . ack 467969 win 3584 After C transmits the first packet shown to D, it takes no action in response to D's ACKs for 461825, because the first packet already reached the advertised window limit of 4096 bytes above 461825. 600 msec after transmitting the first packet, C retransmits 461825:462337, presumably due to a timeout. Its congestion window is now MSS (512 bytes). D acks 465921, indicating that C's retransmission filled a sequence hole. This ACK advances C's cwnd from 512 to 1024. Very shortly after, D acks 465921 again in order to update the offered window from 2048 to 4096. This ACK does not advance cwnd since it is not for new data. Very shortly after, C responds to the newly enlarged window by transmitting two packets. D acks both, advancing cwnd from 1024 to 1536. C in turn transmits three packets. References This problem is documented in [Paxson97]. How to detect Packet loss is common enough in the Internet that generally it is not difficult to find an Internet path that will force retransmission due to packet loss. If the effective window prior to loss is large enough, however, then the TCP may retransmit using the "fast recovery" mechanism described in RFC 2001. In a packet trace, the signature of fast recovery is that the packet retransmission occurs in response to the receipt of three duplicate ACKs, and subsequent duplicate ACKs may lead to the transmission of new data, above both the ack point and the highest sequence transmitted so far. An absence of three duplicate ACKs prior to retransmission suffices to distinguish between timeout and fast recovery retransmissions. In the face of only observing fast recovery retransmissions, generally it is not difficult to repeat the data transfer until observing a timeout retransmission.
Once armed with a trace exhibiting a timeout retransmission, determining whether the TCP follows slow start is done by computing the correct progression of cwnd and comparing it to the amount of data transmitted by the TCP subsequent to the timeout retransmission. How to fix If the root problem is that the implementation lacks a notion of a congestion window, then unfortunately this requires significant work to fix. However, doing so is critical, for reasons outlined above. RFC2414]). One particular form of "No initial slow start", worth separate mention as the bug is fairly widely deployed, is "Uninitialized CWND". That is, while the TCP implements the proper slow start mechanism, it fails to initialize cwnd properly, so slow start in fact fails to occur. One way the bug can occur is if, during the connection establishment handshake, the SYN ACK packet arrives without an MSS option. The faulty implementation uses receipt of the MSS option to initialize cwnd to one segment; if the option fails to arrive, then cwnd is instead initialized to a very large value. Significance In congested environments, detrimental to the performance of other connections, and likely to the connection itself. The burst can be so large (see below) that it has deleterious effects even in uncongested environments. Implications A TCP exhibiting this behavior is stressing the network with a large burst of packets, which can cause loss in the network. Relevant RFCs RFC 1122 requires use of slow start. RFC 2001 gives the specifics of slow start.
Trace file demonstrating it This trace was made using tcpdump running on host A. Host A is the sender and host B is the receiver. The advertised window and timestamp options have been omitted for clarity, except for the first segment sent by host A. Note that A sends an MSS option in its initial SYN but B does not include one in its reply. 16:56:02.226937 A > B: S 237585307:237585307(0) win 8192 <mss 536,nop,wscale 0,nop,nop,timestamp[|tcp]> 16:56:02.557135 B > A: S 1617216000:1617216000(0) ack 237585308 win 16384 16:56:02.557788 A > B: . ack 1 win 8192 16:56:02.566014 A > B: . 1:537(536) ack 1 16:56:02.566557 A > B: . 537:1073(536) ack 1 16:56:02.567120 A > B: . 1073:1609(536) ack 1 16:56:02.567662 A > B: P 1609:2049(440) ack 1 16:56:02.568349 A > B: . 2049:2585(536) ack 1 16:56:02.568909 A > B: . 2585:3121(536) ack 1 [54 additional burst segments deleted for brevity] 16:56:02.936638 A > B: . 32065:32601(536) ack 1 16:56:03.018685 B > A: . ack 1 After the three-way handshake, host A bursts 61 segments into the network, before duplicate ACKs on the first segment cause a retransmission to occur. Since host A did not wait for the ACK on the first segment before sending additional segments, it is exhibiting "Uninitialized CWND" Trace file demonstrating correct behavior See the example for "No initial slow start". References This problem is documented in [Paxson97]. How to detect This problem can be detected by examining a packet trace recorded at either the sender or the receiver. However, the bug can be difficult to induce because it requires finding a remote TCP peer that does not send an MSS option in its SYN ACK. How to fix This problem can be fixed by ensuring that cwnd is initialized upon receipt of a SYN ACK, even if the SYN ACK does not contain an MSS option.
RFC 793, section 1.5, discusses the central role of reliability in TCP operation. Trace file demonstrating it Made using tcpdump recording at the receiving TCP (B). No losses reported by the packet filter. 12:35:53.145503 A > B: FP 90048435:90048461(26) ack 393464682 win 4096 4500 0042 9644 0000 3006 e4c2 86b1 0401 83f3 010a b2a4 0015 055e 07b3 1773 cb6a 5019 1000 68a9 0000 data starts here>504f 5254 2031 3334 2c31 3737*2c34 2c31 2c31 3738 2c31 3635 0d0a 12:35:53.146479 B > A: R 393464682:393464682(0) win 8192 12:35:53.851714 A > B: FP 90048429:90048463(34) ack 393464682 win 4096 4500 004a 965b 0000 3006 e4a3 86b1 0401 83f3 010a b2a4 0015 055e 07ad 1773 cb6a 5019 1000 8bd3 0000 data starts here>5041 5356 0d0a 504f 5254 2031 3334 2c31 3737*2c31 3035 2c31 3431 2c34 2c31 3539 0d0a
The sequence numbers shown in this trace are absolute and not adjusted to reflect the ISN. The 4-digit hex values show a dump of the packet's IP and TCP headers, as well as payload. A first sends to B data for 90048435:90048461. The corresponding data begins with hex words 504f, 5254, etc. B responds with a RST. Since the recording location was local to B, it is unknown whether A received the RST. A then sends 90048429:90048463, which includes six sequence positions below the earlier transmission, all 26 positions of the earlier transmission, and two additional sequence positions. The retransmission disagrees starting just after sequence 90048447, annotated above with a leading '*'. These two bytes were originally transmitted as hex 2c34 but retransmitted as hex 2c31. Subsequent positions disagree as well. This behavior has been observed in other traces involving different hosts. It is unknown how to repeat it. In this instance, no corruption would occur, since B has already indicated it will not accept further packets from A. A second example illustrates a slightly different instance of the problem. The tracing again was made with tcpdump at the receiving TCP (D). 22:23:58.645829 C > D: P 185:212(27) ack 565 win 4096 4500 0043 90a3 0000 3306 0734 cbf1 9eef 83f3 010a 0525 0015 a3a2 faba 578c 70a4 5018 1000 9a53 0000 data starts here>504f 5254 2032 3033 2c32 3431 2c31 3538 2c32 3339 2c35 2c34 330d 0a 22:23:58.646805 D > C: . ack 184 win 8192 4500 0028 beeb 0000 3e06 ce06 83f3 010a cbf1 9eef 0015 0525 578c 70a4 a3a2 fab9 5010 2000 342f 0000 22:31:36.532244 C > D: FP 186:213(27) ack 565 win 4096 4500 0043 9435 0000 3306 03a2 cbf1 9eef 83f3 010a 0525 0015 a3a2 fabb 578c 70a4 5019 1000 9a51 0000 data starts here>504f 5254 2032 3033 2c32 3431 2c31 3538 2c32 3339 2c35 2c34 330d 0a
In this trace, sequence numbers are relative. C sends 185:212, but D only sends an ACK for 184 (so sequence number 184 is missing). C then sends 186:213. The packet payload is identical to the previous payload, but the base sequence number is one higher, resulting in an inconsistent retransmission. Neither trace exhibits checksum errors. Trace file demonstrating correct behavior (Omitted, as presumably correct behavior is obvious.) References None known. How to detect This problem unfortunately can be very difficult to detect, since available experience indicates it is quite rare that it is manifested. No "trigger" has been identified that can be used to reproduce the problem. How to fix In the absence of a known "trigger", we cannot always assess how to fix the problem. In one implementation (not the one illustrated above), the problem manifested itself when (1) the sender received a zero window and stalled; (2) eventually an ACK arrived that offered a window larger than that in effect at the time of the stall; (3) the sender transmitted out of the buffer of data it held at the time of the stall, but (4) failed to limit this transfer to the buffer length, instead using the newly advertised (and larger) offered window. Consequently, in addition to the valid buffer contents, it sent whatever garbage values followed the end of the buffer. If it then retransmitted the corresponding sequence numbers, at that point it sent the correct data, resulting in an inconsistent retransmission. Note that this instance of the problem reflects a more general problem, that of initially transmitting incorrect data.
Description When a TCP receives an "above sequence" segment, meaning one with a sequence number exceeding RCV.NXT but below RCV.NXT+RCV.WND, it SHOULD queue the segment for later delivery (RFC 1122, 184.108.40.206). (See RFC 793 for the definition of RCV.NXT and RCV.WND.) A TCP that fails to do so is said to exhibit "Failure to retain above- sequence data". It may sometimes be appropriate for a TCP to discard above- sequence data to reclaim memory. If they do so only rarely, then we would not consider them to exhibit this problem. Instead, the particular concern is with TCPs that always discard above-sequence data. Significance In environments prone to packet loss, detrimental to the performance of both other connections and the connection itself. Implications In times of congestion, a failure to retain above-sequence data will lead to numerous otherwise-unnecessary retransmissions, aggravating the congestion and potentially reducing performance by a large factor. Relevant RFCs RFC 1122 revises RFC 793 by upgrading the latter's MAY to a SHOULD on this issue. Trace file demonstrating it Made using tcpdump recording at the receiving TCP. No losses reported by the packet filter. B is the TCP sender, A the receiver. A exhibits failure to retain above sequence-data: 10:38:10.164860 B > A: . 221078:221614(536) ack 1 win 33232 [tos 0x8] 10:38:10.170809 B > A: . 221614:222150(536) ack 1 win 33232 [tos 0x8] 10:38:10.177183 B > A: . 222150:222686(536) ack 1 win 33232 [tos 0x8] 10:38:10.225039 A > B: . ack 222686 win 25800 Here B has sent up to (relative) sequence 222686 in-sequence, and A accordingly acknowledges. 10:38:10.268131 B > A: . 223222:223758(536) ack 1 win 33232 [tos 0x8] 10:38:10.337995 B > A: . 223758:224294(536) ack 1 win 33232 [tos 0x8] 10:38:10.344065 B > A: . 224294:224830(536) ack 1 win 33232 [tos 0x8] 10:38:10.350169 B > A: . 224830:225366(536) ack 1 win 33232 [tos 0x8] 10:38:10.356362 B > A: . 225366:225902(536) ack 1 win 33232 [tos 0x8]
10:38:10.362445 B > A: . 225902:226438(536) ack 1 win 33232 [tos 0x8] 10:38:10.368579 B > A: . 226438:226974(536) ack 1 win 33232 [tos 0x8] 10:38:10.374732 B > A: . 226974:227510(536) ack 1 win 33232 [tos 0x8] 10:38:10.380825 B > A: . 227510:228046(536) ack 1 win 33232 [tos 0x8] 10:38:10.387027 B > A: . 228046:228582(536) ack 1 win 33232 [tos 0x8] 10:38:10.393053 B > A: . 228582:229118(536) ack 1 win 33232 [tos 0x8] 10:38:10.399193 B > A: . 229118:229654(536) ack 1 win 33232 [tos 0x8] 10:38:10.405356 B > A: . 229654:230190(536) ack 1 win 33232 [tos 0x8] A now receives 13 additional packets from B. These are above- sequence because 222686:223222 was dropped. The packets do however fit within the offered window of 25800. A does not generate any duplicate ACKs for them. The trace contributor (V. Paxson) verified that these 13 packets had valid IP and TCP checksums. 10:38:11.917728 B > A: . 222686:223222(536) ack 1 win 33232 [tos 0x8] 10:38:11.930925 A > B: . ack 223222 win 32232 B times out for 222686:223222 and retransmits it. Upon receiving it, A only acknowledges 223222. Had it retained the valid above- sequence packets, it would instead have ack'd 230190. 10:38:12.048438 B > A: . 223222:223758(536) ack 1 win 33232 [tos 0x8] 10:38:12.054397 B > A: . 223758:224294(536) ack 1 win 33232 [tos 0x8] 10:38:12.068029 A > B: . ack 224294 win 31696 B retransmits two more packets, and A only acknowledges them. This pattern continues as B retransmits the entire set of previously-received packets. A second trace confirmed that the problem is repeatable. Trace file demonstrating correct behavior Made using tcpdump recording at the receiving TCP (C). No losses reported by the packet filter. 09:11:25.790417 D > C: . 33793:34305(512) ack 1 win 61440 09:11:25.791393 D > C: . 34305:34817(512) ack 1 win 61440 09:11:25.792369 D > C: . 34817:35329(512) ack 1 win 61440 09:11:25.792369 D > C: . 35329:35841(512) ack 1 win 61440 09:11:25.793345 D > C: . 36353:36865(512) ack 1 win 61440 09:11:25.794321 C > D: . ack 35841 win 59904 A sequence hole occurs because 35841:36353 has been dropped.
09:11:25.794321 D > C: . 36865:37377(512) ack 1 win 61440 09:11:25.794321 C > D: . ack 35841 win 59904 09:11:25.795297 D > C: . 37377:37889(512) ack 1 win 61440 09:11:25.795297 C > D: . ack 35841 win 59904 09:11:25.796273 C > D: . ack 35841 win 61440 09:11:25.798225 D > C: . 37889:38401(512) ack 1 win 61440 09:11:25.799201 C > D: . ack 35841 win 61440 09:11:25.807009 D > C: . 38401:38913(512) ack 1 win 61440 09:11:25.807009 C > D: . ack 35841 win 61440 (many additional lines omitted) 09:11:25.884113 D > C: . 52737:53249(512) ack 1 win 61440 09:11:25.884113 C > D: . ack 35841 win 61440 Each additional, above-sequence packet C receives from D elicits a duplicate ACK for 35841. 09:11:25.887041 D > C: . 35841:36353(512) ack 1 win 61440 09:11:25.887041 C > D: . ack 53249 win 44032 D retransmits 35841:36353 and C acknowledges receipt of data all the way up to 53249. References This problem is documented in [Paxson97]. How to detect Packet loss is common enough in the Internet that generally it is not difficult to find an Internet path that will result in some above-sequence packets arriving. A TCP that exhibits "Failure to retain ..." may not generate duplicate ACKs for these packets. However, some TCPs that do retain above-sequence data also do not generate duplicate ACKs, so failure to do so does not definitively identify the problem. Instead, the key observation is whether upon retransmission of the dropped packet, data that was previously above-sequence is acknowledged. Two considerations in detecting this problem using a packet trace are that it is easiest to do so with a trace made at the TCP receiver, in order to unambiguously determine which packets arrived successfully, and that such packets may still be correctly discarded if they arrive with checksum errors. The latter can be tested by capturing the entire packet contents and performing the IP and TCP checksum algorithms to verify their integrity; or by confirming that the packets arrive with the same checksum and contents as that with which they were sent, with a presumption that the sending TCP correctly calculates checksums for the packets it transmits.
It is considerably easier to verify that an implementation does NOT exhibit this problem. This can be done by recording a trace at the data sender, and observing that sometimes after a retransmission the receiver acknowledges a higher sequence number than just that which was retransmitted. How to fix If the root problem is that the implementation lacks buffer, then then unfortunately this requires significant work to fix. However, doing so is important, for reasons outlined above.