When TCP Starts Up With Four Packets Into Only Three Buffers
A simulation showing that a TCP connection starting with four packets is not worse on a slow modem link than starting with one.
Shepard, T. and C. Partridge, "When TCP Starts Up With Four Packets Into Only Three Buffers", RFC 2416, DOI 10.17487/RFC2416, September 1998, <https://www.rfc-editor.org/info/rfc2416>.
@misc{rfc2416,
series = {Request for Comments},
number = 2416,
howpublished = {RFC 2416},
publisher = {RFC Editor},
doi = {10.17487/RFC2416},
url = {https://www.rfc-editor.org/info/rfc2416},
author = {T. Shepard and C. Partridge},
title = {{When TCP Starts Up With Four Packets Into Only Three Buffers}},
pagetotal = 7,
year = 1998,
month = sep,
} RFC text
Contents
Network Working Group T. Shepard
Request for Comments: 2416 C. Partridge
Category: Informational BBN Technologies
September 1998
When TCP Starts Up With Four Packets Into Only Three Buffers
Status of this Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (1998). All Rights Reserved.
Abstract
This memo is to document a simple experiment. The experiment showed
that in the case of a TCP receiver behind a 9600 bps modem link at
the edge of a fast Internet where there are only 3 buffers before the
modem (and the fourth packet of a four-packet start will surely be
dropped), no significant degradation in performance is experienced by
a TCP sending with a four-packet start when compared with a normal
slow start (which starts with just one packet).
Background
Sally Floyd has proposed that TCPs start their initial slow start by
sending as many as four packets (instead of the usual one packet) as
a means of getting TCP up-to-speed faster. (Slow starts instigated
due to timeouts would still start with just one packet.) Starting
with more than one packet might reduce the start-up latency over
long-fat pipes by two round-trip times. This proposal is documented
further in [1], [2], and in [3] and we assume the reader is familiar
with the details of this proposal.
On the end2end-interest mailing list, concern was raised that in the
(allegedly common) case where a slow modem is served by a router
which only allocates three buffers per modem (one buffer being
transmitted while two packets are waiting), that starting with four
packets would not be good because the fourth packet is sure to be
dropped.
Vern Paxson replied with the comment (among other things) that the
four-packet start is no worse than what happens after two round trip
times in normal slow start, hence no new problem is introduced by
starting with as many as four packets. If there is a problem with a
four-packet start, then the problem already exists in a normal slow-
start startup after two round trip times when the slow-start
algorithm will release into the net four closely spaced packets.
The experiment reported here confirmed Vern Paxson's reasoning.
Scenario and experimental setup
+--------+ 100 Mbps +---+ 1.5 Mbps +---+ 9600 bps +----------+
| source +------------+ R +-------------+ R +--------------+ receiver |
+--------+ no delay +---+ 25 ms delay +---+ 150 ms delay +----------+
| |
| |
(we spy here) (this router has only 3 buffers
to hold packets going into the
9600 bps link)
The scenario studied and simulated consists of three links between
the source and sink. The first link is a 100 Mbps link with no
delay. It connects the sender to a router. (It was included to have
a means of logging the returning ACKs at the time they would be seen
by the sender.) The second link is a 1.5 Mbps link with a 25 ms
one-way delay. (This link was included to roughly model traversing
an un-congested, intra-continental piece of the terrestrial
Internet.) The third link is a 9600 bps link with a 150 ms one-way
delay. It connects the edge of the net to a receiver which is behind
the 9600 bps link.
The queue limits for the queues at each end of the first two links
were set to 100 (a value sufficiently large that this limit was never
a factor). The queue limits at each end of the 9600 bps link were
set to 3 packets (which can hold at most two packets while one is
being sent).
Version 1.2a2 of the the NS simulator (available from LBL) was used
to simulate both one-packet and four-packet starts for each of the
available TCP algorithms (tahoe, reno, sack, fack) and the conclusion
reported here is independent of which TCP algorithm is used (in
general, we believe). In this memo, the "tahoe" module will be used
to illustrate what happens. In the 4-packet start cases, the
"window-init" variable was set to 4, and the TCP implementations were
modified to use the value of the window-init variable only on
connection start, but to set cwnd to 1 on other instances of a slow-
start. (The tcp.cc module as shipped with ns-1.2a2 would use the
window-init value in all cases.)
The packets in simulation are 1024 bytes long for purposes of
determining the time it takes to transmit them through the links.
(The TCP modules included with the LBL NS simulator do not simulate
the TCP sequence number mechanisms. They use just packet numbers.)
Observations are made of all packets and acknowledgements crossing
the 100 Mbps no-delay link, near the sender. (All descriptions below
are from this point of view.)
What happens with normal slow start
At time 0.0 packet number 1 is sent.
At time 1.222 an ack is received covering packet number 1, and
packets 2 and 3 are sent.
At time 2.444 an ack is received covering packet number 2, and
packets 4 and 5 are sent.
At time 3.278 an ack is received covering packet number 3, and
packets 6 and 7 are sent.
At time 4.111 an ack is received covering packet number 4, and
packets 8 and 9 are sent.
At time 4.944 an ack is received covering packet number 5, and
packets 10 and 11 are sent.
At time 5.778 an ack is received covering packet number 6, and
packets 12 and 13 are sent.
At time 6.111 a duplicate ack is recieved (covering packet number 6).
At time 7.444 another duplicate ack is received (covering packet
number 6).
At time 8.278 a third duplicate ack is received (covering packet
number 6) and packet number 7 is retransmitted.
(And the trace continues...)
What happens with a four-packet start
At time 0.0, packets 1, 2, 3, and 4 are sent.
At time 1.222 an ack is received covering packet number 1, and
packets 5 and 6 are sent.
At time 2.055 an ack is received covering packet number 2, and
packets 7 and 8 are sent.
At time 2.889 an ack is received covering packet number 3, and
packets 9 and 10 are sent.
At time 3.722 a duplicate ack is received (covering packet number 3).
At time 4.555 another duplicate ack is received (covering packet
number 3).
At time 5.389 a third duplicate ack is received (covering packet
number 3) and packet number 4 is retransmitted.
(And the trace continues...)
Discussion
At the point left off in the two traces above, the two different
systems are in almost identical states. The two traces from that
point on are almost the same, modulo a shift in time of (8.278 -
5.389) = 2.889 seconds and a shift of three packets. If the normal
TCP (with the one-packet start) will deliver packet N at time T, then
the TCP with the four-packet start will deliver packet N - 3 at time
T - 2.889 (seconds).
Note that the time to send three 1024-byte TCP segments through a
9600 bps modem is 2.66 seconds. So at what time does the four-
packet-start TCP deliver packet N? At time T - 2.889 + 2.66 = T -
0.229 in most cases, and in some cases earlier, in some cases later,
because different packets (by number) experience loss in the two
traces.
Thus the four-packet-start TCP is in some sense 0.229 seconds (or
about one fifth of a packet) ahead of where the one-packet-start TCP
would be. (This is due to the extra time the modem sits idle while
waiting for the dally timer to go off in the receiver in the case of
the one-packet-start TCP.)
The states of the two systems are not exactly identical. They differ
slightly in the round-trip-time estimators because the behavior at
the start is not identical. (The observed round trip times may differ
by a small amount due to dally timers and due to that the one-packet
start experiences more round trip times before the first loss.) In
the cases where a retransmit timer did later go off, the additional
difference in timing was much smaller than the 0.229 second
difference discribed above.
Conclusion
In this particular case, the four-packet start is not harmful.
Non-conclusions, opinions, and future work
A four-packet start would be very helpful in situations where a
long-delay link is involved (as it would reduce transfer times for
moderately-sized transfers by as much as two round-trip times). But
it remains (in the authors' opinions at this time) an open question
whether or not the four-packet start would be safe for the network.
It would be nice to see if this result could be duplicated with real
TCPs, real modems, and real three-buffer limits.
Security Considerations
This document discusses a simulation study of the effects of a
proposed change to TCP. Consequently, there are no security
considerations directly related to the document. There are also no
known security considerations associated with the proposed change.
References
1. S. Floyd, Increasing TCP's Initial Window (January 29, 1997).
URL ftp://ftp.ee.lbl.gov/papers/draft.jan29.
2. S. Floyd and M. Allman, Increasing TCP's Initial Window (July,
1997). URL http://gigahertz.lerc.nasa.gov/~mallman/share/draft-
ss.txt
3. Allman, M., Floyd, S., and C. Partridge, "Increasing TCP's
Initial Window", RFC 2414, September 1998.
Authors' Addresses
Tim Shepard
BBN Technologies
10 Moulton Street
Cambridge, MA 02138
EMail: shep@alum.mit.edu
Craig Partridge
BBN Technologies
10 Moulton Street
Cambridge, MA 02138
EMail: craig@bbn.com
Full Copyright Statement
Copyright (C) The Internet Society (1998). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
ISO Plain Language
On this page
§1The Proposal and the Worry
Sally Floyd proposed that TCP begin its initial slow start by sending as many as four packets, instead of the usual one. The aim is to get TCP up to speed faster. A slow start that follows a timeout would still begin with a single packet. Starting with more than one packet might reduce the start-up latency over long, fat pipes by two round-trip times. The proposal is documented separately, and the memo assumes the reader is familiar with it.
On the end2end-interest mailing list, concern was raised about a case said to be common: a slow modem served by a router which allocates only three buffers per modem, one buffer being transmitted while two packets wait. In that arrangement, starting with four packets would not be good, because the fourth packet is sure to be dropped.
Vern Paxson replied that the four-packet start is no worse than what happens after two round-trip times in normal slow start, so no new problem is introduced by starting with as many as four packets. If there is a problem with a four-packet start, then the problem already exists in a normal slow-start start-up after two round-trip times, when the slow-start algorithm releases four closely spaced packets into the network. The experiment reported in the memo confirmed Paxson's reasoning.
Starting a connection with four packets instead of one would save time on long links. The worry was that a slow modem with room for only three packets would simply throw the fourth one away, so the change would make things worse rather than better. The answer was that ordinary slow start already sends four packets close together once it is under way, so the four-packet start only moves that moment earlier.
§2The Experimental Setup
The scenario consists of three links between the source and the sink. The first link is a 100 Mbps link with no delay, connecting the sender to a router. It was included to have a means of logging the returning acknowledgements at the time they would be seen by the sender. The second link is a 1.5 Mbps link with a 25 ms one-way delay, included to roughly model traversing an uncongested, intra-continental piece of the terrestrial Internet. The third link is a 9600 bps link with a 150 ms one-way delay, connecting the edge of the net to a receiver that is behind the modem.
The queue limits for the queues at each end of the first two links were set to 100, a value sufficiently large that this limit was never a factor. The queue limits at each end of the 9600 bps link were set to 3 packets, which can hold at most two packets while one is being sent.
Version 1.2a2 of the NS simulator was used to simulate both one-packet and four-packet starts for each of the available TCP algorithms: tahoe, reno, sack and fack. The conclusion reported is independent of which TCP algorithm is used, in general. In the four-packet start cases the window-init variable was set to 4, and the TCP implementations were modified to use the value of the window-init variable only on connection start, but to set cwnd to 1 on other instances of a slow start. The packets in simulation are 1024 bytes long, for purposes of determining the time it takes to transmit them through the links. The TCP modules included with the simulator do not simulate the TCP sequence number mechanisms; they use just packet numbers. Observations are made of all packets and acknowledgements crossing the 100 Mbps no-delay link, near the sender.
The test used a fast link, then a moderately fast link with a quarter of a second of delay, and finally a 9600 bit per second modem link with a third of a second of delay. The router feeding the modem had room for only three packets. The researchers ran both kinds of start with four different TCP versions in a network simulator, and watched the traffic near the sender.
- 100 Mbps, no delayUsed to log acknowledgements as the sender sees them.
- 1.5 Mbps, 25 ms one-way delayModels an uncongested piece of the terrestrial Internet.
- 9600 bps, 150 ms one-way delayThree buffers here, so the fourth packet is dropped.
§3The Two Traces
With normal slow start, packet number 1 is sent at time 0.0. An acknowledgement covering packet number 1 arrives at time 1.222, and packets 2 and 3 are sent. An acknowledgement covering packet number 2 arrives at time 2.444, and packets 4 and 5 are sent. An acknowledgement covering packet number 3 arrives at time 3.278, and packets 6 and 7 are sent. The pattern continues: acknowledgements covering packet 4 at time 4.111, packet 5 at time 4.944 and packet 6 at time 5.778, each releasing two more packets.
At time 6.111 a duplicate acknowledgement is received, again covering packet number 6. Another arrives at time 7.444. At time 8.278 a third duplicate acknowledgement is received, covering packet number 6, and packet number 7 is retransmitted.
With a four-packet start, packets 1, 2, 3 and 4 are all sent at time 0.0. An acknowledgement covering packet number 1 arrives at time 1.222 and packets 5 and 6 are sent. Acknowledgements covering packet number 2 and packet number 3 arrive at times 2.055 and 2.889, and packets 7 and 8, then 9 and 10, are sent. At time 3.722 a duplicate acknowledgement is received covering packet number 3, another at time 4.555, and at time 5.389 a third duplicate acknowledgement covering packet number 3 is received and packet number 4 is retransmitted.
In the ordinary case the sender sends one packet, waits, then sends two more each time an acknowledgement comes back. In the four-packet case it sends four at once and then carries on in the same way. Both runs lose a packet to the three-buffer queue in front of the modem, and both recover by retransmitting it after three duplicate acknowledgements.
| Point in the trace | One-packet start | Four-packet start |
|---|---|---|
| Packets sent at time 0.0 | Packet 1 | Packets 1, 2, 3 and 4 |
| Acknowledgement covering packet 1 | At 1.222 s | At 1.222 s |
| Third duplicate acknowledgement | At 8.278 s | At 5.389 s |
| Packet retransmitted after it | Packet 7 | Packet 4 |
§4What the Traces Show
At the point where the two traces stop being written out, the two systems are in almost identical states. The traces from that point on are almost the same, modulo a shift in time of 8.278 minus 5.389, which is 2.889 seconds, and a shift of three packets. If the normal TCP with the one-packet start delivers packet N at time T, then the TCP with the four-packet start delivers packet N minus 3 at time T minus 2.889 seconds.
The time to send three 1024-byte TCP segments through a 9600 bps modem is 2.66 seconds. So the four-packet-start TCP delivers packet N at time T minus 2.889 plus 2.66, which is T minus 0.229, in most cases, and in some cases earlier and in some cases later, because different packets, by number, experience loss in the two traces. The four-packet-start TCP is therefore about 0.229 seconds, roughly one fifth of a packet, ahead of where the one-packet-start TCP would be. The memo attributes this to the extra time the modem sits idle while waiting for the dally timer to go off in the receiver in the one-packet-start case.
The states of the two systems are not exactly identical. They differ slightly in the round-trip-time estimators, because the behaviour at the start is not identical. The observed round trip times may differ by a small amount due to dally timers, and due to the one-packet start experiencing more round trip times before the first loss. In the cases where a retransmit timer did later go off, the additional difference in timing was much smaller than the 0.229 second difference described above.
After the first loss the two runs look the same, only shifted: the four-packet run is 2.889 seconds ahead in time and three packets ahead in sequence. Sending those three extra packets over the modem takes 2.66 seconds, so the four-packet run finishes about 0.229 seconds ahead. The difference comes from the modem sitting idle waiting for the receiver's acknowledgement delay, which the four-packet run does not have to wait for.
§5Conclusion and Open Questions
In this particular case, the four-packet start is not harmful.
The authors add that a four-packet start would be very helpful in situations where a long-delay link is involved, because it would reduce transfer times for moderately-sized transfers by as much as two round-trip times. But it remains, in their opinion at the time, an open question whether the four-packet start would be safe for the network.
They also note that it would be good to see whether this result could be duplicated with real TCP implementations, real modems and real three-buffer limits, rather than a simulation.
In the situation tested, starting with four packets did no harm. The authors think it would help a lot on links with long delays, but they say it was still an open question whether it would be safe for the Internet as a whole, and that the result should be repeated with real software and real modems.
§6Security Considerations
The memo's security section explains that it discusses a simulation study of the effects of a proposed change to TCP, so there are no security considerations directly related to the document. It adds that there are also no known security considerations associated with the proposed change.
There is nothing to report here. The memo is a simulation study of a proposed change to TCP, and the change itself has no known security implications.