draft-ietf-tcpm-ecnsyn-06.txt   draft-ietf-tcpm-ecnsyn-07.txt 
Internet Engineering Task Force A. Kuzmanovic Internet Engineering Task Force A. Kuzmanovic
INTERNET-DRAFT A. Mondal INTERNET-DRAFT A. Mondal
Intended status: Proposed Standard Northwestern University Intended status: Proposed Standard Northwestern University
Expires: 22 February 2009 S. Floyd Expires: 3 May 2009 S. Floyd
Updates: 3168 ICIR Updates: 3168 ICIR
K.K. Ramakrishnan K.K. Ramakrishnan
AT&T AT&T
22 August 2008 3 November 2008
Adding Explicit Congestion Notification (ECN) Capability Adding Explicit Congestion Notification (ECN) Capability
to TCP's SYN/ACK Packets to TCP's SYN/ACK Packets
draft-ietf-tcpm-ecnsyn-06.txt draft-ietf-tcpm-ecnsyn-07.txt
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 38 skipping to change at page 1, line 38
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 2008. This Internet-Draft will expire on May 2009.
Copyright Notice Copyright Notice
Copyright (C) The IETF Trust (2008). Copyright (C) The IETF Trust (2008).
Abstract Abstract
This draft specifies a modification to RFC 3168 to allow TCP SYN/ACK This draft specifies a modification to RFC 3168 to allow TCP SYN/ACK
packets to be ECN-Capable. For TCP, RFC 3168 only specifies setting packets to be ECN-Capable. For TCP, RFC 3168 only specifies setting
an ECN-Capable codepoint on data packets, and not on SYN and SYN/ACK an ECN-Capable codepoint on data packets, and not on SYN and SYN/ACK
packets. However, because of the high cost to the TCP transfer of packets. However, because of the high cost to the TCP transfer of
having a SYN/ACK packet dropped, with the resulting retransmit having a SYN/ACK packet dropped, with the resulting retransmit
timeout, this document specifies the use of ECN for the SYN/ACK timeout, this document specifies the use of ECN for the SYN/ACK
packet itself, when sent in response to a SYN packet with the two ECN packet itself, when sent in response to a SYN packet with the two ECN
flags set in the TCP header, indicating a willingness to use ECN. flags set in the TCP header, indicating a willingness to use ECN.
Setting TCP SYN/ACK packets as ECN-Capable can be of great benefit to Setting the initial TCP SYN/ACK packet as ECN-Capable can be of great
the TCP connection, avoiding the severe penalty of a retransmit benefit to the TCP connection, avoiding the severe penalty of a
timeout for a connection that has not yet started placing a load on retransmit timeout for a connection that has not yet started placing
the network. The sender of the SYN/ACK packet must respond to a a load on the network. The TCP responder (the sender of the SYN/ACK
report of an ECN-marked SYN/ACK packet by reducing its initial packet) must reply to a report of an ECN-marked SYN/ACK packet by
congestion window from two, three, or four segments to one segment, resending a SYN/ACK packet that is not ECN-Capable. If the resent
thereby reducing the subsequent load from that connection on the SYN/ACK packet is acknowledged, then the TCP responder reduces its
network. This document updates RFC 3168. initial congestion window from two, three, or four segments to one
segment, thereby reducing the subsequent load from that connection on
the network. If instead the SYN/ACK packet is dropped, or for some
other reason the TCP responder does not receive an acknowledgement in
the specified time, the TCP responder follows TCP standards for a
dropped SYN/ACK packet (setting the retransmit timer). This document
updates RFC 3168.
Table of Contents Table of Contents
1. Introduction ....................................................5 1. Introduction ....................................................5
2. Conventions and Terminology .....................................6 2. Conventions and Terminology .....................................7
3. Specification ...................................................7 3. Specification ...................................................7
3.1. SYN/ACK Packets Dropped in the Network .....................7 3.1. SYN/ACK Packets Dropped in the Network .....................8
3.2. SYN/ACK Packets ECN-Marked in the Network ..................8 3.2. SYN/ACK Packets ECN-Marked in the Network ..................9
3.3. Management Interface ......................................10 3.3. Management Interface ......................................11
4. Discussion .....................................................10 4. Discussion .....................................................12
4.1. Flooding Attacks ..........................................10 4.1. Flooding Attacks ..........................................12
4.2. The TCP SYN Packet ........................................11 4.2. The TCP SYN Packet ........................................12
4.3. SYN/ACK Packets and Packet Size ...........................11 4.3. SYN/ACK Packets and Packet Size ...........................13
4.4. Response to ECN-marking of SYN/ACK Packets ................12 4.4. Response to ECN-marking of SYN/ACK Packets ................13
5. Related Work ...................................................13 5. Related Work ...................................................15
6. Performance Evaluation .........................................14 6. Performance Evaluation .........................................16
6.1. The Costs and Benefit of Adding ECN-Capability ............14 6.1. The Costs and Benefit of Adding ECN-Capability ............16
6.2. An Evaluation of Different Responses to ECN-Marked SYN/ACK 6.2. An Evaluation of Different Responses to ECN-Marked SYN/ACK
Packets ........................................................15 Packets ........................................................17
7. Security Considerations ........................................16 7. Security Considerations ........................................18
7.1. 'Bad' Routers or Middleboxes ..............................16 7.1. 'Bad' Routers or Middleboxes ..............................18
7.2. Congestion Collapse .......................................16 7.2. Congestion Collapse .......................................19
8. Conclusions ....................................................17 8. Conclusions ....................................................19
9. Acknowledgements ...............................................18 9. Acknowledgements ...............................................20
A. Report on Simulations ..........................................18 A. Report on Simulations ..........................................20
A.1. Simulations with RED in Packet Mode .......................19 A.1. Simulations with RED in Packet Mode .......................21
A.2. Simulations with RED in Byte Mode .........................21 A.2. Simulations with RED in Byte Mode .........................25
B. Issues of Incremental Deployment ...............................23 B. Issues of Incremental Deployment ...............................27
Normative References ..............................................26 Normative References ..............................................30
Informative References ............................................26 Informative References ............................................30
IANA Considerations ...............................................27 IANA Considerations ...............................................31
Full Copyright Statement ..........................................28 Full Copyright Statement ..........................................32
Intellectual Property .............................................28 Intellectual Property .............................................32
NOTE TO RFC EDITOR: PLEASE DELETE THIS NOTE UPON PUBLICATION. NOTE TO RFC EDITOR: PLEASE DELETE THIS NOTE UPON PUBLICATION.
Changes from draft-ietf-tcpm-ecnsyn-06:
* Updated text and simulation results to specify ECN+/TryOnce
instead of ECN+. Added tables on CDFs.
* Acknowledged Adam's Linux implementation of ECN+/TryOnce.
Changes from draft-ietf-tcpm-ecnsyn-05: Changes from draft-ietf-tcpm-ecnsyn-05:
* Added "Updates: 3168" to the header. Added a reference * Added "Updates: 3168" to the header. Added a reference
to RFC 4987. Mild editing. to RFC 4987. Mild editing.
Feedback from Lars's Area Director review. Feedback from Lars's Area Director review.
* Updated simulation results with new simulation scripts that * Updated simulation results with new simulation scripts that
don't require any modifications to the ns simulator, and that don't require any modifications to the ns simulator, and that
all use the same seed for generating traffic. The results are all use the same seed for generating traffic. The results are
somewhat different for the very-high-congestion scenarios somewhat different for the very-high-congestion scenarios
skipping to change at page 5, line 44 skipping to change at page 6, line 8
the router detects congestion before buffer overflow, the router can the router detects congestion before buffer overflow, the router can
provide a congestion indication either by dropping a packet, or by provide a congestion indication either by dropping a packet, or by
setting the Congestion Experienced (CE) codepoint in the Explicit setting the Congestion Experienced (CE) codepoint in the Explicit
Congestion Notification (ECN) field in the IP header [RFC3168]. The Congestion Notification (ECN) field in the IP header [RFC3168]. The
IETF has standardized the use of the Congestion Experienced (CE) IETF has standardized the use of the Congestion Experienced (CE)
codepoint in the IP header for routers to indicate congestion. For codepoint in the IP header for routers to indicate congestion. For
incremental deployment and backwards compatibility, the RFC on ECN incremental deployment and backwards compatibility, the RFC on ECN
[RFC3168] specifies that routers may mark ECN-capable packets that [RFC3168] specifies that routers may mark ECN-capable packets that
would otherwise have been dropped, using the Congestion Experienced would otherwise have been dropped, using the Congestion Experienced
codepoint in the ECN field. The use of ECN allows TCP to react to codepoint in the ECN field. The use of ECN allows TCP to react to
congestion while avoiding unnecessary retransmissions and, in some congestion while avoiding unnecessary retransmit timeouts. Thus,
cases, unnecessary retransmit timeouts. Thus, using ECN has several using ECN has several benefits:
benefits:
1) For short transfers, a TCP connection's congestion window may be 1) For short transfers, a TCP connection's congestion window may be
small. For example, if the current window contains only one packet, small. For example, if the current window contains only one packet,
and that packet is dropped, TCP will have to wait for a retransmit and that packet is dropped, TCP will have to wait for a retransmit
timeout to recover, reducing its overall throughput. Similarly, if timeout to recover, reducing its overall throughput. Similarly, if
the current window contains only a few packets and one of those the current window contains only a few packets and one of those
packets is dropped, there might not be enough duplicate packets is dropped, there might not be enough duplicate
acknowledgements for a fast retransmission, and the sender of the acknowledgements for a fast retransmission, and the sender of the
data packet might have to wait for a delay of several round-trip data packet might have to wait for a delay of several round-trip
times using Limited Transmit [RFC3042]. With the use of ECN, short times using Limited Transmit [RFC3042]. With the use of ECN, short
flows are less likely to have packets dropped, sometimes avoiding flows are less likely to have packets dropped, sometimes avoiding
unnecessary delays or costly retransmit timeouts. unnecessary delays or costly retransmit timeouts.
2) While longer flows may not see substantially improved throughput 2) While longer flows may not see substantially improved throughput
with the use of ECN, they experience lower loss. This may benefit TCP with the use of ECN, they may experience lower loss. This may benefit
applications that are latency- and loss-sensitive, because of the TCP applications that are latency- and loss-sensitive, because of the
avoidance of retransmissions. avoidance of retransmissions.
RFC 3168 only specifies marking the Congestion Experienced codepoint RFC 3168 only specifies marking the Congestion Experienced codepoint
on TCP's data packets, and not on SYN and SYN/ACK packets. RFC 3168 on TCP's data packets, and not on SYN and SYN/ACK packets. RFC 3168
specifies the negotiation of the use of ECN between the two TCP end- specifies the negotiation of the use of ECN between the two TCP end-
points in the TCP SYN and SYN-ACK exchange, using flags in the TCP points in the TCP SYN and SYN-ACK exchange, using flags in the TCP
header. Erring on the side of being conservative, RFC 3168 does not header. Erring on the side of being conservative, RFC 3168 does not
specify the use of ECN for the SYN/ACK packet itself. However, specify the use of ECN for the first SYN/ACK packet itself. However,
because of the high cost to the TCP transfer of having a SYN/ACK because of the high cost to the TCP transfer of having a SYN/ACK
packet dropped, with the resulting retransmit timeout, this document packet dropped, with the resulting retransmit timeout, this document
specifies the use of ECN for the SYN/ACK packet itself. This can be specifies the use of ECN for the SYN/ACK packet itself. This can be
of great benefit to the TCP connection, avoiding the severe penalty of great benefit to the TCP connection, avoiding the severe penalty
of a retransmit timeout for a connection that has not yet started of a retransmit timeout for a connection that has not yet started
placing a load on the network. The sender of the SYN/ACK packet must placing a load on the network. The sender of the SYN/ACK packet must
respond to a report of an ECN-marked SYN/ACK packet by reducing its respond to a report of an ECN-marked SYN/ACK packet by sending a non-
initial congestion window from two, three, or four segments to one ECN-Capable SYN/ACK packet, and by reducing its initial congestion
segment, reducing the subsequent load from that connection on the window from two, three, or four segments to one segment, reducing the
network. subsequent load from that connection on the network.
The use of ECN for SYN/ACK packets has the following potential The use of ECN for SYN/ACK packets has the following potential
benefits: benefits:
1) Avoidance of a retransmit timeout; 1) Avoidance of a retransmit timeout;
2) Improvement in the throughput of short connections. 2) Improvement in the throughput of short connections.
This draft specifies ECN+, a modification to RFC 3168 to allow TCP This draft specifies a modification to RFC 3168 to allow TCP SYN/ACK
SYN/ACK packets to be ECN-Capable. Section 3 contains the packets to be ECN-Capable. Section 3 contains the specification of
specification of the change, while Section 4 discusses some of the the change, while Section 4 discusses some of the issues, and Section
issues, and Section 5 discusses related work. Section 6 contains an 5 discusses related work. Section 6 contains an evaluation of the
evaluation of the specified change. specified change.
2. Conventions and Terminology 2. Conventions and Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC 2119]. document are to be interpreted as described in [RFC 2119].
We use the following terminology from RFC 3168: We use the following terminology from RFC 3168:
The ECN field in the IP header: The ECN field in the IP header:
skipping to change at page 7, line 28 skipping to change at page 7, line 39
refer to the sender of the SYN packet and of the SYN-ACK packet, refer to the sender of the SYN packet and of the SYN-ACK packet,
respectively. respectively.
3. Specification 3. Specification
This section specifies the modification to RFC 3168 to allow TCP This section specifies the modification to RFC 3168 to allow TCP
SYN/ACK packets to be ECN-Capable. SYN/ACK packets to be ECN-Capable.
RFC 3168 in Section 6.1.1. states that "A host MUST NOT set ECT on RFC 3168 in Section 6.1.1. states that "A host MUST NOT set ECT on
SYN or SYN-ACK packets." In this section, we specify that a TCP node SYN or SYN-ACK packets." In this section, we specify that a TCP node
MAY respond to an ECN-setup SYN packet by setting ECT in the MAY respond to an initial ECN-setup SYN packet by setting ECT in the
responding ECN-setup SYN/ACK packet, indicating to routers that the responding ECN-setup SYN/ACK packet, indicating to routers that the
SYN/ACK packet is ECN-Capable. This allows a congested router along SYN/ACK packet is ECN-Capable. This allows a congested router along
the path to mark the packet instead of dropping the packet as an the path to mark the packet instead of dropping the packet as an
indication of congestion. indication of congestion.
Assume that TCP node A transmits to TCP node B an ECN-setup SYN Assume that TCP node A transmits to TCP node B an ECN-setup SYN
packet, indicating willingness to use ECN for this connection. As packet, indicating willingness to use ECN for this connection. As
specified by RFC 3168, if TCP node B is willing to use ECN, node B specified by RFC 3168, if TCP node B is willing to use ECN, node B
responds with an ECN-setup SYN-ACK packet. responds with an ECN-setup SYN-ACK packet.
3.1. SYN/ACK Packets Dropped in the Network 3.1. SYN/ACK Packets Dropped in the Network
Figure 1 shows an interchange with the SYN/ACK packet dropped by a Figure 1 shows an interchange with the SYN/ACK packet dropped by a
congested router. Node B waits for a retransmit timeout, and then congested router. Node B waits for a retransmit timeout, and then
retransmits the SYN/ACK packet. retransmits the SYN/ACK packet.
--------------------------------------------------------------- ---------------------------------------------------------------
TCP Node A Router TCP Node B TCP Node A Router TCP Node B
(initiator) (responder)
---------- ------ ---------- ---------- ------ ----------
ECN-setup SYN packet ---> ECN-setup SYN packet --->
ECN-setup SYN packet ---> ECN-setup SYN packet --->
<--- ECN-setup SYN/ACK, possibly ECT <--- ECN-setup SYN/ACK, possibly ECT
3-second timer set 3-second timer set
SYN/ACK dropped . SYN/ACK dropped .
. .
. .
skipping to change at page 9, line 4 skipping to change at page 9, line 9
the exchange in Figure 1, the responder wouldn't set a timer upon the exchange in Figure 1, the responder wouldn't set a timer upon
transmission of the SYN/ACK packet [SYN-COOK] [RFC4987]. In this transmission of the SYN/ACK packet [SYN-COOK] [RFC4987]. In this
case, if the SYN/ACK packet was lost, the initiator (Node A) would case, if the SYN/ACK packet was lost, the initiator (Node A) would
have to timeout and retransmit the SYN packet in order to trigger have to timeout and retransmit the SYN packet in order to trigger
another SYN-ACK. another SYN-ACK.
3.2. SYN/ACK Packets ECN-Marked in the Network 3.2. SYN/ACK Packets ECN-Marked in the Network
Figure 2 shows an interchange with the SYN/ACK packet sent as ECN- Figure 2 shows an interchange with the SYN/ACK packet sent as ECN-
Capable, and ECN-marked instead of dropped at the congested router. Capable, and ECN-marked instead of dropped at the congested router.
This document specifies ECN+/TryOnce, which differs from the original
proposal for ECN+ in [ECN+]; with ECN+/TryOnce, if the TCP responder
is informed that the SYN/ACK was ECN-marked, the TCP responder
immediately sends a SYN/ACK packet that is not ECN-Capable. The TCP
responder is only allowed to send data packets after the TCP
initiator reports the receipt of a SYN/ACK packet that is neither
marked nor dropped.
--------------------------------------------------------------- ---------------------------------------------------------------
TCP Node A Router TCP Node B TCP Node A Router TCP Node B
(initiator) (responder)
---------- ------ ---------- ---------- ------ ----------
ECN-setup SYN packet ---> ECN-setup SYN packet --->
ECN-setup SYN packet ---> ECN-setup SYN packet --->
<--- ECN-setup SYN/ACK, ECT <--- ECN-setup SYN/ACK, ECT
3-second timer set
<--- Sets CE on SYN/ACK <--- Sets CE on SYN/ACK
<--- ECN-setup SYN/ACK, CE <--- ECN-setup SYN/ACK, CE
Data/ACK, ECN-Echo ---> Data/ACK, ECN-Echo --->
Data/ACK, ECN-Echo ---> Data/ACK, ECN-Echo --->
Window reduced to one segment. Window reduced to one segment.
<--- Data, CWR (one segment only) <--- ECN-setup SYN/ACK, CWR, not ECT
<--- ECN-setup SYN/ACK, CWR
Data/ACK --->
Data/ACK --->
<--- Data (one segment only)
--------------------------------------------------------------- ---------------------------------------------------------------
Figure 2: SYN exchange with the SYN/ACK packet marked. Figure 2: SYN exchange with the SYN/ACK packet marked.
ECN+/TryOnce.
If the initiator (node A) receives a SYN/ACK packet that has been If the initiator (node A) receives a SYN/ACK packet that has been
marked by the congested router, with the CE codepoint set, the marked by the congested router, with the CE codepoint set, the
initiator MUST respond by setting the ECN-Echo flag in the TCP header initiator MUST respond by setting the ECN-Echo flag in the TCP header
of the responding ACK packet. As specified in RFC 3168, the of the responding ACK packet. However, with ECN+/TryOnce the
initiator continues to set the ECN-Echo flag in packets until it initiator does not advance from the "SYN-Sent" to the "SYN-Received"
receives a packet with the CWR flag set. state until it receives a SYN/ACK packet that is not ECN-marked. As
specified in RFC 3168, the initiator continues to set the ECN-Echo
flag in packets until it receives a packet with the CWR flag set.
When the responder (node B) receives the ECN-Echo packet reporting When the responder (node B) receives the ECN-Echo packet reporting
the Congestion Experienced indication in the SYN/ACK packet, the the Congestion Experienced indication in the SYN/ACK packet, the
responder MUST set the initial congestion window to one segment, responder MUST set the initial congestion window to one segment,
instead of two segments as allowed by [RFC2581], or three or four instead of two segments as allowed by [RFC2581], or three or four
segments allowed by [RFC3390]. If the responder (node B) was going segments allowed by [RFC3390]. In the original proposal for ECN+, if
to use an initial window of one segment, and receives an ECN-Echo the responder (node B) received an ECN-Echo packet informing it of a
packet informing it of a Congestion Experienced indication on its Congestion Experienced indication on its SYN/ACK packet, the
SYN/ACK packet, the responder MAY continue to send with an initial responder would been able to send data packets using an initial
window of one segment, without waiting for a retransmit timeout. We window of one segment, without waiting for a retransmit timeout. In
note that this updates RFC 3168, which specifies that "the sending contrast, this document specifies ECN+/TryOnce, illustrated in Figure
TCP MUST reset the retransmit timer on receiving the ECN-Echo packet 2; if the responder (node B) receives an ECN-Echo packet informing it
when the congestion window is one." As specified by RFC 3168, the of a Congestion Experienced indication on its SYN/ACK packet, the
responder (node B) also sets the CWR flag in the TCP header of the responder sends a SYN/ACK packet that is not ECN-Capable, in addition
next data packet sent, to acknowledge its receipt of and reaction to to setting the initial window to one segment.
the ECN-Echo flag.
If the data transfer in Figure 2 is entirely from Node A to Node B, We note that this document updates RFC 3168, which specified that
then data packets from Node A continue to set the ECN-Echo flag in "the sending TCP MUST reset the retransmit timer on receiving the
data packets, waiting for the CWR flag from Node B acknowledging a ECN-Echo packet when the congestion window is one." As an update,
response to the ECN-Echo flag. this document specifies the response of a TCP host to receiving an
ECN-Echo packet acknowledging the receipt of an ECN-Capable SYN/ACK
packet.
RFC 3168 specifies that in response to an ECN-Echo packet, the TCP
responder also sets the CWR flag in the TCP header of the next data
packet sent, to acknowledge its receipt of and reaction to the ECN-
Echo flag. This document updates RFC 3168 by specifying that in
response to an ECN-Echo packet acknowledging the receipt of an ECN-
Capable SYN/ACK packet, the responder sets the CWR flag in the TCP
header of the non-ECN-Capable SYN/ACK packet.
---------------------------------------------------------------
TCP Node A Router TCP Node B
(initiator) (responder)
---------- ------ ----------
ECN-setup SYN packet --->
ECN-setup SYN packet --->
<--- ECN-setup SYN/ACK, ECT
<--- Sets CE on SYN/ACK
<--- ECN-setup SYN/ACK, CE
Data/ACK, ECN-Echo --->
Data/ACK, ECN-Echo --->
Window reduced to one segment.
<--- ECN-setup SYN/ACK, CWR, not ECT
3-second timer set
SYN/ACK dropped .
.
.
3-second timer expires
<--- ECN-setup SYN/ACK, CWR, not ECT
<--- ECN-setup SYN/ACK, CWR, not ECT
Data/ACK --->
Data/ACK --->
<--- Data (one segment only)
---------------------------------------------------------------
Figure 3: SYN exchange with the first SYN/ACK packet marked,
and the second SYN/ACK packet dropped. ECN+/TryOnce.
In contrast to Figure 2, Figure 3 shows an interchange where the
first SYN/ACK packet is ECN-marked and the second SYN/ACK packet is
dropped in the network. As in Figure 2, the TCP responder sets a
timer when the second SYN/ACK packet is sent. Figure 3 shows that if
the timer expires before the TCP responder receives an
acknowledgement for the other end, the TCP responder resends the
SYN/ACK packet, following the TCP standards.
3.3. Management Interface 3.3. Management Interface
The TCP implementation using ECN-Capable SYN/ACK packets SHOULD The TCP implementation using ECN-Capable SYN/ACK packets SHOULD
include a management interface to allow the use of ECN to be turned include a management interface to allow the use of ECN to be turned
off for SYN/ACK packets. This is to deal with possible backwards off for SYN/ACK packets. This is to deal with possible backwards
compatibility problems such as those discussed in Appendix B. compatibility problems such as those discussed in Appendix B.
4. Discussion 4. Discussion
skipping to change at page 10, line 32 skipping to change at page 12, line 25
SYN/ACK packets, as is discussed further in [ECN+], and reported SYN/ACK packets, as is discussed further in [ECN+], and reported
briefly in Section 5 below. Congestion is most likely to occur in briefly in Section 5 below. Congestion is most likely to occur in
the server-to-client direction. As a result, setting an ECN-capable the server-to-client direction. As a result, setting an ECN-capable
codepoint in SYN/ACK packets can reduce the occurrence of three- codepoint in SYN/ACK packets can reduce the occurrence of three-
second retransmit timeouts resulting from the drop of SYN/ACK second retransmit timeouts resulting from the drop of SYN/ACK
packets. packets.
4.1. Flooding Attacks 4.1. Flooding Attacks
Setting an ECN-Capable codepoint in the responding TCP SYN/ACK Setting an ECN-Capable codepoint in the responding TCP SYN/ACK
packets does not raise any novel security vulnerabilities. For packets does not raise any new or additional security
example, provoking servers or hosts to send SYN/ACK packets to a vulnerabilities. For example, provoking servers or hosts to send
third party in order to perform a "SYN/ACK flood" attack would be SYN/ACK packets to a third party in order to perform a "SYN/ACK
highly inefficient. Third parties would immediately drop such flood" attack would be highly inefficient. Third parties would
packets, since they would know that they didn't generate the TCP SYN immediately drop such packets, since they would know that they didn't
packets in the first place. Moreover, such SYN/ACK attacks would generate the TCP SYN packets in the first place. Moreover, such
have the same signatures as the existing TCP SYN attacks. Provoking SYN/ACK attacks would have the same signatures as the existing TCP
servers or hosts to reply with SYN/ACK packets in order to congest a SYN attacks. Provoking servers or hosts to reply with SYN/ACK packets
certain link would also be highly inefficient because SYN/ACK packets in order to congest a certain link would also be highly inefficient
are small in size. because SYN/ACK packets are small in size.
However, the addition of ECN-Capability to SYN/ACK packets could However, the addition of ECN-Capability to SYN/ACK packets could
allow SYN/ACK packets to persist for more hops along a network path allow SYN/ACK packets to persist for more hops along a network path
before being dropped, thus adding somewhat to the ability of a before being dropped, thus adding somewhat to the ability of a
SYN/ACK attack to flood a network link. SYN/ACK attack to flood a network link.
4.2. The TCP SYN Packet 4.2. The TCP SYN Packet
There are several reasons why an ECN-Capable codepoint MUST NOT be There are several reasons why an ECN-Capable codepoint MUST NOT be
set in the IP header of the initiating TCP SYN packet. First, when set in the IP header of the initiating TCP SYN packet. First, when
skipping to change at page 12, line 32 skipping to change at page 14, line 25
rate below one packet per round-trip time, by waiting for one RTO rate below one packet per round-trip time, by waiting for one RTO
before sending another packet. If the RTO was set to the average before sending another packet. If the RTO was set to the average
round-trip time, this would result in halving the sending rate; round-trip time, this would result in halving the sending rate;
because the RTO is in fact larger than the average round-trip time, because the RTO is in fact larger than the average round-trip time,
the sending rate is reduced to less than half of its previous value. the sending rate is reduced to less than half of its previous value.
TCP's congestion control response to the *dropping* of a SYN/ACK TCP's congestion control response to the *dropping* of a SYN/ACK
packet is to wait a default time before sending another packet. This packet is to wait a default time before sending another packet. This
document argues that ECN gives end-systems a wider range of possible document argues that ECN gives end-systems a wider range of possible
responses to the *marking* of a SYN/ACK packet, and that waiting a responses to the *marking* of a SYN/ACK packet, and that waiting a
default time before sending a data packet is not the desired default time before sending another packet is not the desired
response. response.
On the conservative end, one could assume an effective congestion On the conservative end, one could assume an effective congestion
window of one packet for the SYN/ACK packet, and respond to an ECN- window of one packet for the SYN/ACK packet, and respond to an ECN-
marked SYN/ACK packet by reducing the sending rate to one packet marked SYN/ACK packet by reducing the sending rate to one packet
every two round-trip times. As an approximation, the TCP end-node every two round-trip times. As an approximation, the TCP end-node
could measure the round-trip time T between the sending of the could measure the round-trip time T between the sending of the
SYN/ACK packet and the receipt of the acknowledgement, and reply to SYN/ACK packet and the receipt of the acknowledgement, and reply to
the acknowledgement of the ECN-marked SYN/ACK packet by waiting T the acknowledgement of the ECN-marked SYN/ACK packet by waiting T
seconds before sending a data packet. seconds before sending a data packet.
However, we note that for an ECN-marked SYN/ACK packet, halving the However, we note that for an ECN-marked SYN/ACK packet, halving the
*congestion window* is not the same as halving the *sending rate*; *congestion window* is not the same as halving the *sending rate*;
there is no `sending rate' associated with an ECN-Capable SYN/ACK there is no `sending rate' associated with an ECN-Capable SYN/ACK
packet, as such packets are only sent as the first packet in a packet, as such packets are only sent as the first packet in a
connection from that host. Further, a router's marking of a SYN/ACK connection from that host. Further, a router's marking of a SYN/ACK
packet is not affected by any past history of that connection. packet is not affected by any past history of that connection.
Adding ECN-Capability to SYN/ACK packets allows the simple response Adding ECN-Capability to SYN/ACK packets allows the response of the
of the responder setting the initial congestion window to one packet, responder setting the initial congestion window to one packet,
instead of its allowed default value of two, three, or four packets, instead of its allowed default value of two, three, or four packets.
with the responder proceeding with a cautious sending rate of one The responder sends a non-ECN-Capable SYN/ACK packet, and proceeds
packet per round-trip time. If that data packet is ECN-marked or with a cautious sending rate of one data packet per round-trip time
dropped, then the responder will wait an RTO before sending another after that SYN/ACK packet is acknowledged. This document argues that
packet. This document argues that this approach is useful to users, this approach is useful to users, with no dangers of congestion
with no dangers of congestion collapse or of starvation of competing collapse or of starvation of competing traffic. This is discussed in
traffic. This is discussed in more detail below in Section 6.2. In more detail below in Section 6.2.
particular, Section 6.2 discusses simulation results that support the
responder's specified behavior of setting the initial congestion
window to one packet in response to an ECN-marked SYN/ACK packet.
We note that if the data transfer is entirely from Node A to Node B, We note that if the data transfer is entirely from Node A to Node B,
then there is no effective difference between the two possible there is still a difference in performance between the original
responses to an ECN-marked SYN/ACK packet outlined above. In either mechanism ECN+ and the mechanism ECN+/TryOnce specified in this
case, Node B sends no data packets, only sending acknowledgement document. In particular, with ECN+/TryOnce the TCP originator does
packets in response to received data packets. not send data packets until it has received a non-ECN-marked SYN/ACK
packet from the other end.
5. Related Work 5. Related Work
The addition of ECN-capability to TCP's SYN/ACK packets was proposed The addition of ECN-capability to TCP's SYN/ACK packets was initially
in [ECN+]. The paper includes an extensive set of simulation and proposed in [ECN+]. The paper includes an extensive set of
testbed experiments to evaluate the effects of the proposal, using simulation and testbed experiments to evaluate the effects of the
several Active Queue Management (AQM) mechanisms, including Random proposal, using several Active Queue Management (AQM) mechanisms,
Early Detection (RED) [RED], Random Exponential Marking (REM) [REM], including Random Early Detection (RED) [RED], Random Exponential
and Proportional Integrator (PI) [PI]. The performance measures were Marking (REM) [REM], and Proportional Integrator (PI) [PI]. The
the end-to-end response times for each request/response pair, and the performance measures were the end-to-end response times for each
aggregate throughput on the bottleneck link. The end-to-end response request/response pair, and the aggregate throughput on the bottleneck
time was computed as the time from the moment when the request for link. The end-to-end response time was computed as the time from the
the file is sent to the server, until that file is successfully moment when the request for the file is sent to the server, until
downloaded by the client. that file is successfully downloaded by the client.
The measurements from [ECN+] show that setting an ECN-Capable The measurements from [ECN+] show that setting an ECN-Capable
codepoint in the IP packet header in TCP SYN/ACK packets codepoint in the IP packet header in TCP SYN/ACK packets
systematically improves performance with all evaluated AQM schemes. systematically improves performance with all evaluated AQM schemes.
When SYN/ACK packets at a congested router are ECN-marked instead of When SYN/ACK packets at a congested router are ECN-marked instead of
dropped, this can avoid a long initial retransmit timeout, improving dropped, this can avoid a long initial retransmit timeout, improving
the response time for the affected flow dramatically. the response time for the affected flow dramatically.
[ECN+] shows that the impact on aggregate throughput can also be [ECN+] shows that the impact on aggregate throughput can also be
quite significant, because marking SYN ACK packets can prevent larger quite significant, because marking SYN ACK packets can prevent larger
skipping to change at page 15, line 41 skipping to change at page 17, line 34
Thus, the degree of benefit of adding ECN-Capability to SYN/ACK Thus, the degree of benefit of adding ECN-Capability to SYN/ACK
packets depends not only on the overall packet drop rate in the packets depends not only on the overall packet drop rate in the
network, but also on the queue management architecture at the network, but also on the queue management architecture at the
congested link. congested link.
6.2. An Evaluation of Different Responses to ECN-Marked SYN/ACK Packets 6.2. An Evaluation of Different Responses to ECN-Marked SYN/ACK Packets
This document specifies that the end-node responds to the report of This document specifies that the end-node responds to the report of
an ECN-marked SYN/ACK packet by setting the initial congestion window an ECN-marked SYN/ACK packet by setting the initial congestion window
to one segment, instead of its possible default value of two to four to one segment, instead of its possible default value of two to four
segments. We call this ECN+ with NoWaiting. However, Section 4 segments, and resending a SYN/ACK packet that is not ECN-Capable. We
discussed another possible response to an ECN-marked SYN/ACK packet, call this ECN+/TryOnce.
of the end-node waiting an RTT before sending a data packet. We call
this approach ECN+ with Waiting. However, Section 4 discussed two other possible responses to an ECN-
marked SYN/ACK packet. In ECN+, the original proposal from [ECN+],
the end node responds to the report of an ECN-marked SYN/ACK packet
by setting the initial congestion window to one segment and
immediately sending a data packet, if it has one to send. In
ECN+/Wait, the end node responds to the report of an ECN-marked
SYN/ACK packet by setting the initial congestion window to one
segment and waiting an RTT before sending a data packet.
Simulations comparing the performance with Standard ECN (without ECN- Simulations comparing the performance with Standard ECN (without ECN-
marked SYN/ACK packets), ECN+ with NoWaiting, and ECN+ with Waiting marked SYN/ACK packets), ECN+, and ECN+/Wait, and ECN/TryOnce show
show little difference, in terms of aggregate congestion, between little difference, in terms of aggregate congestion, between ECN+ and
ECN+ with NoWaiting and ECN+ with Waiting. The details are given in ECN+/Wait. However, for some scenarios with queues that are packet-
Appendix A below. Our conclusions are that ECN+ with NoWaiting is based rather than byte-based, and with packet drop rates above 25%
perfectly safe, and there are no congestion-related reasons for without ECN+, the use of ECN+ or of ECN+/Wait can more than double
preferring ECN+ with Waiting over ECN+ with NoWaiting. That is, the packet drop rates, to greater than 50%. The details are given in
there is no need for the TCP end-node to wait a round-trip time Tables 1 and 3 of Appendix A below. ECN+/TryOnce does not increase
before sending a data packet after receiving an acknowledgement of an the packet drop rate in scenarios of high congestion. Therefore,
ECN-marked SYN/ACK packet. ECN+/TryOnce is superior to ECN+ or to ECN+/Wait, which both
significantly increase the packet drop rate in scenarios of high
congestion. At the same time, ECN+/TryOnce gives a performance
improvement similar to that of ECN+ or ECN+/Wait (Tables 2 and 4 of
Appendix A).
Our conclusions are that ECN+/TryOnce is safe, and has significant
benefits to the user, and avoids the problems of ECN+ or ECN+/Wait
under extreme levels of congestion. As a consequence, this document
specifies the use of ECN+/TryOnce.
[Note: We only discovered the occasional congestion-related problems
of ECN+ and of ECN+/Wait when re-running the simulations with an
updated version of the ns-2 simulator, after the internet-draft had
almost completed the standardization process.]
7. Security Considerations 7. Security Considerations
TCP packets carrying the ECT codepoint in IP headers can be marked TCP packets carrying the ECT codepoint in IP headers can be marked
rather than dropped by ECN-capable routers. This raises several rather than dropped by ECN-capable routers. This raises several
security concerns that we discuss below. security concerns that we discuss below.
7.1. 'Bad' Routers or Middleboxes 7.1. 'Bad' Routers or Middleboxes
There are a number of known deployment problems from using ECN with There are a number of known deployment problems from using ECN with
skipping to change at page 17, line 5 skipping to change at page 19, line 17
already known to crash when a data packet arrives with either ECT(0) already known to crash when a data packet arrives with either ECT(0)
or ECT(1)), but we have not conducted any measurement studies of this or ECT(1)), but we have not conducted any measurement studies of this
[F07]. [F07].
7.2. Congestion Collapse 7.2. Congestion Collapse
Because TCP SYN/ACK packets carrying an ECT codepoint could be ECN- Because TCP SYN/ACK packets carrying an ECT codepoint could be ECN-
marked instead of dropped at an ECN-capable router, the concern is marked instead of dropped at an ECN-capable router, the concern is
whether this can either invoke congestion, or worsen performance in whether this can either invoke congestion, or worsen performance in
highly congested scenarios. However, after learning that a SYN/ACK highly congested scenarios. However, after learning that a SYN/ACK
packet was ECN-marked, the responder will only send one data packet; packet was ECN-marked, the responder sends a SYN/ACK packet that is
if this data packet is ECN-marked, the responder will then wait for a not ECN-Capable; if this SYN/ACK packet is dropped, the responder
retransmission timeout. In addition, routers are free to drop rather then waits for a retransmission timeout, as specified in the TCP
than mark arriving packets in times of high congestion, regardless of standards. In addition, routers are free to drop rather than mark
whether the packets are ECN-capable. When congestion is very high arriving packets in times of high congestion, regardless of whether
and a router's buffer is full, the router has no choice but to drop the packets are ECN-capable. When congestion is very high and a
rather than to mark an arriving packet. router's buffer is full, the router has no choice but to drop rather
than to mark an arriving packet.
The simulations reported in Appendix A show that even with demanding The simulations reported in Appendix A show that even with demanding
traffic mixes dominated by short flows and high levels of congestion, traffic mixes dominated by short flows and high levels of congestion,
the aggregate packet dropping rates are not significantly different the aggregate packet dropping rates are not significantly different
with Standard ECN, ECN+ with NoWaiting, or ECN+ with Waiting. In with Standard ECN or with ECN+/TryOnce. However, in our simulations,
particular, the simulations show that in periods of very high we have one scenario where ECN+ or ECN+/Wait results in a
congestion the packet-marking rate is low with or without ECN+, and significantly higher packet drop rate than ECN or ECN+/TryOnce
the use of ECN+ does not significantly increase the number of dropped (Tables 1 and 3 in Appendix A below).
or marked packets.
The simulations show that ECN+ is most effective in times of moderate
congestion. In these moderate-congested scenarios, the use of ECN+
increases the number of ECN-marked packets, because ECN+ allows
SYN/ACK packets to be ECN-marked. At the same time, in these times
of moderate congestion, the use of ECN+ instead of Standard ECN does
not significantly affect the overall levels of congestion.
The simulations show that the use of ECN+ is less effective in times
of high congestion; the simulations show that in times of high
congestion more packets are dropped instead of marked, both with
Standard ECN and with ECN+. In times of high congestion, the buffer
can overflow, even with Active Queue Management and ECN; when the
buffer is full arriving packets are dropped rather than marked,
whether the packets are ECN-capable or not. Thus while ECN+ is less
effective in times of high congestion, it still doesn't result in a
significant increase in the level of congestion. More details are
given in the appendix.
8. Conclusions 8. Conclusions
This draft specifies a modification to RFC 3168 to allow TCP nodes to This draft specifies a modification to RFC 3168 to allow TCP nodes to
send SYN/ACK packets as being ECN-Capable. Making the SYN/ACK packet send SYN/ACK packets as being ECN-Capable. Making the SYN/ACK packet
ECN-Capable avoids the high cost to a TCP transfer when a SYN/ACK ECN-Capable avoids the high cost to a TCP transfer when a SYN/ACK
packet is dropped by a congested router, by avoiding the resulting packet is dropped by a congested router, by avoiding the resulting
retransmit timeout. This improves the throughput of short retransmit timeout. This improves the throughput of short
connections. The sender of the SYN/ACK packet responds to an ECN connections. This document specifies the ECN+/TryOnce mechanism for
mark by reducing its initial congestion window from two, three, or ECN-Capability for SYN/ACK packets, where the sender of the SYN/ACK
four segments to one segment, reducing the subsequent load from that packet responds to an ECN mark by reducing its initial congestion
connection on the network. The addition of ECN-capability to SYN/ACK window from two, three, or four segments to one segment, and sending
packets is particularly beneficial in the server-to-client direction, a SYN/ACK packet that is not ECN-Capable. The addition of ECN-
where congestion is more likely to occur. In this case, the initial capability to SYN/ACK packets is particularly beneficial in the
information provided by the ECN marking in the SYN/ACK packet enables server-to-client direction, where congestion is more likely to occur.
the server to more appropriately adjust the initial load it places on In this case, the initial information provided by the ECN marking in
the network. the SYN/ACK packet enables the server to appropriately adjust the
initial load it places on the network, while avoiding the delay of a
retransmit timeout.
9. Acknowledgements 9. Acknowledgements
We thank Anil Agarwal, Mark Allman, Remi Denis-Courmont, Wesley Eddy, We thank Anil Agarwal, Mark Allman, Remi Denis-Courmont, Wesley Eddy,
Lars Eggert, Alfred Hoenes, Janardhan Iyengar, and Pasi Sarolahti for Lars Eggert, Alfred Hoenes, Janardhan Iyengar, and Pasi Sarolahti for
feedback on earlier versions of this draft. feedback on earlier versions of this draft. We thank Adam Langley
[L08] for contributing a patch for ECN+/TryOnce for the Linux
development tree.
A. Report on Simulations A. Report on Simulations
This section reports on simulations showing the costs of adding ECN+ This section reports on simulations showing the costs of adding ECN+
in highly-congested scenarios. This section also reports on in highly-congested scenarios. This section also reports on
simulations for a comparative evaluation between ECN+ with NoWaiting simulations for a comparative evaluation between ECN, ECN+,
and ECN+ with Waiting. ECN+/Wait, and ECN+/TryOnce.
The simulations are run with a range of file-size distributions, The simulations are run with a range of file-size distributions,
using the PackMime traffic generator in the ns-2 simulator. They all using the PackMime traffic generator in the ns-2 simulator. They all
use a heavy-tailed distribution of file sizes. The simulations use a heavy-tailed distribution of file sizes. The simulations
reported in the tables below use a mean file size of 3 KBypes, to reported in the tables below use a mean file size of 3 KBypes, to
show the results with a traffic mix with a large number of small show the results with a traffic mix with a large number of small
transfers. Othe simulations were run with mean file sizes of 5 transfers. Other simulations were run with mean file sizes of 5
KBytes, 7 Kbytes, 14 KBytes, and 17 Kbytes. The title of each chart KBytes, 7 Kbytes, 14 KBytes, and 17 Kbytes. The title of each chart
gives the targeted average load from the traffic generator. Because gives the targeted average load from the traffic generator. Because
the simulations use a heavy-tailed distribution of file sizes, and the simulations use a heavy-tailed distribution of file sizes, and
run for only 85 seconds (including ten seconds of warm-up time), the run for only 85 seconds (including ten seconds of warm-up time), the
actual load is often much smaller than the targeted load. The actual load is often much smaller than the targeted load. The
congested link is 100 Mbps. RED is run in gentle mode, and arriving congested link is 100 Mbps. RED is run in gentle mode, and arriving
ECN-Capable packets are only dropped instead of marked if the buffer ECN-Capable packets are only dropped instead of marked if the buffer
is full (and the router has no choice). is full (and the router has no choice).
We explore two alternatives for a TCP node's response to a report of We explore three possible mechanisms for a TCP node's response to a
an ECN-marked SYN/ACK packet. With ECN+ with NoWaiting, the TCP node report of an ECN-marked SYN/ACK packet. With ECN+, the TCP node
sends a data packet immediately (with an initial congestion window of sends a data packet immediately (with an initial congestion window of
one segment). With the alternative ECN+ with Waiting, the TCP node one segment). With ECN+/Wait, the TCP node waits a round-trip time
waits a round-trip time before sending a data packet; the responder before sending a data packet; the responder already has one
already has one measurement of the round-trip time when the measurement of the round-trip time when the acknowledgement for the
acknowledgement for the SYN/ACK packet is received. SYN/ACK packet is received. With ECN+/TryOnce, the mechanism
standardized in this document, the TCP responder replies to a report
In the tables below, ECN+ refers to ECN+ with NoWaiting, where the of an ECN-marked SYN/ACK packet by sending a SYN/ACK packet that is
responder starts transmitting immediately, and ECN+/wait refers to not ECN-Capable, and reducing the initial congestion window to one
ECN+ with Waiting, where the responder waits a round-trip time before segment.
sending a data packet into the network.
The simulation scripts are available on [ECN-SYN]. The simulation scripts are available on [ECN-SYN]. along with graphs
showing the distribution of response times for the TCP connections.
A.1. Simulations with RED in Packet Mode A.1. Simulations with RED in Packet Mode
The simulations with RED in packet mode and with the queue in packets The simulations with RED in packet mode and with the queue in packets
show that ECN+ is useful in times of moderate congestion, though it show that ECN+ is useful in times of moderate or of high congestion.
adds little benefit in times of high congestion. The simulations However, for the simulations with a target load of 125%, with a
show a minimal increase in levels of congestion with either ECN+ with packet loss rate of over 25% for ECN, ECN+ and ECN+/Wait both result
Waiting or ECN+ with NoWaiting, either in terms of packet dropping or in a packet loss rate of over 50%. (In contrast, the packet loss
marking rates or in terms of the distribution of responses times. rate with ECN+/TryOnce is less than that of ECN alone.) For the
Thus, the simulations show no problems with ECN+ in times of high distribution of response times, the simulations show that ECN+,
congestion, and no reason to use ECN+ with Waiting instead of ECN+ ECN+/Wait, and ECN+/TryOnce all significantly improve the response
with NoWaiting. times, compared to the response times with plain ECN.
Table 1 shows the congestion levels for simulations with RED in Table 1 shows the congestion levels for simulations with RED in
packet mode, with a queue in packets. To explore a worst-case packet mode, with a queue in packets. To explore a worst-case
scenario, these simulations use a traffic mix with an unrealistically scenario, these simulations use a traffic mix with an unrealistically
small flow size distribution, with a mean flow size of 3 Kbytes. For small flow size distribution, with a mean flow size of 3 Kbytes. For
each table showing a particular traffic load, the three rows show the each table showing a particular traffic load, the four rows show the
number of packets dropped, the number of packets ECN-marked, and the number of packets dropped, the number of packets ECN-marked, the
aggregate packet drop rate, and the three columns show the aggregate packet drop rate, and the aggregate throughput, and the
simulations with Standard ECN, ECN+ (NoWaiting) and ECN+/wait. four columns show the simulations with Standard ECN, ECN+, ECN+/Wait,
and ECN+/TryOnce.
These simulations were run with RED set to mark instead of drop These simulations were run with RED set to mark instead of drop
packets any time that the queue is not full. For the default packets any time that the queue is not full. This is a worst-case
implementation of RED in the ns-2 simulator, the router drops instead scenario for ECN+ and its variants. For the default implementation
of marks arriving packets when the average queue size exceeds a of RED in the ns-2 simulator, when the average queue size exceeds a
configured threshold. the router drops all arriving packets. For
scenarios with this RED mechanisms, it is less likely that ECN+ or
one of its variants would increase the average queue size above the
configured threshold. configured threshold.
The usefulness of ECN+: The first thing to observe is that for all of The usefulness of ECN+: The first thing to observe is that for all of
the simulations, the use of ECN+ or ECN+/wait significantly increased the simulations, the use of ECN+ or ECN+/Wait significantly increases
the number of packets marked. This indicates that with ECN+ or the number of packets marked. In contrast, the use of ECN+/TryOnce
ECN+/wait, many SYN/ACK packets are marked instead of dropped. significantly increases the number of packets marked in the
simulations with moderate congestion, and gives a more moderate
increase in the number of packets marked for the simulations with
higher levels of congestion. However, the cumulative distribution
function (CDF) in Table 2 shows that ECN+, ECN+/Wait, and
ECN+/TryOnce all improve response times for all of the simulations,
with moderate or with larger levels of congestion.
Little increase in congestion, sometimes: The second thing to observe Little increase in congestion, sometimes: The second thing to observe
is that for the simulations with low or moderate levels of congestion is that for the simulations with low or moderate levels of congestion
(that is, with packet drop rates less than 10%), the use of ECN+ or (that is, with packet drop rates less than 10%), the use of ECN+,
ECN+/wait decreases the aggregate packet drop rate, relative to the ECN+/Wait, and ECN+/TryOnce all decrease the aggregate packet drop
simulations with ECN. This makes sense, since with low or moderate rate, relative to the simulations with ECN. This makes sense, since
levels of congestion, ECN+ allows SYN/ACK packets to be marked with low or moderate levels of congestion, ECN+ allows SYN/ACK
instead of dropped, and the use of ECN+ doesn't add to the aggregate packets to be marked instead of dropped, and the use of ECN+ doesn't
congestion. However, for the simulations with packet drop rates of add to the aggregate congestion. However, for the simulations with
15% or higher with ECN, the use of ECN+ or ECN+/wait increases the packet drop rates of 15% or higher with ECN, the use of ECN+ or
aggregate packet drop rate, sometimes even doubling it. ECN+/Wait increases the aggregate packet drop rate, sometimes even
doubling it.
Comparing ECN+ and ECN+/wait: The third thing to observe is that the Comparing ECN+, ECN+/Wait, and ECN+/TryOnce: The aggregate packet
aggregate packet drop rate is generally higher with ECN+/wait than drop rate is generally higher with ECN+/Wait than with ECN+. Thus,
with ECN+. Thus, there is no congestion-related reason to prefer there is no congestion-related reason to prefer ECN+/Wait over ECN+.
ECN+/wait over ECN+. In contrast, the aggregate packet drop rate with ECN+/TryOnce is
often significantly lower than the aggregate packet drop rate with
either ECN, ECN+, ECN+/Wait.
Target Load = 95%: Target Load = 95%:
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 18,512 11,244 12,135 Dropped 20,516 11,226 11,735 16,446`
Marked 27,026 36,977 38,743 Marked 30,586 37,741 37,425 40,530
Loss rate 1.27% 0.78% 0.84% Loss rate 1.41% 0.78% 0.81% 1.01%
Throughput 81% 81% 81% Throughput 81% 81% 81% 81%
Target Load = 110%: Target Load = 110%:
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 165,866 110,525 144,821 Dropped 165,566 106,083 147,180 218,594
Marked 180,714 290,629 311,233 Marked 179,735 281,306 308,473 242,969
Loss rate 9.04% 6.36% 7.94% Loss rate 9.01% 6.12% 8.02% 7.14%
Throughput 92% 92% 92% Throughput 92% 92% 92% 94%
Target Load = 125%: Target Load = 125%:
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 574,114 1,764,677 2,229,280 Dropped 600,628 1,746,768 2,176,530 650,781
Marked 409,441 1,172,550 1,181,209 Marked 418,433 1,166,450 1,164,932 440,432
Loss rate 24.55% 52.00% 57.64% Loss rate 25.45% 51.73% 56.87% 18.22%
Throughput 94% 98% 97% Throughput 94% 98% 97% 95%
Target Load = 1.50%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 1,449,945 1,565,0517 1,563,0801 1,372,067
Marked 669,840 583,378 591,315 675,290
Loss rate 46.7% 59.0% 59.0% 32.3%
Throughput 88% 94% 94% 93%
Table 1: Simulations with an average flow size of 3 Kbytes, a Table 1: Simulations with an average flow size of 3 Kbytes, a
100 Mbps link, RED in packet mode, queue in packets. 100 Mbps link, RED in packet mode, queue in packets.
Target Load = 95%: Target Load = 95%:
ECN ECN+ ECN+/wait
------- ------- ------- TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
Dropped 8,754 6,719 7,269 ------------------------------------------------------
Marked 10,376 17,637 16,956 ECN: 0.00 0.07 0.26 0.51 0.82 0.96 0.97 0.97 0.97 1.00 1.00
Loss rate 5.68% 4.50% 4.75% ECN+: 0.00 0.07 0.27 0.53 0.85 0.99 1.00 1.00 1.00 1.00 1.00
Throughput 78% 78% 78% Wait: 0.00 0.07 0.26 0.51 0.83 0.97 1.00 1.00 1.00 1.00 1.00
Once: 0.00 0.07 0.24 0.49 0.83 0.97 1.00 1.00 1.00 1.00 1.00
Target Load = 110%: Target Load = 110%:
ECN ECN+ ECN+/wait
------- ------- ------- TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
Dropped 32,110 32,014 48,838 ------------------------------------------------------
Marked 28,476 56,550 62,252 ECN: 0.00 0.05 0.19 0.41 0.67 0.79 0.80 0.80 0.80 0.96 0.96
Loss rate 15.68% 16.11% 21.92% ECN+: 0.00 0.07 0.22 0.48 0.81 0.96 1.00 1.00 1.00 1.00 1.00
Throughput 96% 96% 96% Wait: 0.00 0.05 0.18 0.38 0.64 0.77 0.95 1.00 1.00 1.00 1.00
Once: 0.00 0.06 0.19 0.41 0.70 0.86 0.95 0.96 0.96 0.99 0.99
Target Load = 125%: Target Load = 125%:
ECN ECN+ ECN+/wait
------- ------- -------
Dropped 60,710 174,920 215,001
Marked 43,497 119,620 118,172
Loss rate 25.08% 51.59% 56.27%
Throughput 98% 98% 98%
Target Load = 150%: TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
ECN ECN+ ECN+/wait ------------------------------------------------------
------- ------- ------- ECN: 0.00 0.04 0.13 0.27 0.46 0.56 0.58 0.59 0.59 0.82 0.82
Dropped 133,128 250,762 327,584 ECN+: 0.00 0.06 0.18 0.33 0.58 0.76 0.97 0.99 0.99 1.00 1.00
Marked 63,306 146,581 147,307 Wait: 0.00 0.01 0.06 0.13 0.21 0.27 0.68 0.98 0.99 1.00 1.00
Loss rate 43.34% 61.11% 67.33% Once: 0.00 0.05 0.16 0.34 0.58 0.73 0.85 0.87 0.87 0.95 0.96
Throughput 93% 100% 100%
Table 2: Simulations with an average flow size of 3 Kbytes, a 10 Mbps TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.03 0.08 0.18 0.31 0.39 0.42 0.42 0.43 0.68 0.68
ECN+: 0.00 0.06 0.18 0.39 0.67 0.81 0.83 0.84 0.84 0.93 0.93
Wait: 0.00 0.06 0.18 0.39 0.67 0.81 0.83 0.84 0.84 0.93 0.94
Once: 0.00 0.04 0.13 0.28 0.47 0.60 0.72 0.75 0.76 0.88 0.89
Table 2: The cumulative distribution function (CDF) for transfer
times, for simulations with an average flow size of 3 Kbytes, a
100 Mbps link, RED in packet mode, queue in packets. (The graphs are
available from "http://www.icir.org/floyd/ecn-syn/".)
Target Load = 0.95%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 8,448 6,362 7,740 16,323
Marked 9,891 16,787 17,456 17,186
Loss rate 5.5% 4.3% 5.0% 5.4%
Throughput 78% 78% 78% 82%
Target Load = 1.10%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 31,284 29,773 49,297 42,201
Marked 28,429 54,729 60,383 33,672
Loss rate 15.3% 15.2% 21.9% 13.5%
Throughput 97% 96% 96% 95%
Target Load = 1.25%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 61,433 176,682 214,096 79,463
Marked 44,408 119,728 117,301 48,991
Loss rate 25.4% 51.9% 56.0% 22.5%
Throughput 97% 98% 98% 95%
Target Load = 1.50%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 130,007 251,856 326,845 141,418
Marked 63,066 146,757 147,239 67,772
Loss rate 42.5% 61.3% 67.3% 33.3%
Throughput 93% 99% 99% 94%
Table 3: Simulations with an average flow size of 3 Kbytes, a 10 Mbps
link, RED in packet mode, queue in packets. link, RED in packet mode, queue in packets.
Target Load = 95%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.05 0.18 0.42 0.70 0.86 0.88 0.88 0.88 0.98 0.98
ECN+: 0.00 0.06 0.20 0.45 0.78 0.96 1.00 1.00 1.00 1.00 1.00
Wait: 0.00 0.05 0.18 0.40 0.68 0.84 0.96 1.00 1.00 1.00 1.00
Once: 0.00 0.05 0.18 0.39 0.69 0.87 0.96 0.96 0.96 0.99 0.99
Target Load = 110%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.03 0.13 0.29 0.52 0.66 0.69 0.69 0.69 0.91 0.91
ECN+: 0.00 0.05 0.17 0.36 0.66 0.88 0.98 0.99 1.00 1.00 1.00
Wait: 0.00 0.02 0.08 0.20 0.35 0.47 0.76 0.98 1.00 1.00 1.00
Once: 0.00 0.04 0.15 0.33 0.59 0.76 0.89 0.91 0.91 0.98 0.98
Target Load = 125%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.03 0.10 0.22 0.40 0.52 0.56 0.56 0.57 0.82 0.82
ECN+: 0.00 0.03 0.14 0.27 0.49 0.70 0.96 0.99 0.99 0.99 1.00
Wait: 0.00 0.00 0.03 0.07 0.12 0.18 0.50 0.94 0.99 0.99 1.00
Once: 0.00 0.04 0.13 0.29 0.51 0.66 0.81 0.84 0.84 0.94 0.94
Target Load = 150%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.02 0.07 0.15 0.28 0.38 0.42 0.42 0.43 0.67 0.68
ECN+: 0.00 0.00 0.00 0.00 0.01 0.05 0.68 0.83 0.95 0.97 0.98
Wait: 0.00 0.00 0.00 0.00 0.00 0.00 0.10 0.62 0.83 0.93 0.97
Once: 0.00 0.03 0.11 0.23 0.42 0.56 0.71 0.74 0.74 0.87 0.88
Table 4: The cumulative distribution function (CDF) for transfer
times, for simulations with an average flow size of 3 Kbytes, a
10 Mbps link, RED in packet mode, queue in packets. (The graphs are
available from "http://www.icir.org/floyd/ecn-syn/".)
A.2. Simulations with RED in Byte Mode A.2. Simulations with RED in Byte Mode
Table 3 below shows simulations with RED in byte mode and the queue Table 5 below shows simulations with RED in byte mode and the queue
in bytes. There is no significant increase in aggregate congestion in bytes. There is no significant increase in aggregate congestion
with the use of ECN+ or ECN+/wait, and no congestion-related reason with the use of ECN+, ECN+/Wait, or ECN+/TryOnce.
to prefer ECN+/wait over ECN+.
However, unlike the simulations with RED in packet mode, the However, unlike the simulations with RED in packet mode, the
simulations with RED in byte mode show little benefit from the use of simulations with RED in byte mode show little benefit from the use of
ECN+ or ECN+/wait, in that the packet marking rate with ECN+ or ECN+ or ECN+/Wait, in that the packet marking rate with ECN+ or
ECN+/wait is not much different than the packet marking rate with ECN+/Wait is not much different than the packet marking rate with
Standard ECN. This is because with RED in byte mode, small packets Standard ECN. This is because with RED in byte mode, small packets
like SYN/ACK packets are rarely dropped or marked - that is, there is like SYN/ACK packets are rarely dropped or marked - that is, there is
no drawback from the use of ECN+ in these scenarios, but not much no drawback from the use of ECN+ in these scenarios, but not much
need for ECN+ either, in a scenario where small packets are unlikely need for ECN+ either, in a scenario where small packets are unlikely
to be dropped or marked. to be dropped or marked.
Target Load = 95%: Target Load = 95%
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 739 438 442 Dropped 766 446 427 408
Marked 32,405 34,357 34,000 Marked 32,683 34,289 33,412 31,892
Loss rate 0.05% 0.03% 0.03% Loss rate 0.05% 0.03% 0.03% 0.03%
Throughput 81% 81% 81% Throughput 81% 81% 81% 81%
Target Load = 110%: Target Load = 110%
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 2,473 1,679 3,020 Dropped 2,496 2,110 1,733 2,024
Marked 226,971 222,234 327,608 Marked 220,573 258,696 230,955 224,338
Loss rate 0.15% 0.10% 0.18% Loss rate 0.15% 0.13% 0.11% 0.11%
Throughput 92% 92% 91% Throughput 92% 91% 92% 92%
Target Load = 125%: Target Load = 125%
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 19,358 14,057 14,064 Dropped 20,032 13,555 13,979 19,544
Marked 717,123 728,513 729,001 Marked 725,165 726,992 726,823 627,088
Loss rate 1.07% 0.78% 0.78% Loss rate 1.11% 0.76% 0.78% 0.72%
Throughput 95% 95% 95% Throughput 95% 95% 95% 95%
Table 3: Simulations with an average flow size of 3 Kbytes, a Target Load = 150%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 484,251 483,847 507,727 572,373
Marked 865,905 872,254 873,317 816,841
Loss rate 19.09% 19.13% 19.71% 12.28%
Throughput 99% 98% 99% 99%
Table 5: Simulations with an average flow size of 3 Kbytes, a
100 Mbps link, RED in byte mode, queue in bytes. 100 Mbps link, RED in byte mode, queue in bytes.
Target Load = 95%: Target Load = 0.95%
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 142 81 78 Dropped 142 77 103 99
Marked 11,694 11,812 11,964 Marked 11,694 11,387 11,604 12,129
Loss rate 0.01% 0.06% 0.05% Loss rate 0.1% 0.1% 0.1% 0.1%
Throughput 78% 78% 78% Throughput 78% 78% 78% 78%
Target Load = 110%: Target Load = 1.10%
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 314 215 188 Dropped 338 210 247 292
Marked 39,697 42,388 40,229 Marked 41,676 40,412 44,173 37,527
Loss rate 0.19% 0.13% 0.11% Loss rate 0.2% 0.1% 0.1% 0.1%
Throughput 95% 94% 95% Throughput 94% 94% 94% 95%
Target Load = 125%: Target Load = 1.25%
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 1,599 1,011 985 Dropped 1,559 951 978 1,490
Marked 74,567 75,782 75,528 Marked 74,933 75,499 75,481 57,721
Loss rate 0.87% 0.56% 0.54% Loss rate 0.8% 0.5% 0.5% 0.5%
Throughput 98% 98% 98% Throughput 99% 99% 99% 96%
Target Load = 150%: Target Load = 1.50%
ECN ECN+ ECN+/wait ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ------- ------- ------- ----------
Dropped 2,429 1,538 1,571 Dropped 2,374 1,528 1,515 4,517
Marked 85,312 86,481 86,476 Marked 85,739 86,428 86,144 81,695
Loss rate 1.22% 0.78% 0.79% Loss rate 1.2% 0.8% 0.8% 1.3%
Throughput 98% 98% 98% Throughput 99% 98% 98% 98%
Table 4: Simulations with an average flow size of 3 Kbytes, a 10 Mbps Table 6: Simulations with an average flow size of 3 Kbytes, a 10 Mbps
link, RED in byte mode, queue in bytes. link, RED in byte mode, queue in bytes.
B. Issues of Incremental Deployment B. Issues of Incremental Deployment
In order for TCP node B to send a SYN/ACK packet as ECN-Capable, node In order for TCP node B to send a SYN/ACK packet as ECN-Capable, node
B must have received an ECN-setup SYN packet from node A. However, B must have received an ECN-setup SYN packet from node A. However,
it is possible that node A supports ECN, but either ignores the CE it is possible that node A supports ECN, but either ignores the CE
codepoint on received SYN/ACK packets, or ignores SYN/ACK packets codepoint on received SYN/ACK packets, or ignores SYN/ACK packets
with the ECT or CE codepoint set. If the TCP initiator ignores the with the ECT or CE codepoint set. If the TCP initiator ignores the
CE codepoint on received SYN/ACK packets, this would mean that the CE codepoint on received SYN/ACK packets, this would mean that the
TCP responder would not respond to this congestion indication. TCP responder would not respond to this congestion indication.
However, this seems to us an acceptable cost to pay in the However, this seems to us an acceptable cost to pay in the
incremental deployment of ECN-Capability for TCP's SYN/ACK packets. incremental deployment of ECN-Capability for TCP's SYN/ACK packets.
It would mean that the responder would not reduce the initial It would mean that the responder would not reduce the initial
congestion window from two, three, or four segments down to one congestion window from two, three, or four segments down to one
segment, as it should. However, the TCP end nodes would still segment, as it should. and would not sent a non-ECN-Capable SYN/ACK
respond correctly to any subsequent CE indications on data packets packet to complete the SYN exchange. However, the TCP end nodes
later on in the connection. would still respond correctly to any subsequent CE indications on
data packets later on in the connection.
Figure 3 shows an interchange with the SYN/ACK packet ECN-marked, but Figure 4 shows an interchange with the SYN/ACK packet ECN-marked, but
with the ECN mark ignored by the TCP originator. with the ECN mark ignored by the TCP originator.
--------------------------------------------------------------- ---------------------------------------------------------------
TCP Node A Router TCP Node B TCP Node A Router TCP Node B
(initiator) (responder)
---------- ------ ---------- ---------- ------ ----------
ECN-setup SYN packet ---> ECN-setup SYN packet --->
ECN-setup SYN packet ---> ECN-setup SYN packet --->
<--- ECN-setup SYN/ACK, ECT <--- ECN-setup SYN/ACK, ECT
<--- Sets CE on SYN/ACK <--- Sets CE on SYN/ACK
<--- ECN-setup SYN/ACK, CE <--- ECN-setup SYN/ACK, CE
Data/ACK, No ECN-Echo ---> Data/ACK, No ECN-Echo --->
Data/ACK ---> Data/ACK --->
<--- Data (up to four packets) <--- Data (up to four packets)
--------------------------------------------------------------- ---------------------------------------------------------------
Figure 3: SYN exchange with the SYN/ACK packet marked, Figure 4: SYN exchange with the SYN/ACK packet marked,
but with the ECN mark ignored by the TCP initiator. but with the ECN mark ignored by the TCP initiator.
Thus, to be explicit, when a TCP connection includes an initiator Thus, to be explicit, when a TCP connection includes an initiator
that supports ECN but *does not* support ECN-Capability for SYN/ACK that supports ECN but *does not* support ECN-Capability for SYN/ACK
packets, in combination with a responder that *does* support ECN- packets, in combination with a responder that *does* support ECN-
Capability for SYN/ACK packets, it is possible that the ECN-Capable Capability for SYN/ACK packets, it is possible that the ECN-Capable
SYN/ACK packets will be marked rather than dropped in the network, SYN/ACK packets will be marked rather than dropped in the network,
and that the responder will not learn about the ECN mark on the and that the responder will not learn about the ECN mark on the
SYN/ACK packet. This would not be a problem if most packets from the SYN/ACK packet. This would not be a problem if most packets from the
responder supporting ECN for SYN/ACK packets were in long-lived TCP responder supporting ECN for SYN/ACK packets were in long-lived TCP
skipping to change at page 26, line 30 skipping to change at page 30, line 33
"http://www.icir.org/floyd/ecn-syn". "http://www.icir.org/floyd/ecn-syn".
[F07] S. Floyd, "[BEHAVE] Response of firewalls and middleboxes to [F07] S. Floyd, "[BEHAVE] Response of firewalls and middleboxes to
TCP SYN packets that are ECN-Capable?", August 2, 2007, email sent to TCP SYN packets that are ECN-Capable?", August 2, 2007, email sent to
the BEHAVE mailing list, URL "http://www1.ietf.org/mail- the BEHAVE mailing list, URL "http://www1.ietf.org/mail-
archive/web/behave/current/msg02644.html". archive/web/behave/current/msg02644.html".
[Kelson00] Dax Kelson, note sent to the Linux kernel mailing list, [Kelson00] Dax Kelson, note sent to the Linux kernel mailing list,
September 10, 2000. September 10, 2000.
[L08] A. Landley, "Re: [tcpm] I-D Action:draft-ietf-tcpm-
ecnsyn-06.txt", Email to the tcpm mailing list, August 24, 2008.
[MAF05] A. Medina, M. Allman, and S. Floyd. Measuring the Evolution [MAF05] A. Medina, M. Allman, and S. Floyd. Measuring the Evolution
of Transport Protocols in the Internet, ACM CCR, April 2005. of Transport Protocols in the Internet, ACM CCR, April 2005.
[PI] C. Hollot, V. Misra, W. Gong, and D. Towsley, On Designing [PI] C. Hollot, V. Misra, W. Gong, and D. Towsley, On Designing
Improved Controllers for AQM Routers Supporting TCP Flows, April Improved Controllers for AQM Routers Supporting TCP Flows, April
1998. 1998.
[RED] Floyd, S., and Jacobson, V. Random Early Detection gateways [RED] Floyd, S., and Jacobson, V. Random Early Detection gateways
for Congestion Avoidance . IEEE/ACM Transactions on Networking, V.1 for Congestion Avoidance . IEEE/ACM Transactions on Networking, V.1
N.4, August 1993. N.4, August 1993.
 End of changes. 72 change blocks. 
315 lines changed or deleted 511 lines changed or added

This html diff was produced by rfcdiff 1.35. The latest version is available from http://tools.ietf.org/tools/rfcdiff/