Você está na página 1de 58

Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.

Using Ethereal to Investigate TCP


Performance Issues


Dana J. Dawson
Sr. Staff Engineer, CCIE #1937
Advanced Technical Support CPE
Qwest Services Corporation
Dana.Dawson@qwest.com


July 7, 2006






Summary

Qwest customers sometimes experience data throughput rates with individual TCP
connections that are less than they expect. While there are many possible causes for this,
such as packet loss and TCP window size issues, determining the exact cause of the poor
performance can be difficult. Fortunately, there is a freely available open source software
utility called Ethereal!

that can help. This paper provides an introduction to Ethereal and
describes some of the ways it can be used to investigate TCP performance issues. Various
examples are presented along with descriptions of which Ethereal features were used in
identifying or eliminating possible causes of the observed behavior. Annotated images of
the Ethereal displays are used as necessary to illustrate these features.


Ethereal is a registered trademark of Ethereal, Inc.
Using Ethereal to Investigate TCP Performance Issues Page 2 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Contents
CONTENTS ................................................................................................................2
ABOUT ETHEREAL ......................................................................................................3
ABOUT TCP ...............................................................................................................3
ABOUT THE EXAMPLES................................................................................................4
PACKET CAPTURING BASICS .......................................................................................5
SAVING THE CAPTURE FILE.........................................................................................9
ISOLATING AN INDIVIDUAL TCP CONNECTION ............................................................ 10
TCP STREAM GRAPHS ............................................................................................... 15
EXAMPLE 1 DETERMINING ACTUAL THROUGHPUT OF A TCP CONNECTION ................... 23
EXAMPLE 2 THROUGHPUT LIMITED BY RECEIVE WINDOW SIZE .................................. 27
EXAMPLE 3 THROUGHPUT LIMITED BY SENDER WINDOW SIZE................................... 33
EXAMPLE 4 PACKET LOSS....................................................................................... 36
EXAMPLE 5 OUT OF ORDER PACKETS....................................................................... 44
CONCLUSION .......................................................................................................... 47
TEST TOPOLOGY ...................................................................................................... 48
REFERENCES AND RESOURCES.................................................................................. 49
APPENDIX A: CONFIGURING ETHEREAL ..................................................................... 50
ETHEREAL PREFERENCES: LAYOUT............................................................................ 51
ETHEREAL PREFERENCES: COLUMNS......................................................................... 53
ETHEREAL PREFERENCES: FONTS ............................................................................. 54
ETHEREAL PREFERENCES: CAPTURE.......................................................................... 55
ETHEREAL PREFERENCES: NAME RESOLUTION ........................................................... 56
ETHEREAL PREFERENCES: PACKET COLORING RULES.................................................. 57
SPECIAL NOTE: ETHEREAL AND THE CISCO VPN CLIENT ON WINDOWS......................... 58
Using Ethereal to Investigate TCP Performance Issues Page 3 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
About Ethereal

Ethereal is an open source network protocol analyzer program available at no cost under the
GNU General Public License from http://www.ethereal.com. It runs on most operating
systems, and is capable of both capturing packets and decoding virtually all network
protocols. In addition to the packet capture display features typically available in network
analyzers, such as filtering and colorizing user-specified packets, Ethereal can also display
interactive plots of TCP sessions in a variety of ways, including Time/Sequence, Throughput,
and Round-Trip Time graphs. These TCP plotting features will be the primary focus of this
paper. When I started writing this paper the current version of Ethereal was 0.10.14, which
is the version I used in all the examples, but Version 0.99.0 has recently been released. I
havent noticed any significant differences between the two versions, so this discussion
should apply to the new version and other recent versions of Ethereal as well.

This paper is not intended to be an exhaustive tutorial on Ethereal, but will instead focus on
and illustrate just the features necessary for you to perform the analyses described in the
examples. I assume you will know or be able to easily learn how to install Ethereal, as well
as navigate the Ethereal menus and GUI interface. Ethereal has many options and features,
some of which can be complex and counter-intuitive. Fortunately for us, the features being
discussed here are relatively straightforward. The visual appearance of Ethereal is quite
uniform across most platforms, however it is possible that some of the screen images in this
paper may be slightly different from what you may see on your computer screen. The
screen images in this paper were taken with a default installation of Ethereal on both
Windows 2000 SP4 and Mac OS X 10.4.6 and should be quite similar to how any recent
version of Ethereal looks on other platforms. In any event, the important concepts of the
examples should be clear regardless of the appearance of your version of Ethereal.


About TCP

While this paper is also not intended to be a complete tutorial on TCP, it is likely that the
discussions in the examples will provide a more thorough understanding of some of the
specifics of TCP behavior, especially in response to congestion and packet loss. Even so,
you should be familiar with the basics of TCP and its concepts, including the three-way
handshake used for connection establishment, sequence numbers, and the TCP Sliding
Window scheme. Ill describe any complex or obscure concepts that are important to the
situation as necessary.

Its worth noting that there are different implementations of TCP in general use, and their
default behaviors can vary. Different platforms also use different default values for some
parameters, and include different sets of options. For example, Windows XP enables the
Selective ACK option by default, whereas other platforms dont support it all. While Ill try
to make any conclusions in this paper as general as possible, there may be individual cases
in the real world that will appear to conflict with those conclusions. These situations are
more likely to occur under extreme or unusual conditions and so I expect theyll be rare. In
such cases it will likely be necessary to consult detailed references of the TCP
implementations involved to try to determine the expected behavior. Additionally, some
TCP parameters, such as the Congestion Window, are not easily seen or derived from
packet traces, so the best that can be done is to speculate on their impact on performance
based on an understanding of how they are used by TCP.

Using Ethereal to Investigate TCP Performance Issues Page 4 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.

About the Examples

All of the examples described in this paper involve situations where the observed data
throughput for a single TCP connection is less than expected. For example, a customer may
be using FTP to transfer large files between two sites and seeing throughput thats
substantially below the available bandwidth of the WAN. Ill assume that other more
obvious possible causes of the low throughput have already been eliminated, such as high
traffic levels resulting in full link utilization and congestion of the circuit(s) involved, bad
QoS or other configuration settings in routers or other CPE in the data path, and physical
errors on the LAN or WAN. While Ethereal can sometimes be useful in finding these types of
problems, other methods are usually more effective and should be pursued first. When
those other approaches fail to explain the observed behavior, then it becomes more likely
that some aspect of TCPs behavior may be the cause and Ethereal can be used to help
identify it.

Some of the examples may have been taken from actual Qwest customer issues, in which
case I have removed any customer-private information. In these cases the information
obscured is not pertinent to the discussion. Most of the examples are from simulations. In
both cases I chose the examples to illustrate a specific aspect of TCP behavior and how it
appears in Ethereal, such as the effect of a too small window size on throughput. However,
TCP is not a deterministic protocol, so no two connections are exactly the same and you
should not expect to see identical results for the various situations. The point of the
examples is to expose identifiable characteristics in a way that applies to similar situations
in the real world.
Using Ethereal to Investigate TCP Performance Issues Page 5 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Packet Capturing Basics

Before Ethereal can be used to analyze a packet capture, the packets have to actually be
captured. Ethereal can do this directly, but since it can also open a wide variety of capture
file formats, other utilities or network analyzers can also be used. Ideally, the packets
would be captured at both the source and destination of the traffic, and should include the
complete TCP connection, since that provides the most complete information. If thats not
possible, its better to capture traffic at the transmitting end rather than the receiving end,
since in TCP its the transmitter that controls the traffic rate based on information it has
available. Its usually, but not always, more useful to be able to see the original traffic
before its been affected by any intermediate devices or phenomena. In the examples in
this paper all the captures were taken on the transmitting host unless otherwise stated.

Its also better to configure Ethereal to only capture the traffic to be analyzed, since that
reduces the CPU load as well as the size of the capture file. This is often as simple as
specifying a single host IP address and capturing all the traffic to or from that host. In
Ethereal, this is done by opening the Capture Options dialog in one of a few ways by
clicking on the second toolbar icon from the left (the one with the little wrench on it), by
selecting Options from the Capture menu, or by using the Ctrl-K keyboard shortcut:



Using Ethereal to Investigate TCP Performance Issues Page 6 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.

This will open the Capture Option dialog window where you can enter the Capture Filter
expression you want, in addition to choosing other capture settings:





There are a few pre-configured capture filters that you can open and use as examples for
your own filters by clicking on the Capture Filter button, and you can create and save your
own in case you find that you use the same capture filters repeatedly. The filters I use
most often are host <ip address> and port <number>. You can also combine individual
specifications with the logical operators and, or, and not, or their symbolic equivalents,
&&, ||, and !, and you can group parts of your expression together with parentheses to
create complex and wonderful filter expressions. Finally, you can also use hostnames and
keywords instead of numbers, but I find theres less confusion if I specify things by number.
Using Ethereal to Investigate TCP Performance Issues Page 7 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Once you get your capture filter configured and the other options set the way you want,
click on the Start button to start capturing packets. You should do this before the traffic
to be captured is initiated, since its better to capture all the packets in the session. Unless
youve chosen to hide the Capture Info dialog window, you should get a new window similar
to the one below that gives you a continuously updated view of how many and which types
of packets have been captured:





If youve set a good capture filter that includes only the traffic to be studied, you can tell
when the connection closes because the packet counts will stop incrementing. To stop the
capture, click on the Stop button. This will close the Capture Info window and the
captured packets will then be displayed in the main Ethereal window, which might take a
few to several seconds, depending on how much data was captured and how fast your
computer is. Because Ethereal captures the entire packet by default, the size of the capture
file will be larger than the amount of data transferred. Be careful about attempting to
capture too much data, as very large capture files can take a long time to process. Ive
worked with capture files as large as 10 to 15 MBytes on older computers. Ethereal has
options for splitting and saving capture files by size or at regular time intervals, though
doing that can complicate the analysis.
Using Ethereal to Investigate TCP Performance Issues Page 8 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Once youve run a capture, you dont need to go back to the Capture Options to run it again
unless you want to change something, such as the capture filter. The three icon buttons in
the upper left of the main Ethereal window starting just below the Ethereal View menu
allow you to quickly start, stop, and restart a capture. I find these especially convenient
when used with the Capture Info window hidden, since I usually dont need to see the live
packet counts. If the icons arent obvious, you can hover the mouse over them to see a
popup description of them. Buttons that arent meaningful in Ethereals current operational
state are grayed out (see below).







Using Ethereal to Investigate TCP Performance Issues Page 9 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Saving the Capture File

Unless youre just looking for quick answers to simple questions, youll probably want to
save your captured data for later analysis. The Ethereal dialog for doing so looks different
from the standard Windows file save dialogs, but the functionality is similar. You can
choose the name of the saved file and navigate to the desired destination folder, though you
may have to click on Browse for other folders to do that (its highlighted in the screenshot
below). There is no default file extension, but I recommend using .pcap. You can choose
between a variety of different file formats in which to save the capture file, but I suggest
using the libpcap (tcpdump, Ethereal, etc.) option, since its the most commonly used
Ethereal format. Finally, you can save all or just some of the packets in the capture, which
is very useful for reducing the size of the saved files. This is most useful if youve isolated
individual sessions or otherwise filtered the display results. See the next section for more
details on doing that. The default is to save all the packets in the current capture. If you
start a new capture before saving the current one, youll be prompted about whether or not
you want to save the current capture data.



Using Ethereal to Investigate TCP Performance Issues Page 10 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Isolating an Individual TCP Connection

Even if youve configured a very specific capture filter, youll probably end up capturing all
or parts of multiple sessions. Ethereal provides a few ways to filter the packet display so
you can see just the packets related to a specific connection.

The first and most obvious way, but not the easiest way, is to just type a filter expression in
the Filter field in the upper left area of the main packet display window and click the
Apply button to the right. If your expression is valid the background of that field will turn
green. You can use the Clear button to return to the unfiltered packet display, and you
can click the Expression button to use a GUI window to add filter elements to your
expression. The details of doing that are beyond the scope of this paper, but playing with
this feature is a good way to become familiar with Ethereals filter expression syntax. The
pull-down menu at the right end of the filter expression field will display previously used
expressions, so you can quickly go back to other filter results.








A quicker and easier way to isolate an individual TCP connection is to right-click on one of
the packets in the desired connection and then select Follow TCP Stream from the menu
that pops up:



Using Ethereal to Investigate TCP Performance Issues Page 11 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
This will do two things. First, it will open a new window that shows the text of the TCP
connection color coded so you can identify the traffic flowing in each direction:





This can be useful for text-based applications, but usually you wont be interested in the
text of the session. All we really want is the display filter that it creates for us, so you can
just click the Close button to get rid of this window and return to the main packet display
window, which should now show all the packets for this TCP connection and no others:





If there was a lot of data transferred over this connection it can take quite a while for this
window to open. This is one reason to use the next method for isolating and individual
connection.
Using Ethereal to Investigate TCP Performance Issues Page 12 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Ethereal has a nifty feature that allows you to view all the conversations present in the
capture, where a conversation is all the traffic between a given source and destination
address on unique source and destination ports, depending on the protocol layer youre
investigating. You can see a new window with all the conversations by selecting the
Statistics ! Conversations menu item, or you can get just the TCP conversations if you
select Statistics ! Conversation List ! TCP (IPv4 & IPv6):



Using Ethereal to Investigate TCP Performance Issues Page 13 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
The first option opens a new window with a tab for all the different protocol layers for which
you can view conversations (see below). To see the TCP sessions, click on the TCP tab.
The second option will open a new window that only shows the TCP conversations, so its a
little quicker, but I find the first method results in a more reliable window. These new
windows often open behind the main window, in which case youll have to move it out of the
way to see the new window. The conversation window shows the source and destination
addresses and ports, the total number of bytes and packets transferred in both directions
combined, and the number of bytes and packets transferred in each direction. You can click
on any of the column headers to sort by that column, and reverse the sort order by clicking
on that column header again. This is handy for helping to identify the connection youre
interested in. Once youve identified the connection you want to isolate, right-click on it and
select Apply as Filter ! Selected ! A <-> B from the resulting pop-up menus:





Once youve done this you can close the Conversations window.
Using Ethereal to Investigate TCP Performance Issues Page 14 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
At this point the main packet display window should show just the packets for the TCP
connection youve selected. If you captured the entire TCP connection the first packet in
the display will be the SYN packet that initiates the TCP three-way handshake:








If you scroll down to the end of the capture you should also see the two FIN packets
associated with the FIN/ACK, ACK, FIN/ACK, ACK sequence of four packets that typically
closes a TCP connection:





You can save this session to a separate file by selecting File ! Save As and clicking on
the Displayed button instead of the default Captured button, and then save the file as
usual. To return to the entire list of packets just click on the Clear button by the filter
expression field.

Isolating a single TCP connection is a very useful thing to do, and all the examples in this
paper assume thats already been done. They also make heavy use of the TCP Stream
Graph feature in Ethereal, which well discuss next.
Using Ethereal to Investigate TCP Performance Issues Page 15 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
TCP Stream Graphs

Ethereal can produce interactive graphs of a few different characteristics of a TCP
connection, including a Round Trip Time graph, a Throughput graph, and two different plots
of TCP Sequence numbers over time called, appropriately enough, Time/Sequence graphs.
These graphs are the Ethereal feature I use most often when attempting to diagnose TCP
performance problems, so itll be useful to learn more about how to work with them before
we get to specific examples.

All of the graph types are accessed in essentially the same way: select a TCP packet in the
main packet display window by single-clicking on it and then choose one of the graph types
from the Statistic ! TCP Stream Graph menus. Heres an example showing the
Throughput Graph being selected (see below). I usually find it more convenient to first
isolate the TCP connection in question as described in the previous section before opening a
graph, but thats not strictly necessary.



Using Ethereal to Investigate TCP Performance Issues Page 16 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
This will produce two windows a Graph window and a Graph Control window, which always
seems to appear behind the Graph window for some reason. Since it looks and behaves the
same for all the different graph windows, lets start with the Graph Control window. If you
move the Graph Window out of the way you should see the Control window. You cant just
close the Graph Window, since closing either of the Graph or Control windows also closes
the other one. This window has five tabs across the top for accessing different aspects of
the graph window controls. Of those tabs, I use Graph Type and Cross most often, and
Zoom every once in a while. I dont recall ever using the other two tabs, Magnify and
Origin, though you should feel free to experiment with them to see if they provide
features you find useful.

The only feature in the Zoom screen Ive ever used is Zoom Lock, which allows you to
zoom just the height or width without affecting the other, which is sometimes useful. The
in and out radio buttons allow you to zoom in and out without holding down the shift
key, but youll probably never use them. The Horizontal and Vertical fields show the
current zoom level, which I also always ignore. Finally, the Horizontal step and Vertical
step settings let you specify the amount of zoom in the different directions that happens
each time you zoom in. If you wanted to zoom in smaller increments you could reduce
these values, but thats another thing Ive never used. The two check boxes below the step
settings let you put restrictions on how one step setting changes if you change the other.



Using Ethereal to Investigate TCP Performance Issues Page 17 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
The Cross screen does only one thing: it lets you turn on or off crosshairs in the graph
window so you can more accurately see the axis coordinates associated with any point in
the graph. This is sometimes handy, since the axes are only on the left and bottom of the
graph so it can be hard to see where data points on the other side of the graph line up. The
default for the crosshairs is off.



Using Ethereal to Investigate TCP Performance Issues Page 18 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
The Graph Type screen is the one I use most. It lets you quickly change the graph type on
the fly without having to close the window and start over. The new graph will be zoomed to
the same zoom level and position as the previous graph, which I seldom find useful. To
avoid this, check the box by Init on change before you select a new graph type. This will
cause the new graph to be displayed completely un-zoomed so you can see the entire
graph.

There are four graph types: Two varieties of a Time/Sequence graph, one of which is
extremely useful and will be discussed later in much greater detail, the Throughput graph,
and a Round-trip Time graph. The Round-trip Time graph is both simple and obvious. Its
just a plot of the round trip times Ethereal computes based on the timestamps and
sequence numbers in each packet. While we wont discuss the Round-trip Time graph in
detail, we will include the concept of round-trip times when we talk about TCP window size
and how it can limit throughput.



Using Ethereal to Investigate TCP Performance Issues Page 19 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Now that we understand the Graph Control window, lets look at a couple of actual Graph
Windows. Heres the Throughput graph window for the packet previously selected when we
first opened the graph window:





As usual there are a few things to keep in mind. First, it matters which packet you select
before you open any graph window, since if you pick the wrong one youll probably get a
graph with no data in it. Because data can flow in both directions of a TCP connection and
each direction can have different throughput rates, you need to make sure you select a
packet thats going in the appropriate direction. Since the data is usually only flowing in
one direction anyway, this means youll typically just need to select any packet with data in
it, which is why I add a packet length field to the main Ethereal display see the Appendix
for more details. Assuming youre looking at a data throughput issue in the first place, itll
usually be a large packet because TCP always tries to use the largest packet it can for
efficiency reasons. For example, the highlighted packet when we opened this graph has a
length of 1314 bytes. You can, however, pick any packet going in the direction you want
even if it doesnt contain any data, such as a SYN or a FIN packet.
Using Ethereal to Investigate TCP Performance Issues Page 20 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
You may remember that I described these graphs as interactive. What I mean by this is
that you can resize the window, zoom in and out, and scroll the contents of the window to
view the data at whatever level of detail you want. This is a key feature in the other
examples in this paper, so its useful to become familiar with it. On Windows platforms you
zoom in by just left-clicking on the part of the graph youd like to see closer, which can be
slow if there youre viewing a large TCP connection. Every time you click, the zoom level
increases. To zoom back out, hold the Shift key down while you left-click. Ethereal
remembers the zoom levels as you zoom in so you can zoom back out to previous views,
sort of like the way the Back button works in a web browser. To scroll the window after
youve zoomed in, just click and drag with the right mouse button. You can resize the
window just as youd resize any other window just drag any window border or corner to
the desired size. You can also Control-Left Click on a data point in the graph window to
scroll the main packet display window and select the corresponding packet, though it can be
a bit of a challenge to click on the desired data point accurately enough to invoke this
feature. When you do manage to click in the right place you can see the packet display
scroll to the newly selected packet. This isnt quite as useful in the Throughput graph, but
can be quite useful in Time/Sequence graphs where individual packet phenomena are of
more interest. On the Mac platform, the Option key must be added to the Zoom key
sequences (e.g. Option-Click to zoom in and Shift-Option-Click to zoom out), and the
Control key is optional for the Go to Packet feature. If you have a Mac but dont have a
multi-button mouse, you can scroll by holding the Command key while you click and drag.
Heres a quick summary of the Windows keyboard commands:


Zoom In Left Click
Zoom Out Shift + Left Click
Scroll Right Click + Drag
Go to Packet in Main Window Control + Left Click


The axes of the Throughput graph should be obvious. The X axis is time in seconds and the
Y axis is Throughput in Bytes per second, not bits per second! The individual data points
are the instantaneous throughput rates per packet as computed by Ethereal, so this graph is
mostly useful for identifying peak high or low throughput levels rather than the overall rate
that users typically see and care about. Because these graphs tend to look similar to this
example with lots of dots scattered all over the place, I dont use them very often. For
example, you can see that this session had many instances of throughput in the 5 to 6
MBytes/sec range, but also regular samples well below 1 MBytes/sec. While this can be
useful for seeing how the throughput rate changed during the course of the TCP connection,
it can be very difficult to deduce a good estimate of the overall average throughput from
such a graph. This is why I generally use a different method, which will be described later,
based on the TCP sequence numbers and packet times to get the overall throughput.
However, sometimes the Throughput graph is more consistent than this sample and
provides more obvious and useful information. Since its so easy to view these graphs, I
usually at least glance at them just in case they show something interesting.
Using Ethereal to Investigate TCP Performance Issues Page 21 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Now that we know how to open and control graph windows, lets look at the tcptrace version
of the Time/Sequence graph in particular. This graph is, in my opinion, one of the most
useful features in Ethereal because it can make so many aspects of TCP behavior visible,
but first its necessary to understand the different parts of the graph and how they relate to
TCP.

Heres a Time/Sequence graph zoomed in to show the different parts of the graph in greater
detail:





In the Time/Sequence graphs packets appear as vertical lines with very small bars at both
ends, like a letter I There are six packets in the sample above. The position of each
packet along the X axis indicates the time that packet was captured. The vertical length of
each packet indicates how much data was in that packet, and its position along the Y axis
indicates exactly which data bytes of the entire connection were contained in that packet.
Headers are not included in these lengths, since TCP sequence numbers apply only to data.
The lower end of each packet always corresponds to the TCP sequence number of that
packet. (Note: If your graph looks very strange, as if it contains multiple disjointed packet
traces, check to make sure you dont have any packets in the main display window with the
time reference toggled on, since that confuses Ethereal when it creates these graphs.)

Using Ethereal to Investigate TCP Performance Issues Page 22 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
The step-like line labeled ACK represents the amount of data thats been acknowledged by
the receiver. Since TCP uses a cumulative ACK algorithm, this means that at any given
time all the data with sequence numbers lower than the ACK line have been successfully
received, and any data above the ACK line may not have been received. The vertical
sections of the ACK line represent ACK packets from the receiver, with the upper end of the
step representing the new sequence number being ACKed. The packet being ACKed is
whichever previous packet has an upper end at the same sequence number value as the
new value of the ACK curve. ACK packets that dont acknowledge new data, such as
duplicate ACKs, are indicated by tiny little downward tick marks in the ACK line.

The step-like line labeled RWIN represents the value of the receivers Receive Window as
advertised in the ACK packets, so upward steps in the RWIN line will usually correspond to a
step in the ACK line, but not always, since TCP can increase the Receive Window without
acknowledging new data and vice versa. The vertical distance between the ACK line and the
RWIN line is the current size of the receivers Receive Window. Since TCP does not allow
the sender to transmit data beyond the current value of the receivers RWIN, you should
never see packets above the RWIN line, though, due to timing anomalies or out-of order
packet delivery you actually might occasionally see packets above the RWIN line. If you
capture at the sender, you should also never see any packets below the ACK line, since the
sender knows theyve already been successfully received.

Finally, if you see a packet repeated at the same vertical position but at a later time, thats
a retransmitted packet. These can sometimes appear below the ACK line depending on
where the data was captured. Usually this is seen when you capture at or near the
receiver.

To summarize, the vertical segments represent data packets sent by the transmitter, and
the ACK and RWIN lines represent the corresponding values from the ACK packets returned
by the receiver.

This completes our overview of the basic features of Ethereal well be using in the
subsequent examples in this paper.
Using Ethereal to Investigate TCP Performance Issues Page 23 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Example 1 Determining Actual Throughput of a TCP Connection

Since were looking at TCP performance issues, one of the first things youll probably want
to know is what the actual throughput was for a given TCP connection. Weve already seen
the Throughput graph and how you can use it to see instantaneous throughput rates for a
TCP connection, but users usually see the overall throughput reported by the application at
the end of the file transfer, and the Throughput graph doesnt show that. However, since
TCP sequentially numbers every byte transferred in both directions, and since Ethereal puts
a timestamp on each captured packet, its possible to compute the effective data throughput
of the connection. If youve captured the entire connection, and have isolated it as
described in the previous section, its actually pretty easy to do. To compute the total
throughput, we need to know how much data was transferred and how long it took.

First, you should go to the View menu and set the Time Display Format to Seconds
Since Beginning of Capture (see below). You can also specify the displayed time
resolution, but I usually just use the default Automatic setting, which is typically
microsecond or better. This setting shows the packet timestamp in a cumulative way
instead of as an incremental time between sequential packets, which matches the way the
graph time scale appears.



Using Ethereal to Investigate TCP Performance Issues Page 24 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
The next step is to set the time reference to be the first packet of the connection, which
should be a SYN packet, and should be the first packet in the display. To do this, right-
click on the first packet and select Time Reference ! Set Time Reference (toggle) from
the menu that pops up. If you previously saved this individual session to its own capture
file and have re-opened it you wont have to do this:





This will cause the timestamp of the first packet to change to *REF* and all subsequent
timestamps to be relative to that first packet:





Now the timestamp on the last packet of the connection is the elapsed time of the
connection. This is the how long did it take part of the throughput calculation. Remember
to toggle off any time references when youre done with them if you want to later view
graphs of the capture, since setting a time reference can confuse Ethereal when it computes
various graph values.

To answer the how much data was transferred part of the question we need to go to the
end of the connection and look at the sequence number of the last packet. Recall that TCP
sequence numbers are incrementing numbers assigned to the individual bytes transferred in
each direction. That is, each data byte is individually numbered, and the number of the first
byte of data in each packet is the sequence number for that packet. TCP is supposed to
choose those initial sequence numbers at random, and most TCP implementations today
actually do that to at least some degree. In reality, theyre not always truly random, but
they are virtually always large and awkward to deal with. There are two series of
incrementing sequence numbers in every TCP connection, one in each direction, and the
initial sequence numbers in each direction are chosen independently by the two endpoints.
Luckily for us, by default Ethereal displays all TCP sequence numbers in a relative way as if
they were always numbered starting at 1. This makes determining the total amount of user
data transferred in each direction very easy, since the value we need is simply the last
sequence number of the connection in the direction were interested in: the sequence
numbers in the FIN packets at the end of the connection. (Note: Actually, those last
sequence numbers will be off by one, since the initial SYN packet is counted as a byte even
Using Ethereal to Investigate TCP Performance Issues Page 25 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
though there is no data in that packet, and the same is true of the closing FIN packet. For
our purposes that one byte is not significant and can be safely ignored, since the amount of
data being transferred will typically be very large so one or two bytes either way wont
affect the computed throughput in any significant way.) In the case of file transfers, its
almost always the case that data is only sent in one direction, and the only packets that are
sent in the other direction are ACK packets that contain no data. So, of the two FIN
packets at the end of the connection, one will typically have a sequence number of 1 and
the other will have a large sequence number that is one larger than the amount of data
transferred. If large amounts of data were transferred in both directions then you may
have to identify the transmitter of the flow in question by IP address or source port number
instead.

In our example, the first FIN packet has the large sequence number. This will usually be
the case, since its usually the transmitter who decides to close the connection. The
sequence number is 10573163, or a little over 10.5 MBytes, and the elapsed time for this
amount of data is about 4.39 seconds:







Therefore, the average user data throughput can be computed as follows:

(10,573,163 Bytes * 8 bits/Byte) / 4.39 seconds = 19,267,723 bits/second

There are a few things to keep in mind when computing throughput this way. The first is
that this analysis ignores all the IP and TCP headers (usually 40 bytes per packet), so the
actual bandwidth consumed on the network will be a bit higher. If one assumes the data
transfer used maximally sized packets of 1500 Bytes and that 40 bytes of each packet were
used for headers, the overhead percentage is (40 / 1500) * 100 = 2.67%. This is a best-
case value, and the actual overhead may be slightly larger. However, most customers
experiencing throughput issues would be ecstatic to get within 3% of the theoretical
maximum, so header overhead can usually be ignored. On the plus side, all applications
also ignore packet header overhead, so this calculation is consistent with what the customer
is likely to see.

Another thing to remember is that most applications report throughput in Bytes/second, but
virtually all network elements specify bandwidth in bits/second. This is complicated by the
fact that computers and applications almost universally use the K and M prefixes for
Kilo and Mega, where K = 1024 and M = K
2
= 1024
2
= 1,048,576. However, network
bandwidth numbers always use 1,000 and 1,000,000 for Kilo and Mega, so a 56K
Using Ethereal to Investigate TCP Performance Issues Page 26 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
circuit really does have a bandwidth of 56,000 bits/second, not 57,344 bits/sec (56 * 1024
= 57,344). Again, this is usually not a significant enough factor to explain a customers
throughput concerns, but it can become an issue when doing computations and its good to
know about and be able to explain to customers.
Using Ethereal to Investigate TCP Performance Issues Page 27 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Example 2 Throughput Limited by Receive Window Size

In this example Ill use the Time/Sequence graph to illustrate what a TCP connection looks
like when the throughput is limited by a too-small Receive Window. Before we get to the
graphs, a brief discussion of the theory behind TCP performance is appropriate.

Recall that TCP uses a Sliding Window scheme to allow each end of a connection to send a
limited amount of data beyond what has been acknowledged by the other end. Because the
sender has to be able to transmit new data continuously without having to stop if its going
to fully utilize the available bandwidth, this implies that the sliding window has to be at least
large enough to allow the sender to transmit for as long as it takes for a packet to get to the
receiver and the acknowledgement to get back, which is the Round Trip Time (RTT)
between the endpoints. This minimum window size is called the Bandwidth-Delay Product,
and is equal to the RTT of the path in seconds times the effective bandwidth of the path,
which is the bandwidth of the slowest link of the path. There are a few things that can limit
the size of the TCP sliding window, and one of them is the Receive Window (RWIN)
advertised by the receiver to the sender. If the size of the RWIN is smaller than the
Bandwidth-Delay Product of the path, then TCP cannot fully utilize the bandwidth of the
path. Since its seldom possible to reduce the RTT between the endpoints, the only other
option is to increase the RWIN if you want to improve TCP throughput. In most operating
systems its possible to change the TCP window sizes if the defaults dont allow adequate
throughput. In recent versions of Microsoft Windows, the default RWIN is usually 17,520
Bytes. (Note: Its actually 16 KBytes rounded up to the next even multiple of the MSS
received from the far end, which is typically 1460. Since 16,384 / 1460 = 11.22, the typical
RWIN is 12 * 1460 = 17,520 Bytes.)

An additional implication of the Bandwidth-Delay Product is that if you know the RWIN size
and the RTT between two particular hosts, you can compute the maximum theoretical
throughput of a TCP connection between them. Since the equation for the limiting case of
how big the sliding window needs to be is RWIN = Bandwidth * RTT, simple algebra yields
this equation:

Bandwidth = RWIN / RTT

For example, if you know a pair of Windows PCs are using the default RWIN of 17,520
Bytes and the RTT between them is 35 milliseconds, the maximum usable bandwidth
between the hosts by a single TCP connection is 17,520 Bytes / 0.035 seconds = 500,571
Bytes/second = 4,004,568 bits/second. This equation gives a theoretical upper bound that
ignores packet headers and other performance impacting factors, so the real-world
maximum throughput will always be a few percent less. However, its still a useful
computation to be able to make, since you know TCP cannot achieve higher throughput
than the result. So, if a customer has a fractional DS3 circuit provisioned at 20 Mbits/sec
and sees typical round trip times of around 35 msec, any individual TCP file transfers
between Windows PCs will be limited to about 4 Mbits/sec unless they increase the TCP
Window size of hosts. If they are actually seeing throughput close to that, then the RWIN is
likely the limiting factor. However, if they see throughput substantially less than 4
Mbits/sec then something else is the cause and increasing the TCP Window size will not
improve throughput.

Now that we understand RWIN and how it can limit TCP throughput, lets see what it looks
like in Ethereal.
Using Ethereal to Investigate TCP Performance Issues Page 28 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
The useful graph type for this analysis is the Time-Sequence Graph (tcptrace). There are
actually two styles of Time/Sequence graph in Ethereal, the Stevens style and the
tcptrace style. The Stevens style is named after W. Richard Stevens, who used this type
of graph to illustrate various aspects of TCP behavior in his most excellent book, TCP/IP
Illustrated, Volume 1: The Protocols. The tcptrace style of Time/Sequence graph is named
after the free utility of the same name that can be used to post-process packet capture files
and produce statistics and graphs of several different characteristics of a TCP connection,
one of which is a Time/Sequence graph. In Ethereal these two graph styles are very
similar, but since the tcptrace style includes everything the Stevens style does along with
additional information, its the only one I use, and from now on thats the one I mean when
I refer to Time/Sequence graphs. To open it, select a packet carrying data in the direction
of the file transfer, then select the Statistics ! TCP Stream Graph ! Time-Sequence Graph
(tcptrace) menu item:



Using Ethereal to Investigate TCP Performance Issues Page 29 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
For this example I captured a TCP connection to a Windows 2000 PC configured with an
RWIN of 8760, which is the default for older versions of Microsoft Windows. The test
transferred 250,000 Bytes of data and was run through an intermediate FreeBSD computer
configured to add an artificial delay of about 30 msec in each direction for an RTT of at least
60 msec. The packets were captured on the sending host, since thats where the most
useful information is. Later well see what a capture at the receiving host looks like. Heres
the Time/Sequence graph for this connection:





As with all the other graphs, the X axis is time in seconds. With the Time/Sequence graphs
the Y axis is the normalized TCP Sequence Numbers, which is also the number of Bytes
transferred. Because this is essentially a graph of data over time, the general slope of the
graph can give you a visual indication of how consistent the throughput was. In this case,
because the graph is quite straight the data was transferred at a relatively constant rate. In
general, however, steeper parts of a graph indicate higher throughput, and flatter parts of a
graph are areas of lower throughput. Well see examples of that later.
Using Ethereal to Investigate TCP Performance Issues Page 30 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Now that we know what the various parts of the tcptrace graph show, lets look at the
current example in a bit more detail:





The single packet at A is the first data packet of the connection, but not the very first
packet, which is always a SYN packet and virtually never carries data. Note also that the
RWIN line is at 8760 bytes, which is the initial value advertised by the receiver even before
any data has been sent. TCP connections start in a mode called slow start where the
transmitter first sends one packet and waits for an ACK from the receiver, and then
increases the number of packets it can send without waiting by one packet for each
acknowledged packet. This typically results in an exponential increase in the number of
packets the transmitter can send, though in this case with such a small RWIN size thats not
very evident. TCP also has a feature called Delayed ACK which causes TCP to wait a short
time, usually 200 msec, before sending an acknowledgement in hopes of being able to
include the ACK with data going the other direction. This means that its not unusual for the
ACK of the first data packet to be delayed, which can be seen in the graph. This can cause
performance issues with some Microsoft Windows protocols so theyve modified their version
of slow start so that it starts with two packets instead of one. Since TCP usually sends an
ACK for every other received packet, this eliminates the initial Delayed ACK and increases
efficiency.

Using Ethereal to Investigate TCP Performance Issues Page 31 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
By point B the transmitter is sending three packets at a time, and you can see that the
gap between the upper end of each burst of packets and the RWIN line is narrowing. At
point C the upper end of the three packets in the burst is right at the RWIN line. This
means the transmitter has filled the receivers RWIN and must stop transmitting until an
ACK is received. If we zoom out a bit to see more of the graph we can see very easily that
the throughput is being limited by the receivers RWIN because all the packet transmissions
stop at the RWIN line:





This is also visible in the un-zoomed version of the graph, so its usually quite easy to
identify when TCP throughput is being limited by the receivers RWIN. Ethereal will
sometimes indicate this in the main packet display by putting the text [TCP Window Full]
in the Info column, so that can be another indication that a TCP connection is limited by a
RWIN thats too small:



Using Ethereal to Investigate TCP Performance Issues Page 32 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Its very important to remember that this technique of identifying this issue only works if
the packets were captured on or very near the transmitter. If the session is captured at the
receiver the RWIN throughput limitation is virtually impossible to identify. Heres what the
Time/Sequence graph looks like when captured at the receiver. This is the exact same TCP
connection as before, slightly zoomed to show more detail:





Notice how all the packets now hug the ACK line instead of the RWIN line. This is because
this capture was taken at the receiving host, so there is virtually no delay between when a
packet is received and when the corresponding ACK for that packet is sent. Were
effectively on the wrong side of the round-trip time, so we cant see the interval between
the transmission time of a data packet we receive and the receipt of the corresponding ACK
we send, which is the round-trip time TCP uses as part of its flow control scheme. If this is
the only capture file available, the best alternative is to just do the math to determine if the
realized throughput is near what would be predicted from the known value of RWIN and the
average round-trip time between the endpoints using the equation discussed previously.

Finally, notice how every other packet aligns with a step in the ACK line, which fits the
typical TCP behavior of only ACKing every other received packet.
Using Ethereal to Investigate TCP Performance Issues Page 33 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Example 3 Throughput Limited by Sender Window Size

In addition to the receivers RWIN, there are other parameters that can limit TCP
throughput, including the Send Window (SWIN), which is static and does not change during
a connection, and the Congestion Window (CWIN), which is dynamic and does change in
complex ways. Both of these parameters are maintained by the transmitter. While we cant
see either of these values in actual packets, the fact that they are coming into play tends to
be easily recognizable in the Time/Sequence graph:





This is another nuttcp connection through the same simulation setup as the previous
example, but in this case I reconfigured the receiver to use an RWIN of 32,768 Bytes and
the transmitter to use a Send Window (SWIN) of 8192 Bytes everything else was
unchanged. In the graph the RWIN line is farther away from the ACK line, which reflects
that the receivers RWIN was, indeed, larger than before. However, the data packets never
approach the RWIN line. Instead, they stay what appears to be a constant distance short of
it. When there is an obvious gap like this between the RWIN line and the packets it means
the receivers RWIN was absolutely not a factor in limiting the throughput of the connection,
and increasing it will not make a difference. Something else is limiting throughput: the
transmitters SWIN.
Using Ethereal to Investigate TCP Performance Issues Page 34 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
The transmitters SWIN is analogous to the receivers RWIN, and can limit throughput in
exactly the same way RWIN can. Unfortunately, because the recipient doesnt need to
know how big the transmitters SWIN is, its never advertised, and, therefore, we cant
explicitly see what it is by looking at a packet capture. On the plus side, most operating
systems set both RWIN and SWIN to the same value, so if you look at the RWIN advertised
by the transmitter, that will usually be what its SWIN is also.

Lets look a bit closer at the graph and see what we can learn. Heres a zoomed view of the
above graph, along with some annotations:





I used the control-click graph feature to go to the corresponding packets in the main
Ethereal window to determine the values on the graph from the ACK and Seq values in
the packets, but you can also zoom way in on the graph until the Y-axis resolution shows
individual Sequence numbers and find the data values that way.

So what do these numbers mean? Remember that I set the SWIN to 8192, so why did the
transmitter stop after sending 7300 Bytes? It turns out that TCP really likes to send full-
sized segments if at all possible, where the segment size is the MSS advertised by the
receiver. In this case the MSS is 1460 (1460 = 1500 Byte MTU 40 Bytes of standard TCP
and IP headers), and 7300 is an even multiple of 1460. Rather than send a somewhat
smaller packet and have the maximum possible 8192 Bytes of outstanding data, TCP, for
Using Ethereal to Investigate TCP Performance Issues Page 35 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
some reason, decided to just stop at 7300 Bytes. This may be because the data stream
from the application had already been segmented into 1460 Bytes packets by the TCP stack,
or possibly an optimization by the operating system similar to the way Microsoft rounds the
RWIN to an even multiple of the MSS, but its hard to say for sure. The resulting average
throughput would then be limited by this effective SWIN, with the theoretical limit given
by 7300 * 8 / RTT. This is the formula discussed previously, but with SWIN in place of
RWIN.

One thing to keep in mind regarding the SWIN, and to some extent, RWIN, is that the
application programmer has some control over it and can override the default system
values. So, even if you conclude that the throughput of a particular TCP connection appears
to be limited by the transmitters SWIN, changing the system settings for the SWIN may not
fix the problem. In this case the only solution may be to update or replace the application.

In summary, and this is the key point of this example, when you see a graph like the first
one in this example where the data packets never approach the RWIN line but instead stay
what appears to be a relatively constant distance from it, that usually indicates the
throughput was limited by the senders SWIN rather than the recipients RWIN. A possible
fix for this is to increase the SWIN on the sending host rather than the RWIN on the
receiving host, or both the RWIN and SWIN since theyre usually the same. Because the
application programmer can override the system settings, however, this may not always fix
the problem. In any case, increasing the receivers RWIN will absolutely not improve
throughput.
Using Ethereal to Investigate TCP Performance Issues Page 36 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Example 4 Packet Loss

The other common cause of slow TCP performance is packet loss, but the two are not as
directly related as you might think. TCP assumes that packet loss is a result of congestion,
which is virtually always the case, so it reacts by slowing down. In reality, the event that
directly triggers TCPs reaction to congestion is the retransmission of a packet, not packet
loss. Retransmission, in turn, is caused by either the expiration of the retransmission timer
for the original packet, or else the receipt of three Duplicate ACKs (Dup ACKs) in addition
to the expected ACK, for a total of four identical ACKs. (Note: Recent versions of Microsoft
Windows use a limit of two duplicate ACKs instead of three, so retransmissions can happen
sooner in that case.) If the retransmission timer expires it is very likely the original packet
was actually lost, since its quite rare for a packet to be delayed for such a long time and
still make it to the destination. Dup ACKs are also usually caused by dropped packets
because TCP is required to ACK each packet that arrives out of order, which is what each
packet after the dropped packet looks like to the recipient. Once three packets after the
dropped packet are received, the transmitter will get three Dup ACKs, assume the
corresponding packet was dropped, and retransmit it. A large number of out of order
packets can also trigger packet retransmission even though no packets are actually
dropped, but this is less common. Because non-dropped packets are generally not delayed
much, retransmission is most often triggered by Dup ACKs rather than timer expiration, at
least with file transfers where there is a constant flow of packets. Heres a Time/Sequence
graph for a TCP connection experiencing packet loss:



Using Ethereal to Investigate TCP Performance Issues Page 37 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.

This was an nuttcp test connection that transferred 2,000,000 Bytes with a simulated
packet loss rate of 0.5 percent. The most obvious thing about this graph is how uneven it
looks its not nearly as straight as the other examples weve looked at so far, and is
actually more typical of real-world connections. Lets look at this curve in more detail,
starting at the beginning of the connection:





For the first second or so we see the classic TCP Slow Start behavior where the number of
unacknowledged packets sent at any point roughly doubles until the RWIN is filled and the
transmitter has to stop and wait for an ACK. Because the packet transmission rate is
increasing exponentially, Slow Start generally appears as a sharply upward curving shape in
the Time/Sequence graph. In this case, however, just as the RWIN limit is about to be
reached we can also see a retransmitted packet, which is circled in the above graph.
Using Ethereal to Investigate TCP Performance Issues Page 38 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Heres a much closer view of that packet, on the right, along with the original packet that
was dropped on the left:





We know this is a retransmitted packet because it appears at the same vertical position
along the Y-Axis as a previous packet, and itll almost certainly be labeled as a
retransmission in the main Ethereal packet display. Even from this view, however, its not
completely clear why this packet was retransmitted, so lets look even closer at the
retransmitted packet:





This is an extremely close view of the ACK line shortly before the retransmitted packet. The
vertical section of the ACK line at the left corner of that line is a normal ACK packet, and the
tiny downward tic marks circled in red are Dup ACKs of that normal ACK. You can tell from
the time scale that these packets arrived in a very short time interval, which is why there
are more than three Dup ACKs the transmitting host just didnt have time to process the
three Dup ACKs and retransmit the lost packet before a fifth ACK arrived. Because TCP has
to ACK every out of order packet, and since the packets after the dropped packet arrived at
the recipient in a very short time interval, the corresponding ACKs for those packets were
also sent in a very short interval. So, this retransmitted packet was triggered by Dup
ACKs, which were the result of a dropped packet. Note that its generally easier to see Dup
ACKs in the Ethereal packet display window, since they are usually labeled with TCP Dup
ACK followed by the packet number of the original ACK and number indicating how many
duplicates the current packet makes. For example, itll say TCP Dup ACK 104#3 if this is
the 3
rd
duplicate of the ACK in packet 104. If you enable the packet coloring rules you can
Using Ethereal to Investigate TCP Performance Issues Page 39 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
make it even easier, since you can make the Dup ACKs stand out from the other packets
with a different color. Heres an example for the part of the sample trace weve been
looking at. The Dup ACKs have a pink background, and the retransmitted packet has a
black background. You can also see that there were several additional Dup ACKs that
arrived after the retransmitted packet was sent. This is because the RWIN was almost full
when the packet was dropped and the dropped packet was one of the earlier ones. So,
there were quite a few subsequent packets that were already on the way and were received
at the far end, and each one of them was considered to be an out of order packet and had
to be individually ACKd:





There is yet another way to quickly see this type of information the Expert Info option in
the Analyze menu:



Using Ethereal to Investigate TCP Performance Issues Page 40 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
This will bring up a window containing a nice list of all the various Expert analyses
Ethereal does on the captured data, most of which also appear as comments in the Info
column in the main packet display. Heres what the Expert Info looks like for our captured
data:





The lines are color-coded by severity, and you can filter the individual messages to see just
those at and above any particular severity with the Severity filter pull-down menu in the
upper right part of the window. Clicking on a message in the Expert Info window selects the
corresponding packet in the main display window, so this is a quick way to get to individual
problem areas in a large capture, since scrolling through all those packets in the main
window can take a while. Notice all the TCP Duplicate ACK messages with incrementing
numbers, along with the Fast retransmission message. This is an easy way to see what
we just covered, and can save some time if you already suspect retransmissions may be an
issue.
Using Ethereal to Investigate TCP Performance Issues Page 41 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Another way to identify that retransmissions are happening is to look at the Time/Sequence
graph, since the changes in throughput that retransmissions cause are often, but not
always, relatively easy to see. Heres the same graph shown at the beginning of this
section, but with the most noticeable retransmissions circled:





The key indicator of retransmission is an abrupt change in the slope of the graph when
theres a retransmission the sender reduces its rate of transmission in order to avoid
congestion, which appears as an abrupt change in the slope of the graph to a less steep
angle. As TCP recovers and increases the transmit rate, the slope gradually increases,
which appears as a gentle upward curve. If the retransmissions are not too frequent the
overall result is a scalloped look to the graph, which is clearly visible in this example. In
more severe cases of packet loss the phenomenon is less obvious and one of the previously
discussed methods of identifying retransmissions is necessary.
Using Ethereal to Investigate TCP Performance Issues Page 42 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
At this point we know how to spot retransmissions, and we can identify if the cause of the
retransmission was too many Dup ACKs. Now lets look at the performance impact of these
retransmissions. Here are six Time/Sequence graphs of the same data transfer test but
with different levels of packet loss:





The most visible difference between these graphs is the increasing levels of jaggedness as
the packet loss rate increases. However, because Ethereal scales the graphs to fit the
window, unless you look at the time scale its not obvious just how much slower the higher
loss transfers are compared to the lower loss transfers. To make this comparison easier, on
the next page is a larger composite graph that shows all the individual curves using the
same scale (I used Photoshop to create this, as theres no option for doing this in Ethereal).
Using Ethereal to Investigate TCP Performance Issues Page 43 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.




Its now very clear that relatively small levels of packet loss can have quite a dramatic effect
on throughput. Even just 1.0% packet loss resulted in throughput that was about one third
that of the zero-loss rate. The point of this graph is not to indicate what performance you
should expect if you know what your packet loss rate is. Thats not a valid generalization to
make, since the details of the effect of packet loss on performance are quite complex. The
point is that even small amounts of packet loss can have a large impact on throughput.

However, if throughput is not limited by RWIN or SWIN, then there will always be at least
some packet loss, since TCP will continuously speed up until it experiences congestion
resulting in packet loss. This is the normal mechanism TCP uses to detect available
bandwidth and is not inherently bad. Its only when packet loss is excessive and causes
unnecessary slowness that it becomes a problem. Normal amounts of packet loss tend to
appear in the Time/Sequence graphs as more scalloped and less jagged, but even thats not
always true.

Finally, its important to remember that its not actually the loss of a packet that directly
causes TCP to reduce its transmission rate its the retransmission of a packet that triggers
the rate reduction. While dropping a packet will always result in a retransmission, its also
possible for packets to be retransmitted even when no packets are lost, and this will also
cause TCP to slow down. The next example will go into this in a more detail.
Using Ethereal to Investigate TCP Performance Issues Page 44 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Example 5 Out of Order Packets

With the recent popularity of Qwest NxT1 services and Multilink PPP (MLPPP) the question of
out of order packets and their impact on performance has become much more common.
The short answer is that TCP doesnt care about out of order packets and they have no
impact on performance at all, but thats only true up to a point. If packets are severely out
of order TCP can erroneously interpret them as a sign of dropped packets and slow down as
if packets were actually lost. Heres a Time/Sequence graph from a customer issue I
recently worked on:





This was an FTP data transfer that was experiencing slow throughput, and this data was
captured at the transmitting host. Ive numbered the individual packets with the order in
which they were sent, except for the retransmitted packet, which is labeled 1R, and circled
the Dup ACKs that triggered the retransmission. From this graph its not possible to tell
with certainty why Packet 1 was retransmitted all we can do is assume that it was
dropped, since we got two Dup ACKs for Packets 2 and 3. Note that this was a Microsoft
Windows server so the Dup ACK Threshold was two instead of three. Fortunately, this same
connection was also captured at the receiving host so we can see more details by looking
there.
Using Ethereal to Investigate TCP Performance Issues Page 45 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Heres a Time/Sequence graph for the same part of the connection, but from the data
captured on the receiving host instead:





The individual packets have the same labels as on the previous graph, and the Dup ACKs
are circled as before. With this additional information we can now see more accurately what
actually happened. First, notice that Packet 1 was not dropped at all. In fact, it arrived
only slightly after Packets 2 and 3, which happened to arrive in reverse order but thats not
a factor in this particular issue. We already know the two Dup ACKs in the previous graph
triggered the retransmission, so packet delay was not the cause. Those same Dup ACKs
are present in this graph, too, and now we can see that they were caused by the receipt of
Packet 3 and then Packet 2 because they appear at virtually the same time as those
corresponding packets, though this is only true because this capture was taken on the
receiving host. Remember that TCP is required to immediately send an ACK for every
packet that arrives which is not the next expected packet, which is the definition of an out
of order packet. From the ACK line in the graph we can see that the next expected packet
was Packet 1. When Packets 3 and 2 arrive they each trigger an ACK, each of which is a
duplicate of the original ACK represented by the ACK line. Because the sender was a
Microsoft Windows machine, two Dup ACKs are enough to cause a retransmission, which
also causes TCP to slow down. There were about 40 similar instances of retransmissions
caused by out of order packets in this TCP connection out of a total of about 1100 packets,
which is a retransmission rate of just under 4 percent. There were no dropped packets
Using Ethereal to Investigate TCP Performance Issues Page 46 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
whatsoever. Weve already seen what a dramatic impact such a seemingly small rate of
retransmissions can have on TCP performance, so it hardly surprising that the customer was
not happy with the performance they were seeing. The details of the cause of the out of
order packets are beyond the scope of this paper, but the gist of it is they were the result of
an unfortunate combination of features and options with Multilink PPP across the NxT1
bundle. One possible workaround in a situation like this is to modify the Windows Registry
on the transmitting host so it uses a larger Dup ACK threshold (3 instead of 2, for example),
thereby making Windows less sensitive to them and reducing the number of
retransmissions.
Using Ethereal to Investigate TCP Performance Issues Page 47 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Conclusion

Ethereal is an exceptionally powerful tool for investigating various network phenomena.
While it has many features that make it applicable to a wide variety of network applications,
such as VoIP, Ive attempted to describe the features Ive found to be most useful when
investigating TCP performance issues. I described how to create and open packet capture
files and isolate individual connections to make further analysis simpler. I paid particular
attention to what I think is one of Ethereals strongest features, the TCP Time/Sequence
graphs, which provide a surprising amount of information. When coupled with a basic
understanding of the fundamentals of TCP, these graphs are often all you need to identify
the specific cause(s) of a variety of TCP performance problems. I tried to go into some
detail about a few of the more common performance issues Ive seen over the years.
However, since there are always new problems and variations of old problems, its useful to
learn as much as you can about the technology involved and the tools and resources
available for working with and testing that technology. Ethereal is certainly one of the more
useful of those tools, and its effective use can be a tremendous aid to the network
troubleshooter.
Using Ethereal to Investigate TCP Performance Issues Page 48 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Test Topology

This is a simple diagram of the network topology I used to generate most of the packet
captures I used in this paper:





The system in the center was running FreeBSD 6.0-Release with the ipfw and dummynet
features configured to allow for arbitrary amounts of bandwidth, delay, and packet loss.
This system, which served as the WAN simulator, routed between its two Ethernet
interfaces, which were connected via cross-over cables to the other two test systems. While
the ipfw/dummynet software provides a lot of flexibility for simulating various aspects of
network performance, due to feature and system clock resolution limits the one thing it
cannot easily do is accurately simulate the out of order packet phenomena Ive seen occur
across MLPPP links. This is why I used a packet capture from an actual customer for that
example.

The test system on the left usually served as the nuttcp server and also had Ethereal
available to capture the destination end of test traffic. The system on the right was usually
the nuttcp client and was the source of the test traffic. It was also running Ethereal, and
is where most of the test sessions were captured. This was especially convenient since this
is also the system where I wrote this paper, which reduced the need to transfer capture files
between systems. I specifically wanted one of the two test systems to be a non-Windows
platform, since I wanted to be able to investigate differences in the TCP implementations.
Since Mac OS X is based on FreeBSD, that provided a nice alternative OS. A side benefit is
that Unix-based systems generally have built-in commands for adjusting different TCP
parameters, which made creating the test scenarios a bit easier.

All of these systems are rather old (at least 6 years old, in fact) and not particularly
powerful, but that turned out to not be an issue since I was not attempting to maximize
throughput across a high bandwidth link. This is a good reminder of just how efficient
modern TCP stacks are.
Using Ethereal to Investigate TCP Performance Issues Page 49 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
References and Resources

Heres a list of the various books, web links, and software tools I used or consulted while
writing this paper, or that I think would be generally useful to someone interested in
learning more about TCP performance analysis:


Stevens, W. Richard. TCP/IP Illustrated, Volume 1 The Protocols. Reading, MA: Addison-
Wesley Publishing Company, 1994. ISBN 0-201-63346-9. This is absolutely the finest book
on the internal workings of TCP/IP Ive ever seen. It covers TCP in great detail but is still
quite readable. I highly recommend it.

Ethereal. http://ethereal.com/ This is where you can download the latest version of
Ethereal, as well as learn more about it. This free, open source software is a must-have for
investigating a wide variety of networking issues. The more recent versions include the
winpcap software needed to actually capture network traffic.

ttcp and variants. http://sd.wareonearth.com/~phil/net/ttcp/ TTCP is a TCP throughput
testing utility, and there are many different versions of it available. This page lists most of
the better ones. I like nuttcp, but there are others that are also useful. I used nuttcp on
two systems connected through a third system serving as a WAN simulator to generate the
capture files used in most of the examples in this paper.

tcptrace and xplot. http://www.tcptrace.com/ I didnt talk about tcptrace in this paper,
but it can be quite useful for extracting even more information from a TCP capture file when
used with the xplot package. Theres a page with links to other companion programs on
this site as well, including xplot, and documentation on how to use the many options.

TCP Optimizer. http://www.speedguide.net/downloads.php This is a handy little free
utility from SpeedGuide.net for setting various TCP parameters on Microsoft Windows
systems, including things like TCP Window sizes, Dup ACK threshold, MTU, etc. Its the
nicest such utility Ive found so far, and is what I used on the Windows test system to help
create the test conditions I needed for the various examples.

ipfw and dummynet. http://www.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8
Ipfw is one of the firewall packages supported in FreeBSD, and it now includes the
dummynet traffic shaping feature that allows the configuration of arbitrary bandwidth,
delay, and packet loss settings. This is the software I used on an old laptop to simulate
various types of WAN links in order to capture more realistic sessions for most of the
examples in this paper.

tcpdump. http://www.ethereal.com/docs/man-pages/tcpdump.8.html The tcpdump
command is available in virtually all versions of Unix and Linux. Since Ethereal can open
tcpdump capture files, it can be a useful alternative for capturing traffic if Ethereal doesnt
happen to be installed.
Using Ethereal to Investigate TCP Performance Issues Page 50 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Appendix A: Configuring Ethereal

The Ethereal settings used in the examples are largely the defaults, though I changed a few
of them for the sake of convenience and they are illustrated below. None of these settings
are critical to understanding or following the examples in this paper. The pages that follow
show all the Preferences screens for the Ethereal setup I used in this paper along with a
brief description of the changes from the defaults. Its important to remember to use the
Save button in order for any changes made to apply to subsequent Ethereal sessions.

Using Ethereal to Investigate TCP Performance Issues Page 51 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Ethereal Preferences: Layout

Ethereal defaults to the first layout shown below, but Ive found the second layout to be
more convenient, as it allows for a larger Packet List in Pane 1. A reasonable font
selection still allows for full visibility of the other two panes.



Using Ethereal to Investigate TCP Performance Issues Page 52 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
This is a sample of what the above layout looks like. Notice that having the lower two
panels side-by-side allows you to see more entries in the Packet List and still see the full
width of the Packet Details and Packet Bytes panels. The price for this is that it requires
a smaller font than some people may prefer.



Using Ethereal to Investigate TCP Performance Issues Page 53 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Ethereal Preferences: Columns

The default Packet List view in Ethereal does not include columns for the source and
destination port numbers, nor for the packet length in bytes, but I really like seeing them.
To add them, click on the New button to the left of the list of columns, type an appropriate
name for the column being added in the Title filed below the column list, and then select
the corresponding field from the Format pull-down list. The new column will appear at the
end of the list, but the Up and Down buttons can be used to move it to the desired
location. Note that changes to the Columns preferences do not take effect until Ethereal
is restarted. Also, there is no way to save the width of the individual columns, so you have
to adjust them in the main Ethereal window by dragging the column header borders to the
desired width every time you start Ethereal if you dont like the defaults.



Using Ethereal to Investigate TCP Performance Issues Page 54 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Ethereal Preferences: Fonts

I prefer a fixed-width sans serif font in Ethereal, even though variable-width fonts allow for
more text on individual lines. The available fonts will vary by platform, but there are
usually several good options. There are also text zoom controls in the main Ethereal
window, so its easy to change the font size dynamically if desired.



Using Ethereal to Investigate TCP Performance Issues Page 55 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Ethereal Preferences: Capture

I find it convenient to specify the default interface Ethereal will use for capturing packets, as
well as the behavior of the Capture Info dialog box. The list of available interfaces will
vary by platform, and will include any virtual interfaces that might exist on the host, such as
VPN interfaces (see the Special Note later in this paper for more details on Ethereal and
the Cisco VPN Client). The Edit button can be used to remove any interfaces from the list
of available interfaces when starting a new capture. In order to capture packets not
sourced or destined to the host running Ethereal, promiscuous mode must be enabled. I
also usually prefer to see the Packet List updated in real time, but this can cause
increased CPU load on the host and sometimes has to be disabled if Ethereal cant keep up.
None of these settings affect the packet analysis or decodes at all, but they can make the
process of performing repeated captures easier, since all you usually have to do is click on a
button or two.



Using Ethereal to Investigate TCP Performance Issues Page 56 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Ethereal Preferences: Name Resolution

I prefer to see the actual numeric values of the various header fields in the packet list view,
since it eliminates name lookup delays, thereby increasing performance, and it helps
prevent long host names from overflowing the column widths. However, if you prefer to see
names for hosts and a few other things, you can specify that here.



Using Ethereal to Investigate TCP Performance Issues Page 57 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Ethereal Preferences: Packet Coloring Rules

Ethereal also includes a default set of Packet Coloring Rules. For this paper most of those
rules have been deleted or disabled in order to simplify the appearance of the screen
captures. Theres also a button in the Ethereal toolbar to toggle all packet coloring on and
off. The syntax for creating the rules can be quite complex, but it basically involves
selecting the desired packet header and/or Ethereal analysis field(s) from a list and
combining them with the keywords and and or, and optionally including parentheses to
create complex rules. The foreground text color and background color can be chosen so
that any packets that match the rule will stand out in the packet list highlighted with the
selected colors. This can be quite useful when looking for a particular type of packet in a
large capture. Its also possible to import and export saved coloring rules so that different
rule sets and be used for different applications. The details of creating and using these
rules are beyond the scope of this paper, but a good way to get started with them is to
experiment with the Expression feature in the main packet list window. As a reference,
the coloring rules used in this paper are shown below.

This setting is located in the View ! Coloring Rules menu rather than the Edit !
Preferences menu.



Using Ethereal to Investigate TCP Performance Issues Page 58 of 58
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Special Note: Ethereal and the Cisco VPN Client on Windows

The Cisco VPN Client installs a virtual interface on Microsoft Windows systems. If you use
Ethereal to capture packets on such a system, you may get duplicates of all the packets
sent by the local system (i.e. the PC running Ethereal). The work around for this is to
temporarily disable the virtual interface by opening the properties for the Ethernet interface
youre using and un-checking the box for the Deterministic Network Enhancer (see below).
Remember to re-enable the Deterministic Network Enhancer when youre done, since the
Cisco VPN Client software requires it.

Você também pode gostar