Qwest customers sometimes experience data throughput rates with individual TCP connections that are less than they expect. There is a freely available open source software utility called Ethereal! that can help. This paper provides an introduction to Ethereal and describes some of the ways it can be used to investigate TCP performance issues.
Qwest customers sometimes experience data throughput rates with individual TCP connections that are less than they expect. There is a freely available open source software utility called Ethereal! that can help. This paper provides an introduction to Ethereal and describes some of the ways it can be used to investigate TCP performance issues.
Qwest customers sometimes experience data throughput rates with individual TCP connections that are less than they expect. There is a freely available open source software utility called Ethereal! that can help. This paper provides an introduction to Ethereal and describes some of the ways it can be used to investigate TCP performance issues.
Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Using Ethereal to Investigate TCP
Performance Issues
Dana J. Dawson Sr. Staff Engineer, CCIE #1937 Advanced Technical Support CPE Qwest Services Corporation Dana.Dawson@qwest.com
July 7, 2006
Summary
Qwest customers sometimes experience data throughput rates with individual TCP connections that are less than they expect. While there are many possible causes for this, such as packet loss and TCP window size issues, determining the exact cause of the poor performance can be difficult. Fortunately, there is a freely available open source software utility called Ethereal!
that can help. This paper provides an introduction to Ethereal and describes some of the ways it can be used to investigate TCP performance issues. Various examples are presented along with descriptions of which Ethereal features were used in identifying or eliminating possible causes of the observed behavior. Annotated images of the Ethereal displays are used as necessary to illustrate these features.
Ethereal is a registered trademark of Ethereal, Inc. Using Ethereal to Investigate TCP Performance Issues Page 2 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Contents CONTENTS ................................................................................................................2 ABOUT ETHEREAL ......................................................................................................3 ABOUT TCP ...............................................................................................................3 ABOUT THE EXAMPLES................................................................................................4 PACKET CAPTURING BASICS .......................................................................................5 SAVING THE CAPTURE FILE.........................................................................................9 ISOLATING AN INDIVIDUAL TCP CONNECTION ............................................................ 10 TCP STREAM GRAPHS ............................................................................................... 15 EXAMPLE 1 DETERMINING ACTUAL THROUGHPUT OF A TCP CONNECTION ................... 23 EXAMPLE 2 THROUGHPUT LIMITED BY RECEIVE WINDOW SIZE .................................. 27 EXAMPLE 3 THROUGHPUT LIMITED BY SENDER WINDOW SIZE................................... 33 EXAMPLE 4 PACKET LOSS....................................................................................... 36 EXAMPLE 5 OUT OF ORDER PACKETS....................................................................... 44 CONCLUSION .......................................................................................................... 47 TEST TOPOLOGY ...................................................................................................... 48 REFERENCES AND RESOURCES.................................................................................. 49 APPENDIX A: CONFIGURING ETHEREAL ..................................................................... 50 ETHEREAL PREFERENCES: LAYOUT............................................................................ 51 ETHEREAL PREFERENCES: COLUMNS......................................................................... 53 ETHEREAL PREFERENCES: FONTS ............................................................................. 54 ETHEREAL PREFERENCES: CAPTURE.......................................................................... 55 ETHEREAL PREFERENCES: NAME RESOLUTION ........................................................... 56 ETHEREAL PREFERENCES: PACKET COLORING RULES.................................................. 57 SPECIAL NOTE: ETHEREAL AND THE CISCO VPN CLIENT ON WINDOWS......................... 58 Using Ethereal to Investigate TCP Performance Issues Page 3 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. About Ethereal
Ethereal is an open source network protocol analyzer program available at no cost under the GNU General Public License from http://www.ethereal.com. It runs on most operating systems, and is capable of both capturing packets and decoding virtually all network protocols. In addition to the packet capture display features typically available in network analyzers, such as filtering and colorizing user-specified packets, Ethereal can also display interactive plots of TCP sessions in a variety of ways, including Time/Sequence, Throughput, and Round-Trip Time graphs. These TCP plotting features will be the primary focus of this paper. When I started writing this paper the current version of Ethereal was 0.10.14, which is the version I used in all the examples, but Version 0.99.0 has recently been released. I havent noticed any significant differences between the two versions, so this discussion should apply to the new version and other recent versions of Ethereal as well.
This paper is not intended to be an exhaustive tutorial on Ethereal, but will instead focus on and illustrate just the features necessary for you to perform the analyses described in the examples. I assume you will know or be able to easily learn how to install Ethereal, as well as navigate the Ethereal menus and GUI interface. Ethereal has many options and features, some of which can be complex and counter-intuitive. Fortunately for us, the features being discussed here are relatively straightforward. The visual appearance of Ethereal is quite uniform across most platforms, however it is possible that some of the screen images in this paper may be slightly different from what you may see on your computer screen. The screen images in this paper were taken with a default installation of Ethereal on both Windows 2000 SP4 and Mac OS X 10.4.6 and should be quite similar to how any recent version of Ethereal looks on other platforms. In any event, the important concepts of the examples should be clear regardless of the appearance of your version of Ethereal.
About TCP
While this paper is also not intended to be a complete tutorial on TCP, it is likely that the discussions in the examples will provide a more thorough understanding of some of the specifics of TCP behavior, especially in response to congestion and packet loss. Even so, you should be familiar with the basics of TCP and its concepts, including the three-way handshake used for connection establishment, sequence numbers, and the TCP Sliding Window scheme. Ill describe any complex or obscure concepts that are important to the situation as necessary.
Its worth noting that there are different implementations of TCP in general use, and their default behaviors can vary. Different platforms also use different default values for some parameters, and include different sets of options. For example, Windows XP enables the Selective ACK option by default, whereas other platforms dont support it all. While Ill try to make any conclusions in this paper as general as possible, there may be individual cases in the real world that will appear to conflict with those conclusions. These situations are more likely to occur under extreme or unusual conditions and so I expect theyll be rare. In such cases it will likely be necessary to consult detailed references of the TCP implementations involved to try to determine the expected behavior. Additionally, some TCP parameters, such as the Congestion Window, are not easily seen or derived from packet traces, so the best that can be done is to speculate on their impact on performance based on an understanding of how they are used by TCP.
Using Ethereal to Investigate TCP Performance Issues Page 4 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
About the Examples
All of the examples described in this paper involve situations where the observed data throughput for a single TCP connection is less than expected. For example, a customer may be using FTP to transfer large files between two sites and seeing throughput thats substantially below the available bandwidth of the WAN. Ill assume that other more obvious possible causes of the low throughput have already been eliminated, such as high traffic levels resulting in full link utilization and congestion of the circuit(s) involved, bad QoS or other configuration settings in routers or other CPE in the data path, and physical errors on the LAN or WAN. While Ethereal can sometimes be useful in finding these types of problems, other methods are usually more effective and should be pursued first. When those other approaches fail to explain the observed behavior, then it becomes more likely that some aspect of TCPs behavior may be the cause and Ethereal can be used to help identify it.
Some of the examples may have been taken from actual Qwest customer issues, in which case I have removed any customer-private information. In these cases the information obscured is not pertinent to the discussion. Most of the examples are from simulations. In both cases I chose the examples to illustrate a specific aspect of TCP behavior and how it appears in Ethereal, such as the effect of a too small window size on throughput. However, TCP is not a deterministic protocol, so no two connections are exactly the same and you should not expect to see identical results for the various situations. The point of the examples is to expose identifiable characteristics in a way that applies to similar situations in the real world. Using Ethereal to Investigate TCP Performance Issues Page 5 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Packet Capturing Basics
Before Ethereal can be used to analyze a packet capture, the packets have to actually be captured. Ethereal can do this directly, but since it can also open a wide variety of capture file formats, other utilities or network analyzers can also be used. Ideally, the packets would be captured at both the source and destination of the traffic, and should include the complete TCP connection, since that provides the most complete information. If thats not possible, its better to capture traffic at the transmitting end rather than the receiving end, since in TCP its the transmitter that controls the traffic rate based on information it has available. Its usually, but not always, more useful to be able to see the original traffic before its been affected by any intermediate devices or phenomena. In the examples in this paper all the captures were taken on the transmitting host unless otherwise stated.
Its also better to configure Ethereal to only capture the traffic to be analyzed, since that reduces the CPU load as well as the size of the capture file. This is often as simple as specifying a single host IP address and capturing all the traffic to or from that host. In Ethereal, this is done by opening the Capture Options dialog in one of a few ways by clicking on the second toolbar icon from the left (the one with the little wrench on it), by selecting Options from the Capture menu, or by using the Ctrl-K keyboard shortcut:
Using Ethereal to Investigate TCP Performance Issues Page 6 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
This will open the Capture Option dialog window where you can enter the Capture Filter expression you want, in addition to choosing other capture settings:
There are a few pre-configured capture filters that you can open and use as examples for your own filters by clicking on the Capture Filter button, and you can create and save your own in case you find that you use the same capture filters repeatedly. The filters I use most often are host <ip address> and port <number>. You can also combine individual specifications with the logical operators and, or, and not, or their symbolic equivalents, &&, ||, and !, and you can group parts of your expression together with parentheses to create complex and wonderful filter expressions. Finally, you can also use hostnames and keywords instead of numbers, but I find theres less confusion if I specify things by number. Using Ethereal to Investigate TCP Performance Issues Page 7 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Once you get your capture filter configured and the other options set the way you want, click on the Start button to start capturing packets. You should do this before the traffic to be captured is initiated, since its better to capture all the packets in the session. Unless youve chosen to hide the Capture Info dialog window, you should get a new window similar to the one below that gives you a continuously updated view of how many and which types of packets have been captured:
If youve set a good capture filter that includes only the traffic to be studied, you can tell when the connection closes because the packet counts will stop incrementing. To stop the capture, click on the Stop button. This will close the Capture Info window and the captured packets will then be displayed in the main Ethereal window, which might take a few to several seconds, depending on how much data was captured and how fast your computer is. Because Ethereal captures the entire packet by default, the size of the capture file will be larger than the amount of data transferred. Be careful about attempting to capture too much data, as very large capture files can take a long time to process. Ive worked with capture files as large as 10 to 15 MBytes on older computers. Ethereal has options for splitting and saving capture files by size or at regular time intervals, though doing that can complicate the analysis. Using Ethereal to Investigate TCP Performance Issues Page 8 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Once youve run a capture, you dont need to go back to the Capture Options to run it again unless you want to change something, such as the capture filter. The three icon buttons in the upper left of the main Ethereal window starting just below the Ethereal View menu allow you to quickly start, stop, and restart a capture. I find these especially convenient when used with the Capture Info window hidden, since I usually dont need to see the live packet counts. If the icons arent obvious, you can hover the mouse over them to see a popup description of them. Buttons that arent meaningful in Ethereals current operational state are grayed out (see below).
Using Ethereal to Investigate TCP Performance Issues Page 9 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Saving the Capture File
Unless youre just looking for quick answers to simple questions, youll probably want to save your captured data for later analysis. The Ethereal dialog for doing so looks different from the standard Windows file save dialogs, but the functionality is similar. You can choose the name of the saved file and navigate to the desired destination folder, though you may have to click on Browse for other folders to do that (its highlighted in the screenshot below). There is no default file extension, but I recommend using .pcap. You can choose between a variety of different file formats in which to save the capture file, but I suggest using the libpcap (tcpdump, Ethereal, etc.) option, since its the most commonly used Ethereal format. Finally, you can save all or just some of the packets in the capture, which is very useful for reducing the size of the saved files. This is most useful if youve isolated individual sessions or otherwise filtered the display results. See the next section for more details on doing that. The default is to save all the packets in the current capture. If you start a new capture before saving the current one, youll be prompted about whether or not you want to save the current capture data.
Using Ethereal to Investigate TCP Performance Issues Page 10 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Isolating an Individual TCP Connection
Even if youve configured a very specific capture filter, youll probably end up capturing all or parts of multiple sessions. Ethereal provides a few ways to filter the packet display so you can see just the packets related to a specific connection.
The first and most obvious way, but not the easiest way, is to just type a filter expression in the Filter field in the upper left area of the main packet display window and click the Apply button to the right. If your expression is valid the background of that field will turn green. You can use the Clear button to return to the unfiltered packet display, and you can click the Expression button to use a GUI window to add filter elements to your expression. The details of doing that are beyond the scope of this paper, but playing with this feature is a good way to become familiar with Ethereals filter expression syntax. The pull-down menu at the right end of the filter expression field will display previously used expressions, so you can quickly go back to other filter results.
A quicker and easier way to isolate an individual TCP connection is to right-click on one of the packets in the desired connection and then select Follow TCP Stream from the menu that pops up:
Using Ethereal to Investigate TCP Performance Issues Page 11 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. This will do two things. First, it will open a new window that shows the text of the TCP connection color coded so you can identify the traffic flowing in each direction:
This can be useful for text-based applications, but usually you wont be interested in the text of the session. All we really want is the display filter that it creates for us, so you can just click the Close button to get rid of this window and return to the main packet display window, which should now show all the packets for this TCP connection and no others:
If there was a lot of data transferred over this connection it can take quite a while for this window to open. This is one reason to use the next method for isolating and individual connection. Using Ethereal to Investigate TCP Performance Issues Page 12 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Ethereal has a nifty feature that allows you to view all the conversations present in the capture, where a conversation is all the traffic between a given source and destination address on unique source and destination ports, depending on the protocol layer youre investigating. You can see a new window with all the conversations by selecting the Statistics ! Conversations menu item, or you can get just the TCP conversations if you select Statistics ! Conversation List ! TCP (IPv4 & IPv6):
Using Ethereal to Investigate TCP Performance Issues Page 13 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. The first option opens a new window with a tab for all the different protocol layers for which you can view conversations (see below). To see the TCP sessions, click on the TCP tab. The second option will open a new window that only shows the TCP conversations, so its a little quicker, but I find the first method results in a more reliable window. These new windows often open behind the main window, in which case youll have to move it out of the way to see the new window. The conversation window shows the source and destination addresses and ports, the total number of bytes and packets transferred in both directions combined, and the number of bytes and packets transferred in each direction. You can click on any of the column headers to sort by that column, and reverse the sort order by clicking on that column header again. This is handy for helping to identify the connection youre interested in. Once youve identified the connection you want to isolate, right-click on it and select Apply as Filter ! Selected ! A <-> B from the resulting pop-up menus:
Once youve done this you can close the Conversations window. Using Ethereal to Investigate TCP Performance Issues Page 14 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. At this point the main packet display window should show just the packets for the TCP connection youve selected. If you captured the entire TCP connection the first packet in the display will be the SYN packet that initiates the TCP three-way handshake:
If you scroll down to the end of the capture you should also see the two FIN packets associated with the FIN/ACK, ACK, FIN/ACK, ACK sequence of four packets that typically closes a TCP connection:
You can save this session to a separate file by selecting File ! Save As and clicking on the Displayed button instead of the default Captured button, and then save the file as usual. To return to the entire list of packets just click on the Clear button by the filter expression field.
Isolating a single TCP connection is a very useful thing to do, and all the examples in this paper assume thats already been done. They also make heavy use of the TCP Stream Graph feature in Ethereal, which well discuss next. Using Ethereal to Investigate TCP Performance Issues Page 15 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. TCP Stream Graphs
Ethereal can produce interactive graphs of a few different characteristics of a TCP connection, including a Round Trip Time graph, a Throughput graph, and two different plots of TCP Sequence numbers over time called, appropriately enough, Time/Sequence graphs. These graphs are the Ethereal feature I use most often when attempting to diagnose TCP performance problems, so itll be useful to learn more about how to work with them before we get to specific examples.
All of the graph types are accessed in essentially the same way: select a TCP packet in the main packet display window by single-clicking on it and then choose one of the graph types from the Statistic ! TCP Stream Graph menus. Heres an example showing the Throughput Graph being selected (see below). I usually find it more convenient to first isolate the TCP connection in question as described in the previous section before opening a graph, but thats not strictly necessary.
Using Ethereal to Investigate TCP Performance Issues Page 16 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. This will produce two windows a Graph window and a Graph Control window, which always seems to appear behind the Graph window for some reason. Since it looks and behaves the same for all the different graph windows, lets start with the Graph Control window. If you move the Graph Window out of the way you should see the Control window. You cant just close the Graph Window, since closing either of the Graph or Control windows also closes the other one. This window has five tabs across the top for accessing different aspects of the graph window controls. Of those tabs, I use Graph Type and Cross most often, and Zoom every once in a while. I dont recall ever using the other two tabs, Magnify and Origin, though you should feel free to experiment with them to see if they provide features you find useful.
The only feature in the Zoom screen Ive ever used is Zoom Lock, which allows you to zoom just the height or width without affecting the other, which is sometimes useful. The in and out radio buttons allow you to zoom in and out without holding down the shift key, but youll probably never use them. The Horizontal and Vertical fields show the current zoom level, which I also always ignore. Finally, the Horizontal step and Vertical step settings let you specify the amount of zoom in the different directions that happens each time you zoom in. If you wanted to zoom in smaller increments you could reduce these values, but thats another thing Ive never used. The two check boxes below the step settings let you put restrictions on how one step setting changes if you change the other.
Using Ethereal to Investigate TCP Performance Issues Page 17 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. The Cross screen does only one thing: it lets you turn on or off crosshairs in the graph window so you can more accurately see the axis coordinates associated with any point in the graph. This is sometimes handy, since the axes are only on the left and bottom of the graph so it can be hard to see where data points on the other side of the graph line up. The default for the crosshairs is off.
Using Ethereal to Investigate TCP Performance Issues Page 18 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. The Graph Type screen is the one I use most. It lets you quickly change the graph type on the fly without having to close the window and start over. The new graph will be zoomed to the same zoom level and position as the previous graph, which I seldom find useful. To avoid this, check the box by Init on change before you select a new graph type. This will cause the new graph to be displayed completely un-zoomed so you can see the entire graph.
There are four graph types: Two varieties of a Time/Sequence graph, one of which is extremely useful and will be discussed later in much greater detail, the Throughput graph, and a Round-trip Time graph. The Round-trip Time graph is both simple and obvious. Its just a plot of the round trip times Ethereal computes based on the timestamps and sequence numbers in each packet. While we wont discuss the Round-trip Time graph in detail, we will include the concept of round-trip times when we talk about TCP window size and how it can limit throughput.
Using Ethereal to Investigate TCP Performance Issues Page 19 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Now that we understand the Graph Control window, lets look at a couple of actual Graph Windows. Heres the Throughput graph window for the packet previously selected when we first opened the graph window:
As usual there are a few things to keep in mind. First, it matters which packet you select before you open any graph window, since if you pick the wrong one youll probably get a graph with no data in it. Because data can flow in both directions of a TCP connection and each direction can have different throughput rates, you need to make sure you select a packet thats going in the appropriate direction. Since the data is usually only flowing in one direction anyway, this means youll typically just need to select any packet with data in it, which is why I add a packet length field to the main Ethereal display see the Appendix for more details. Assuming youre looking at a data throughput issue in the first place, itll usually be a large packet because TCP always tries to use the largest packet it can for efficiency reasons. For example, the highlighted packet when we opened this graph has a length of 1314 bytes. You can, however, pick any packet going in the direction you want even if it doesnt contain any data, such as a SYN or a FIN packet. Using Ethereal to Investigate TCP Performance Issues Page 20 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. You may remember that I described these graphs as interactive. What I mean by this is that you can resize the window, zoom in and out, and scroll the contents of the window to view the data at whatever level of detail you want. This is a key feature in the other examples in this paper, so its useful to become familiar with it. On Windows platforms you zoom in by just left-clicking on the part of the graph youd like to see closer, which can be slow if there youre viewing a large TCP connection. Every time you click, the zoom level increases. To zoom back out, hold the Shift key down while you left-click. Ethereal remembers the zoom levels as you zoom in so you can zoom back out to previous views, sort of like the way the Back button works in a web browser. To scroll the window after youve zoomed in, just click and drag with the right mouse button. You can resize the window just as youd resize any other window just drag any window border or corner to the desired size. You can also Control-Left Click on a data point in the graph window to scroll the main packet display window and select the corresponding packet, though it can be a bit of a challenge to click on the desired data point accurately enough to invoke this feature. When you do manage to click in the right place you can see the packet display scroll to the newly selected packet. This isnt quite as useful in the Throughput graph, but can be quite useful in Time/Sequence graphs where individual packet phenomena are of more interest. On the Mac platform, the Option key must be added to the Zoom key sequences (e.g. Option-Click to zoom in and Shift-Option-Click to zoom out), and the Control key is optional for the Go to Packet feature. If you have a Mac but dont have a multi-button mouse, you can scroll by holding the Command key while you click and drag. Heres a quick summary of the Windows keyboard commands:
Zoom In Left Click Zoom Out Shift + Left Click Scroll Right Click + Drag Go to Packet in Main Window Control + Left Click
The axes of the Throughput graph should be obvious. The X axis is time in seconds and the Y axis is Throughput in Bytes per second, not bits per second! The individual data points are the instantaneous throughput rates per packet as computed by Ethereal, so this graph is mostly useful for identifying peak high or low throughput levels rather than the overall rate that users typically see and care about. Because these graphs tend to look similar to this example with lots of dots scattered all over the place, I dont use them very often. For example, you can see that this session had many instances of throughput in the 5 to 6 MBytes/sec range, but also regular samples well below 1 MBytes/sec. While this can be useful for seeing how the throughput rate changed during the course of the TCP connection, it can be very difficult to deduce a good estimate of the overall average throughput from such a graph. This is why I generally use a different method, which will be described later, based on the TCP sequence numbers and packet times to get the overall throughput. However, sometimes the Throughput graph is more consistent than this sample and provides more obvious and useful information. Since its so easy to view these graphs, I usually at least glance at them just in case they show something interesting. Using Ethereal to Investigate TCP Performance Issues Page 21 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Now that we know how to open and control graph windows, lets look at the tcptrace version of the Time/Sequence graph in particular. This graph is, in my opinion, one of the most useful features in Ethereal because it can make so many aspects of TCP behavior visible, but first its necessary to understand the different parts of the graph and how they relate to TCP.
Heres a Time/Sequence graph zoomed in to show the different parts of the graph in greater detail:
In the Time/Sequence graphs packets appear as vertical lines with very small bars at both ends, like a letter I There are six packets in the sample above. The position of each packet along the X axis indicates the time that packet was captured. The vertical length of each packet indicates how much data was in that packet, and its position along the Y axis indicates exactly which data bytes of the entire connection were contained in that packet. Headers are not included in these lengths, since TCP sequence numbers apply only to data. The lower end of each packet always corresponds to the TCP sequence number of that packet. (Note: If your graph looks very strange, as if it contains multiple disjointed packet traces, check to make sure you dont have any packets in the main display window with the time reference toggled on, since that confuses Ethereal when it creates these graphs.)
Using Ethereal to Investigate TCP Performance Issues Page 22 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. The step-like line labeled ACK represents the amount of data thats been acknowledged by the receiver. Since TCP uses a cumulative ACK algorithm, this means that at any given time all the data with sequence numbers lower than the ACK line have been successfully received, and any data above the ACK line may not have been received. The vertical sections of the ACK line represent ACK packets from the receiver, with the upper end of the step representing the new sequence number being ACKed. The packet being ACKed is whichever previous packet has an upper end at the same sequence number value as the new value of the ACK curve. ACK packets that dont acknowledge new data, such as duplicate ACKs, are indicated by tiny little downward tick marks in the ACK line.
The step-like line labeled RWIN represents the value of the receivers Receive Window as advertised in the ACK packets, so upward steps in the RWIN line will usually correspond to a step in the ACK line, but not always, since TCP can increase the Receive Window without acknowledging new data and vice versa. The vertical distance between the ACK line and the RWIN line is the current size of the receivers Receive Window. Since TCP does not allow the sender to transmit data beyond the current value of the receivers RWIN, you should never see packets above the RWIN line, though, due to timing anomalies or out-of order packet delivery you actually might occasionally see packets above the RWIN line. If you capture at the sender, you should also never see any packets below the ACK line, since the sender knows theyve already been successfully received.
Finally, if you see a packet repeated at the same vertical position but at a later time, thats a retransmitted packet. These can sometimes appear below the ACK line depending on where the data was captured. Usually this is seen when you capture at or near the receiver.
To summarize, the vertical segments represent data packets sent by the transmitter, and the ACK and RWIN lines represent the corresponding values from the ACK packets returned by the receiver.
This completes our overview of the basic features of Ethereal well be using in the subsequent examples in this paper. Using Ethereal to Investigate TCP Performance Issues Page 23 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Example 1 Determining Actual Throughput of a TCP Connection
Since were looking at TCP performance issues, one of the first things youll probably want to know is what the actual throughput was for a given TCP connection. Weve already seen the Throughput graph and how you can use it to see instantaneous throughput rates for a TCP connection, but users usually see the overall throughput reported by the application at the end of the file transfer, and the Throughput graph doesnt show that. However, since TCP sequentially numbers every byte transferred in both directions, and since Ethereal puts a timestamp on each captured packet, its possible to compute the effective data throughput of the connection. If youve captured the entire connection, and have isolated it as described in the previous section, its actually pretty easy to do. To compute the total throughput, we need to know how much data was transferred and how long it took.
First, you should go to the View menu and set the Time Display Format to Seconds Since Beginning of Capture (see below). You can also specify the displayed time resolution, but I usually just use the default Automatic setting, which is typically microsecond or better. This setting shows the packet timestamp in a cumulative way instead of as an incremental time between sequential packets, which matches the way the graph time scale appears.
Using Ethereal to Investigate TCP Performance Issues Page 24 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. The next step is to set the time reference to be the first packet of the connection, which should be a SYN packet, and should be the first packet in the display. To do this, right- click on the first packet and select Time Reference ! Set Time Reference (toggle) from the menu that pops up. If you previously saved this individual session to its own capture file and have re-opened it you wont have to do this:
This will cause the timestamp of the first packet to change to *REF* and all subsequent timestamps to be relative to that first packet:
Now the timestamp on the last packet of the connection is the elapsed time of the connection. This is the how long did it take part of the throughput calculation. Remember to toggle off any time references when youre done with them if you want to later view graphs of the capture, since setting a time reference can confuse Ethereal when it computes various graph values.
To answer the how much data was transferred part of the question we need to go to the end of the connection and look at the sequence number of the last packet. Recall that TCP sequence numbers are incrementing numbers assigned to the individual bytes transferred in each direction. That is, each data byte is individually numbered, and the number of the first byte of data in each packet is the sequence number for that packet. TCP is supposed to choose those initial sequence numbers at random, and most TCP implementations today actually do that to at least some degree. In reality, theyre not always truly random, but they are virtually always large and awkward to deal with. There are two series of incrementing sequence numbers in every TCP connection, one in each direction, and the initial sequence numbers in each direction are chosen independently by the two endpoints. Luckily for us, by default Ethereal displays all TCP sequence numbers in a relative way as if they were always numbered starting at 1. This makes determining the total amount of user data transferred in each direction very easy, since the value we need is simply the last sequence number of the connection in the direction were interested in: the sequence numbers in the FIN packets at the end of the connection. (Note: Actually, those last sequence numbers will be off by one, since the initial SYN packet is counted as a byte even Using Ethereal to Investigate TCP Performance Issues Page 25 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. though there is no data in that packet, and the same is true of the closing FIN packet. For our purposes that one byte is not significant and can be safely ignored, since the amount of data being transferred will typically be very large so one or two bytes either way wont affect the computed throughput in any significant way.) In the case of file transfers, its almost always the case that data is only sent in one direction, and the only packets that are sent in the other direction are ACK packets that contain no data. So, of the two FIN packets at the end of the connection, one will typically have a sequence number of 1 and the other will have a large sequence number that is one larger than the amount of data transferred. If large amounts of data were transferred in both directions then you may have to identify the transmitter of the flow in question by IP address or source port number instead.
In our example, the first FIN packet has the large sequence number. This will usually be the case, since its usually the transmitter who decides to close the connection. The sequence number is 10573163, or a little over 10.5 MBytes, and the elapsed time for this amount of data is about 4.39 seconds:
Therefore, the average user data throughput can be computed as follows:
There are a few things to keep in mind when computing throughput this way. The first is that this analysis ignores all the IP and TCP headers (usually 40 bytes per packet), so the actual bandwidth consumed on the network will be a bit higher. If one assumes the data transfer used maximally sized packets of 1500 Bytes and that 40 bytes of each packet were used for headers, the overhead percentage is (40 / 1500) * 100 = 2.67%. This is a best- case value, and the actual overhead may be slightly larger. However, most customers experiencing throughput issues would be ecstatic to get within 3% of the theoretical maximum, so header overhead can usually be ignored. On the plus side, all applications also ignore packet header overhead, so this calculation is consistent with what the customer is likely to see.
Another thing to remember is that most applications report throughput in Bytes/second, but virtually all network elements specify bandwidth in bits/second. This is complicated by the fact that computers and applications almost universally use the K and M prefixes for Kilo and Mega, where K = 1024 and M = K 2 = 1024 2 = 1,048,576. However, network bandwidth numbers always use 1,000 and 1,000,000 for Kilo and Mega, so a 56K Using Ethereal to Investigate TCP Performance Issues Page 26 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. circuit really does have a bandwidth of 56,000 bits/second, not 57,344 bits/sec (56 * 1024 = 57,344). Again, this is usually not a significant enough factor to explain a customers throughput concerns, but it can become an issue when doing computations and its good to know about and be able to explain to customers. Using Ethereal to Investigate TCP Performance Issues Page 27 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Example 2 Throughput Limited by Receive Window Size
In this example Ill use the Time/Sequence graph to illustrate what a TCP connection looks like when the throughput is limited by a too-small Receive Window. Before we get to the graphs, a brief discussion of the theory behind TCP performance is appropriate.
Recall that TCP uses a Sliding Window scheme to allow each end of a connection to send a limited amount of data beyond what has been acknowledged by the other end. Because the sender has to be able to transmit new data continuously without having to stop if its going to fully utilize the available bandwidth, this implies that the sliding window has to be at least large enough to allow the sender to transmit for as long as it takes for a packet to get to the receiver and the acknowledgement to get back, which is the Round Trip Time (RTT) between the endpoints. This minimum window size is called the Bandwidth-Delay Product, and is equal to the RTT of the path in seconds times the effective bandwidth of the path, which is the bandwidth of the slowest link of the path. There are a few things that can limit the size of the TCP sliding window, and one of them is the Receive Window (RWIN) advertised by the receiver to the sender. If the size of the RWIN is smaller than the Bandwidth-Delay Product of the path, then TCP cannot fully utilize the bandwidth of the path. Since its seldom possible to reduce the RTT between the endpoints, the only other option is to increase the RWIN if you want to improve TCP throughput. In most operating systems its possible to change the TCP window sizes if the defaults dont allow adequate throughput. In recent versions of Microsoft Windows, the default RWIN is usually 17,520 Bytes. (Note: Its actually 16 KBytes rounded up to the next even multiple of the MSS received from the far end, which is typically 1460. Since 16,384 / 1460 = 11.22, the typical RWIN is 12 * 1460 = 17,520 Bytes.)
An additional implication of the Bandwidth-Delay Product is that if you know the RWIN size and the RTT between two particular hosts, you can compute the maximum theoretical throughput of a TCP connection between them. Since the equation for the limiting case of how big the sliding window needs to be is RWIN = Bandwidth * RTT, simple algebra yields this equation:
Bandwidth = RWIN / RTT
For example, if you know a pair of Windows PCs are using the default RWIN of 17,520 Bytes and the RTT between them is 35 milliseconds, the maximum usable bandwidth between the hosts by a single TCP connection is 17,520 Bytes / 0.035 seconds = 500,571 Bytes/second = 4,004,568 bits/second. This equation gives a theoretical upper bound that ignores packet headers and other performance impacting factors, so the real-world maximum throughput will always be a few percent less. However, its still a useful computation to be able to make, since you know TCP cannot achieve higher throughput than the result. So, if a customer has a fractional DS3 circuit provisioned at 20 Mbits/sec and sees typical round trip times of around 35 msec, any individual TCP file transfers between Windows PCs will be limited to about 4 Mbits/sec unless they increase the TCP Window size of hosts. If they are actually seeing throughput close to that, then the RWIN is likely the limiting factor. However, if they see throughput substantially less than 4 Mbits/sec then something else is the cause and increasing the TCP Window size will not improve throughput.
Now that we understand RWIN and how it can limit TCP throughput, lets see what it looks like in Ethereal. Using Ethereal to Investigate TCP Performance Issues Page 28 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. The useful graph type for this analysis is the Time-Sequence Graph (tcptrace). There are actually two styles of Time/Sequence graph in Ethereal, the Stevens style and the tcptrace style. The Stevens style is named after W. Richard Stevens, who used this type of graph to illustrate various aspects of TCP behavior in his most excellent book, TCP/IP Illustrated, Volume 1: The Protocols. The tcptrace style of Time/Sequence graph is named after the free utility of the same name that can be used to post-process packet capture files and produce statistics and graphs of several different characteristics of a TCP connection, one of which is a Time/Sequence graph. In Ethereal these two graph styles are very similar, but since the tcptrace style includes everything the Stevens style does along with additional information, its the only one I use, and from now on thats the one I mean when I refer to Time/Sequence graphs. To open it, select a packet carrying data in the direction of the file transfer, then select the Statistics ! TCP Stream Graph ! Time-Sequence Graph (tcptrace) menu item:
Using Ethereal to Investigate TCP Performance Issues Page 29 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. For this example I captured a TCP connection to a Windows 2000 PC configured with an RWIN of 8760, which is the default for older versions of Microsoft Windows. The test transferred 250,000 Bytes of data and was run through an intermediate FreeBSD computer configured to add an artificial delay of about 30 msec in each direction for an RTT of at least 60 msec. The packets were captured on the sending host, since thats where the most useful information is. Later well see what a capture at the receiving host looks like. Heres the Time/Sequence graph for this connection:
As with all the other graphs, the X axis is time in seconds. With the Time/Sequence graphs the Y axis is the normalized TCP Sequence Numbers, which is also the number of Bytes transferred. Because this is essentially a graph of data over time, the general slope of the graph can give you a visual indication of how consistent the throughput was. In this case, because the graph is quite straight the data was transferred at a relatively constant rate. In general, however, steeper parts of a graph indicate higher throughput, and flatter parts of a graph are areas of lower throughput. Well see examples of that later. Using Ethereal to Investigate TCP Performance Issues Page 30 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Now that we know what the various parts of the tcptrace graph show, lets look at the current example in a bit more detail:
The single packet at A is the first data packet of the connection, but not the very first packet, which is always a SYN packet and virtually never carries data. Note also that the RWIN line is at 8760 bytes, which is the initial value advertised by the receiver even before any data has been sent. TCP connections start in a mode called slow start where the transmitter first sends one packet and waits for an ACK from the receiver, and then increases the number of packets it can send without waiting by one packet for each acknowledged packet. This typically results in an exponential increase in the number of packets the transmitter can send, though in this case with such a small RWIN size thats not very evident. TCP also has a feature called Delayed ACK which causes TCP to wait a short time, usually 200 msec, before sending an acknowledgement in hopes of being able to include the ACK with data going the other direction. This means that its not unusual for the ACK of the first data packet to be delayed, which can be seen in the graph. This can cause performance issues with some Microsoft Windows protocols so theyve modified their version of slow start so that it starts with two packets instead of one. Since TCP usually sends an ACK for every other received packet, this eliminates the initial Delayed ACK and increases efficiency.
Using Ethereal to Investigate TCP Performance Issues Page 31 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. By point B the transmitter is sending three packets at a time, and you can see that the gap between the upper end of each burst of packets and the RWIN line is narrowing. At point C the upper end of the three packets in the burst is right at the RWIN line. This means the transmitter has filled the receivers RWIN and must stop transmitting until an ACK is received. If we zoom out a bit to see more of the graph we can see very easily that the throughput is being limited by the receivers RWIN because all the packet transmissions stop at the RWIN line:
This is also visible in the un-zoomed version of the graph, so its usually quite easy to identify when TCP throughput is being limited by the receivers RWIN. Ethereal will sometimes indicate this in the main packet display by putting the text [TCP Window Full] in the Info column, so that can be another indication that a TCP connection is limited by a RWIN thats too small:
Using Ethereal to Investigate TCP Performance Issues Page 32 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Its very important to remember that this technique of identifying this issue only works if the packets were captured on or very near the transmitter. If the session is captured at the receiver the RWIN throughput limitation is virtually impossible to identify. Heres what the Time/Sequence graph looks like when captured at the receiver. This is the exact same TCP connection as before, slightly zoomed to show more detail:
Notice how all the packets now hug the ACK line instead of the RWIN line. This is because this capture was taken at the receiving host, so there is virtually no delay between when a packet is received and when the corresponding ACK for that packet is sent. Were effectively on the wrong side of the round-trip time, so we cant see the interval between the transmission time of a data packet we receive and the receipt of the corresponding ACK we send, which is the round-trip time TCP uses as part of its flow control scheme. If this is the only capture file available, the best alternative is to just do the math to determine if the realized throughput is near what would be predicted from the known value of RWIN and the average round-trip time between the endpoints using the equation discussed previously.
Finally, notice how every other packet aligns with a step in the ACK line, which fits the typical TCP behavior of only ACKing every other received packet. Using Ethereal to Investigate TCP Performance Issues Page 33 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Example 3 Throughput Limited by Sender Window Size
In addition to the receivers RWIN, there are other parameters that can limit TCP throughput, including the Send Window (SWIN), which is static and does not change during a connection, and the Congestion Window (CWIN), which is dynamic and does change in complex ways. Both of these parameters are maintained by the transmitter. While we cant see either of these values in actual packets, the fact that they are coming into play tends to be easily recognizable in the Time/Sequence graph:
This is another nuttcp connection through the same simulation setup as the previous example, but in this case I reconfigured the receiver to use an RWIN of 32,768 Bytes and the transmitter to use a Send Window (SWIN) of 8192 Bytes everything else was unchanged. In the graph the RWIN line is farther away from the ACK line, which reflects that the receivers RWIN was, indeed, larger than before. However, the data packets never approach the RWIN line. Instead, they stay what appears to be a constant distance short of it. When there is an obvious gap like this between the RWIN line and the packets it means the receivers RWIN was absolutely not a factor in limiting the throughput of the connection, and increasing it will not make a difference. Something else is limiting throughput: the transmitters SWIN. Using Ethereal to Investigate TCP Performance Issues Page 34 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. The transmitters SWIN is analogous to the receivers RWIN, and can limit throughput in exactly the same way RWIN can. Unfortunately, because the recipient doesnt need to know how big the transmitters SWIN is, its never advertised, and, therefore, we cant explicitly see what it is by looking at a packet capture. On the plus side, most operating systems set both RWIN and SWIN to the same value, so if you look at the RWIN advertised by the transmitter, that will usually be what its SWIN is also.
Lets look a bit closer at the graph and see what we can learn. Heres a zoomed view of the above graph, along with some annotations:
I used the control-click graph feature to go to the corresponding packets in the main Ethereal window to determine the values on the graph from the ACK and Seq values in the packets, but you can also zoom way in on the graph until the Y-axis resolution shows individual Sequence numbers and find the data values that way.
So what do these numbers mean? Remember that I set the SWIN to 8192, so why did the transmitter stop after sending 7300 Bytes? It turns out that TCP really likes to send full- sized segments if at all possible, where the segment size is the MSS advertised by the receiver. In this case the MSS is 1460 (1460 = 1500 Byte MTU 40 Bytes of standard TCP and IP headers), and 7300 is an even multiple of 1460. Rather than send a somewhat smaller packet and have the maximum possible 8192 Bytes of outstanding data, TCP, for Using Ethereal to Investigate TCP Performance Issues Page 35 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. some reason, decided to just stop at 7300 Bytes. This may be because the data stream from the application had already been segmented into 1460 Bytes packets by the TCP stack, or possibly an optimization by the operating system similar to the way Microsoft rounds the RWIN to an even multiple of the MSS, but its hard to say for sure. The resulting average throughput would then be limited by this effective SWIN, with the theoretical limit given by 7300 * 8 / RTT. This is the formula discussed previously, but with SWIN in place of RWIN.
One thing to keep in mind regarding the SWIN, and to some extent, RWIN, is that the application programmer has some control over it and can override the default system values. So, even if you conclude that the throughput of a particular TCP connection appears to be limited by the transmitters SWIN, changing the system settings for the SWIN may not fix the problem. In this case the only solution may be to update or replace the application.
In summary, and this is the key point of this example, when you see a graph like the first one in this example where the data packets never approach the RWIN line but instead stay what appears to be a relatively constant distance from it, that usually indicates the throughput was limited by the senders SWIN rather than the recipients RWIN. A possible fix for this is to increase the SWIN on the sending host rather than the RWIN on the receiving host, or both the RWIN and SWIN since theyre usually the same. Because the application programmer can override the system settings, however, this may not always fix the problem. In any case, increasing the receivers RWIN will absolutely not improve throughput. Using Ethereal to Investigate TCP Performance Issues Page 36 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Example 4 Packet Loss
The other common cause of slow TCP performance is packet loss, but the two are not as directly related as you might think. TCP assumes that packet loss is a result of congestion, which is virtually always the case, so it reacts by slowing down. In reality, the event that directly triggers TCPs reaction to congestion is the retransmission of a packet, not packet loss. Retransmission, in turn, is caused by either the expiration of the retransmission timer for the original packet, or else the receipt of three Duplicate ACKs (Dup ACKs) in addition to the expected ACK, for a total of four identical ACKs. (Note: Recent versions of Microsoft Windows use a limit of two duplicate ACKs instead of three, so retransmissions can happen sooner in that case.) If the retransmission timer expires it is very likely the original packet was actually lost, since its quite rare for a packet to be delayed for such a long time and still make it to the destination. Dup ACKs are also usually caused by dropped packets because TCP is required to ACK each packet that arrives out of order, which is what each packet after the dropped packet looks like to the recipient. Once three packets after the dropped packet are received, the transmitter will get three Dup ACKs, assume the corresponding packet was dropped, and retransmit it. A large number of out of order packets can also trigger packet retransmission even though no packets are actually dropped, but this is less common. Because non-dropped packets are generally not delayed much, retransmission is most often triggered by Dup ACKs rather than timer expiration, at least with file transfers where there is a constant flow of packets. Heres a Time/Sequence graph for a TCP connection experiencing packet loss:
Using Ethereal to Investigate TCP Performance Issues Page 37 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
This was an nuttcp test connection that transferred 2,000,000 Bytes with a simulated packet loss rate of 0.5 percent. The most obvious thing about this graph is how uneven it looks its not nearly as straight as the other examples weve looked at so far, and is actually more typical of real-world connections. Lets look at this curve in more detail, starting at the beginning of the connection:
For the first second or so we see the classic TCP Slow Start behavior where the number of unacknowledged packets sent at any point roughly doubles until the RWIN is filled and the transmitter has to stop and wait for an ACK. Because the packet transmission rate is increasing exponentially, Slow Start generally appears as a sharply upward curving shape in the Time/Sequence graph. In this case, however, just as the RWIN limit is about to be reached we can also see a retransmitted packet, which is circled in the above graph. Using Ethereal to Investigate TCP Performance Issues Page 38 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Heres a much closer view of that packet, on the right, along with the original packet that was dropped on the left:
We know this is a retransmitted packet because it appears at the same vertical position along the Y-Axis as a previous packet, and itll almost certainly be labeled as a retransmission in the main Ethereal packet display. Even from this view, however, its not completely clear why this packet was retransmitted, so lets look even closer at the retransmitted packet:
This is an extremely close view of the ACK line shortly before the retransmitted packet. The vertical section of the ACK line at the left corner of that line is a normal ACK packet, and the tiny downward tic marks circled in red are Dup ACKs of that normal ACK. You can tell from the time scale that these packets arrived in a very short time interval, which is why there are more than three Dup ACKs the transmitting host just didnt have time to process the three Dup ACKs and retransmit the lost packet before a fifth ACK arrived. Because TCP has to ACK every out of order packet, and since the packets after the dropped packet arrived at the recipient in a very short time interval, the corresponding ACKs for those packets were also sent in a very short interval. So, this retransmitted packet was triggered by Dup ACKs, which were the result of a dropped packet. Note that its generally easier to see Dup ACKs in the Ethereal packet display window, since they are usually labeled with TCP Dup ACK followed by the packet number of the original ACK and number indicating how many duplicates the current packet makes. For example, itll say TCP Dup ACK 104#3 if this is the 3 rd duplicate of the ACK in packet 104. If you enable the packet coloring rules you can Using Ethereal to Investigate TCP Performance Issues Page 39 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. make it even easier, since you can make the Dup ACKs stand out from the other packets with a different color. Heres an example for the part of the sample trace weve been looking at. The Dup ACKs have a pink background, and the retransmitted packet has a black background. You can also see that there were several additional Dup ACKs that arrived after the retransmitted packet was sent. This is because the RWIN was almost full when the packet was dropped and the dropped packet was one of the earlier ones. So, there were quite a few subsequent packets that were already on the way and were received at the far end, and each one of them was considered to be an out of order packet and had to be individually ACKd:
There is yet another way to quickly see this type of information the Expert Info option in the Analyze menu:
Using Ethereal to Investigate TCP Performance Issues Page 40 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. This will bring up a window containing a nice list of all the various Expert analyses Ethereal does on the captured data, most of which also appear as comments in the Info column in the main packet display. Heres what the Expert Info looks like for our captured data:
The lines are color-coded by severity, and you can filter the individual messages to see just those at and above any particular severity with the Severity filter pull-down menu in the upper right part of the window. Clicking on a message in the Expert Info window selects the corresponding packet in the main display window, so this is a quick way to get to individual problem areas in a large capture, since scrolling through all those packets in the main window can take a while. Notice all the TCP Duplicate ACK messages with incrementing numbers, along with the Fast retransmission message. This is an easy way to see what we just covered, and can save some time if you already suspect retransmissions may be an issue. Using Ethereal to Investigate TCP Performance Issues Page 41 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Another way to identify that retransmissions are happening is to look at the Time/Sequence graph, since the changes in throughput that retransmissions cause are often, but not always, relatively easy to see. Heres the same graph shown at the beginning of this section, but with the most noticeable retransmissions circled:
The key indicator of retransmission is an abrupt change in the slope of the graph when theres a retransmission the sender reduces its rate of transmission in order to avoid congestion, which appears as an abrupt change in the slope of the graph to a less steep angle. As TCP recovers and increases the transmit rate, the slope gradually increases, which appears as a gentle upward curve. If the retransmissions are not too frequent the overall result is a scalloped look to the graph, which is clearly visible in this example. In more severe cases of packet loss the phenomenon is less obvious and one of the previously discussed methods of identifying retransmissions is necessary. Using Ethereal to Investigate TCP Performance Issues Page 42 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. At this point we know how to spot retransmissions, and we can identify if the cause of the retransmission was too many Dup ACKs. Now lets look at the performance impact of these retransmissions. Here are six Time/Sequence graphs of the same data transfer test but with different levels of packet loss:
The most visible difference between these graphs is the increasing levels of jaggedness as the packet loss rate increases. However, because Ethereal scales the graphs to fit the window, unless you look at the time scale its not obvious just how much slower the higher loss transfers are compared to the lower loss transfers. To make this comparison easier, on the next page is a larger composite graph that shows all the individual curves using the same scale (I used Photoshop to create this, as theres no option for doing this in Ethereal). Using Ethereal to Investigate TCP Performance Issues Page 43 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees.
Its now very clear that relatively small levels of packet loss can have quite a dramatic effect on throughput. Even just 1.0% packet loss resulted in throughput that was about one third that of the zero-loss rate. The point of this graph is not to indicate what performance you should expect if you know what your packet loss rate is. Thats not a valid generalization to make, since the details of the effect of packet loss on performance are quite complex. The point is that even small amounts of packet loss can have a large impact on throughput.
However, if throughput is not limited by RWIN or SWIN, then there will always be at least some packet loss, since TCP will continuously speed up until it experiences congestion resulting in packet loss. This is the normal mechanism TCP uses to detect available bandwidth and is not inherently bad. Its only when packet loss is excessive and causes unnecessary slowness that it becomes a problem. Normal amounts of packet loss tend to appear in the Time/Sequence graphs as more scalloped and less jagged, but even thats not always true.
Finally, its important to remember that its not actually the loss of a packet that directly causes TCP to reduce its transmission rate its the retransmission of a packet that triggers the rate reduction. While dropping a packet will always result in a retransmission, its also possible for packets to be retransmitted even when no packets are lost, and this will also cause TCP to slow down. The next example will go into this in a more detail. Using Ethereal to Investigate TCP Performance Issues Page 44 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Example 5 Out of Order Packets
With the recent popularity of Qwest NxT1 services and Multilink PPP (MLPPP) the question of out of order packets and their impact on performance has become much more common. The short answer is that TCP doesnt care about out of order packets and they have no impact on performance at all, but thats only true up to a point. If packets are severely out of order TCP can erroneously interpret them as a sign of dropped packets and slow down as if packets were actually lost. Heres a Time/Sequence graph from a customer issue I recently worked on:
This was an FTP data transfer that was experiencing slow throughput, and this data was captured at the transmitting host. Ive numbered the individual packets with the order in which they were sent, except for the retransmitted packet, which is labeled 1R, and circled the Dup ACKs that triggered the retransmission. From this graph its not possible to tell with certainty why Packet 1 was retransmitted all we can do is assume that it was dropped, since we got two Dup ACKs for Packets 2 and 3. Note that this was a Microsoft Windows server so the Dup ACK Threshold was two instead of three. Fortunately, this same connection was also captured at the receiving host so we can see more details by looking there. Using Ethereal to Investigate TCP Performance Issues Page 45 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Heres a Time/Sequence graph for the same part of the connection, but from the data captured on the receiving host instead:
The individual packets have the same labels as on the previous graph, and the Dup ACKs are circled as before. With this additional information we can now see more accurately what actually happened. First, notice that Packet 1 was not dropped at all. In fact, it arrived only slightly after Packets 2 and 3, which happened to arrive in reverse order but thats not a factor in this particular issue. We already know the two Dup ACKs in the previous graph triggered the retransmission, so packet delay was not the cause. Those same Dup ACKs are present in this graph, too, and now we can see that they were caused by the receipt of Packet 3 and then Packet 2 because they appear at virtually the same time as those corresponding packets, though this is only true because this capture was taken on the receiving host. Remember that TCP is required to immediately send an ACK for every packet that arrives which is not the next expected packet, which is the definition of an out of order packet. From the ACK line in the graph we can see that the next expected packet was Packet 1. When Packets 3 and 2 arrive they each trigger an ACK, each of which is a duplicate of the original ACK represented by the ACK line. Because the sender was a Microsoft Windows machine, two Dup ACKs are enough to cause a retransmission, which also causes TCP to slow down. There were about 40 similar instances of retransmissions caused by out of order packets in this TCP connection out of a total of about 1100 packets, which is a retransmission rate of just under 4 percent. There were no dropped packets Using Ethereal to Investigate TCP Performance Issues Page 46 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. whatsoever. Weve already seen what a dramatic impact such a seemingly small rate of retransmissions can have on TCP performance, so it hardly surprising that the customer was not happy with the performance they were seeing. The details of the cause of the out of order packets are beyond the scope of this paper, but the gist of it is they were the result of an unfortunate combination of features and options with Multilink PPP across the NxT1 bundle. One possible workaround in a situation like this is to modify the Windows Registry on the transmitting host so it uses a larger Dup ACK threshold (3 instead of 2, for example), thereby making Windows less sensitive to them and reducing the number of retransmissions. Using Ethereal to Investigate TCP Performance Issues Page 47 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Conclusion
Ethereal is an exceptionally powerful tool for investigating various network phenomena. While it has many features that make it applicable to a wide variety of network applications, such as VoIP, Ive attempted to describe the features Ive found to be most useful when investigating TCP performance issues. I described how to create and open packet capture files and isolate individual connections to make further analysis simpler. I paid particular attention to what I think is one of Ethereals strongest features, the TCP Time/Sequence graphs, which provide a surprising amount of information. When coupled with a basic understanding of the fundamentals of TCP, these graphs are often all you need to identify the specific cause(s) of a variety of TCP performance problems. I tried to go into some detail about a few of the more common performance issues Ive seen over the years. However, since there are always new problems and variations of old problems, its useful to learn as much as you can about the technology involved and the tools and resources available for working with and testing that technology. Ethereal is certainly one of the more useful of those tools, and its effective use can be a tremendous aid to the network troubleshooter. Using Ethereal to Investigate TCP Performance Issues Page 48 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Test Topology
This is a simple diagram of the network topology I used to generate most of the packet captures I used in this paper:
The system in the center was running FreeBSD 6.0-Release with the ipfw and dummynet features configured to allow for arbitrary amounts of bandwidth, delay, and packet loss. This system, which served as the WAN simulator, routed between its two Ethernet interfaces, which were connected via cross-over cables to the other two test systems. While the ipfw/dummynet software provides a lot of flexibility for simulating various aspects of network performance, due to feature and system clock resolution limits the one thing it cannot easily do is accurately simulate the out of order packet phenomena Ive seen occur across MLPPP links. This is why I used a packet capture from an actual customer for that example.
The test system on the left usually served as the nuttcp server and also had Ethereal available to capture the destination end of test traffic. The system on the right was usually the nuttcp client and was the source of the test traffic. It was also running Ethereal, and is where most of the test sessions were captured. This was especially convenient since this is also the system where I wrote this paper, which reduced the need to transfer capture files between systems. I specifically wanted one of the two test systems to be a non-Windows platform, since I wanted to be able to investigate differences in the TCP implementations. Since Mac OS X is based on FreeBSD, that provided a nice alternative OS. A side benefit is that Unix-based systems generally have built-in commands for adjusting different TCP parameters, which made creating the test scenarios a bit easier.
All of these systems are rather old (at least 6 years old, in fact) and not particularly powerful, but that turned out to not be an issue since I was not attempting to maximize throughput across a high bandwidth link. This is a good reminder of just how efficient modern TCP stacks are. Using Ethereal to Investigate TCP Performance Issues Page 49 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. References and Resources
Heres a list of the various books, web links, and software tools I used or consulted while writing this paper, or that I think would be generally useful to someone interested in learning more about TCP performance analysis:
Stevens, W. Richard. TCP/IP Illustrated, Volume 1 The Protocols. Reading, MA: Addison- Wesley Publishing Company, 1994. ISBN 0-201-63346-9. This is absolutely the finest book on the internal workings of TCP/IP Ive ever seen. It covers TCP in great detail but is still quite readable. I highly recommend it.
Ethereal. http://ethereal.com/ This is where you can download the latest version of Ethereal, as well as learn more about it. This free, open source software is a must-have for investigating a wide variety of networking issues. The more recent versions include the winpcap software needed to actually capture network traffic.
ttcp and variants. http://sd.wareonearth.com/~phil/net/ttcp/ TTCP is a TCP throughput testing utility, and there are many different versions of it available. This page lists most of the better ones. I like nuttcp, but there are others that are also useful. I used nuttcp on two systems connected through a third system serving as a WAN simulator to generate the capture files used in most of the examples in this paper.
tcptrace and xplot. http://www.tcptrace.com/ I didnt talk about tcptrace in this paper, but it can be quite useful for extracting even more information from a TCP capture file when used with the xplot package. Theres a page with links to other companion programs on this site as well, including xplot, and documentation on how to use the many options.
TCP Optimizer. http://www.speedguide.net/downloads.php This is a handy little free utility from SpeedGuide.net for setting various TCP parameters on Microsoft Windows systems, including things like TCP Window sizes, Dup ACK threshold, MTU, etc. Its the nicest such utility Ive found so far, and is what I used on the Windows test system to help create the test conditions I needed for the various examples.
ipfw and dummynet. http://www.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8 Ipfw is one of the firewall packages supported in FreeBSD, and it now includes the dummynet traffic shaping feature that allows the configuration of arbitrary bandwidth, delay, and packet loss settings. This is the software I used on an old laptop to simulate various types of WAN links in order to capture more realistic sessions for most of the examples in this paper.
tcpdump. http://www.ethereal.com/docs/man-pages/tcpdump.8.html The tcpdump command is available in virtually all versions of Unix and Linux. Since Ethereal can open tcpdump capture files, it can be a useful alternative for capturing traffic if Ethereal doesnt happen to be installed. Using Ethereal to Investigate TCP Performance Issues Page 50 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Appendix A: Configuring Ethereal
The Ethereal settings used in the examples are largely the defaults, though I changed a few of them for the sake of convenience and they are illustrated below. None of these settings are critical to understanding or following the examples in this paper. The pages that follow show all the Preferences screens for the Ethereal setup I used in this paper along with a brief description of the changes from the defaults. Its important to remember to use the Save button in order for any changes made to apply to subsequent Ethereal sessions.
Using Ethereal to Investigate TCP Performance Issues Page 51 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Ethereal Preferences: Layout
Ethereal defaults to the first layout shown below, but Ive found the second layout to be more convenient, as it allows for a larger Packet List in Pane 1. A reasonable font selection still allows for full visibility of the other two panes.
Using Ethereal to Investigate TCP Performance Issues Page 52 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. This is a sample of what the above layout looks like. Notice that having the lower two panels side-by-side allows you to see more entries in the Packet List and still see the full width of the Packet Details and Packet Bytes panels. The price for this is that it requires a smaller font than some people may prefer.
Using Ethereal to Investigate TCP Performance Issues Page 53 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Ethereal Preferences: Columns
The default Packet List view in Ethereal does not include columns for the source and destination port numbers, nor for the packet length in bytes, but I really like seeing them. To add them, click on the New button to the left of the list of columns, type an appropriate name for the column being added in the Title filed below the column list, and then select the corresponding field from the Format pull-down list. The new column will appear at the end of the list, but the Up and Down buttons can be used to move it to the desired location. Note that changes to the Columns preferences do not take effect until Ethereal is restarted. Also, there is no way to save the width of the individual columns, so you have to adjust them in the main Ethereal window by dragging the column header borders to the desired width every time you start Ethereal if you dont like the defaults.
Using Ethereal to Investigate TCP Performance Issues Page 54 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Ethereal Preferences: Fonts
I prefer a fixed-width sans serif font in Ethereal, even though variable-width fonts allow for more text on individual lines. The available fonts will vary by platform, but there are usually several good options. There are also text zoom controls in the main Ethereal window, so its easy to change the font size dynamically if desired.
Using Ethereal to Investigate TCP Performance Issues Page 55 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Ethereal Preferences: Capture
I find it convenient to specify the default interface Ethereal will use for capturing packets, as well as the behavior of the Capture Info dialog box. The list of available interfaces will vary by platform, and will include any virtual interfaces that might exist on the host, such as VPN interfaces (see the Special Note later in this paper for more details on Ethereal and the Cisco VPN Client). The Edit button can be used to remove any interfaces from the list of available interfaces when starting a new capture. In order to capture packets not sourced or destined to the host running Ethereal, promiscuous mode must be enabled. I also usually prefer to see the Packet List updated in real time, but this can cause increased CPU load on the host and sometimes has to be disabled if Ethereal cant keep up. None of these settings affect the packet analysis or decodes at all, but they can make the process of performing repeated captures easier, since all you usually have to do is click on a button or two.
Using Ethereal to Investigate TCP Performance Issues Page 56 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Ethereal Preferences: Name Resolution
I prefer to see the actual numeric values of the various header fields in the packet list view, since it eliminates name lookup delays, thereby increasing performance, and it helps prevent long host names from overflowing the column widths. However, if you prefer to see names for hosts and a few other things, you can specify that here.
Using Ethereal to Investigate TCP Performance Issues Page 57 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Ethereal Preferences: Packet Coloring Rules
Ethereal also includes a default set of Packet Coloring Rules. For this paper most of those rules have been deleted or disabled in order to simplify the appearance of the screen captures. Theres also a button in the Ethereal toolbar to toggle all packet coloring on and off. The syntax for creating the rules can be quite complex, but it basically involves selecting the desired packet header and/or Ethereal analysis field(s) from a list and combining them with the keywords and and or, and optionally including parentheses to create complex rules. The foreground text color and background color can be chosen so that any packets that match the rule will stand out in the packet list highlighted with the selected colors. This can be quite useful when looking for a particular type of packet in a large capture. Its also possible to import and export saved coloring rules so that different rule sets and be used for different applications. The details of creating and using these rules are beyond the scope of this paper, but a good way to get started with them is to experiment with the Expression feature in the main packet list window. As a reference, the coloring rules used in this paper are shown below.
This setting is located in the View ! Coloring Rules menu rather than the Edit ! Preferences menu.
Using Ethereal to Investigate TCP Performance Issues Page 58 of 58 Confidential Internal Use Only. Disclose and Distribute Solely to Qwest Employees. Special Note: Ethereal and the Cisco VPN Client on Windows
The Cisco VPN Client installs a virtual interface on Microsoft Windows systems. If you use Ethereal to capture packets on such a system, you may get duplicates of all the packets sent by the local system (i.e. the PC running Ethereal). The work around for this is to temporarily disable the virtual interface by opening the properties for the Ethernet interface youre using and un-checking the box for the Deterministic Network Enhancer (see below). Remember to re-enable the Deterministic Network Enhancer when youre done, since the Cisco VPN Client software requires it.