Research Activity

The following are links to various troubleshooting tools.

Software Tools
Network Management Systems:
http://www.ipswitch.com/products/whatsup/index.asp?t=demo
http://www.solarwinds.com/products/network_tools.aspx
Baselining Tools:
http://www.networkuptime.com/tools/enterprise/
Knowledge Bases:
http://www.cisco.com
Protocol Analyzers:
http://www.flukenetworks.com/fnet/en-us/products/OptiView+Protocol+Expert/
Hardware Tools
Cisco Network Analyzer Module (NAM):
http://www.cisco.com/en/US/docs/net_mgmt/network_analysis_module_software/3.5/
user/guide/user.html
Cable Testers:
http://www.flukenetworks.com/fnet/enus/products/CableIQ+Qualification+Tester/Demo.htm
Network Analyzers:
http://www.flukenetworks.com/fnet/enus/products/OptiView+Series+III+Integrated+Network+Analyzer/Demos.htm

Troubleshooting Layer 2 - STP Loops

If you suspect that an STP loop is causing a Layer 2 problem, verify if the Spanning
Tree Protocol is running on each of the switches. A switch should only have STP
disabled if it is not part of a physically looped topology. To verify STP operation, use
the show spanning-tree command on each switch. If you discover that STP is not
operating, you can enable it using the spanning-tree vlan ID command.
Use these steps to troubleshoot forwarding loops:
Step 1. Identify that an STP loop is occurring.
When a forwarding loop has developed in the network, these are the usual
symptoms:
Loss of connectivity to, from, and through the affected network regions
High CPU utilization on routers connected to affected segments or VLANs
High link utilization (often 100 percent)
High switch backplane utilization (compared to the baseline utilization)

Syslog messages that indicate packet looping in the network (for example, Hot
Standby Router Protocol duplicate IP address messages)
Syslog messages that indicate constant address relearning or MAC address flapping
messages
Increasing number of output drops on many interfaces
Step 2. Discover the topology (scope) of the loop.
The highest priority is to stop the loop and restore network operation. To stop the
loop, you must know which ports are involved. Look at the ports with the highest
link utilization (packets per second). The show interface command displays the
utilization for each interface. Make sure that you record this information before
proceeding to the next step. Otherwise, it could be difficult later on to determine the
cause of the loop.
Step 3. Break the loop.
Shut down or disconnect the involved ports one at a time. After you disable or
disconnect each port, check whether the switch backplane utilization is back to a
normal level. Document your findings. Keep in mind that some ports may not be
sustaining the loop but rather are flooding the traffic arriving with the loop. When
you shut down such flooding ports, you only reduce backplane utilization a small
amount, but you do not stop the loop.
Step 4. Find and fix the cause of the loop.
Determining why the loop began is often the most difficult part of the process,
because the reasons can vary. It is also difficult to formalize an exact procedure that
works in every case. First, investigate the topology diagram to find a redundant
path.
For every switch on the redundant path, check for these issues:
Does the switch know the correct STP root?
Is the root port identified correctly?
Are Bridge Protocol Data Units (BPDUs) received regularly on the root port and on
ports that are supposed to be blocking?
Are BPDUs sent regularly on non-root, designated ports?
Step 5. Restore the redundancy.
After the device or link that is causing the loop has been found and the problem has
been resolved, restore the redundant links that were disconnected.
We have only touched lightly on the subject of troubleshooting STP loops.
Troubleshooting loops and other STP problems is complex, and a detailed discussion
is beyond the scope of this course. However, if you want to learn more about
troubleshooting STP problems, an excellent techinical note is available at:

http://cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a008013667
3.shtml#troubleshoot.

Troubleshooting Layer 3 Problems

In most networks, static routes are used in combination with dynamic routing
protocols. Improper configuration of static routes can lead to less than optimal
routing and, in some cases, create routing loops or parts of the network to become
unreachable.
Troubleshooting dynamic routing protocols requires a thorough understanding of
how the specific routing protocol functions. Some problems are common to all
routing protocols, while other problems are particular to the individual routing
protocol.
There is no single template for solving Layer 3 problems. Routing problems are
solved with a methodical process, using a series of commands to isolate and
diagnose the problem.
Here are some areas to explore when diagnosing a possible problem involving
routing protocols:
General network issues
Often a change in the topology, such as a down link, may have other affects on
other areas of the network that might not be obvious at the time. This may include
the installation of new routes, static or dynamic, removal of other routes, and so on.
Some of the things to consider include:
Has anything in the network changed recently?
Is there anyone currently working on the network infrastructure?
Connectivity issues
Check for any equipment and connectivity problems, including power problems such
as outages and environmental problems such as overheating. Also check for Layer 1
problems, such as cabling problems, bad ports, and ISP problems.
Neighbor issues
If the routing protocol establishes an adjacency with a neighbor, check to see if
there are any problems with the routers forming neighbor relationships.
Topology database
If the routing protocol uses a topology table or database, check the table for
anything unexpected, such as missing entries or unexpected entries.

Routing table
Check the routing table for anything unexpected, such as missing routes or
unexpected routes. Use debug commands to view routing updates and routing table
maintenance.

Common NAT Issues

The biggest problem with all NAT technologies is interoperability with other network
technologies, especially those that contain or derive information from host network
addressing in the packet. Some of these technologies include:
BOOTP and DHCP - Both protocols manage the automatic assignment of IP
addresses to clients. Recall that the first packet that a new client sends is a DHCPRequest broadcast IP packet. The DHCP-Request packet has a source IP address of
0.0.0.0. Because NAT requires both a valid destination and source IP address,
BOOTP and DHCP can have difficulty operating over a router running either static or
dynamic NAT. Configuring the IP helper feature can help solve this problem.
DNS and WINS - Because a router running dynamic NAT is changing the relationship
between inside and outside addresses regularly as table entries expire and are
recreated, a DNS or WINS server outside the NAT router does not have an accurate
representation of the network inside the router. Configuring the IP helper feature
can help solve this problem.
SNMP - Similar to DNS packets, NAT is not able to alter the addressing information
stored in the data payload of the packet. Because of this, an SNMP management
station on one side of a NAT router may not be able to contact SNMP agents on the
other side of the NAT router. Configuring the IP helper feature can help solve this
problem.
Tunneling and encryption protocols - Encryption and tunneling protocols often
require that traffic be sourced from a specific UDP or TCP port, or use a protocol at
the Transport layer that cannot be processed by NAT. For example, IPsec tunneling
protocols and generic routing encapsulation protocols used by VPN implementations
cannot be processed by NAT. If encryption or tunneling protocols must be run
through a NAT router, the network administrator can create a static NAT entry for
the required port for a single IP address on the inside of the NAT router.

Troubleshooting Application Layer Problems

The same general troubleshooting process that is used to isolate problems at the
lower layers can also be used to isolate problems at the Application layer. The
concepts are the same, but the technological focus has shifted to involve things
such as refused or timed out connections, access lists, and DNS issues.
The steps for troubleshooting Application layer problems are as follows:
Step 1. Ping the default gateway.
If successful, Layer 1 and Layer 2 services are functioning properly.
Step 2. Verify end-to-end connectivity.

Use an extended ping if attempting the ping from a Cisco router. If successful, Layer
3 is operating correctly. If Layers 1-3 are functioning properly, the issue must exist
at a higher layer.
Step 3. Verify access list and NAT operation.
To troubleshoot access control lists, use the following steps:
Use the show access-list command. Are there any ACLs that could be stopping
traffic? Notice which access lists have matches.
Clear the access-list counters with the clear access-list counters command and try
to establish a connection again.
Verify the access-list counters. Have any increased? Should they increase?
To troubleshoot NAT, use the following steps:
Use the show ip nat translations command. Are there any translations? Are the
translations as expected?
Clear the NAT translations with the clear ip nat translation * command and try to
access the external resource again.
Use the debug ip nat command and examine the output.
Look at the running configuration file. Are the ip nat inside and ip nat outside
commands located on the right interfaces? Is the NAT pool correctly configured? Is
the ACL correctly identifying the hosts?
If the ACLs and NAT are functioning as expected, the problem must lie in a higher
layer.
Step 4. Troubleshoot upper layer protocol connectivity.
Even though there may be IP connectivity between a source and a destination,
problems may still exist for a specific upper layer protocol, such as FTP, HTTP, or
Telnet. These protocols ride on top of the basic IP transport but are subject to
protocol-specific problems relating to packet filters and firewalls. It is possible that
everything except mail works between a given source and destination.
Troubleshooting an upper layer protocol connectivity problem requires
understanding the process of the protocol. This information is usually found in the
latest RFC for the protocol or on the developer web page.