Tshoot SG

TSHOOT
Version 2 ILT
Troubleshooting and
Maintaining Cisco IP
Networks
Copyright Notices
Americas Headquarters Asia Pacific Headquarters Europe Headquarters

Cisco Systems, Inc. Cisco Systems (USA) Pte. Ltd. Cisco Systems International BV
San Jose, CA Singapore Amsterdam,
The Netherlands
Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco Website at
www.cisco.com/go/offices.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To
view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third party trademarks mentioned are the
property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any
other company. (1110R)
DISCLAIMER WARRANTY: THIS CONTENT IS BEING PROVIDED “AS IS.” CISCO MAKES AND YOU RECEIVE NO
WARRANTIES IN CONNECTION WITH THE CONTENT PROVIDED HEREUNDER, EXPRESS, IMPLIED, STATUTORY OR IN
ANY OTHER PROVISION OF THIS CONTENT OR COMMUNICATION BETWEEN CISCO AND YOU. CISCO SPECIFICALLY
DISCLAIMS ALL IMPLIED WARRANTIES, INCLUDING WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT AND
FITNESS FOR A PARTICULAR PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE.
This learning product may contain early release content, and while Cisco believes it to be accurate, it falls subject to the disclaimer
above.
Welcome Students
Students, this letter describes important course evaluation access information!
Welcome to Cisco Systems Learning. Through the Cisco Learning Partner Program, Cisco is
committed to bringing you the highest-quality training in the industry. Cisco learning products are
designed to advance your professional goals and give you the expertise that you need to build
and maintain strategic networks.
Cisco relies on customer feedback to guide business decisions. Therefore, your valuable input will
help shape future Cisco course curricula, products, and training offerings. Please complete a brief
Cisco online course evaluation of your instructor and the course materials in this student kit. On
the final day of class, your instructor will provide you with a URL, directing you to a short post-
course evaluation. If there is no Internet access in the classroom, please complete the evaluation
within the next 48 hours or as soon as you can access the web.
On behalf of Cisco, thank you for choosing Cisco Learning Partners for your Internet technology
training.
Sincerely,
Cisco Systems Learning
© 2014 Cisco Systems, Inc.

Table of Contents
Course Introduction .............................................................................................................. I
Overview ................................................................................................................................... I
Course Goal and Objectives ....................................................................................................... III
Course Flow ............................................................................................................................. IV
Your Training Curriculum ........................................................................................................... V
General Administration .............................................................................................................. VI
Module 1: Tools and Methodologies of Troubleshooting ....................................................... 9
Lesson 1: Describing Troubleshooting Methodologies ...................................................................... 10
What Is Troubleshooting? ......................................................................................................... 12
Diagnostic Principles ................................................................................................................ 13
Troubleshooting Methods.......................................................................................................... 14
Structured Network Troubleshooting .......................................................................................... 15
Common Troubleshooting Approaches ....................................................................................... 17
Top-Down Method ................................................................................................................... 19
Bottom-Up Method .................................................................................................................. 21
Divide-and-Conquer Method ..................................................................................................... 22
Following the Traffic Path......................................................................................................... 23
Performing Comparison ............................................................................................................ 24
Swapping Components ............................................................................................................. 26
Case Study: Troubleshooting Approaches ................................................................................... 28
Summary ................................................................................................................................ 30
Lesson 2: Using Troubleshooting Procedures .................................................................................. 31
Network Troubleshooting Procedures ......................................................................................... 33
Defining the Problem................................................................................................................ 34
Gathering Information .............................................................................................................. 36
Analyzing the Gathered Information ........................................................................................... 38
Proposing and Eliminating Potential Problem Causes ................................................................... 39
Proposing a Hypothesis............................................................................................................. 40
Testing and Verifying a Hypothesis ............................................................................................ 41
Solving and Documenting the Problem ....................................................................................... 42
Case Study: Troubleshooting Procedures .................................................................................... 43
Summary ................................................................................................................................ 49
Lesson 3: Following Recommended Practices During Routine Network Maintenance .......................... 51
Common Maintenance Tasks ..................................................................................................... 53
Troubleshooting as Part of Maintenance...................................................................................... 55
Maintenance Planning............................................................................................................... 56
Change Control ........................................................................................................................ 57
Saving Configurations .............................................................................................................. 59
Restoring Configurations .......................................................................................................... 62
Archiving Configurations .......................................................................................................... 65
Discovery 1: Maintaining and Documenting a Network ................................................................ 67
Labeling Interfaces and Cables .................................................................................................. 83
Documentation ........................................................................................................................ 84
Implementing Time Services ..................................................................................................... 85
Implementing Logging Services ................................................................................................. 86
Creating a Baseline................................................................................................................... 87
Communication ....................................................................................................................... 88
Summary ................................................................................................................................ 90
© 2014 Cisco Systems, Inc. Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT i
Lesson 4: Using Basic IOS Troubleshooting Tools .......................................................................... 91
Layer 2 Switching Process ........................................................................................................ 92
Layer 3 Routing Process ........................................................................................................... 98
Filtering show Commands ....................................................................................................... 101
Filtering show Command Output Using Regular Expressions ...................................................... 104
Redirecting show Command Output to a File............................................................................. 106
Discovery 2: Troubleshooting Connectivity ............................................................................... 108
Basic Hardware Diagnostic ..................................................................................................... 117
Debug Commands .................................................................................................................. 123
Summary .............................................................................................................................. 125
Lesson 5: Using Specialized Troubleshooting Tools ....................................................................... 127
Troubleshooting Tools Overview ............................................................................................. 128
Troubleshooting Tools Categories ............................................................................................ 129
Case Study: Syslog................................................................................................................. 131
Case Study: Troubleshooting with SPAN .................................................................................. 132
Case Study: Troubleshooting with SNMP ................................................................................. 134
Case Study: Netflow ............................................................................................................... 136
Introducing EEM ................................................................................................................... 138
EEM Example: Logging when Configuration Mode Is Entered .................................................... 140
EEM Example: Bring up a Disabled Interface ............................................................................ 141
Summary .............................................................................................................................. 142
Lesson 6: Module Summary ........................................................................................................ 143
Lesson 7: Module Self-Check...................................................................................................... 145
Module Self-Check Answers ....................................................................................................... 150
Answer Key .......................................................................................................................... 150
Module 2: Troubleshooting at SECHNIK Networking Ltd. ............................................... 151
Lesson 1: Debrief of the First Troubleshooting at SECHNIK Networking Ltd. .................................. 152
Trouble Tickets Overview ....................................................................................................... 153
Example Troubleshooting Flow: PC1 Unable to Access Data on the Server .................................. 154
Troubleshooting Trunks .......................................................................................................... 161
Example Troubleshooting Flow: PC2 Unable to Access the Internet ............................................. 163
Troubleshooting NAT ............................................................................................................. 171
Example Troubleshooting Flow: PC3 Unable to Use SSH to Connect to the Server ........................ 174
Troubleshooting Interfaces ...................................................................................................... 180
Example Troubleshooting Flow: PC4 Unable to Access Internet Through IPv6 ............................. 181
Troubleshooting IPv6 Address Assignment on Clients ................................................................ 187
Summary .............................................................................................................................. 189
Lesson 2: Debrief of the Second Troubleshooting at SECHNIK Networking Ltd. .............................. 191
Example Troubleshooting Flow: PC1 Unable to Access the Internet Host ..................................... 193
Troubleshooting Network Layer Connectivity ........................................................................... 204
Example Troubleshooting Flow: PC2 Cannot Use SSH to Connect to Internal Server ..................... 208
TCP Handshake ..................................................................................................................... 220
Example Troubleshooting Flow: PC4 Does Not Acquire IP Address Via DHCP After Port Security Is
Implemented ......................................................................................................................... 223
Troubleshooting Error-Disabled Port ........................................................................................ 231
Summary .............................................................................................................................. 234
Lesson 3: Debrief of the Third Troubleshooting at SECHNIK Networking Ltd. ................................ 235
Example Troubleshooting Flow: PC1 and PC2 Cannot Ping Internet Host..................................... 237
Troubleshooting DHCP .......................................................................................................... 252
Passive Interfaces with Different Routing Protocols ................................................................... 254
Example Troubleshooting Flow: PC3 Cannot Connect to the Internet ........................................... 255
ii Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT © 2014 Cisco Systems, Inc.
IPv6 Review .......................................................................................................................... 260
Summary .............................................................................................................................. 261
Answer Key .......................................................................................................................... 269
Module 3: Troubleshooting at TINC Garbage Disposal Ltd. ............................................. 271
Lesson 1: Debrief of the First Troubleshooting Lab at TINC Garbage Disposal Ltd. .......................... 272
Example Troubleshooting Flow: GW2 Does Not Serve as the Backup to the Internet ..................... 274
Troubleshooting BGP Neighborship ......................................................................................... 285
Example Troubleshooting Flow: PC1 and PC2 Do Not Have Internet Connectivity ........................ 287
Troubleshooting Port Security ................................................................................................. 302
Example Troubleshooting Flow: Classroom PC2 Does Not Have Internet Connectivity .................. 304
Troubleshooting VLANs ......................................................................................................... 315
Troubleshooting Native VLAN ................................................................................................ 317
Summary .............................................................................................................................. 319
Lesson 2: Debrief of the Second Troubleshooting Lab at TINC Garbage Disposal Ltd........................ 321
Example Troubleshooting Flow: GW1 Only Has OSPF Adjacency With GW2 .............................. 323
Troubleshooting OSPF Adjacency............................................................................................ 337
Example Troubleshooting Flow: R2 Is Not Accessible Via SSH Version 2.................................... 341
Troubleshooting Management Access ....................................................................................... 352
Example Troubleshooting Flow: Duplicate IP Addresses on Routers R1 and R2 ............................ 354
Troubleshooting HSRP ........................................................................................................... 363
Summary .............................................................................................................................. 365
Lesson 3: Debrief of the Third Troubleshooting Lab at TINC Garbage Disposal Ltd. ......................... 367
Trouble Ticket Overview ........................................................................................................ 368
Example Troubleshooting Flow: Sporadic Access To Internet ..................................................... 369
Troubleshooting Problems With Routing Sources ...................................................................... 385
Example Troubleshooting Flow: Multiple Masters In A VRRP Group .......................................... 389
Troubleshooting VRRP ........................................................................................................... 397
Example Troubleshooting Flow: Non-Functional EtherChannel ................................................... 400
Troubleshooting EtherChannel................................................................................................. 407
Summary .............................................................................................................................. 411
Lesson 4: Debrief of the Fourth Troubleshooting Lab at TINC Garbage Disposal Ltd. ....................... 413
Example Troubleshooting Flow: Occasional Lack of Network Connectivity For PCs 1 And 2 ......... 415
Troubleshooting GLBP ........................................................................................................... 431
Troubleshooting FHRPs .......................................................................................................... 433
Example Troubleshooting Flow: Sporadic Loss of Connectivity on PC4 ....................................... 435
DHCP Snooping .................................................................................................................... 450
Cisco TAC ............................................................................................................................ 452
Example Troubleshooting Flow: No SSH Connectivity to GW2 From PC4 ................................... 454
Summary .............................................................................................................................. 464
Answer Key .......................................................................................................................... 473
Module 4: Troubleshooting at PILE Forensic Accounting Ltd........................................... 475
Lesson 1: Debrief of the First Troubleshooting at PILE Forensic Accounting Ltd. ............................. 476
Example Troubleshooting Flow: Branch Without Internet Connectivity ........................................ 479
© 2014 Cisco Systems, Inc. Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT iii
Troubleshooting EIGRP Adjacency .......................................................................................... 492
Example Troubleshooting Flow: ISP2 Not Serving as a Backup................................................... 494
Summary .............................................................................................................................. 505
Lesson 2: Debrief of the Second Troubleshooting at PILE Forensic Accounting Ltd. ......................... 507
Example Troubleshooting Flow: PC3 Unable To Remotely Access The Branch Router .................. 510
Example Troubleshooting Flow: No Internet Connectivity .......................................................... 515
BGP Filtering ........................................................................................................................ 531
BGP Transit Area................................................................................................................... 533
Troubleshooting BGP ............................................................................................................. 534
Example Troubleshooting Flow: HQ1 Does Not Synchronize With the Primary NTP Server ........... 535
Troubleshooting NTP ............................................................................................................. 544
Summary .............................................................................................................................. 547
Lesson 3: Debrief of the Third Troubleshooting at PILE Forensic Accounting Ltd. ............................ 549
Example Troubleshooting Flow: Connectivity Issue After Disaster .............................................. 551
Disaster Recovery .................................................................................................................. 569
Troubleshooting Inter-VLAN Routing ...................................................................................... 572
Example Troubleshooting Flow: Connectivity Issue When Using Domain Names ......................... 576
Troubleshooting DNS ............................................................................................................. 585
Remote Device Management Challenges .................................................................................. 587
Summary .............................................................................................................................. 590
Lesson 4: Debrief of the Fourth Troubleshooting at PILE Forensic Accounting Ltd. .......................... 591
Example Troubleshooting Flow: EIGRP Reconfiguration Issue ................................................... 593
EIGRP Named Configuration .................................................................................................. 610
Troubleshooting EIGRP Stub .................................................................................................. 612
Example Troubleshooting Flow: Lack of Management Access .................................................... 614
Providing Default Route On Layer 2 And Multi-layer Devices .................................................... 623
Summary .............................................................................................................................. 625
Lesson 5: Debrief of the Fifth Troubleshooting at PILE Forensic Accounting Ltd. ............................. 627
Example Troubleshooting Flow: Internet Access Via Router HQ0 Does Not Work ........................ 629
Troubleshooting BGP Route Selection ...................................................................................... 652
Example Troubleshooting Flow: PC3 Is Able to Telnet to the Router BR ...................................... 654
Securing the Management Plane .............................................................................................. 661
Summary .............................................................................................................................. 663
Answer Key .......................................................................................................................... 674
Module 5: Troubleshooting at Bank of POLONA Ltd. ...................................................... 677
Lesson 1: Debrief of the First Troubleshooting at Bank of POLONA Ltd.......................................... 678
Example Troubleshooting Flow: Lack of Connectivity ............................................................... 681
Troubleshooting Redistribution ................................................................................................ 693
Example Troubleshooting Flow: Suboptimal Routing ................................................................. 697
Troubleshooting FHRP Tracking ............................................................................................. 705
Example Troubleshooting Flow: IP SLA Does Not Start ............................................................. 707
Troubleshooting IP SLA ......................................................................................................... 714
Summary .............................................................................................................................. 715
Lesson 2: Debrief of the Second Troubleshooting at Bank of POLONA Ltd. ..................................... 717
iv Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT © 2014 Cisco Systems, Inc.
Example Troubleshooting Flow: Incorrect EIGRP Summarization ............................................... 720
Troubleshooting EIGRP Summarization ................................................................................... 727
Example Troubleshooting Flow: IPv4/IPv6 Internet Access Issue ................................................ 729
Troubleshooting Basic RIPng .................................................................................................. 743
Example Troubleshooting Flow: Internet Connectivity Lost ........................................................ 745
Troubleshooting Access Lists .................................................................................................. 753
Summary .............................................................................................................................. 755
Lesson 3: Debrief of the Third Troubleshooting at Bank of POLONA Ltd. ....................................... 757
Example Troubleshooting Flow: Branch 1 Cannot Reach Headquarter.......................................... 760
Troubleshoot GRE Tunnels ..................................................................................................... 772
Example Troubleshooting Flow: Route Summarization From Branch 3 Does Not Work ................. 774
Troubleshoot OSPF Summarization.......................................................................................... 783
Example Troubleshooting Flow: AAA Does Not Work on the Router BR1 ................................... 785
Troubleshoot AAA ................................................................................................................. 791
Summary .............................................................................................................................. 794
Lesson 4: Debrief of the Fourth Troubleshooting at Bank of POLONA Ltd. ...................................... 795
Example Troubleshooting Flow: PC0 Does Not Have Connectivity to IPv6 Internet Sites ............... 798
Troubleshooting OSPF For IPv6 .............................................................................................. 816
Example Troubleshooting Flow: Totally Stubby Area on the Branch Is Not Working ..................... 819
Troubleshooting OSPF Stubby Areas........................................................................................ 829
Summary .............................................................................................................................. 831
Answer Key .......................................................................................................................... 841
Module 6: Troubleshooting at RADULKO Transport Ltd. ............................................... 843
Lesson 1: Debrief of the First Troubleshooting at RADULKO Transport Ltd. ................................... 844
Example Troubleshooting Flow: A Layer 2 Loop in the Network ................................................. 847
Troubleshooting Spanning-Tree Protocol .................................................................................. 851
Example Troubleshooting Flow: Configuring Route-Map Causes Local Connectivity Issue ............ 856
Troubleshoot Policy Based Routing .......................................................................................... 863
Example Troubleshooting Flow: CDP Neighboring Issue ............................................................ 865
Troubleshooting CDP And LLDP ............................................................................................ 868
Summary .............................................................................................................................. 870
Lesson 2: Debrief of the Second Troubleshooting at RADULKO Transport Ltd. ............................... 871
Example Troubleshooting Flow: PC1 and PC2 Do Not Have Connectivity to the Internet ............... 874
Troubleshooting VTP ............................................................................................................. 884
Example Troubleshooting Flow: BR Does Not Have Connectivity to the Internet via IPv6 .............. 886
Troubleshooting EIGRP for IPv6 ............................................................................................. 894
Example Troubleshooting Flow: IPv6 BGP Is Not Established to the ISP2 .................................... 895
Troubleshooting MP-BGP ....................................................................................................... 900
Summary .............................................................................................................................. 902
Lesson 3: Debrief of the Third Troubleshooting at RADULKO Transport Ltd. .................................. 903
Example Troubleshooting Flow: Lack of Connectivity ............................................................... 906
Troubleshooting OSPFv3 Address Families Feature ................................................................... 912
Example Troubleshooting Flow: Authentication Problem ............................................................ 914
Summary .............................................................................................................................. 919
© 2014 Cisco Systems, Inc. Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT v
Lesson 4: Debrief of the Fourth Troubleshooting at RADULKO Transport Ltd. ................................ 921
Example Troubleshooting Flow: External OSPF Routes on Router DST ....................................... 924
Example Troubleshooting Flow: PC1 and PC2 Cannot Access the Internet via IPv6 ....................... 931
Summary .............................................................................................................................. 937
Answer Key .......................................................................................................................... 946
vi Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT © 2014 Cisco Systems, Inc.
© 2014 Cisco Systems, Inc. Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT vii
Course Introduction
Overview
Troubleshooting and Maintaining Cisco IP Networks (TSHOOT) v2.0 is an instructor-led training course
presented by Cisco training partners to their end customers. This five-day course help network professionals
improve the skills and knowledge that they need to maintain their network and to diagnose and resolve
network problems quickly and effectively. It also assists the network professional in preparing for Cisco
CCNP R&S® certification. This course is one of the three courses in the CCNP Routing and Switching
curriculum.
© 2014 Cisco Systems, Inc. Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT 1
Learner Skills and Knowledge
Students should have the knowledge of and experience with the implementation and verification of
enterprise routing and switching technologies as offered by the Implementing Cisco Switched Networks
(SWITCH) and Implementing Cisco IP Routing (ROUTE) courses or equivalent skills and knowledge. This
includes knowledge and experience of the following technologies:
 Layer 2 switching
− VLANs, VLAN access control lists, port security
− Switch security issues
 Link aggregation protocols
 Spanning Tree Protocol (STP)
 Inter-VLAN routing solutions
 First Hop Redundancy Protocols (FHRPs) - HSRP, VRRP, and GLBP
 Branch office operations
 Enhanced Interior Gateway Routing Protocol (EIGRP)
 Open Shortest Path First (OSPF)
 Layer 3 path control
 Redistribution
 Internal and External Border Gateway Protocol (BGP)
 IPv6 Networking
2 Version 2 ILT Troubleshooting and Maintaining Cisco IP Networks TSHOOT © 2014 Cisco Systems, Inc.
Course Goal and Objectives
Upon completing this course, you will be able to isolate and fix routing and switching issues.
Course Flow
The schedule reflects the approximate structure for this course. This structure allows enough time for the
instructor to present the course information and for you to work through the lab activities. The exact timing
of the subject materials and labs depends on the pace of your specific class.
At the beginning of this course you will be familiarized with the tools and methodologies that you will need
to quickly find and isolate network issues. Then you will meet five different topologies that reflect real-life
networks. You will sharpen your troubleshooting skills on multiple different scenarios by bringing together
troubleshooting tools, methodologies, and the knowledge of routing and switching. And just like with real-
life cases, you will not be told what kind of issue is hampering the network performance. For example, a
customer reports that "Internet is not working". What does that mean? Is this a VLAN issue? Routing
protocol issue? Perhaps a security problem? After fixing the issue the debriefs will be provided, describing a
sample troubleshooting flow. Very often an expansion topic with similar troubleshooting cases will be
presented.
Module 6 (Troubleshooting at RADULKO Transport Ltd.) is optional. Students will have access to the
materials after the course is finished.
Your Training Curriculum
Cisco provides three levels of general certifications for IT professionals with several different tracks to meet
individual needs. There are many paths to Cisco certification, but only one requirement—passing one or
more exams demonstrating knowledge and skill. For details, go to
http://www.cisco.com/web/learning/certifications.
You are encouraged to join The Cisco Learning Network, a discussion forum open to anyone holding a
valid Cisco Career Certification. It provides a gathering place for Cisco certified professionals to share
questions, suggestions, and information about Cisco Career Certification programs and other certification-
related topics. For more information, visit https://learningnetwork.cisco.com.
General Administration
The instructor will discuss the following administrative issues so that you know exactly what to expect from
the class:
 Sign-in process
 Start and anticipated end times of each class day
 Class break and lunch facilities
 Appropriate attire during class
 Materials you can expect to receive during class
 What to do in the event of an emergency
 Location of the restrooms
 How to send and receive telephone and fax messages
Student Introductions
Module 1: Tools and
Methodologies of
Troubleshooting
Introduction
Smooth network operation and network high availability are crucial to organizations. Unplanned downtime
can quickly lead to loss of productivity and, therefore, money. To maximize the availability of the network,
it is important to plan network maintenance processes and procedures carefully. Troubleshooting is a part of
network maintenance that every network engineer needs to be capable of performing.
In complex environments, troubleshooting can be a daunting task, and the only way to diagnose and resolve
problems quickly and effectively is by following a structured approach. This approach involves having well-
defined and documented troubleshooting procedures and aligning troubleshooting procedures to general
network maintenance procedures.
Additionally, troubleshooting can be a very time-consuming process. The use of tools built into the Cisco
IOS Software can help shorten the time to diagnose and resolve problems quickly. There are many
technologies and protocols that can be leveraged in combination with specialized tools and applications to
support troubleshooting and maintenance processes
Upon completing this module, you will be able to:

• Describe troubleshooting methodologies
• Describe troubleshooting procedures
• Describe the fundamental subprocesses of the troubleshooting process
• Describe how basic Cisco IOS tools can be used to troubleshoot network issues
• Describe how to Syslog, SPAN and RSPAN, Netflow, SNMP and EEM can be used to troubleshoot
network issues
Lesson 1: Describing
Troubleshooting
Methodologies
Overview
In an ideal world, things always work and problems never happen. However, in reality, people make
mistakes and devices break. Diagnosing and resolving problems is an essential skill that network engineers
use as a part of their many different job tasks.
There are no recipes for troubleshooting. A particular problem can be diagnosed and sometimes even solved
in many different ways. However, by employing a structured approach to the troubleshooting process, you
can greatly reduce the average amount of time that it takes to diagnose and solve a problem.
There are many different structured troubleshooting approaches. For some problems, a specific method
might work better, while for others, a different method may be more suitable. Therefore, the toolbox of a
troubleshooter should contain various structured approaches that a network engineer can choose from to
select the best method or combination of methods for a particular problem.
Upon completing this lesson, you will be able to:

• Identify what troubleshooting is
• Identify diagnostic principles
• Identify troubleshooting methods
• Describe the advantages of a structured network troubleshooting method
• List common troubleshooting approaches
• Evaluate and assess the top-down troubleshooting method
• Evaluate and assess the bottom-up troubleshooting method
• Evaluate and assess the divide-and-conquer troubleshooting method
• Evaluate and assess the follow-the-traffic-path troubleshooting approach
• Evaluate and assess the comparing-configurations troubleshooting approach
• Evaluate and assess the swap-component troubleshooting approach
• In an example troubleshooting case, discuss which troubleshooting method would be most appropriate
What Is Troubleshooting?
Troubleshooting is a process of responding to a problem that leads to the diagnosis and its resolution.
In general, a troubleshooting process starts when someone reports a problem. In a way, you could say that a
problem does not exist until it is noticed, considered a problem, and reported. You need to differentiate
between a problem, as experienced by the user, and the cause of that problem.
Consequently, the time that a problem was reported is not necessarily the same as the time at which the
event that caused that problem occurred. Another consequence is that the reporting user generally equates
the problem with the symptoms while the troubleshooter equates the problem with the root cause.
If the Internet connection fails on Saturday in a small company, is that a problem? Probably not, but you can
be sure that it will turn into a problem on Monday morning if it is not fixed by then.
Although this distinction between symptoms and the cause may seem philosophical, it is good to be aware
of the potential communication issues that can arise.
A troubleshooting process starts with reporting and defining a problem. It is followed by the process of
diagnosing the problem. During this process, information is gathered, the problem definition is refined, and
possible causes for the problem are proposed. Eventually this process should lead to a diagnosis of the root
cause of the problem.
When the root cause has been found, possible solutions need to be proposed and evaluated. After the best
solution is selected, that solution should be implemented. In some cases, the solution cannot immediately be
implemented, and you will need to propose a workaround until the actual solution can be implemented. The
difference between a solution and a workaround is that a solution resolves the root cause of the problem,
and a workaround only remedies or alleviates the symptoms of the problem.
Once the problem is fixed, it is essential that any changes are well documented. This information will be
helpful next time someone needs to resolve similar issues.
Diagnostic Principles
Although problem reporting and resolution are definitely essential elements of the troubleshooting process,
most of the time is spent in the diagnostic phase.
Diagnosis is the process in which you identify the nature and the cause of a problem. These are the essential
elements of this process:
 Gathered information: Gathering information about what is happening is essential to the
troubleshooting process. Usually, the problem report does not contain enough information for you to
formulate a good hypothesis without first gathering more information. You can gather information and
symptoms either directly by observing processes or indirectly by executing tests.
 Analysis: The gathered information is analyzed. You compare the symptoms against your knowledge of
the system, processes, and baselines to separate the normal behavior from the abnormal behavior.
 Elimination: By comparing the observed behavior against expected behavior, you can eliminate
possible problem causes.
 Proposed hypotheses: After gathering and analyzing information and eliminating the possible causes,
you will be left with one or more potential problem causes. You need to assess the probability of each
of these causes, so you can propose the most likely cause as the hypothetical cause of the problem.
 Testing: Test the hypothetical cause to confirm or deny that it is the actual cause. The simplest way to
perform testing is to propose a solution that is based on this hypothesis, implement that solution, and
verify if it solves the problem. If this method is impossible or disruptive, the hypothesis can be
strengthened or invalidated by gathering and analyzing more information.
Troubleshooting Methods
A troubleshooting method is a guiding principle that determines how you move through the phases of the
troubleshooting process.
All troubleshooting processes include the elements of gathering and analyzing information, eliminating
possible causes, and formulating and testing hypotheses. However, the time one spends on each of those
phases, and how one moves from phase to phase, can be significantly different from person to person and is
a key differentiator between effective and less-effective troubleshooters.
In a typical troubleshooting process for a complex problem, you would continually move between the
different processes: gather some information, analyze it, eliminate some possibilities, gather more
information, analyze again, formulate a hypothesis, test it, reject it, eliminate some more possibilities, gather
more information, and so on.
If you do not use a structured approach but move between the phases randomly, you might eventually find
the solution but the process will be very inefficient. In addition, if your approach has no structure, it is
practically impossible to hand it over to someone else without losing all the progress that was made up to
that point. You also may need to resume your own troubleshooting process.
A structured approach to troubleshooting (no matter what the exact method is) will yield more predictable
results in the end and will make it easier to pick up the process where you left off in a later stage or to hand
it over to someone else.
Structured Network Troubleshooting
Quickly formulating a first hypothesis based on common problem causes and corresponding solutions can
be very effective in the short run.
A troubleshooting method that is commonly deployed by both inexperienced and experienced

troubleshooters is the “shoot-from-the-hip” method, where, after a very short period of gathering
information, the troubleshooter quickly makes a change to see if it solves the problem. This action might
seem like random troubleshooting, but usually the guiding principle for this method is knowing common
symptoms and corresponding causes.
Look at the following example: A user reports a LAN performance problem to you. In 90 percent of similar
problems in the past in this environment, the problem was caused by a duplex mismatch, and the solution
was to configure the switch port for 100 Mb/s full duplex. An obvious thing to do is to quickly verify the
duplex setting of the switch port to which the user connects and to change it to 100 Mb/s full duplex to see
if that fixes the problem.
When it works, this method can be very effective, because very little time is spent on gathering data,
analyzing, and eliminating possible causes. However, the downside is that if it does not work, you have not
come any closer to a possible solution.
Experienced troubleshooters can use this method effectively, and it also can be a useful tool for an
inexperienced troubleshooter. However, the main factor in using this method effectively is knowing when to
stop and switch to a more methodical approach.
A structured troubleshooting method is a guideline that helps you move through the different phases of the
troubleshooting process. The key to all structured troubleshooting methods is the elimination of the causes.
By systematically eliminating possible problem causes, you can reduce the scope of the problem until you
manage to isolate and solve the problem. If it turns out that you lack the knowledge or experience to solve
the problem yourself, you can hand it over as a better-defined problem. So, even if you do not manage to
solve the problem, you will increase the chances that someone else can find and resolve it quickly and
efficiently.
Common Troubleshooting Approaches
Several different structured troubleshooting approaches exist and it depends on the problem, which one to
use.
 Top-down method: Work from the application layer in the OSI model down to the physical layer.
 Bottom-up method: Work from the physical layer in the OSI model up to the application layer.
 Divide-and-conquer method: Start in the middle of the OSI layers (usually the network layer) and then
go up or down, depending on the results.
 Perform comparison method: Compare devices or processes of the network that are operating
correctly to devices or processes that are not operating as expected. Gather clues by spotting significant
differences.
 Follow-the-path method: Determine the path that packets follow through the network from source to
destination and track the packets along the path.
 Swap components method: Physically move components and observe if the problem moves with the
components or not.
Top-Down Method
This method follows the layers of the OSI model, starting from the application layer and moving down to
the physical layer.
The top-down method uses the OSI model as a guiding principle. One of the most important characteristics
of the OSI model is that each layer depends on the underlying layers for its operation. This structure implies
that if you find a layer to be operational, you can safely assume that all underlying layers are fully
operational as well.
For example, if you are researching a problem of a user who cannot browse a particular website and you
find that you can establish a TCP connection on port 80 from this host to the server and get a response from
the server, you can typically draw the conclusion that the transport layer and all layers below must be fully
functional between the client and the server. It is most likely a client or server problem and not a network
problem.
Be aware that, in the example above, it is reasonable to conclude that Layers 1 through 4 must be fully
operational, but this idea is not definitively proved. For example, unfragmented packets might be routed
correctly, while fragmented packets are dropped. The TCP connection to port 80 might not uncover such a
problem.
Therefore, the goal of this method is to find the highest OSI layer that is still working. All devices and
processes that work on that layer or on the layers below it are then eliminated from the scope of your
problem. It might be clear that this method is most effective if the problem is on one of the higher OSI
layers.
The top-down method is one of the most straightforward troubleshooting methods, because problems
reported by users are typically defined as application layer problems, so starting the troubleshooting process
at that layer is an obvious thing to do.
A drawback or impediment to this method is that you need to access the application layer software on the
machine of the client in order to initiate the troubleshooting process. If the software is installed only on few
machines, it might be hard to test properly.
Bottom-Up Method
The bottom-up method follows the layers of the OSI model, starting from the physical layer and moving up
to the application layer.
The bottom-up approach also uses the OSI model as the guiding principle, but this time you start on the
physical layer and work your way up to the application layer. By verifying layer by layer that the network is
operating correctly, you steadily eliminate more and more potential problem causes and narrow the scope of
the potential problems.
For example, if you are researching the problem of a user who cannot browse a particular website, you
would first verify physical connectivity. You would log in to the switch and verify the port status. After
each test or verification step, you would move up through the layers of the OSI model.
A benefit of this method is that all the initial troubleshooting takes place on the network, so access to
clients, servers, or applications is not necessary until later in the troubleshooting process.
Also, the thoroughness and steady progress of this method will give you a relatively high probability of
eventual success or, at the very least, a decent reduction of the problem scope.
A disadvantage of this method is that, in large networks, it can be a very time-consuming process, because a
lot of effort will be spent on gathering and analyzing data.
Therefore, the best use of this method is to first reduce the problem scope by using a different strategy and
then switching to this method for clearly bounded parts of the network topology.
Divide-and-Conquer Method
The divide-and-conquer method strikes a balance between the top-down and bottom-up approaches.
If it is not clear whether the top-down or the bottom-up approach would be most effective, it can be helpful
to start in the middle (typically the network layer) and run an end-to-end test such as ping. If the ping
succeeds, you can assume that all lower layers are good, and you can start bottom-up troubleshooting.
Alternatively, if the test fails, you can start a top-down troubleshooting process.
Whether the result of the initial test is positive or negative, this method usually results in a faster elimination
of potential problems than what you would achieve by implementing a full top-down or bottom-up
approach, which makes the divide-and-conquer method a very effective strategy.
Following the Traffic Path
Following the path is one of the most basic troubleshooting methods, and it usually complements one of the
other troubleshooting methods, like the top-down or bottom-up approaches.
Tracing the path of packets through the network eliminates irrelevant links and devices from the
troubleshooting process.
The objective of a troubleshooting method is to isolate the problem by eliminating potential problem areas
from the scope of the troubleshooting process. By analyzing and verifying the path that packets and frames
take through the network as they travel from the source to the destination, you can reduce the scope of your
troubleshooting to just those links and devices that are actually in the forwarding path.
Performing Comparison
Another common troubleshooting method is performing comparison. Comparing the functioning devices or
processes to the malfunctioning ones and spotting the differences can enable you to implement a solution or
a workaround to a problem without even understanding the underlying cause.
By comparing configurations, software versions, hardware or other device properties, links, or processes
between working and nonworking situations and then seeing significant differences between them, you
might be able to resolve the problem by changing the nonoperational situation to be consistent with the
working situation.
The biggest disadvantage of this method is that it can lead to a working situation, but not to an
understanding of the root cause of the problem. In some cases you cannot even be sure if you have
implemented a real solution or only a workaround.
Here is an example. You are troubleshooting a connectivity problem with a branch office router. You have
managed to narrow down the problem to some issue with the default routing, but you cannot seem to find
the cause. You notice that this router is an older type that was phased out in most of the other branch offices.
You have one of the newer types of routers in the trunk of your car, because you plan to install that in
another branch office next week. You decide to copy the configuration of the existing branch router to the
newer router and replace it. Now everything starts to work as expected.
So what do you do? Do you consider the problem fixed? What was the root cause? What should you do with
the old and new routers now?
As you can see, this method has a number of drawbacks, but it is still a useful technique because you can
use it even when you lack the background to troubleshoot based on knowledge of the technology.
The effectiveness of this method depends on how easy it is to compare the working and the nonworking
devices, situations, or processes. Having a good baseline of what constitutes the normal behavior on the
network makes it easier to notice the abnormal behavior. Also, the use of consistent configuration templates
makes it easier to see the significant differences between functioning and malfunctioning devices.
Consequently, the effectiveness of this method depends on the quality of the overall network maintenance
process.
Like the follow-the-path method, this approach is best used as a supporting method in combination with
other methods such as top-down or bottom-up troubleshooting.
Swapping Components
An elementary troubleshooting technique is to swap components and observe whether the problem stays,
moves, or disappears.
Another very elementary troubleshooting technique that can be used to isolate a problem is to physically
swap components and see if the problem stays in place, moves with the component, or disappears entirely.
Look at the figure. One approach would be to start gathering data. You can check settings on the laptop,
examine the settings and the statistics on the switch, compare the settings on other laptops and switch ports,
and so on.
However, what if you do not have the passwords for the PC and the switch? The only data that you can
gather is the status of the link LEDs on the switch, the laptops, and the PCs. What can you do? A common
way to at least isolate the problem (if it is not solved outright) is to start swapping cables:
 Swap the cables that are connected to switch port 2 to one of the other ports, such as port 3. If the link
LED on port 2 stays off and port 3 is still on, the problem was not related to the PC or the cable because
they were both changed, and therefore the problem must be on the switch. If you have unused ports on
the switch, you could try to change to one of those ports and see if that fixes the problem.
 If, after swapping the cable from port 2 to port 3, the situation changed and the LED on port 3 is now
off, you can conclude that the problem must be with the cable or the PC, not with the switch. So now
you can swap the cables between the NIC of laptop B and the NIC of PC C. Again, see if the problem
moves. If the LED on port 3 stays off and the LED on port 2 stays on, you must conclude that the
problem is with the cable, because the PCs were swapped and the cable stayed in place. You should
swap the cables and confirm that the problem moves with the cable. Conversely, if the link status
changed after swapping the cable from the NIC of B to the NIC of C, the conclusion must be that the
problem is with the PC, not with the cable.
As you can see, this method allows you to isolate the problem, even if the information that you can gather is
minimal, just by executing simple tests in a methodical way. Even if you do not solve the problem, you have
scoped it to a single element, and further troubleshooting can now be focused on that element. (If you
determined that the problem was with the cable, not the switch or PC, it is unnecessary to obtain the switch
and PC passwords for further examination.)
What are the drawbacks of this method?

 You are isolating the problem to only a limited set of physical elements and you are not gaining any real
insight into what is happening, because you are gathering only very limited, indirect information.
 This method assumes that the problem is with a single component. If the problem is with a particular
combination of elements, you might not be able to isolate the problem correctly.
Case Study: Troubleshooting Approaches
An external financial consultant has come in to help the controller at your company with an accounting
problem. He needs to access the finance server. An account has been created for him on the server, and the
client software has been installed on the consultant laptop. You happen to walk past the office of the
controller and you are called in: “The consultant cannot connect to the finance server.”
You are a network support engineer and have access to all the network devices but not to the servers.
What are the possible approaches for troubleshooting this issue?
This case lends itself to many different approaches, but certain characteristics can help in deciding an
appropriate approach:
 You have access to the network devices but not to the server. Therefore, you will likely be able to
handle the Layer 1–4 problems by yourself, but for the Layer 5–7 problems you will probably have to
escalate to a different person.
 You have access to the client device, so it is possible to start troubleshooting from it.
 The controller has the same software and access rights on his machine, so it is possible to compare
between these two devices.
What are the benefits and the drawbacks of each of the possible approaches?
 Top-down: You have the opportunity to start testing at the application layer. It is good troubleshooting
practice to confirm the reported problem, so starting from the application layer is an obvious choice.
The only possible drawback is that you will not discover simple problems, such as the cable being
plugged into a wrong outlet, until later in the process.
 Bottom-up: A full bottom-up check of the whole network is not a very useful approach because it takes
too much time, and at this point there is no reason to assume that the network beyond the first access
switch is causing the issue. You could consider starting with a bottom-up approach for the first stretch
of the network, from the consultant laptop to the access switch, to uncover the potential cabling
problems.
 Divide-and-conquer: This approach is very viable. Ping from the consultant laptop to the finance
server. If that succeeds, the problem is more likely to be with the application (although you have to
consider the potential firewall problems as well). If the ping fails, you are definitely dealing with a
network issue and you are responsible for fixing it. The advantage of this method is that you can quickly
decide on the scope of the problem and whether escalation would be necessary or not.
 Follow-the-path: Similar to the bottom-up approach, a full follow-the-path approach is not efficient
under the circumstances, but tracing the cabling to the first switch can be a good start if it turns out that
the link LED is off on the consultant PC. This method might come into play after other techniques have
been used to scope the problem down to where it seems to be an issue somewhere farther up the path in
the network.
 Perform comparison: You have access to both the controller PC and the consultant laptop, so
comparing them is a possible strategy. However, because these machines are not under the control of a
single IT department, there will be many differences, and therefore it might be hard to notice the
significant differences. This method can be useful later in the process, after it has been determined that
the problem is likely to be on the client.
 Swap components: Also known as the move-the-problem approach. Using this approach alone is
unlikely to solve the problem, but if following any of the other methods indicates a potential hardware
issue between the consultant PC and the access switch, this method may come into play. Also, as a first
step you could consider swapping the cable that is connected to the consultant laptop and the controller
PC to establish whether the problem is related to the laptop or not.
Many combinations of these different approaches could be considered. The most likely approaches are top-
down and divide-and-conquer, possibly switching to follow-the-path or performing comparison after the
scope of the problem has been properly reduced. As an initial step in any approach, the move-the-problem
approach could be used to quickly separate client-related issues from network-related issues, or the bottom-
up approach could be used to verify the first stretch of cabling.
Summary
This topic summarizes the key points that were discussed in this lesson.
Lesson 2: Using
Troubleshooting Procedures
Overview
It is impossible to write out a set of troubleshooting procedures that will solve any problem. The
troubleshooting process can be guided by structured methods, but the exact steps you should take at each
point along the way cannot be prescribed because they depend on many different factors. Each network is
different, each problem is different, and the skill set and experience of each engineer who is involved in a
troubleshooting process are different.
However, to guarantee a certain level of consistency in the way that problems are diagnosed and solved in
an organization, it is still important to evaluate the common subprocesses that are a part of the
troubleshooting process and to define the procedures that outline how certain elements of these processes
should be handled.
This lesson reviews the generic troubleshooting process and its subprocesses: defining a problem, gathering
information, analyzing the information, eliminating possible problem causes, formulating a hypothesis
about the likely cause of the problem, testing that hypothesis, and solving the problem. This lesson analyzes
the typical actions and decisions that you should take during each of those subprocesses and describes how
you should plan and implement troubleshooting procedures.

• Identify the subprocesses of the generic troubleshooting process
• Clearly define the problem
• Gather information in a structured manner
• Interpret and analyze the gathered information
• Isolate a problem through a process of elimination
• Formulate a hypothesis and evaluate the necessary actions to take after you have formulated a
hypothesis
• Test a hypothesis and restart the process if a hypothesis is not confirmed
• Integrate a solution into the existing network
• Using an example issue, discuss the troubleshooting procedure flow
Network Troubleshooting Procedures
Each time when you troubleshoot, you will have to go through major elementary processes.
The troubleshooting process can be reduced to a number of elementary subprocesses. These subprocesses
are not strictly sequential in nature, and in many cases you will go through many of these subprocesses
several times before you eventually reach the “solve-the-problem” phase.
A troubleshooting method provides a guiding principle that helps you move through these processes in a
structured way.
There is no recipe for troubleshooting. Every problem that you will be faced with will be different, and it is
impossible to create a script that will solve all the possible problem scenarios. Troubleshooting is a skill that
you acquire through practice. By practicing different methods, you will become more effective at selecting
the right methods for the problem, gathering the most relevant information, and analyzing problems quickly
and efficiently.
Although a complete script of troubleshooting actions and decisions cannot be created, you can have a
closer look at each of the subprocesses and look at the best common practices that apply to each of those
processes.
Defining the Problem
Usually, you receive a written or verbal description of a mix of different aspects of the problem. This
description can include a symptom (“When I try to go to this location on the intranet, I get a page that says I
do not have permission”); a partial diagnosis (“The mail server is not working”); or a consequence for the
user (“I cannot file my expense report”). To prevent wasting a lot of time, based on false assumptions
during the troubleshooting process, you must clearly define the problem.
The first step is to verify that the problem report is an accurate description of the problem and that the
reported condition still exists. Sometimes a problem is intermittent in nature, and troubleshooting a problem
when it is not occurring is nearly impossible. A good problem description consists of accurate descriptions
of the symptoms, not interpretations or conclusions. Strictly speaking, consequences for the user are not part
of the problem description itself but can help you assess the urgency of the issue.
When a problem is reported as “The mail server is not working,” this description is not very useful. What
does it mean when someone says that the mail server is not working? Does it mean that the person just
walked into the server room and noticed that all LEDs were off on the server? Does it mean that a ping to
the server is unsuccessful from anywhere? Or does it mean that the mail client reports that the server cannot
be reached?
In a situation like that, you want to get back to the person who reported the problem and ask what it means
when the report says that the mail server is not working. What is not working? Can you tell me or show me
what you are doing and how it is not working? A better problem definition for the mail server problem
could be as follows: “When user X starts his email client he gets an error message saying that the client
cannot connect to the server. He can still access his network drives and browse the Internet.”
After you have created a ticket that clearly states what needs to fixed, you have one more step to take before
starting the actual troubleshooting process. You need to determine whether this problem is your
responsibility or does it need to be escalated to another department or person.
What if the problem definition is: “When user Y tries to access the corporate directory on the company
intranet, she gets a message saying that the permission is denied. She can access all other intranet pages?
You are a network engineer and you do not have access to the servers. A separate department in your
company manages the intranet servers. What do you do when this problem is reported to you as a network
problem? Do you start troubleshooting or do you escalate it to the server department?
As a part of the definition of your troubleshooting procedures, you need to answer these questions:
 Which types of problems are your responsibility to act on?
 Which minimal actions do you need to take before you escalate a problem?
 How do you escalate a problem?
Gathering Information
The first thing to do, after you have defined the problem and determined that it is your responsibility, is to
start gathering more information about the problem. But which information do you need? Just randomly
gathering information and hoping that at some point a hypothesis might materialize is not very effective.
Before gathering information, you should select your initial troubleshooting method and develop an
information-gathering plan. The information that you gather will be used as the input for the troubleshooting
approach that you select.
As a part of this plan, you need to identify what are the targets for the information-gathering process. From
which network devices, clients, or servers do you want to get the information? Which corresponding tools
do you intend to use to gather that information?
When you have identified your targets and the tools that you intend to use to gather the information, you
must acquire access to these targets. In many cases, you might have access to these systems as a normal part
of your job role, but in some cases you might need to get information from systems that you cannot
normally access. You might need to escalate the issue to a different department or person, either to obtain
access or to get someone else to gather the information for you.
If the escalation process would slow the procedure down and the problem is urgent, you might want to
reconsider the troubleshooting method that you selected and first try a method that uses different targets and
would not require you to escalate.
For example, consider the following issue: The sales manager at your company reports that he cannot send
or receive email from one of the branch offices, from which he is working this morning. His request is
urgent, because he needs to send out an important email later in the afternoon. Your initial thought is to start
a top-down troubleshooting method by calling him and running a couple of tests. However, he does not
answer his phone, and when you check his calendar, it turns out that he is in a meeting until 4:30 p.m.
(16:30). One of your colleagues in that branch office confirms that the sales manager is in a meeting, but he
left his laptop on his desk. The customer needs to receive the email before 5:00 p.m. (17:00). It is 1:00 p.m.
(13:00). What do you do?
Even though a top-down troubleshooting approach might be the obvious choice, it would mean that you
need to wait until 4:30 p.m. (16:30) before you can start troubleshooting, because you will not have access
to the laptop before then. Waiting that long would put you under a lot of pressure to solve the problem in a
half hour. In this case you could consider a combination of the bottom-up and follow-the-path approaches to
verify that there are no Layer 1–3 problems between the laptop and the mail server of the company. Even if
you do not find an issue, you have eliminated many potential problem causes, and when you start a top-
down approach at 4:30 p.m. (16:30) you will be able to work more efficiently.
Analyzing the Gathered Information
After gathering information from various devices and tools, the next step is to interpret and analyze the
information.
To interpret the raw information that you have gathered (for example, the output from show commands and
debugs, packet captures, and device logs), you might need to research commands, protocols, and
technologies. You might also need to consult network documentation to be able to interpret the information
in the context of the actual network implementation.
During the analysis of the gathered information, you typically try to determine two things: what is
happening on the network, and what should be happening. If you discover differences between these two,
they will usually give you clues about what is going wrong or at least a direction for further information
gathering.
Your image of what is actually happening will mostly be formed based on interpretation of the raw data,
supported by research and documentation. But how do you know what should have been happening? You
need to have a good understanding of the operational processes of protocols and technologies. So, if you are
troubleshooting protocols and technologies that you are not very familiar with, you need to invest some time
in researching how they operate under ideal conditions.
The second way to know what should have been happening under normal circumstances is by having a good
baseline of the behavior of your network. If you know how your network performs and how things work
normally, this information will allow you to spot abnormalities in the behavior of the network and derive
clues from those abnormalities. As part of a proactive network maintenance approach, it is important to
compile a baseline of the behavior of your network that you can refer to while troubleshooting.
Finally, in this phase of the troubleshooting process, experience is important. An experienced network
engineer needs to spend less time than an inexperienced engineer on researching processes, interpreting raw
data, and distilling the relevant information.
Proposing and Eliminating Potential Problem
Causes
After you have interpreted and analyzed the information that you have gathered, you can start to draw
conclusions from the results.
After analyzing the collected information, you might discover clues that point toward certain issues that
could be causing the problem. These clues add items to your list of potential problem causes. For example,
you might have observed high CPU loads on your multilayer switches, which might point to a bridging
loop. On the other hand, you might have observed behavior that rules out potential problem causes. For
example, you successfully pinged the default gateway from the client, ruling out Layer 2 problems between
the client and the default gateway.
Although the elimination process seems to be a rational, scientific procedure, you need to be aware that
assumptions also play a role in this process. You need to be willing to go back and re-examine and verify
your assumptions. If you do not re-examine and verify, you might end up eliminating the actual root cause
of the problem as a potential cause and therefore be unable to solve the problem.
Proposing a Hypothesis
After you propose and eliminate potential problem causes, you now have a list of those causes. Based on
experience, you might even be able to assign a certain measure of probability to each of these causes.
If there are many different possible problem causes and none of them clearly stands out as the most likely
cause, you might need to go back and gather more information and eliminate more problem causes before
you can propose a good problem hypothesis.
After you have reduced the list of potential causes to just a few, ideally just one, that emerge as the most
likely causes, select one of these as your problem hypothesis. You will assume that this is the cause of the
problem. Attempt to solve the problem, based on this assumption.
You will now need to reassess whether the proposed problem cause is your responsibility. If the issue that
you are proposing as your hypothesis causes the problem, is it your responsibility to solve it, or will you
need to escalate it to some other person or department?
If you decide to escalate the problem, you should ask yourself if this action ends your involvement in the
process, because the escalation by itself does not immediately solve the problem. How long do you think it
will take the other person to solve the problem? And how urgent is this problem? Can you afford to wait for
someone else to fix it?
If you cannot solve the problem, but it is too urgent to wait for the problem to be solved through an
escalation, you might need to come up with a workaround—a temporary fix that alleviates or remedies the
symptoms that the user experiences, even if it does not address the root cause of the problem.
Testing and Verifying a Hypothesis
After you have proposed a hypothesis about the cause of the problem, the next step is to come up with a
possible solution to that problem and start planning the implementation of this solution.
Usually, implementing a possible solution involves making changes to the network. Therefore, if your
organization has defined procedures for regular network maintenance, you must follow those regular change
procedures.
The next step is to assess the impact of the change on the network and balance that against the urgency of
the problem. If the urgency outweighs the impact and you decide to go ahead with the change, it is
important to ensure that you have a way to revert to the original situation after you make the change. Even
though you have determined that your hypothesis is the most likely cause of the problem and your solution
is intended to fix it, you can never be entirely sure that your proposed solution will actually solve the
problem. If it does not solve it, you need to have a way to undo your changes and revert to the original
situation.
After you have created a rollback plan, implement your proposed solution according to the change
procedures and verify that it has solved the problem. You should verify that the change that you made really
did what you expected it to do, that it solved the root cause and problem symptoms, and that it has not
introduced any new problems.
After you verify that your hypothesis was correct and the problem symptoms have disappeared, you can
move to the final stage of the troubleshooting process.
But what do you do if it turns out that the problem was not fixed, the symptoms did not disappear, or new
problems have been introduced by the change that you made? In those cases, you should execute your
rollback plan, revert to the original situation, and resume the troubleshooting process. If the problem was
not fixed, a good first step is to determine whether the root cause hypothesis was invalid or if it was simply
the proposed solution that did not work.
Solving and Documenting the Problem
When you confirm your hypothesis and verify that the symptoms have disappeared, you have essentially
solved the problem. Now, all you need to do is to make sure that the changes are integrated into the regular
implementation of the network and that the maintenance procedures, that are associated with the changes
that you made, are executed.
You need to create backups of any elements that were changed, such as device configurations and software.
Document all changes to make sure that the network documentation still accurately describes the current
state of the network. In addition, you need to perform any other actions that are prescribed by the regular
change control procedures that are in use on the network.
Finally, you need to communicate that the problem has been resolved and close the ticket. At a minimum,
you must communicate back to the original user who reported the problem, but if you have involved others
as part of an escalation process, you need to update them as well.
For any of the processes and procedures described in this lesson, each organization must make its own
choices regarding how much of these procedures should be described, formalized, and followed. However,
for anyone involved in troubleshooting, it is beneficial to review these processes, compare them to current
troubleshooting habits, and note down lessons learned.
It is very important that you document your solution after solving the problem.
Case Study: Troubleshooting Procedures
This case study presents a scenario where you are taken through all the troubleshooting procedure steps:
defining the problem, gathering information, analyzing the gathered information, eliminating problem
causes, proposing a hypothesis, testing a hypothesis, and solving the problem.
Defining the problem: On your second day at the new job, the phone rings. It is Kimberly from
Accounting. The Internet is not working for her. Since Kimberly is just down the hall, you walk to her. It
turns out she is trying to get to Cisco.com. You successfully replicate the problem. You also find out that
Kimberly was able to access Cisco.com a day ago. Because you are the only IT person at the company, you
will have to try to solve the problem yourself.
Every time before you troubleshoot, make sure that you have an accurate description of the issue. Whenever
possible, also try to replicate the issue.
Gathering information: You first test whether you can access Cisco.com from your own computer. It
works fine. You decide to troubleshoot on the laptop that Kimberly uses by using the bottom-up approach.
First, you check that the laptop has its wireless connection turned on. Then you verify that it has
connectivity to its default gateway and to the resources on the local network. All of these tests are
successful. Ping to an IP address of a public DNS server also works fine. However, browsing to some other
Internet sites also does not work. You try resolving Cisco.com on the computer by using nslookup. The
name resolution does not work.
There is not only one correct way of gathering information. One person might gather all possible
information and only then move to analyze it. Another troubleshooter might come up with a hypothesis
sooner. If the hypothesis proves to be incorrect, this person would then return to gathering information.
Analyzing information: The laptop cannot connect to Cisco.com or any other site, because the DNS
resolution is not working. According to the documentation you received on your first day, the IP address of
the DNS server should be acquired through DHCP.
In a best-case scenario you have good documentation available. In reality, the documentation will be
inaccurate, outdated, or even nonexistent. It is in your interest to create and update it.
Eliminating potential causes: Your first thought is that the IP addresses are manually configured. It turns
out that DHCP on the laptop is turned on.
Proposing a hypothesis: The DNS address in the documentation and the DNS address on the laptop are
different. It might be that last night somebody was tweaking the DHCP configuration, and the DNS server
got mistyped for the Accounting DHCP pool.
Success at proposing a hypothesis that is actually the solution, depends largely on your level of knowledge,
experience, and familiarity with the system.
Testing a hypothesis: You manually change the DNS server IP address on the laptop to the IP address that
is listed in the documentation. Now, Kimberly can access Cisco.com. You change the laptop back to
acquiring the DNS IP address through DHCP.
Solving the problem and documenting the solution: According to the documentation, all user laptops
should acquire IP addresses through DHCP, so manually setting the IP address of the DNS server on the PC
is not an option. You change the DNS address for the Accounting DHCP pool to the correct address and test
that Kimberly can now access Cisco.com and any other site she is eligible to access.
Of course, do not forget to save the configuration of the DHCP server and make sure to document the
changes that you have made.
Summary
Lesson 3: Following
Recommended Practices
During Routine Network
Maintenance
Overview
Network maintenance is one of network administrator’s responsibilities. Network maintenance includes
necessary tasks and operations which keep the network functional with the final intention to fulfill company
business needs. While performing routine maintenance tasks, network administrators sometimes also need
to troubleshoot different problems that might occur during network day to day operations. If you have a
maintenance plan, you may be able to avoid many problems before they occur, thus, reducing network
downtime and improving availability.

• Describe the common maintenance tasks
• Evaluate the benefits gained by aligning the troubleshooting procedures to the network maintenance
procedures
• Describe the advantages of scheduled maintenance
• Implement the change procedures that are flexible enough to support the changes that need to be made
during troubleshooting, but also controlled enough so that changes are integrated into the standard
maintenance and documentation procedures
• Identify and evaluate backup and restore services
• Implement backup and restore services
• Configure archiving of the device configuration
• Discover network devices and how they are connected
• Label cables and add interface descriptions to support and keep the troubleshooting process easier
• Create and update the documentation as part of routine maintenance to support the troubleshooting
process and routinely update documentation as part of the troubleshooting process to keep the
documentation accurate and up-to-date
• Evaluate time services
• Identify and evaluate logging services
• Identify abnormal network behavior through the comparison of actual behavior to a baseline created as
part of the network maintenance process
• Implement communication processes that increase the effectiveness of the troubleshooting process
Common Maintenance Tasks
Essentially, all network maintenance plans need to include procedures that are used to manage maintenance
tasks.
 Minor configuration and cabling changes: Networks are always undergoing changes. As people
move and offices are changed and restructured, network devices such as PCs, printers, and servers may
need to be moved. These tasks are often referred to as “adds”, “moves”, and “changes”, and are a
normal part of network maintenance.
 Installation and configuration of new devices: Even if the implementation of new technologies is
handled by a different group within your organization or by an external party, adding ports, link
capacity, or otherwise upgrading the existing network is usually considered a part of network
maintenance.
 Replacement of failed devices: Whether you manage this task via service contracts or by having spare
equipment on a shelf, as part of your maintenance plan you have to determine how you will handle
equipment replacement when devices fail.
 Backup of device configurations and software: In a way, this item is linked to the previous item—
replacing the failed devices. Without good backups of both software and configurations, the time that it
takes to replace the failed equipment or recover from other severe device failures will be much longer.
 Troubleshooting link and device failures: Inevitably, network components, links, or service provider
connections experience failures. Diagnosing and resolving these failures is an essential part of the job of
a network engineer.
 Software upgrading or patching: You need to stay informed of critical security vulnerabilities and
patch the devices of your organization when the devices are at risk. Regular patching intervals are
advised.
 Network monitoring: Monitoring operation of the devices and user activity on the network is generally
part of a network maintenance plan. This task may be done with very simple tools, such as the
collection of router and firewall logs, or with more complex methods, such as specialized network
monitoring systems.
 Performance measurement and capacity planning: Since the demand for bandwidth is continually
increasing, you have to perform at least some basic measurements to decide when it is time to scale up
your network and to justify the cost of the corresponding investments.
 Writing and updating documentation: It is important to maintain documentation that describes the
current state of the network—for reference during implementation, administration, and troubleshooting
tasks.
Troubleshooting as Part of Maintenance
Troubleshooting is an essential skill that plays an important role in many different types of network
maintenance tasks.
Network maintenance involves many different tasks. For some of these tasks, such as supporting users,
responding to network failures, and disaster recovery, troubleshooting is a major component. Tasks that do
not revolve around fault management, such as adding or replacing equipment, moving servers and users,
and performing software upgrades, also regularly include troubleshooting processes.
To troubleshoot effectively, you are dependent on many processes and resources that are part of the network
maintenance process.
 You need to have access to documentation that is up-to-date and accurate.
 You are dependent on good backup and restore procedures to be able to roll back changes if needed.
 You need to have a good baseline of the network's behavior so you know which conditions are
considered normal in your network.
 You need an access to logs that are properly time-stamped to find out when particular events happened.
So in many ways, the quality of your troubleshooting processes is very dependent on the quality of your
network maintenance processes. Therefore, it makes sense to plan and implement troubleshooting activities
as part of the overall network maintenance process.
Maintenance Planning
With scheduled maintenance, you will reduce the number of planned outages and their duration.
When you have determined the tasks and processes that are part of network maintenance, you can assign
priorities to each of the tasks. You can also determine which of these tasks are interrupt-driven by nature
(hardware failures, outages, etc.), and which tasks are part of a long-term maintenance cycle (software
patching, backups, etc.). For the long-term tasks, you have to work out a schedule that guarantees that these
tasks will be done regularly.
For some tasks, such as adds, moves, and changes, you can adopt a procedure that is partly interrupt-based
(incoming change requests) and partly scheduled. Change requests do not need to be handled immediately,
but can be handled during the next scheduled time frame. This practice allows you to prioritize tasks
properly, while still providing a predictable lead-time during which the requesting party can expect the
problem to be fixed.
Another advantage of a scheduled maintenance cycle is that you can schedule the tasks that are disruptive to
the network during off hours. You can select maintenance windows during evenings or weekends when
outages will be acceptable, thereby reducing unnecessary outages during office hours.
Change Control
Troubleshooting and change processes are strongly related.
Change control is one of the most fundamental processes in network maintenance. By strictly controlling
when the changes are made, and defining what type of authorization is required, and what actions need to be
taken as part of that process, you can reduce the frequency and duration of unplanned outages and thereby
increase the overall uptime of your network.
There is a difference between making a change as part of the maintenance process and making a change as
part of troubleshooting. Most of the actions that you take are the same: you implement the change, verify
that it achieved the desired results, roll back if it did not, back up the changed configurations or software,
document your changes, and so on.
The main differences between regular changes and emergency changes regard the required authorization to
make a change, and the scheduling of the change. When you are creating change control procedures, there is
always an aspect of balancing urgency, necessity, impact, and risk. The outcome of this assessment
determines whether a change can be executed immediately or needs to be scheduled at a later time.
It is uncommon for devices or links to simply fail from one moment to the next. In most cases, problems are
triggered or caused by some sort of change.
 The change can be a simple direct change, such as changing a cable or reconfiguring a setting.
 The change can also be more subtle, like a change in traffic patterns due to the outbreak of a new worm
or virus.
 A problem could be caused by a combination of changes, where the first change is the underlying cause
of the problem, but the problem is not triggered until you make another change. For example, consider a
situation where someone accidentally erases the router software from its flash. This does not cause the
router to fail immediately, because it is running the software from its RAM. However, if that router
reboots due to a short power failure a month later, it will not boot, because of the missing software in
flash. In this example, the underlying cause of the failure is the erased software, but the trigger is the
power failure. This type of problem is very hard to catch, and you will be able to find the underlying
cause or prevent this type of a problem only in the most tightly controlled environments. In this case, a
log of all the privileged EXEC commands that were executed on the router might have revealed that the
software had been erased.
For the troubleshooting process, this means that when a problem is reported, the first question you should
always ask yourself is “Has anything been changed?”. The more thoroughly the changes are reported and
documented on your network, the higher is the chance that you will be able to get an answer to that question
and find the cause or trigger of the problem.
Saving Configurations
One of essential maintenance tasks is to create backups of device configurations. Backup server is a device
where configurations are saved and from which they can be restored. A simple and frequently implemented
service is TFTP.
TFTP usage is simple because it does not require any configuration on network devices. With TFTP you
only need to specify the configuration (startup, running, or from file)—there is no authentication. However,
without authentication capabilities and transferring files in clear text, TFTP is considered insecure protocol.
More secure protocols in use are FTP, SCP, HTTP, and HTTPS.
When you use protocols that require authentication, you must specify the username and the password. The
credentials can be part of configuration, or they can be stated as a part of the URL used with the copy
command.
When using FTP to save running configuration to a server, credentials can be entered with the copy
command, as part of the URL.
You can add the username and the password into configuration and then back up the running configuration.
You will need to enter the username and the password in the global configuration mode using the following
commands: ip ftp username and ip ftp password. This is also possible for HTTP and HTTPS. It is the
same for both HTTP and HTTPS configuration, the only difference is protocol identifier in the URL.
When using SCP, you can use the local user database instead of writing the username and the password in
the command line. For SCP to work, SSH must be configured properly.
R1(config)# username cisco privilege 15 password 0 cisco123

R1(config)# ! SSH must be configured and functioning properly.
R1(config)# crypto key generate rsa
The name for the keys will be: R1.lab.local
Choose the size of the key modulus in the range of 360 to 2048 for your
General Purpose Keys. Choosing a key modulus greater than 512 may take
a few minutes.
How many bits in the modulus [512]: 768

% Generating 768 bit RSA keys, keys will be non-exportable...[OK]
R1(config)# ip ssh time-out 120

R1(config)# ip ssh authentication-retries 5
R1(config)# ip scp server enable
After completing the SSH configuration, you can use an SCP client to securely copy files to and from the
IOS filesystem.
R1# copy flash: scp:

Source filename []? test-scp.txt
Address or name of remote host []? 10.10.10.3
Destination username [Router]? cisco
Destination filename [test-scp.txt]?
Writing test-scp.txt
Password:
!
30 bytes copied in 13.404 secs (2 bytes/sec)
FTP and HTTP require authentication but they send credentials in clear text. HTTPS and SCP are more
secure, using encryption for confidentiality of both credentials and the transferred file.
Restoring Configurations
With a backup configuration you have a point to return to in case of a problem. This is valid either in a case
of a hardware failure, device replacement, or human error. You can directly restore the latest archived
configuration or manually copy a file to device's NVRAM and reboot it to run the configuration that is
saved in the archive.
Restoration of an archived configuration might also be necessary for other reasons. For example you have
unsuccessfully tried to change you configuration through one or more configuration attempts. If they turn
out with unexpected or even undesired results, you can always return to a known working configuration by
restoring it from the archive.
In best case scenario, these changes are performed during maintenance windows and will not disrupt normal
working activities. However, if you are forced to do these changes during the normal working hours and
normal network operation, restoring an archived configuration and then reloading a device might be a
disruptive action, which is only acceptable if no other options are left.
To help manage situations when you need to replace the running configuration that is not working correctly,
but reloading a device is not an option, you can use the configure replace option. This allows you to
replace the currently running configuration on the router with a saved configuration. It compares the
running configuration with the one in the file that you appointed with the configure replace command. It
then creates a list of differences between this two files and generates the appropriate Cisco IOS
configuration commands that will replace existing running configuration with the saved one. Only the parts
of the configuration that are different will be changed. The device will not need to reload. This method will
perform a rollback of an archived configuration with the least possible disruption of the operation.
The configure replace command has some restrictions and cannot be always used to restore a
configuration. A major restriction is that it cannot add or remove commands affecting the existing physical
interfaces in the device's configuration. Another restriction is that the device's free memory space must be
larger than the combined size of the running and the replacing configuration.
In the example, the archive is located at device's flash. A part of a device's configuration is changed and
then the configuration is rolled back to the most current archived configuration. The command option list is
added to show the configuration commands that are being applied during the configuration replacement. As
you can see from the example, the change that was made is undone, without affecting other parts of the
configuration.
Note It is a common practice among network engineers to copy and paste configuration directly
into the console. While this method can be handy to replace parts of configuration it is not
particularly useful when replacing the complete configuration as it adds to, rather than
replaces the current configuration.
Note Although this command was designed with the intention to complement the configuration
archiving feature, the configure replace command can be used with any complete
Cisco IOS configuration file.
Archiving Configurations
You can create an archive by periodically copying device configurations to a chosen destination. You will
usually use a server for creating an archive, but if you have sufficient space in device´s flash memory, you
can also use it for storing the archive—although this is not a recommended practice.
First you need to enter the archive command in the global configuration mode, so you enter the archive
configuration mode. There is one mandatory parameter that you need to set—the base file path. The base
file path is specified by using the URL notation form. It can denote either a local or a network path. There
are two variables that you can use with the path command. $h will be replaced with device hostname. $t
will be replaced with the date and the time of the archive. If you do not use $t, the names of new files will
have the version number appended to differentiate from the previous configurations of the same device.
Note While specifying the path to a local flash storage, check if your flash card is supported.
After specifying the location of the archive, you can create archive copies of the configuration, either
manually or automatically.
 To create an archive copy manually, use the archive config command from EXEC mode.
 To create archived copies automatically, you have to add the write-memory option to the archive
configuration section. This option will trigger the creation of an archived copy of the configuration each
time the running configuration is copied into NVRAM. Alternatively, you can create archived copies
periodically by specifying the time-period option followed by a time period in minutes.
You can list the archives with the show archive command. Notice the version number which is
automatically appended.
Note Archiving function is not available prior to Cisco IOS Release 12.3(7)T.
Discovery 1: Maintaining and Documenting a
Network
Overview
You have just joined a new company and your first job is to document the company's network. You are
asked to investigate and document which devices are in your network and how are they interconnected.
You only have access to one PC. You will have to use Telnet to connect to different devices and discover
how these are interconnected.
After discovering how devices are interconnected, you are asked to shut down the unused ports, and verify
logging and time services.
Note The device access information is a part of network documentation that you were handed and
you can find it in the Job Aids section of this lab.
Topology
Job Aids
Note If you shut down an interface on a real router or switch, the connected device will see it as
"down/down". Due to virtualization specifics, IOL behaviour is slightly different. If you shut
down an interface on a router or switch, the connected device will see it as "up/up". In IOL,
the status of an interface can only be "up/up" or "administratively down/down".
Devices management IP addresses

Device IP address Telnet/enable password
R1 10.10.20.1 cisco/cisco
DSW1 192.168.50.253 cisco/cisco
ASW1 192.168.50.100 cisco/cisco
ASW2 192.168.50.110 cisco/cisco
PC1 192.168.50.10 /
PC2 192.168.50.20 /
PC3 192.168.50.30 /
Maintaining a Documented Network
Step 1 From PC1, connect to R1 using telnet.
PC1# telnet 10.10.20.1

Trying 10.10.20.1 ... Open
User Access Verification
Password:
R1> enable
Password:
R1#
NOTE: This is only an example of how to start the discovery of the network.
You can find the IP address information and credentials for remote access in the Job Aids
section.
Step 2 On R1 issue the show version command to investigate the uptime, the IOS version, the software
release, the hardware, and the value of the configuration register.
R1# show version
Cisco IOS Software, Solaris Software (I86BI_LINUX-ADVENTERPRISE-M), Experimental
Version 15.1(20130726:213425) [dstivers-july26-2013-team_track 106]
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Fri 26-Jul-13 16:24 by dstivers
ROM: Bootstrap program is Linux
R1 uptime is 3 minutes
System returned to ROM by reload at 0
System image file is "unix:/iou_root/iou_images/IOL/i86bi_linux-adventerprise-
ms.july26-201"
Last reload reason: Unknown reason
Linux Unix (Intel-x86) processor with 265566K bytes of memory.

Processor board ID 1032602
4 Ethernet interfaces
16K bytes of NVRAM.
Configuration register is 0x0
The show version command shows the IOS version of the device, the bootstrap software, and
the image version. You can also see the system uptime and why the system has restarted the last
time. For example, it may indicate a normal system startup, or an error which has caused the
system restart. You can also see the information about the hardware of the device such as the
CPU type, the memory size, the installed controllers, and the nonstandard software options. The
last line will show you the value of the configuration register in the hexadecimal format. In
essence, this value tells you how will the system reboot next time. The default value for the
configuration register is 0x2102, which means that the system will attempt to load an IOS image
from flash memory and then load the startup configuration. It will boot to ROM if the initial boot
fails. Values that are different from 0x2102, might indicate a different set of boot options or a
different console set (baud rate).
NOTE: In this lab Cisco IOL (IOS On Linux) is used, so the output is slightly different from the
one you will experience on real Cisco IOS devices.
Step 3 Investigate R1's CDP neighbors.
R1# show cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID

DSW1 Eth 0/1 157 R S I Linux Uni Eth 0/2
R1# show cdp neighbors detail
-------------------------
Device ID: DSW1
Entry address(es):
IP address: 192.168.10.253
Platform: Linux Unix, Capabilities: Router Switch IGMP
Interface: Ethernet0/1, Port ID (outgoing port): Ethernet0/2
Holdtime : 139 sec
Version :
Cisco IOS Software, Solaris Software (I86BI_LINUXL2-ADVENTERPRISEK9-M),
Experimental Version 15.1(20130726:213425) [dstivers-july26-2013-team_track 104]
<... output omitted ...>
The output of the command shows that local interface Ethernet0/1 is connected to a device
named DSW1. Capabilities of this device are route, switch, and IGMP, so it is a multilayer
switch. DSW1 is connected to R1 on the interface Ethernet0/2. Also note IP address
192.168.10.253 of DSW1.
Cisco Discovery Protocol

CDP is a Cisco proprietary protocol that works on the Data Link Layer. It is used primarily for information
sharing between directly connected Cisco equipment. It can also be used for On-Demand Routing to
substitute dynamic routing protocols in simpler networks. The show cdp neighbors command is used to
show information about the directly connected Cisco devices. CDP must be enabled in all devices to be
included in the displayed output of the command. By default, CDP is enabled in Cisco devices. You can
disable it by using the no cdp run command. You should disable CDP on the links that connect to external
networks. You do not want your ISP to see your devices through CDP.
The command output of show cdp neighbors lists the device IDs, the local and the remote ports through
which the devices are connected, the device IOS versions, and capabilites.
The show cdp neighbors detail command displays more information about the neighbors, including the
management addresses.
LLDP is another protocol that is used by network devices to advertise their identity, capabilities, and
neighbors. It performs similarly to CDP, but is vendor neutral and it also identifies the devices of other
vendors. You should use show lldp neighbors to investigate LLDP-enabled devices in a network where
LLDP is configured. By default, LLDP is not enabled on Cisco devices. You can enable it using global
configuration command lldp run.
Step 4 On R1, investigate the ARP table in order to discover IP to MAC mapping of devices in your
network.
R1# show arp

Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.1.254 - aabb.cc00.be20 ARPA Ethernet0/2
Internet 192.168.10.254 - aabb.cc00.be10 ARPA Ethernet0/1.10
Internet 192.168.50.10 2 aabb.cc00.bf00 ARPA Ethernet0/1.50
Internet 192.168.50.100 2 aabb.cc80.bb00 ARPA Ethernet0/1.50
Internet 192.168.50.110 2 aabb.cc80.bc00 ARPA Ethernet0/1.50
Internet 192.168.50.253 2 aabb.cc80.bd00 ARPA Ethernet0/1.50
Internet 192.168.50.254 - aabb.cc00.be10 ARPA Ethernet0/1.50
Internet 209.165.200.2 - aabb.cc00.be00 ARPA Ethernet0/0
NOTE: The ARP table might not have the same entries in your case.
The router has subinterfaces on interface Ethernet0/1 that is connected to DSW1. You can also
see that router´s interfaces Ethernet0/0 and Ethernet0/2 are connected to non-Cisco devices, or to
Cisco devices with disabled CDP protocol, as show cdp neighbors did not discover them.
If a router or a switch communicated with a device, it will retain its IP to MAC address mapping
in its ARP cache until the ARP cache timeout expires. The default ARP cache timeout is 4 hours,
but you can change it using the arp timeout command.
Address Resolution Protocol

ARP is an essential part of IP networks. For a network device to communicate with another network
device, it is necessary to have the Ethernet MAC address for that device. If that device is not in your LAN,
you must go through your default gateway. In that case, your default gateway will have the destination
address that your device will communicate with.
The ARP protocol is used to map your IP addressing to your Ethernet MAC addressing. There are two types
of ARP entries, static and dynamic. Dynamic ARP entries are learned and kept in the so called ARP cache
as long as they are in use. Static ARP entries are entered manually to map IP addresses to Ethernet
addresses.
Step 5 On R1, list interfaces descriptions.
R1# show interfaces description

Interface Status Protocol Description
Et0/0 up up Connection to Internet
Et0/1 up up Connection to DSW1
Et0/1.10 up up
Et0/1.50 up up
Et0/2 up up Connection to Server
Et0/3 admin down down
Lo1 up up
The show interfaces description command will list router's interfaces, their status and
description. Description of an interface explains which device the interfaces is connected to.
These descriptions are configured by the administrator and so they may contain errors.
NOTE: You can also see interfaces descriptions if you issue the show running-config
command.
Step 6 Based on what was discovered so far, verify partial topology of the network.
Step 7 From PC1, connect to DSW1 and investigate its connections.
To investigate device's connections, use the CDP neighbors, device's MAC address table, and the
ARP table feature of Cisco devices.
PC1# telnet 192.168.10.253

Trying 192.168.10.253 ... Open
Password:
DSW1> enable
Password:
DSW1#
DSW1# show cdp neighbors


R1 Eth 0/2 145 R Linux Uni Eth 0/1
ASW1 Eth 0/0 130 R S I Linux Uni Eth 0/2
Directly connected neighbors of DSW1 are R1, ASW1, and ASW2.
DSW1# show mac address-table

Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports

---- ----------- -------- -----
1 aabb.cc00.b510 DYNAMIC Et0/2
Total Mac Addresses for this criterion: 5
The output of the command shows that the switch has learned three MAC addresses from
interface Ethernet 0/2, for which you know it is connected to R1. There are device's connected to
Ethernet0/0 that connects to ASW1. This means that frames with shown source MAC addresses
were flooded from ASW1.
Note that MAC addresses in your lab might be different. The contents of the MAC address table
depend on which MAC addresses were already learned by DSW1.
MAC Address Database

Switches maintain a database of MAC addresses, both manually configured and dynamically learned.
Switches dynamically learn MAC addresses from the source field of the frames entering its ports,
originating either from directly connected devices or from frames that are flooded through the broadcast
domain. When troubleshooting, it may be necessary to investigate the entries in the MAC address table. To
view the whole table, containing all the addresses, use the show mac address-table command. If you want
to list only the dynamically learned addresses, use the show mac address-table dynamic command.
Switches have a default MAC address aging time of 300 seconds. After expiring, entries are removed from
the table. It is important to know that each STP topology change will trigger flushing of the MAC address
table. Frequent appearance and removal of MAC address entries from the MAC address table may indicate a
frequent topology change, probably a flapping link.
Step 8 Document the new identified connections.
It is important that you keep track of what you learned along the way. Document anything that is
valuable for the network documentation of the company. Better documentation means less time
troubleshooting when issues arise. Shorter troubleshooting time means less down time.
Step 9 From PC1, connect to ASW1 and investigate its connection.
PC1# telnet 192.168.50.100

Trying 192.168.50.100 ... Open
Password:
ASW1> enable
Password:
ASW1#
ASW1# show cdp neighbors


There are two neighbors, DSW1 on Ethernet 0/2 and ASW2 on Ethernet 0/3.
ASW1# show mac address-table
Mac Address Table
-------------------------------------------

---- ----------- -------- -----
From the MAC address table, you can see that more than one device is connected to the
Ethernet0/3 interface. CDP has discovered ASW2 connected to this port, so the other MAC
address belongs to a host that is connected to the ASW2. There are devices that CDP did not
discover that are connected to Ethernet0/0 and Ethernet0/1.
However in your lab MAC addresses will probably be different. Also the number of entries in
the MAC address table might be different since it all depends which addresses has ASW1
already learned about.
Step 10 Investigate ASW1's connections.
You know that all PCs belong to the 192.168.50.0/24 subnet, however, you do not know how
they are connected to the switches. If you want to map the MAC address to the IP address, you
should observe the ARP table. Host's MAC address will be in the ARP table of ASW1 only if the
host and ASW1 have exchanged traffic. Ping the subnet broadcast address. Once all devices in
the subnet reply, the ARP table is updated.
In a real network these PCs would be Windows, MAC, or Linux devices. As long these are
turned on, they generate traffic. So on a real network ping to a broadcast IP would not be
necessary.
ASW1# ping 192.168.50.255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.50.255, timeout is 2 seconds:
Reply to request 0 from 192.168.50.254, 2 ms

After about a minute stop the pinging process using the "Ctrl+Shift+6" key combination and then
hitting "X" on your keyboard.
ASW1# show arp
Internet 192.168.50.10 2 aabb.cc00.b600 ARPA Vlan50
Internet 192.168.50.100 - aabb.cc80.b200 ARPA Vlan50
Compare the MAC addresses of these devices with their respective IP mapping in the MAC table
from the previous step. Two PC hosts are connected to the ASW1. The PC with IP address
192.168.50.10 is connected to the Ethernet0/0 interface, and the PC with IP address
192.168.50.20 is connected to the Ethernet0/1 interface.
Step 11 From PC1, connect to ASW2 and investigate its connections.
PC1# telnet 192.168.50.110

Trying 192.168.50.110 ... Open
Password:
ASW2> enable
Password:
ASW2#
ASW2# show cdp neighbors


ASW2 has two neighbors, DSW1 on Ethernet 0/1 and ASW1 on Ethernet 0/3.
ASW2# show mac address-table

Mac Address Table
-------------------------------------------

---- ----------- -------- -----
From the MAC address table, you can see a device that is connected to interface Ethernet0/0.
Again, CDP has not discovered other switches that are connected to that port, so MAC address
belongs to a directly connected host or any other device that does not have CDP capabilities.
Step 12 On ASW2, ping the subnet broadcast address, to update the ARP table. Then, verify ASW2´s
ARP table.
ASW2# ping 192.168.50.255


ASW2# show arp

Internet 192.168.50.110 - aabb.cc80.b300 ARPA Vlan50
Comparing the MAC address of the device on Ethernet0/0 with its respective IP mapping in the
ARP table, you can conclude that a PC host is connected to ASW2. The PC with IP address
192.168.50.30 is connected to the Ethernet0/0 interface.
Step 13 What does your complete network topology look like?
Step 14 Go back to ASW1 and shut down all the unused ports.
The investigation of ASW1's interfaces and connections showed that the only interfaces in use
were Ethernet 0/0, 0/1, 0/2 and 0/3. With show ip interface brief you can get the list of all
interfaces. You should shut down all other interfaces.
ASW1(config)# interface range ethernet1/0-3, ethernet2/0-3, ethernet3/0-3,

ethernet4/0-3, ethernet5/0-3
ASW1(config-if)# shutdown
You can shut down individual interfaces or shut down ranges of interfaces with one command.
To shut down an interface, select the interface from configuration mode and issue the shutdown
command.
It is one of Cisco´s recommended practices to shut down all the unused ports. This is preferred
primarily for security purposes. Some security threats that can take advantage of unsecured ports
to gain unauthorized access to the network resources perform man-in-the-middle attacks, and so
on.
Unused ports on ASW2, DSW1, and R1 are already shut-down.
Step 15 Investigate the configured interfaces descriptions on DSW1.
There are configured interface descriptions on DSW1.
DSW1# show interfaces description

Interface Status Protocol Description
Et0/0 up up
Et0/1 up up
Et0/2 up up
The purpose of interface descriptions is very simple. If you or somebody else looks at your
device's configuration, they should understand the purpose of the network connection and where
it is connected. You can also add other relevant information, such as whom to call or what to do
if that particular interface is down.
Interface descriptions are important part of network documentation and can greatly simplify the
understanding and troubleshooting of a network.
Step 16 Add missing descriptions to interfaces on DSW1.
DSW1(config)# interface Ethernet 0/0

DSW1(config-if)# description Connection to ASW1
DSW1(config-if)# interface Ethernet 0/1
DSW1(config-if)# description Connection to ASW2
DSW1(config-if)# interface Ethernet 0/2
DSW1(config-if)# description Connection to R1
To configure interface description, enter the global configuration mode, enter the configuration
mode of the desired interface, and use the description command to add a description which may
be helpful in identifying the interface and its connection.
Step 17 Verify clock synchronization on all network devices.
Accurate time is essential for logging events in you network. Only when devices are time-
synchronized, you can track the problem from one device to another.
Utilize the Job Aids section to figure out how to telnet to different devices.
DSW1# show ntp status

Clock is synchronized, stratum 9, reference is 192.168.50.254
< ... output omitted...>
R1# show ntp status

ASW1# show ntp status

ASW2# show ntp status

The show ntp status command shows that R1 serves as master NTP for this network. Other
devices DSW1, ASW1, and ASW2 are synchronized to R1 through its IP address
192.168.50.254.
Step 18 Verify logging on R1.
R1# show logging
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes,
0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
Console logging: level debugging, 15 messages logged, xml disabled,

filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 15 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
Trap logging: level informational, 20 message lines logged
Logging Source-Interface: VRF Name:
Log Buffer (4096 bytes):
*Dec 17 12:38:49.774: %SYS-5-CONFIG_I: Configured from memory by console

*Dec 17 12:38:49.804: %SYS-5-RESTART: System restarted --
Cisco IOS Software, Solaris Software (I86BI_LINUX-ADVENTERPRISE-M), Experimental
Version 15.1(20130726:213425) [dstivers-july26-2013-team_track 106]
*Dec 17 12:38:51.528: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Dec 17 12:38:52.534: %LINEPROTO-5-UPDOWN: Line protocol on Interface
Ethernet0/0, changed state to up
*Dec 17 12:58:54.233: %SYS-5-CONFIG_I: Configured from console by console
*Dec 17 12:59:23.055: %LINK-5-CHANGED: Interface Ethernet0/3, changed state to
administratively down
Ethernet0/3, changed state to down
*Dec 17 14:44:01.979: %AMDP2_FE-6-EXCESSCOLL: Ethernet0/1 TDR=0, TRC=0
*Dec 17 14:53:13.704: %AMDP2_FE-6-EXCESSCOLL: Ethernet0/1 TDR=0, TRC=0
R1#
During operation, routers and switches generate different logging messages that are sent to a
logging process. This process is responsible for sending these messages to various destinations,
as directed by device configuration. Logging messages are also sent to the console. Even if
logging process is disabled, logging messages are nevertheless sent to the console. You can
decide about the severity level of the logged messages and their destination.
From the output of the command you can chronologically see the events that have triggered
logging messages.
First, you can see that the syslog logging is enabled. Severity level of logging is at its default—
debugging, which means that all messages are logged.
The logging messages are sent to the console, the monitor, and the memory buffer, which has a
size of 4096 bytes. Altogether, there were 15 types of messages. Timestamps show the time
when each event occurred. System was restarted once. After the restart, the interfaces and the
line protocols changed the state to up. This was logged as a notification message—level 5.
Logging to monitor is off. If you usually access your devices using telnet, you need to enable
logging to monitor with terminal monitor.
Step 19 Reconfigure logging on R1. Set the destination, the severity, and time stamps of the logging
messages.
Set the logging messages to be buffered, and severity level for the logged messages critical.
R1(config)# logging buffered critical
You can also set an external device, such as Syslog server, as a destination for the logs.
R1(config)# logging host 192.168.10.1
By default, log messages do not include timestamps, you need to explicitly configure this
feature.
R1(config)# service timestamps log uptime
Step 20 Verify reconfigured logging on R1.
R1# show logging
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes,
0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
Console logging: level debugging, 13 messages logged, xml disabled,

filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level critical, 0 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
Trap logging: level informational, 16 message lines logged
Logging to 192.168.10.1 (udp port 514, audit disabled,
link up),
2 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging Source-Interface: VRF Name:
Log Buffer (4096 bytes):
NOTE: Remember that after completing a job you need to save configurations on all devices
you made changes to. It is very important to document changes that were made.
Labeling Interfaces and Cables
You will make your maintenance and troubleshooting process much easier if you clearly label cables and
interfaces. A nonlabeled or mislabeled cable can lead to difficulties in identifying a network problem and
the device responsible for the problem.
When labeling cables, try to follow a strict labeling scheme. This way, you reduce the time to identify and
trace cables when troubleshooting a problem.
The labeling scheme must be documented. A copy of documentation should be placed within the wire
cabinet, so you can access it during the maintenance or troubleshooting.
To make your troubleshooting and maintenance easier, use interface descriptions. When you add description
to interfaces, you usually describe where it is connected. You can also add other information that you
consider important during maintenance, such as remote interface, IP address, or even a tech support phone
number to be called if that interface goes down.
Documentation
One of the most essential parts of network maintenance is creating and updating network documentation.
With updated network documentation troubleshooting is easier and less time consuming.
Documentation is first created during the design and implementation of the network. Keeping it updated is a
part of network maintenance plan. A maintenance or change control procedure should therefore include
update of documentation after changes are made.
A typical network documentation includes:

 Network drawings: Diagrams showing physical and logical structure of the network.
 Device interconnections: A spreadsheet or a database listing all physical connections, connection to
service providers and also power circuits.
 Equipment inventory: A document in form of a spreadsheet or a database of all devices, including
their part numbers, serial numbers, software versions, software licenses, and, if applicable, their expiry
dates.
 Addressing scheme: A document listing all IP addresses that are in use.
 Device configurations: Includes current device configurations, and possibly an archive of previous
configurations.
 Design documentation: Describes the implementation solution, and reasoning behind the choices
made.
It can also include a more detailed documents describing network implementation, change procedures, and
so on.
Implementing Time Services
From troubleshooting perspective it is very important that all the all network devices are synchronized so
they all have the correct timestamps in their logging messages.
To obtain correct time, you can use NTP to set up and synchronize clock between network devices. Using
this protocol, your devices connect to previously configured NTP servers. NTP server is synchronized to
another server higher in the hierarchy of this protocol. Position of a device within this hierarchy is called
stratum which is an NTP hop count.
You can set up multiple stratum 1 servers as your primary clock sources in your network, if you have access
to atomic or radio clock. Alternatively you can use public stratum 1 and stratum 2 servers from your service
provider or the internet. It is usual to have redundant NTP servers on your network.
You can configure timeservers by using the ntp server command. You can configure multiple timeservers.
NTP protocol will select the most reliable server among the servers with the lowest stratum number and
synchronize to that server. You may configure the prefered NTP servers with the prefer keyword.
After setting up the NTP servers, the network devices must be set up accordingly to add time properties to
their logging messages.
Implementing Logging Services
During operation, network devices generate messages about different events. These messages are sent to an
operating system process which further proceeds them to the destination.
Default destination for the logging messages is the console, but you can also send logging messages to:
monitor (vty/AUX), memory buffers, SNMP traps, flash memory or external syslog server.
There are 8 levels of severity of logging messages. Levels are numbered from 0 to 7, from most severe to
debugging messages, namely: emergency, alert, critical, error, warning, notification, informational, and
debugging.
By default, system logging is on and the default severity level is debugging, which means that all messages
are logged.
Creating a Baseline
Creating a baseline is important for evaluation and eventual troubleshooting of the network. By creating a
baseline, you create a reference point. Based on the reference point you can decide whether your network is
performing normally.
For instance, a CPU load of 40% is not considered high for a Cisco IOS device. However, if your baseline
value under normal circumstances is 10%, a CPU load of 40% might indicate an unusual behavior of your
network, which needs to be diagnosed and corrected if needed.
Tools which can be used to establish baseline are:
 SNMP: Used for network management and monitoring of network devices. It works by employing an
agent software running on managed-monitored devices and a software which runs on the manager
(NMS).
 NetFlow: A feature on Cisco devices that collects and analyzes traffic entering devices interfaces.
Collected traffic is sent to a traffic collector which is usually a server where traffic statistics is
performed. NetFlow works by analyzing network flows, which are defined by 7 values: Ingress
interface, source IP address, destination IP address, IP protocol, source port, destination port, IP Type of
Service.
 Cisco IP SLA: A network measurement feature. Devices that support IP SLA can perform network
tests or measurements such as FTP downloads, DNS lookups, TCP connections, HTTP GETs etc. IP
SLA can be configured either through a console or via SNMP.
An important part of documenting expected behavior of your network is to note types of interfaces used to
interconnect devices.
Communication
Communication is an essential element of the troubleshooting process.
Elementary phases of the troubleshooting process through communication:

 Define problem: When someone reports a problem, the problem definition is often too vague to act on
it immediately. You need to verify the problem and gather as much information as you can about the
symptoms from the person that reported the problem. Asking good questions and carefully listening to
the answers is essential in this phase. Some of the questions that you can ask during this phase are:
What do you mean exactly when you say that something is failing? Did you make any changes before
the problem started? Did you notice anything special before this problem started? When did it last
work? Has it ever worked?
 Gather information: During this phase of the process, you will often depend on other engineers or
users to gather information for you. You might need to obtain information that is contained in server or
application logs, configurations of devices that you do not manage, information about outages from a
service provider, or information from users in different locations to compare against the location that is
experiencing the problem.
 Analyze information and eliminate potential causes: Interpretation and analysis are mostly solitary
processes, but there are still some communication aspects to this phase. First, you cannot be
experienced in every aspect of networking, so if you find that you are having trouble interpreting certain
results or if you lack knowledge about certain processes, you can ask specialists on your team to help
you out. Also, there is always a chance that you are misinterpreting results, misreading information,
making wrong assumptions, or having other flaws in your interpretation and analysis. A different
viewpoint can often help in these situations, so discussing your reasoning and results with teammates to
validate your assumptions and conclusions can be very helpful.
 Propose and test a hypothesis: Most of the time, testing a hypothesis involves making changes to the
network. These changes might be disruptive, and users might be impacted. Even if you have decided
that the urgency of the problem outweighs the impact and that the change needs to be made, you should
still communicate clearly what you are doing and why you are doing it.
 Solve the problem and document solution: You will need to report back to the person who originally
reported the problem that it has been resolved. Also, you will need to communicate this to the other
people that were involved during the process. And finally, you will need to go through any
communication that is involved in the normal change processes, to ensure that the changes that you
made are properly integrated in the standard network maintenance processes.
Communication process must be effective and well structured. This is especially important during the
troubleshooting operations. The changes that were made, the results of the changes, and the conclusions that
were drawn must be communicated between the team members. Otherwise, they may disrupt the work of
other team members and generate new problems.
If there is a case where troubleshooting and resolution must be done by a team of people or it requires
multiple sessions of work, a good log of performed actions, test results, team communication, and drawn
conclusions is most essential.
Summary
Lesson 4: Using Basic IOS
Troubleshooting Tools
Overview
You need to think about troubleshooting before connectivity issues arise. You need to create a baseline,
write-up documentation, so you know what is normal behavior in your network. This way troubleshooting
will be much easier.
Network Administrators spend a lot of their time troubleshooting the network. The tools that are used for
troubleshooting are capable to generate outputs with a lot of information.
In the process of troubleshooting, one of the challenges is to know how and what to look for in an output
command because you want to check only for specific information that is relevant to the case. This can be
achieved with the Cisco IOS Troubleshooting tools.

• Describe Layer 2 switching process
• Describe Layer 3 routing process
• Apply filtering to Cisco IOS commands to select relevant output
• Apply filtering to Cisco IOS commands using regular expressions to select relevant output
• Apply redirecting of show command output to a file
• Test network connectivity using Cisco IOS commands
• Determine the path that a packet takes from the source to the destination
• Test the transport layer using Telnet
• Diagnose basic hardware-related problems
• Explain how to use the CLI debug commands to show real-time information while actively
troubleshooting a network
Layer 2 Switching Process
A good understanding of Layer 2 Switching is essential with every troubleshooting process. Thorough
knowledge of the core processes performed by hosts and network devices is some of the most important
knowledge that a network engineer can have.
When things break down and devices are not functioning as they should, a good understanding of process
helps you to determine where exactly a process breaks down and, consequently, it helps you determine
which parts of the network are functioning correctly and which parts are not functioning correctly. So this
topic starts by asking a simple question: “What are the processes that take place when two hosts
communicate using IP over a switched LAN?”
Application being used is irrelevant in this context, this lesson uses ping as the sample application, as shown
in the figure.
This process can be broken down into the following steps:
1. Host A checks the destination (Host B) IP address and compares it against its own IP address and mask.
It concludes that Host B is on the same subnet.
2. As Host B is on the same subnet, Host A consults its ARP cache to find the MAC address of Host B. If
the cache contains an entry for Host B, Host A skips the ARP process, encapsulates the IP packet in an
Ethernet frame destined for Host A, and transmits the frame.
3. If the ARP cache on Host A does not contain an entry for the IP address of Host B, Host A sends out an
ARP request as a broadcast (destination address ff:ff:ff:ff:ff:ff) to obtain the MAC address of Host B.
4. Switch C checks the VLAN of the port it receives the frame on, records the source MAC address in its
MAC address table, and associates it to that port and VLAN. As ARP frame is a broadcast frame,
Switch C floods the frame on all ports that belong to the same VLAN, including all trunks that this
VLAN is allowed on. Switches D and E repeat this process as they receive the frame.
5. Host B receives the ARP request. As ARP request frame also includes sender's IP address, Host B
records the IP address and MAC address of Host A in its own ARP cache, and then proceeds to send an
ARP reply as a unicast back to Host A.
6. The switches check the VLAN of the port they received the frame on, and as all switches now have an
entry in their MAC address table for the MAC address of Host A, they forward the frame containing the
ARP reply on the path to Host A only, not flooding it out on any other port. At the same time, they
record Host B’s MAC address and corresponding interface and VLAN in their MAC address table if
they do not already have that entry.
7. Host A receives the ARP reply and records the IP and MAC address of Host B in its ARP cache. Now
Host A is ready to send the original IP packet.
8. Host A encapsulates the IP packet (ICMP echo request) in a unicast frame destined for Host B and
sends it out.
9. The switches again consult their MAC address tables, find an entry for the Host B MAC address, and
forward it on the path toward Host B.
10. Host B receives the packet and responds to Host A with an ICMP echo reply packet.
11. The switches again consult their MAC address tables and forward the frame straight to Host A, without
any flooding.
12. Host A receives the packet. This step concludes this simple packet exchange.
Layer 3 Routing Process
When troubleshooting IP network connectivity issues, Layer 3 is an appropriate layer to begin your
troubleshooting efforts. For that you need a good Layer 3 routing process knowledge.
Understanding this process and knowing it from memory is fundamental to IP routing troubleshooting. If
the source host or any of the routers in the chain are not capable of forwarding packets, because the host or
routers are missing the required information, the packets will be dropped and Layer 3 connectivity is lost.
A sequence of the major processes, decisions, and actions is presented in this topic.
1. When host A needs to send a packet to host B it will start by comparing the destination IP address
(10.1.4.2) to its own IP address and subnet mask (10.1.1.1/24) to determine if the destination is on its
local subnet or not. It concludes that the destination is not local and therefore it will attempt to forward
the packet to its default gateway, which can be either manually configured or learned via DHCP. In
order to encapsulate the packet in an Ethernet frame, it will need the MAC address of the default
gateway. This can be resolved via the ARP. The host will either already have an entry in its ARP cache
for the default gateway IP address, or alternatively, it will send out an ARP request to obtain the
information and populate the ARP cache.
2. Router C decapsulates the received Ethernet frame and examines the IP header of the packet contained
in the frame. It decrements the TTL field in the IP header of the packet by one. If this causes the TTL
field to be set to zero, it will discard the packet and send an ICMP “time exceeded” message back to the
source. If the TTL of the packet is not reduced to zero the router performs a routing table lookup to find
the longest prefix that matches the destination IP address contained in the packet header. In this
example, router C finds the entry 10.1.4.0/24 as the best match for the destination address of the packet
(10.1.4.2). Two important parameters are associated with this entry: The next hop IP address 10.1.2.2
and the egress interface FastEthernet 1. Router performs an ARP cache lookup to find the MAC address
of the next hop router 10.1.2.2. If it finds an entry in the cache it will encapsulate the packet in an
Ethernet frame and forward it to next hop router D. Similar to the process on a host, it will send out an
ARP request if no entry in the cache is found. However, the packet that triggers this ARP request will
be dropped. The router will not keep the packet in a buffer until an ARP reply is received. For any
subsequent packets that need to be forwarded to router D, router C will have an entry in the ARP cache
after it processes the ARP reply and these packets will be forwarded correctly.
3. Router D will go through the same general process as router C.
4. The process on router E is similar to the process on routers C and D. The most important difference is
that the entry for prefix 10.1.4.0/24, which router D finds in the routing table, is listed as being directly
connected to the FastEthernet 1 interface. This means that instead of forwarding the packets to a next-
hop router, this router will forward them directly to the destination host on that interface. Router E will
consult its ARP cache to find the MAC address for host B (10.1.4.2) and encapsulate the packet in an
Ethernet frame destined for host B. When host B receives the packet, this concludes the transmission of
a packet from host A to host B.
5. The process in sending return packets from host B to host A is similar. However, the information used
in the lookups in the routing table and Layer 3 to Layer 2 mapping tables, such as the ARP cache, is
reversed. For the return packets, the destination IP address is 10.1.1.1, so instead of entries for subnet
10.1.4.0/24, entries for subnet 10.1.1.0/24 will need to be present in all routers. These entries will have
different associated egress interfaces and next-hop IP addresses. As a result, the corresponding ARP
cache entries for those next hops will need to be present and these are different from the entries that
were used on the path from host A to host B. Consequently, you cannot conclude that if packets are
successfully forwarded from host A to host B, return packets from host B to host A will automatically
be successful as well.
So what does this mean from a troubleshooting standpoint?
When you find that there is no Layer 3 connectivity between two hosts, a good method to troubleshoot the
problem is to track the path of the packet from router to router, similar to the method of tracking the path of
a frame from switch to switch to diagnose Layer 2 problems. Along the way, you need to verify the
availability of a matching route in the routing table for the destination of the packet and, subsequently, the
availability of a Layer 3 to Layer 2 address mapping for the next hop for those technologies that require a
Layer 2 address, such as Ethernet.
For any type of application that requires two-way communication, you need to track the packets in both
directions. Availability of the correct routing information and Layer 3 to Layer 2 mappings for packets
traveling in one direction does not imply that the correct information is available for packets traveling in the
other direction as well.
Filtering show Commands
The show command is one of the primary troubleshooting tools on Cisco IOS networking devices. If not
used properly, these show commands can provide too much information.
As you become familiar with these commands, you will find that the capabilities offered by the show
commands are invaluable and help you sort through large amount of output information.
All outputs of the show commands can be filtered. Filtering is done by using the pipe “|” symbol, followed
by a filter word. Filters allow you to quickly identify the part of configuration that is of your interest for
troubleshooting. It is possible to use the full power of the Regular Expressions to match a string.
Common filters that can be used are include, exclude, begin and section.
In this example, a word or a line fragment is used to select the lines in the text that are interesting for the
case. By using the show ip route | include 10.10 command, you show only the lines that contain string
“10.10”.
You can exclude lines from different outputs by using the exclude filter. This can be useful when you are
only interested in operational interfaces.
The begin filter allows you to skip all output down to the first occurrence of the regular expression pattern.
The section keyword in the example, matches the lines that include the expression “router” and shows them
together with their configuration sections. In this case, it selects all the routing protocol configuration
sections, but leaves out the rest of the configuration.
Filtering show Command Output Using Regular
Expressions
Regular expressions are patterns that can be used to match strings in a piece of text. They can be used to
match words or text fragments in a line of text, but the full use of the regular expression syntax allows you
to build complex expressions that match very specific text patterns.
The example shows the information for the GigabitEthernet0/0 interface,. The output includes the
information on the number of packets per second in the last 5 minutes, input/output errors, and the number
of input and output packets in total. The filtering of the show command is done with the help of Regular
Expressions to match only the relevant information.
Regular Expressions
RegEx are strings of special characters that can be used to filter character patterns. They are made up of
symbols, numbers, and letters and are used in show commands as well as in BGP (AS-PATH) ACL's to
match a BGP route based on the information that is on its AS-PATH.
Some of Cisco IOS regular expression characters and their functions:
Regular Expression Function Example
Character
^ Matches the character at the beginning of the ^234 matches 2345, but not 12345
string.
| Matches one of the characters or character 10|20|30 matches 10, 20 or 30

patterns on either side of the pipe. This is similar to
the logical OR.
$ Matches the character or null string at the end of a 123$ matches 0123, but not 1234
regular expression.
* Matches zero or more sequences of the character 2345* matches 2345, 2345 2345, 2345
preceding the asterisk. Also acts as a wildcard for 2345 2345, etc.
matching any number of characters.
Note The text match is sensitive to capitals and spacing. This becomes important when
referencing interfaces, which are always capitalized.
Redirecting show Command Output to a File
Instead of filtering the output of a show command, the output can also be redirected, copied, or appended to
a file.
You can redirect the output by using the pipe character "|", followed by the options redirect, tee, or append
and a URL that denotes the file.
When you use the redirect option on a show command, the output is not displayed on the screen but is
redirected to a text file instead. This file can be stored locally on the device’s flash memory or on a network
server, such as a TFTP or FTP server.
The tee option is similar to the redirect. The difference is that this command copies the output to a text file
and also displays it on the screen.
The append filter allows you to append the output to a file instead of replacing that file. The use of this
command makes it very simple to collect the outputs of several show commands in a text file, either directly
on a server, or first on the device itself and then copied to a server.
A prerequisite for this option is that the file system that you are writing to supports “append” operations.
TFTP, for instance, does not support appending.
Discovery 2: Troubleshooting Connectivity
Overview
In this discovery lab, you will learn how to troubleshoot connectivity on Cisco devices, using Ping,
Extended Ping, Traceroute, and Telnet tools.
Topology
Job Aids
Note If you shut down an interface on a real router or switch, the connected device will see it as
"down/down". Due to virtualization specifics, IOL behaviour is slightly different. If you shut
down an interface on a router or switch, the connected device will see it as "up/up". In IOL,
the status of an interface can only be "up/up" or "administratively down/down".
Device Information
Device Information
Device Interface Neighbor IP Address
PC1 Ethernet 0/0 R1 10.10.10.1/24
R1 Ethernet 0/0 PC1 10.10.10.2/24
R1 Ethernet 0/1 R2 10.10.20.1/24
R2 Ethernet 0/0 R1 10.10.20.2/24
R2 Ethernet 0/1 R3 10.10.30.1/24
R2 Ethernet 0/2 RA 10.10.70.1/24
R2 Tunnel 0 RA 172.16.0.1/24
RA Ethernet 0/0 R2 10.10.70.2/24
RA Ethernet 0/3 R3 10.10.80.1/24
RA Tunnel 0 R2 172.16.0.2/24
R3 Ethernet 0/1 RA 10.10.80.2/24
R3 Ethernet 0/0 R2 10.10.30.2/24
R3 Ethernet 0/2 R4 10.10.40.1/24
R4 Ethernet 0/0 R3 10.10.40.2/24
R4 Ethernet 0/1 SW 10.10.50.1/24
SW Ethernet 0/0 R4 N/A
SW Ethernet 0/1 PC2 N/A
PC2 Ethernet 0/0 SW 10.10.50.2/24
Routing Information
Troubleshooting Connectivity
Step 1 Test end-to-end connectivity from PC1 to PC2.
PC1# ping 10.10.50.2

.....
Success rate is 0 percent (0/5)
PC1 does not have connectivity to PC2.
The Purpose of Ping

The basic purpose of ping is to check:
 Reachability
 RTT
 Packet loss
After sending ICMP echo requests, if an ICMP echo reply packet is received within the default 2-second
(configurable) timeout, an "!" is printed, meaning the reply was received before the timeout expired, or a "."
is printed meaning the reply was not received before the timeout expired.
The router as a result prints the min/avg/max RTT in milliseconds.
Note When pinging, processing delays can be significant, because the router considers
responding to a ping a low priority task.
Step 2 From R1's LAN interface, source ping PC2.
Usually you will not have the luxury to test connectivity from an end-user device. Plus, this is
not really convenient. Another way to test if the end-user device has connectivity is to access a
first hop device and specify the LAN interface as the source interface for ping operation.
By default the source IP address in the IP header is the IP address of the outgoing interface.
R1# ping 10.10.50.2 source ethernet 0/0

Packet sent with a source address of 10.10.10.2
.....
In this case, the ping fails because at least one router does not have a return address to PC1
subnet.
Step 3 From R1, ping PC2.
R1# ping 10.10.50.2

.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
When you do not have connectivity to an end-device, move one step back and ask yourself for
example: "I know that ping is not successful from PC1 to PC2, but can I ping from R1 to PC2?".
Why is all of a sudden the ping successful? The network in this discovery is using a dynamic
routing protocol to ensure Layer 3 connectivity. In this case the reason for the ping failure is that
R1 is not advertising 10.10.10.0/24—the network in which PC1 is in. Because R1 is not
advertising the network, R4 does not know about it.
Step 4 Verify that R4 does not have PC1's network in routing table.
R4# show ip route 10.10.10.1

% Subnet not in table
Step 5 On R1, configure OSPF to advertise the 10.10.10.0/24 network.
R1(config)# router ospf 1

R1(config-router)# network 10.10.10.0 0.0.0.255 area 0
You can issue show ip protocols on R4 to verify that R4 is now advertising the network.
Step 6 Perform traceroute from PC1 to PC2.
PC1# traceroute 10.10.50.2
Tracing the route to 10.10.50.2
VRF info: (vrf in name/id, vrf out name/id)
1 10.10.10.2 0 msec 0 msec 1 msec
2 10.10.20.2 0 msec 1 msec 0 msec
3 172.16.0.2 1 msec 0 msec 0 msec
4 10.10.80.2 0 msec 1 msec 0 msec
5 10.10.40.2 1 msec 1 msec 0 msec
6 *
10.10.50.2 1 msec 1 msec
End to end connectivity now works.
You can test the connectivity using ping or traceroute. The traceroute tool is very useful, if you
want to determine the specific path that a frame takes to its destination. In case of unreachable
destination, you can determine where on the path the issue lies.
NOTE: To interrupt the traceroute operation, use Ctrl+Shift+6 combination and then hit the X
button on your keyboard.
Traceroute Tool
Traceroute works by sending the remote host a sequence of three UDP datagrams with a TTL of 1 in the IP
header and the destination ports 33434 (first packet), 33435 (second packet) and 33436 (third packet). The
TTL of 1 causes the datagram to "timeout" as soon as it hits the first router in the path, which router
responds with an ICMP "time exceeded" message, meaning the datagram has expired.
The next three UDP datagrams are sent with TTL of 2 to destination ports 33437, 33438 and 33439.
After passing the first router, the datagram arrives at the ingress interface of the second router, which router
responds with an ICMP "time exceeded" message.
This process continues until the packet reaches the final destination and the ICMP "time exceeded",
messages are sent by all the routers along the path.
When the packet reaches the final destination, the device responds with an ICMP "port unreachable".
Step 7 Perform ping from R1 to PC2 with df-bit set and datagram size of 1476 bytes.
Testing with df-bit set is useful to see if the packet can make it to the destination, without
fragmenting. By setting the df-bit, the routers will not fragment the packets.
R1# ping 10.10.50.2 size 1476 df-bit

Packet sent with the DF bit set
!!!!!
Ping succeeds. When routers receive IPv4 packets larger than MTU (Ethernet packets can
transport only 1500 bytes of payload using default encapsulation) and the packets do not have
the df-bit set, the routers fragment the packets into smaller data chunks.
You may want to avoid fragmentation of IPv4 packets in your network because it can cause high
CPU spikes on the router, it increases the overhead by adding the IP header to each data chunk,
and it is also linked with the performance of an application. For example, if a packet is
fragmented into multiple data chunks and if only one of those data chunks is lost on its way to
destination, then the TCP must resend the whole packet.
Step 8 Perform ping from R1 to PC2 with df-bit set and datagram size of 1477.
To detect errors, you must sometimes test connectivity with various packet sizes. The default
ICMP packet size can be routed or switched without problems whereas larger packets can be
dropped due to MTU, encoding errors in WAN links, hardware issues, security policies in the
network, and so on.
R1# ping 10.10.50.2 size 1477 df-bit

M.M.M
Apparently, there is a link in the path that has an MTU of 1476 bytes instead of 1500 bytes. Ping
performed with df-bit set and datagram size 1476 bytes succeeds, whereas the same testing fails
when changing the datagram size to 1477 bytes. The result "M", means the router could not
fragment the packet and this makes you think that this is caused by the use of GRE tunnels as
they have 24 bytes overhead.
Step 9 Perform an interactive ping from R1 to PC2. Determine MTU by sweeping packets between
1400 bytes and 1500 bytes.
This method of ping testing and determining the MTU size across the network, is more efficient
than manually increasing the datagram size bit by bit in a single-line ping test like you did in the
previous steps.
R1# ping
Protocol [ip]:
Target IP address: 10.10.50.2
Repeat count [5]: 1
Datagram size [100]:
Timeout in seconds [2]: 1
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]: y
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]: y
Sweep min size [36]: 1400
Sweep max size [18024]: 1500
Sweep interval [1]:
Sending 101, [1400..1500]-byte ICMP Echos to 10.10.50.2, timeout is 1 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!M.M.M.M.M.M.M.M.M.M.M.M.
Ping succeeds with datagram size from 1400-1476 bytes and df-bit set, for the rest the result is
that the packets cannot be fragmented. For testing, you can sweep packets at different size
(minimum, maximum), set the sweeping interval, and determine the MTU by seeing which
packets are passing through the links and which packets need to be fragmented since you already
have set df-bit for all the packets.
Step 10 Perform telnet from R1 to PC2 through HTTP port TCP 80.
Even though you have Layer 3 connectivity between PC1 and PC2, this does not mean that users
have full functionality. The end-user not only needs functional Layer 3, but all the layers above.
Cisco IOS allows you to test the transport layer by using Telnet. For example you can use Telnet
to connect to a device through port 80. If connection is successful, you can be reasonably sure
that the user does not have HTTP traffic blocked. If your attempt is unsuccessful, you might
want to investigate if the firewall in your network is blocking HTTP traffic.
After you successfully connect to PC2, disconnect using Ctrl+C keyboard key combination.
R1# telnet 10.10.50.2 80

Trying 10.10.50.2, 80 ... Open
^C
HTTP/1.1 400 Bad Request
Date: Wed, 12 Feb 2014 10:00:32 GMT
Server: cisco-IOS
Accept-Ranges: none
400 Bad Request

[Connection to 10.10.50.2 closed by foreign host]
As you see, there is a connection from R1 to PC2's HTTP server open on port 80. The output
"HTTP/1.1 400 Bad Request", is a result of closing the connection with the Ctrl-C.
Telnet uses the well-known port number 23 by default. When testing from the command line,
you can specify a different port number (in the case of the HTTP server, you specify port number
80) and see if the connection to the destination is accepted, refused, or timed out.
Basic Hardware Diagnostic
Any network device is a specialized computer, which at a minimum consists of a CPU, RAM, and storage,
allowing it to boot and run the operating system and interfaces, which allow the reception and transmission
of network traffic.
The show processes cpu command displays the active processes running on the router and the CPU
utilization corresponding to each process in the last 5 seconds, last minute, and last 5 minutes. Each process
has its own name and process ID.
Both routers and switches have a main CPU that executes the processes that constitute the Cisco IOS
Software.
Routers can be classified into three categories:
 Software-based routers: Have a shared control and data plane and a single CPU responsible for all
operations.
 Hardware-based routers: Have a separated control and data plane, a single CPU to manage the control
plane, and an ASIC designed with the purpose to only forward packets very fast. When the ASIC cannot
forward a packet it punts it to the CPU.
 Hardware assisted routers: Have a separated control and data plane, a single CPU to manage the
control plane, and a NP significantly increasing the throughput of the device. If the NP can not forward
a packet, it punts it to the CPU.
In the example, from the show processes cpu sorted 1min command, the five second CPU utilization is
made up from two numbers 7% and 4%. The first number is the total CPU utilization and the second
number is the interrupt utilization, which is used only for forwarding the traffic.
Interrupt utilization indicates the volume of network traffic the device is receiving. If you see a high
interrupt percentage, it means that the device is receiving too many network packets.
As you can see, the difference between the CPU utilization and the interrupt utilization is 3% which makes
the sum of each individual process utilized.
When there is a constant CPU utilization of more than 70% or 80%, troubleshooting is required. Some of
the reasons punting the traffic to the CPU are Fragmentation, IP options, TTL=1, ACL log, Forwarding path
issues, Multicast RPF drops, Multicast path setup, and so on.
You may find cases where CPU utilization is 55% and interrupt utilization is 54%. This is perfectly fine
because 54% is being used only for forwarding the traffic.
In the example, the output shows the processor memory and I/O memory. This is a useful command for both
Cisco routers and switches to check the amount of RAM memory that is used by processes and the free
amount. I/O memory is also used for temporary packet buffering.
When device's performance is a concern during network troubleshooting, it is advised to check it's free
memory. As a result of lack of memory, device can be rebooted, the BGP route processing on a router can
be very slow, etc. You should always plan and take into consideration the free amount of memory on your
device.
When a process cannot allocate the needed memory due to unavailability of enough memory, the router or
the switch will log a memory allocation failure with a %SYS-2-MALLOCFAIL message.
The show interface command is essential for troubleshooting. It shows you the status of the interfaces
(Interface status/Line protocol status), MTU, interface operation mode as well as statistics about Input
Queue/Output Queue drops and errors, which shows how many packets are in Input Queue. It also shows
the queue depth and the number of dropped packets since the interface counters were last cleared.
 Input Queue Drops: Signify that the traffic is dropping because the router was receiving traffic more
than it can process.
 Output Queue Drops: This is usually a result of congested link.
 Input errors: This may be a result of interface problems, duplex errors, and CRC errors.
 Output errors: This is usually related to duplex issues.
It is essential to have a proper MTU configured in your network, as there are different types of MTU (MTU
for each protocol and physical interface) - Ethernet, PPP, IP, MPLS.
Your device may have GigabitEthernet, FastEthernet, T1, T3, E1, E3 controllers. Dependent on the
interface hardware, the show controllers command gives you a more detailed packet and error statistics for
that particular type of hardware.
Some information to look for and compare from this output command are Received Good Bytes, Received
Bad Bytes, Received Good Frames, Received Bad Frames, Transmitted Good Bytes and Transmitted Good
Frames. Using show controllers is also useful to see if device is DTE or DCE when using serial links.
Usually the information of the show platform command can be helpful to troubleshoot a router crash. In
case you have to open a Cisco TAC Service Request to troubleshoot a device's crash, be sure to collect and
include this information prior to opening the case.
Note The show platform can be used on both, routers and switches.
Debug Commands
In the example, ICMP traffic is generated. Packets that are sourced from 10.20.1.1 are routed via FIB,
whereas packets sourced from 10.20.1.2 are routed via RIB.
Note Every time that you ping a router, that traffic is Process Switched, and every time you ping a
server behind that router, the traffic is CEF switched.
Some common debug commands:
Command Description
debug interface interface slot/number Provides debug messages for specific physical port on the device.
debug ip icmp Used to troubleshoot connectivity issues, from the output you can see
whether the device is sending or receiving ICMP messages.
debug ip packet Used to troubleshooting end to end communication. It should always be

used with an ACL.
debug eigrp packets hello Used to troubleshoot neighbor establishment, it shows the frequency of
the sent and the received hello packets.
debug ip ospf adjacency Provides information about events concerning adjacency relationships
with other OSPF routers.
debug ip ospf events Provides information about all OSPF events.
debug ip bgp updates Provides information about routes you have advertised and/or received
from your BGP peer.
debug ip bgp events Provides information about any BGP event, such as neighbor state
changes.
debug spanning-tree bpdu receive Used to confirm the BPDU flow on switches.
The debug ip packet command helps you to better understand the IP packet forwarding process. It is
advised to use it carefully in a production network. For example, if you enable it on a router where a high
volume of IP traffic is being switched through a router, the tool will utilize all CPU cycles, and as a result it
will prevent the router from doing its primary job—forwarding packets as fast as possible. So before you
use debug ip packet in a production network, remember to issue the show processes cpu command, to see
how much the CPU is being utilized.
Note The debug ip packet output will show information for process-switched packets only.
Another command that needs special attention is the debug all command. This command should be used
with extreme caution. If you want to disable any debugging command that is flooding your screen with a lot
of information, you can use the undebug all command. To check, if there is any debugging left activated,
you can check that by using the show debug command.
Note The undebug all command does not stop the process of the debug interface interface
slot/number command.
Summary
Lesson 5: Using Specialized
Troubleshooting Tools
Overview
Access to information is one of the most important aspects of troubleshooting and maintenance. Information
is gathered in many ways: on demand, during troubleshooting processes, continuously, as part of baseline
creation, and triggered by network events.
In addition to the tools that are available in the Cisco IOS CLI, there are many specialized network
maintenance and troubleshooting tools that you can use to support these information-gathering processes.
These tools and applications typically require communication with the network devices. Several different
underlying technologies can be used to transfer the information between the devices and the tools.
Upon completing this lesson, you will be able to meet the following objectives:
• List and describe useful troubleshooting tools
• List and describe the troubleshooting tools categories
• Explain how to use syslog to log information for monitoring and troubleshooting
• Enable SPAN, RSPAN to facilitate the use of packet sniffers
• Explain how SNMP can be used for troubleshooting
• Explain how Netflow can be used for troubleshooting
• Explain how to use EEM for system management from within the device itself
• Explain how to use EEM for logging when configuration mode is entered
• Explain how to use EEM for detecting the disabled interface
Troubleshooting Tools Overview
A generic troubleshooting process consists of several phases, or subprocesses. Some of these processes are
primarily mental, such as the elimination process. Some of these processes are administrative in nature, such
as documenting and reporting changes and solutions. And some of these processes are more technical in
nature, such as gathering and the analysis of information.
The processes that benefit the most from the deployment of network maintenance and troubleshooting tools
are the processes that are technical in nature. Therefore, the focus here will be primarily on the use of those
tools and how to prepare the network to support those tools.
The following processes have elements that you can optimize by the use of tools:
 Define the problem: One of the main objectives of deploying a proactive network management
strategy is to be aware of potential problems before users report that they are experiencing outages or
performance degradation. Network monitoring and event reporting systems can notify the network
support team of events as they happen, giving them time to respond to the problem before the users
notice and report them.
 Gather information: This is one of the essential steps in the troubleshooting process, and you can
leverage any tool to obtain detailed information about events in an effective way.
 Analyze: A major component of the interpretation and analysis of the gathered information is the
comparison against a baseline. The ability to differentiate between normal and abnormal behavior can
yield important clues about the potential problem cause. Collecting statistics about network behavior
and network traffic is therefore a key process to support the troubleshooting data analysis.
 Test your hypothesis: Testing a hypothesis commonly involves making changes to the network, and
you might need to roll back those changes if they did not resolve the problem. Tools that enable easy
rollback of changes are therefore important to an efficient troubleshooting process.
Troubleshooting Tools Categories
Except for the configuration rollback, which is more a generic change management tool than a specific
troubleshooting tool, most of the troubleshooting mechanisms fall into one of the three categories.
 Collection of information on demand, driven by incidents: This is the typical information gathering
that you do during the troubleshooting process itself. You gather, interpret, and analyze information,
and based on the outcome of this process you gather more information. Examples of this are the
capturing of network traffic and debugging of the device processes.
 Continuous collection of information to establish a baseline: A set of major network performance
indicators is established, and based on those indicators, statistics about the behavior of the network are
collected over a long time period. These statistics form a baseline that you can use to judge whether the
behavior that you observe is normal or not. This process also provides historical data that you can
correlate to the events. Examples of this are the collection of statistics through the use of SNMP and
traffic accounting by using the NetFlow technology.
 Notification of network events: Information is reported by network elements. The notifications are
triggered by the occurrence of specific events. Examples of this are the reporting of events via syslog
log messages or SNMP traps and the definition and the reporting of specific events by using the
Embedded Event Manager that is part of the Cisco IOS Software.
What these categories have in common is that their functionality depends on the interaction between a tool
or application running on a host, and the network devices. In the first two categories, the information is
pulled from the network elements to the application or tool. In the last category, the information is pushed to
the application or tool by the network devices.
A very broad spectrum of tools and applications can perform the mentioned processes, and it is virtually
impossible to list them all, let alone compare and contrast them. However, many of these tools depend on
the same underlying technologies and protocols for the communication between the application and the
network. This includes protocols such as syslog, SNMP, and NetFlow, and also the technology that allows
you to forward packets that are received on a port to a specific system for analysis.
An engineer should understand the main benefits that a particular tool or application brings to the network
troubleshooting process. For this reason it is also important to know how to enable the communication
between these tools and applications, and the network devices.
Case Study: Syslog
A user in your network complains that he is unable to open any web site on his workstation. You need to
verify whether it is blocked by any of the ACLs on any of the routers.
Your branch network consists of multiple routers, switches, and workstations. You have a central syslog
server, collecting syslog messages from all the Cisco IOS devices in the network. Basic security is
implemented on routers by using ACLs. ACLs are configured with the log option at the end, reporting ACL
matches to a syslog server.
With a central Syslog server, finding an ACL match is simple. Filter the syslog messages by the message
type „SEC-6-IPACCESSLOGP“ and user‘s workstation IP. From the results, you can see that user‘s HTTP
traffic was blocked by access-list PREVENT-HTTP on router 10.10.1.1.
Note The case study shows only one of the possible syslog use cases. Depending on the
configuration, Cisco network devices can send all types and levels of syslog messages to an
external syslog server, and specialized software packages can offer additional features such
as event-based alarming, event analysis, and so on.
Case Study: Troubleshooting with SPAN
Monitoring shows that interface FastEthernet 0/0 on a local switch is experiencing regular output queue
drops. Output drops are caused by a congested interface. For example, the traffic rate on the outgoing
interface cannot accept all the packets that should be sent out. NMS shows a maximum of 40 MBps traffic
over a 100 MBps Fast Ethernet link. NMS data is gathered via SNMP with a sampling time of 30 seconds,
therefore you suspect there might be short bursts of traffic that reach 100 MBps.
You install a packet sniffer (a workstation with the Wireshark software installed) and configure the SPAN
functionality on a switch. The SPAN session is identified by a session number, “1” in this example. The
source ports or VLANs are identified with the monitor session number source command and the
destination ports are identified with the monitor session number destination command. The session
number is what binds the commands together to form a single session.
All traffic coming from, or going to the rest of the network through the interface FastEthernet 0/0 is copied
to the interface FastEthernet 0/2 and inspected by Wireshark. Verify that SPAN is properly configured by
issuing the show monitor command.
Use Wireshark to capture all traffic over a span of 5 minutes. Once the capture is done, the Wireshark‘s
Statistics > IO Graphs tool will chart all the captured traffic. The traffic graph confirms there are short
bursts of traffic, utilizing the interface to it‘s maximum, hence forcing the switch to drop some packets once
the output queue is full.
Additionally, you can apply filters to isolate traffic originating from, or going to the IP address of the client,
confirming the source of the traffic. You can use additional filters to distinguish the type of the traffic (for
instance HTTP, non-HTTP), allowing you to find the root cause of the excessive traffic.
There are specific hardware-based limitations for SPAN feature on each of the different switching
platforms. You should check the documentation for the platform that you are working with to find out
exactly which capabilities are supported and what the limitations are for that particular platform.
Note SPAN feature is limited, as it only allows for local copy on a single switch. A typical switched
network usually consists of multiple switches and it is practical to monitor ports spread all
over the switched network with a single packet sniffer. This is possible with the Remote
SPAN feature. RSPAN does not copy traffic to a destination port, but rather floods it into a
special RSPAN VLAN. The destination port can be located anywhere in this RSPAN VLAN.
Case Study: Troubleshooting with SNMP
Router R1 in your production network has restarted over the weekend. This outage was unplanned and it is
your responsibility to investigate why the restart occurred. Using syslog, you notice "SYS-2-
MALLOCFAIL" message prior to the router restart, indicating that the router has run out of memory.
The causes that result in "SYS-2-MALLOCFAIL" message are:

 The memory in the router is insufficient to support the Cisco IOS Software or the size of the routing
table.
 A memory leak occurs.
 The memory becomes fragmented.
Note A memory leak occurs when a process requests or allocates memory and then forgets to
free (deallocate) the memory when it is finished with that task. As a result, the memory block
is reserved until the router is reloaded. Over time, more and more memory blocks are
allocated by that process until there is no free memory available.
The use of the show processes memory command displays that the current memory utilization is around
50% of total processes memory that is available when the router operates normally. You suspect that a
memory leak occurred, but you cannot confirm it as there is no historical data available on the router itself.
You have a NMS installed in your network, periodically querying the network devices using the SNMP
protocol. Parameters are gathered, collected over time, and graphically presented. NMS also gathers R1‘s
processes memory utilization.
Looking at the R1‘s processes memory utilization graph over the span of last 6 weeks you notice a linear
increase of processes memory utilization up to a point of free memory exhaustion. At that point Cisco IOS
was unable to allocate additional memory to the running processes and the router was restarted.
Router currently only utilizes 50% of it‘s processes memory, but the increasing trend continues. The
utilization trend that is shown on the graph indicates a memory leak bug.
Note Case study shows only one of the possible SNMP use cases. Depending on the
configuration and MiB support, Cisco network devices expose various environment and
performance counters. NMS software packages collect that data and offer additional
features such as permanent data storage, parameter history and trending, alarming,
parameter correlation, overview of the entire network, and so on.
Case Study: Netflow
NMS that is monitoring the interface traffic on router R1 sets off an alarm, showing unusual increases of
ingress traffic on interface Fast Ethernet 0/0 and egress traffic on interface FastEthernet 0/1. Based on the
network scheme you understand that there must be an unusual traffic flow, or a number of unusual traffic
flows, entering your network from the internet.
You need to identify the traffic flow that is causing the increase and find the type of traffic and its
source/destination. Unfortunately, the standard CLI commands or the SNMP counters offer no insight into
the IP traffic flows.
Enable NetFlow and configure it to collect ingress and egress traffic information on either Fast Ethernet 0/0
or Fast Ethernet 0/1 interface. Since you are not looking for detailed traffic data, rather, you need to identify
the flow that is causing the unusual increase, you don‘t configure NetFlow to export data to an external
NetFlow collected.
Cisco IOS provides the NetFlow Top Talkers feature. It uses the NetFlow functionality to obtain
information regarding heaviest traffic patterns and most-used applications in the network.
Top talkers can be useful for analyzing the network traffic in any of the following ways:
 Security: You can view the list of top talkers to see if traffic patterns that are consistent with a DoS
attack are present in your network.
 Load balancing: You can identify the most heavily used parts of the system and move the network
traffic over to less-used parts of the system.
 Traffic analysis: Consulting the data that you retrieved from the NetFlow MIB and Top Talkers feature
can assist you in general traffic study and planning for your network.
Enable the NetFlow Top Talkers feature. Configure it to show top 3 traffic flows sorted by bytes of traffic.
Set cache timeout to 600000 milliseconds to retain the NetFlow data for 10 minutes.
After a minute issue the show ip flow top-talkers verbose command. Notice that on the first place there is
a flow with 549 Mbytes of traffic. Flow‘s source is a public web server (IP 72.163.4.161, port 0x0050 = 80)
and it‘s destination is a local IP address.
Introducing EEM
A key element of a proactive network management strategy is fault notification. When a significant event
happens on your network, you do not want to wait for users to start reporting the problems that were caused
by the event. You want the network devices to report that event to a central system, so you can take actions
in response to the event as soon as possible.
The EEM framework enables the creation of custom policies that trigger actions based on notified network
events.
Events can be triggered based on various Cisco IOS subsystems such as Syslog messages, Cisco IOS
counter changes, SNMP MIB object changes or traps, CLI command execution, timers, IP SLA, NetFlow
events, and many others.
Actions can consist of sending SNMP traps and Syslog messages, executing or disabling CLI commands,
sending email, reloading the device, or even running TCL scripts.
EEM allows the creation of very powerful and complex policies. For example:
 The EEM applet for disabling specific commands disables the desired command to be executed from
the command line. For example, if you do not want the device to be manually reloaded, you can disable
the reload command.
 The EEM applet for turning on an interface and assigning it the IP address can automatically turn on the
interface when you connect the device to it and assign the IP address from the prespecified subnet.
 The EEM applet for saving the configuration and automatically reloading the device can automatically
save the configuration in case that it has been changed. It also can notify you about the changes.
 The EEM applet for reporting the device configuration changes can inform you in case that changes
have been made to the configuration. For example, you have a device, whose configuration should not
be changed. If this happens, you have to take quick actions in response, so you have to be immediately
informed about the changes.
 The EEM applet for reporting the routing table changes can inform you when the number of routes in
the routing table changes. For example, if one route is no longer present in the routing table, this could
indicate that packets are traversing a backup link, which may be more costly. If this happens, you want
to be informed so you can quickly recover the failed link.
 The EEM applet for periodically checking the reachability of certain devices can check the reachability
of certain devices in specific time intervals. Normally you want all devices to be reachable all the time.
In case there is a failure on the link, the EEM applet can immediately notify you about it so you can
take further actions.
EEM Example: Logging when Configuration Mode
Is Entered
Imagine that you have the following policy in your company. All network engineers get privileged access to
the network equipment and can make changes if necessary. However, only level 3 support engineers are
allowed to make emergency changes if required. All other engineers always need to obtain authorization
before making any change to the system. Whenever an engineer configures a router or a switch, a %SYS-5-
CONFIG_I message is logged to the syslog server. However, this message is logged as a syslog level 5
“notification” message and does not show up in the logs as a high-priority item. You want to change the
behavior of the router as follows. A message should be logged as soon as anybody enters configuration
mode, in addition to the %SYS-5-CONFIG_I message that is logged after leaving configuration mode.
This message should be logged as a critical message. Second, an informational message should be logged,
reminding the engineer of the existing change control policies. You can achieve that with programmed EEM
policy.
The EEM applet achieves this as follows:

1 The applet is created and named "CONFIG-STARTED".
2 The event that should trigger this applet is defined.
3 Corresponding actions are defined.
EEM Example: Bring up a Disabled Interface
Imagine you have a critical link and there is no good reason for anyone to shut it down. If someone
accidentally shut it down, you want to be informed about it and the interface must immediately be brought
up. You can use EEM applet to achieve that. As soon as the interface is down, the applet will turn it back
on, and an informational message will be logged.
EEM supports many other event triggers and actions. All these actions are programmed using either simple
command-line interface or using a scripting language, Tool Command Language.
Summary
References
For additional information, refer to these references:
 http://www.cisco.com/web/about/security/intelligence/identify-incidents-via-syslog.html
 http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/span.html
 http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_cfg_snmp_sup_ps6350_TSD_
Products_Configuration_Guide_Chapter.html
 http://www.cisco.com/go/netflow
 http://www.cisco.com/go/eem
Lesson 6: Module Summary
Overview
This topic summarizes the key points that were discussed in this module.
Lesson 7: Module Self-
Check
Use the questions here to review what you learned in this module. The correct answers and solutions are
found in the Module Self-Check Answer Key.
1. An experienced troubleshooter might skip the "analyze information" and "eliminate potential causes"
steps, and instead of that relay on his own insight to determine the cause of the problem. What approach
of troubleshooting would he use? (Source: Describing Troubleshooting Methodologies)
A. Ad hoc
B. Shoot from the hip
C. Independent path
D. Random troubleshooting
2. Match the troubleshooting method with the statement that best describes it. (Source: Describing
Troubleshooting Methodologies)
Traces packets through the network to isolate the
A. Follow the path
problem.
Might provide a solution even without understanding
B. Spot the
why.
differences
Swap components to observe what happens.
C. Divide and
conquer
Starting at the network layer first.
D. Move the problem
3. What are the primary goals of collecting information? (Choose two.) (Source: Using Troubleshooting
Procedures)
A. Eliminate potential causes.
B. Identify indicators pointing to the underlying cause of the problem.
C. Form a hypothesis for the most likely cause of the problem.
D. Find evidence that can be used to eliminate potential cause.
4. What troubleshooting step should be performed after a problem has been reported and clearly defined?
(Source: Using Troubleshooting Procedures)
A. Eliminate potential causes.
B. Propose hypothesis.
C. Gather information.
D. Analyze gathered information.
5. Rollback plan is necessary when testing a hypothesis. True or false? (Source: Using Troubleshooting
Procedures)
A. True
B. False
6. Updating documentation after changes are made is a maintenance or change control method. True or
false? (Source: Using Troubleshooting Procedures)
A. True
B. False
7. During configuration of archiving, which of the following parameters is mandatory? (Source: Following
Recommended Practices During Routine Network Maintenance)
A. Username and password
B. Path for archived file
C. Time period between archives
D. There are no mandatory parameters for configuration of archiving
8. Which of the following commands shows the value of the configuration register? (Source: Following
A. show running-config
B. show version
C. show configuration
D. show startup-config
9. Which of the following is the default severity level for logging messages? (Source: Following
A. Debugging
B. Emergency
C. Notification
D. There is no default level - it has to be configured.
10. Which of the following filters skips the output up to the first occurrence of the regular expression
pattern? (Source: Using Basic IOS Troubleshooting Tools)
A. include
B. exclude
C. begin
D. section
11. Which of the following counter increases of the show interface command output signifies congested
interface? (Source: Using Basic IOS Troubleshooting Tools)
A. Output errors
B. Input errors
C. Output queue drops
D. Input queue drops
12. How is the ping traffic processed on a Cisco IOS router, when pinging from a remote host to any of the
router's local interfaces is performed? (Source: Using Basic IOS Troubleshooting Tools)
A. Fast switched
B. Process switched
C. CEF switched
D. None of the above
13. Which two of the following technologies are suitable to monitor device interface status? (Choose two.)
(Source: Using Specialized Troubleshooting Tools)
A. Syslog
B. (R)SPAN and Traffic sniffer
C. SNMP
D. Netflow
14. Which two of the following technologies are suitable to identify the type of traffic passing through a
network device? (Choose two.) (Source: Using Specialized Troubleshooting Tools)
A. Syslog
C. SNMP
D. Netflow
15. Which of the following technologies provides event detection and onboard automation? (Source: Using
Specialized Troubleshooting Tools)
A. Syslog
C. SNMP
D. Netflow
E. EEM
Module Self-Check Answers
Answer Key
1 B
2
A. Follow the path — Traces packets through the network to isolate the problem.
B. Spot the differences — Might provide a solution even without understanding why.
C. Move the problem — Swap components to observe what happens.
D. Divide and conquer — Starting at the network layer first.
3 B, D
4 C
5 A
6 A
7 B
8 B
9 A
10 C
11 C
12 B
13 A, C
14 B, D
15 E
Module 2: Troubleshooting at
SECHNIK Networking Ltd.
Introduction
You got a job at SECHNIK Network Ltd. as a network technician. Company users and fellow technicians
will came to you with problems that you will need to resolve as quickly as possible.
In this module you will be faced with three challenge labs. Each lab has multiple troubleshooting tickets that
you need to investigate, analyze, and finally resolve.
Upon completing this module, you will be able to:

• Solve troubleshooting tasks for the first challenge lab at SECHNIK Networking Ltd.
• Describe how you solved first challenge lab
• Solve troubleshooting tasks for the second challenge lab at SECHNIK Networking Ltd.
• Describe how you solved second challenge lab
• Solve troubleshooting tasks for the third challenge lab at SECHNIK Networking Ltd.
• Describe how you solved third challenge lab
Lesson 1: Debrief of the First
Troubleshooting at
Overview
This lesson serves as a debrief for the first troubleshooting lab at SECHNIK Networking Ltd.
Example troubleshooting flows are provided, however keep in mind there are multiple ways to approach
troubleshooting problems.
Upon completing this lesson, you will be able to meet these objectives:
• Describe issues that you had to solve in the challenge lab
• Describe how you solved PC1's connectivity problems to the internal server
• Describe the possible issues for non-functioning trunk link
• Describe how you solved PC2's Internet connectivity issues
• Describe the possible issues for non-functioning network address translation
• Describe how you solved PC3's SSH connectivity issues to the internal server
• Describe how interface status can be interpreted
• Describe how you solved PC4's IPv6 Internet connectivity issues
• Describe IPv6 address assignment methods
Trouble Tickets Overview
This topic reviews the problems that were introduced in lab.
The text introducing the trouble tickets was the following:
On Monday you arrive at your workplace at SECHNIK Networking Ltd., grab a cup of coffee and stroll to
your desk. There are already four people waiting for you.
 Kimberly, the user on PC1, cannot access data on the server at 172.16.200.10.
 Andrew, the user on PC2, cannot access the Internet. He is trying to access a server at 209.165.200.2.
 Carol, the user on PC3, cannot use SSH to connect to the server at 172.16.200.10.
 Mithun, the user on PC4, said that he would like to access the Internet through IPv6. He adds that since
it works for Andrew, it should also work for him. Use IPv6 address 2001:DB8:D1:A5:C8::2 to test the
connectivity.
The four employees are telling you that all of these things worked fine on Friday. Senior engineers are all
out and it seems that you are the only one that can fix these issues.
Example Troubleshooting Flow: PC1 Unable to
Access Data on the Server
This topic offers an example troubleshooting flow for solving the issue of PC1 not being able to access data
on the internal server.
Kimberly, the user on PC1, cannot access data on the server at 172.16.200.10.
Verifying the Problem
Always first verify the reported problem.

In this case, the ping from PC1 to Server is not working, therefore the problem is real and still present.
Troubleshooting Plan
Note This is one possible troubleshooting plan. There are many different approaches to this
problem.
Before you start troubleshooting, first verify the reported problem.

When you are faced with this typical problem of resolving network connectivity, it’s always a good choice
to start with the “bottom-up” approach. With this approach, you start at Layer 1 of the OSI model and make
your way up.
Information Gathering
Working with the "bottom-up" approach, first check the interface status and IP address with the show ip
interface brief command. the interface connecting to ASW1, Ethernet0/0, is „up/up“, which leads you to
conclude that Layers 1 and 2 are operational. However, there is no Layer 3 address assigned to the interface.
DHCP is configured as an IP address assignment method to this interface.
On ASW1 you can discover that PC1 is connected to a port configured as „access“ port, member of VLAN
10.
You already checked that the PC1-ASW1 link is operational on Layers 1 and 2, but not Layer 3. You might
want to apply the same „bottom-up“ approach on the ASW1-DSW1 link.
It is a good practice to add a description to all ports in the network. That way somebody that is not familiar
can use show interfaces description to discover how devices are interconnected. From the show interfaces
description you can conclude that the ASW1-DSW1 link is functional on Layers 1 and 2.
Since PC1 and PC2 are in different VLANs, you can assume that ASW1-DSW1 ports should be trunking.
Information Gathering and Proposing the Hypothesis
If you verify trunking on ASW1, you can see that Ethernet 0/0, the port that connects to DSW1, indeed has
status of „trunking“. However, only VLANs 1 and 20 are allowed to traverse the trunk link. Investigation of
trunking on DSW1 confirms the suspicions of misconfiguration on ASW1. DSW1 has two trunk links, one
to R1 and the other to ASW1. Both are trunking and allowing traffic for VLAN 10.
Testing the Hypothesis
To test the hypothesis, add VLAN 10 to the allowed VLAN list on ASW1‘s Ethernet 0/0. Using the show
interfaces trunk command, verify that now VLAN 10 traffic is allowed to traverse the link.
It will take some time for PC1 to acquire IP address from the DHCP server. One way to speed up this
process is to shut down and then bring back up PC1‘s Ethernet0/0 interface.
Using the ping tool, you should now be able to test the connectivity from PC1 to the internal server at
172.16.200.10. You can be fairly confident that Kimberly, the user on PC1, will now be able to access data
on the internal server.
Note At this point you should inform the user that the problem is solved. Call Kimberly and let her
know that she can access data on the Server now. Save the configuration on ASW1,
document changes made and close the ticket.
Troubleshooting Trunks
To allow a switch port that connects to switches to carry more than one VLAN, you must configure a trunk.
If you do not configure it properly, the trunk will not establish or will not work the way that it should.
First of the possible mistakes is the encapsulation type mismatch. There are two trunking technologies, ISL
which is a Cisco proprietary, and 802.1Q which is the IEEE standard. ISL is not widely used anymore, but it
does not matter which one you configure, the encapsulation type must match on both sides of a trunk link.
For example if you configure 802.1Q on one side, make sure that it is configured on the other side of the
link.
Other possible issue occurs when using DTP. Cisco switch ports can run DTP, which can automatically
negotiate a trunk link. Protocol can determine an operational trunking mode and protocol on a connected
switch. You can configure DTP mode to turn off the protocol or to instruct it to negotiate a trunk link only
under certain conditions as described in the table. To avoid the misconfiguration, manual configuration of a
trunk link with disabled negotiation is recommended.
Mode Function
dynamic auto Creates the trunk, based on the DTP request from the neighboring switch.
dynamic desirable Communicates to the neighboring switch via DTP that the interface is attempting to become
a trunk if the neighboring switch interface is able to become a trunk.
trunk Automatically enables trunking regardless of the state of the neighboring switch and
regardless of any DTP requests sent from the neighboring switch.
access Trunking not allowed on this port regardless of the state of the neighboring switch interface
and regardless of any DTP requests sent from the neighboring switch.
nonegotiate Prevents the interface from generating DTP frames. This command can be used only when
the interface switch port mode is access or trunk. You must manually configure the
neighboring interface as a trunk interface to establish a trunk link.
When you configure an 802.1Q trunk, matching native VLAN must be configured on both sides of the
trunk.
Also, do not forget to allow all the VLANs whose traffic you want to be carried within a trunk link. Or
exclude all the VLANs whose traffic you do not want to be carried.
Using the show interface trunk command, you can verify all of the possible issues—configured trunk
mode, encapsulation type, native VLAN, and allowed VLANs on trunk.
The other way to check all of these parameters is by using show interface slot/number switchport.
Access the Internet
This topic offers an example troubleshooting flow for solving the issue of PC2 not being able to access the
Internet.
Andrew, the user on PC2, cannot access the Internet. He is trying to access the server at 209.165.200.2.
Andrew says that he was able to access the Internet site last week.
First, you want to verify the problem. Ping from PC2 to Internet site at 209.165.200.2.
After confirming that the problem exists, you can start the troubleshooting procedure.
problem.
When a client looses the Internet access, it is always a good idea to check other clients. This is the
"swapping components" troubleshooting technique. There is a good possibility that the whole network is cut
away from the Internet. If other clients are able to access the Internet, then you dramatically narrowed down
the problem right there.
Swapping components does not necessarily mean swapping PC2 itself. Checking other PCs for Internet
connectivity will help you isolate the problem. If you fixed the connectivity problem from troubleshooting
ticket 1, you can check the connectivity to the Internet from there. The user on PC4 reported an issue with
IPv6, so you can also use PC4 to check for IPv4 Internet connectivity.
If other PCs are able to access the Internet site, than the problem is probably local to PC1—disconnected
cable, misconfigured port on local switch, and so on. If other PCs are unable to access the Internet, then the
issue is probably closer to the network edge or on the ISP side.
In this troubleshooting ticket, you can verify that other PCs are also unable to access the Internet site.
Information Gathering and Eliminating Possible Causes
After confirming that Internet connectivity problem is not specific to PC2, you can use "follow the path"
troubleshooting technique to further narrow down the problem.
Following the path can be done from the PC2's side or by going to the network edge and starting there. In
this example the later is done.
On PC2 use show ip interface brief to discover it's IP address. Then ping that IP address from R1.
Since there is connectivity on segments R1-PC2 and R1-Internet, you can assume that the problem is
routing in nature.
Trying to connect from a private IP address (PC2) to public IP address (209.165.200.2) is not possible
without network address translation. So the next action would be to check the NAT configuration.
By issuing show ip nat statistics you can verify that R1's Ethernet 0/1 is configured as outside NAT
interface and Ethernet 0/2 is configured as the inside interface. By issuing show interfaces description or
show cdp neighbors you can deduce that the port that connects to DSW1 is configured as "inside" and port
that connects to the Internet is configured as "outside". Inside and outside NAT interfaces are correctly
identified.
The show ip nat statistics output also tells you that access list 1 is being used as the source list of subnets
that are eligible to be translated. You should verify this access list. Is it allowing VLAN 10
(192.168.10.0/24) and VLAN 20 (192.168.20.0/24) subnets to be translated?
If you try to investigate access list "1" on R1, you will see it is not actually defined. If you check other
access lists defined you will see that there is one ACL - ACL number 21. This access list permits
192.168.0.0/16 and 172.16.0.0/16 subnets. Even though this access list would work for NAT inside source
list, it would not be wise just to use it for NAT. It might be used for something else.
Check if access list 21 is being used on any interface:
R1# show ip interface | include Outgoing|Inbound
Outgoing access list is not set
Inbound access list is not set
By investigating the NAT configuration, you can now be fairly sure that the misconfigured NAT is the
source of connectivity issues.
There are multiple ways that you can go about solving the identified problem. One of the ways is to delete
the original NAT statement and replace it with a new one that uses ACL number 21.
The connectivity test from PC2 to the Internet site at 209.165.200.2 should now be successful. Likewise this
connectivity test should be successful from PC1 and PC4 if you successfully solved the first trouble ticket.
Note Fixing PC2's connectivity to the Internet should be succeeded by saving the configuration on
R1 and informing the user on PC2 that the issue was fixed. Documenting the changes that
were made in the network is a crucial last step of every troubleshooting.
Troubleshooting NAT
Private IP addresses are not routable through the Internet. For devices with private IP addresses to be able to
access an Internet site, you need to translate them into public addresses.
Besides pure NAT misconfigurations there are some caveats that you should be aware of:
 Some applications are not NAT compatible: Certain applications can choose ports in a way that is
incompatible with how NAT operates. Good examples could be found with some VoIP protocols. If
application communicates in such a way that when communicating with a remote device, it includes the
IP address in the payload of the packet, then the remote device might want to return the traffic to the IP
address embedded in the payload. IP address will be unreachable because of the NAT translation. You
will have to avoid using NAT for some applications.
 NAT can cause processing delays: NAT works at Layer 3 of the OSI model, thus translating packets
can have a bit more delay than they would without NAT. If you are experiencing significant delays due
to translations, this is most probably because NAT device is doing numerous NAT translations—more
than it was designed to do.
 Using NAT over a VPN: Some VPN protocols verify the checksum of a packet. The checksum of a
packet before and after NAT translation will be different since NAT changes the IP information. In this
case the VPN protocol will reject the packet since the checksums do not match. In such a case
workarounds are available, such as NAT Transparency, IPsec over TCP/UDP, and NAT Traversal.
 NAT will hide the IP address information: End-to-end troubleshooting can be challenging with NAT
because a packet will get it's IP addresses changed along the way. A good understanding of the NAT
process is crucial before starting the troubleshooting process.
There are two ways to configure NAT. Besides the regular NAT syntax there is one newer that is called
NAT NVI.
Arguably, the most common issue with NAT is pure misconfiguration. Be especially careful that you are not
combining NAT and NAT NVI configurations since the syntax is a little bit different.
With NAT NVI you do not need to define NAT inside and NAT outside interfaces, you only need to enable
interfaces for NAT. Also the command that translates the addresses differs a little bit. In the example, notice
that the difference is only in the keyword inside, everything else is the same.
The NAT NVI order of operation is different from the legacy NAT order of operation. When the IP packet
enters the NAT router on any NAT enabled interface it gets matched against the NAT translation table. If
there's a match, it gets routed to a NAT Virtual Interface where the IP address gets translated. After the
address translation, the packet is routed again and forwarded to an egress interface. As you can see, NVI
differs from legacy NAT in that that the routing decision is now taken twice: before and after the translation
and the process is symmetrical - packet goes through the same order on the way from the client to the server
and on the way back.
The show ip nat translations command shows which interfaces are acting as the inside and outside
interfaces and the current number of static and dynamic translations.
When troubleshooting NAT, clear ip nat translation * is a useful command. It will remove all active
dynamic translations from the NAT table.
"Inside global" address is an IP address of an internal device as it appears to the external network. "Inside
local" address is an IP adress assigned to a device on internal network. "Outside local" address is an IP
address of an external device as it appears to the internal network. "Outside global" address is an IP address
assigned to an external device.
The debug ip nat command shows translations as they occur. The IP identification number can be used to
match the packets in the output with the packets captured with a protocol analyzer.
When looking at the NAT debug output, "s=" shows source address and "d=" shows destination address.
The "->" sign indicates a translation. The "*" sign tells you that all subsequent packets have been translated
and fast switched.
Use SSH to Connect to the Server
This topic offers an example troubleshooting flow for solving the issue of PC3 not being able to use SSH to
connect to the server at 172.16.200.10.
Carol, the user on PC3, cannot use SSH to connect to the server at 172.16.200.10.
Carol says that she was able to access the Internet site last week.
To verify the problem, try to connect from PC3 to the server at 172.16.200.10. Credentials were provided in
the Job Aids section.
problem.
Since you already verified that an SSH session cannot be established from PC3 to the internal server, you
should make your way down the OSI layers. By doing this you are using the "top-bottom" troubleshooting
method of narrowing down the problem.
With "top-bottom" or "top-down"approach you are seeking the highest functional OSI layer. Layers 3 and 4
are not working so the next step should be checking if Layer 2 is operational all the way from PC3 to the
internal server.
Information Gathering and Hypothesis Proposal
Issuing show ip interface brief on PC3, you can see that PC3's interface is operational at Layers 1 and 2.
On ASW2, you can use a combination of show cdp neighbors, show mac address-table, and show ip
interfaces brief to verify interface statuses. Alternatively you can use show interface description, if
interfaces are already configured with descriptions.
On ASW2, the port that connects to DSW1 (Ethernet 0/0) is seen as operational at Layers 1 and 2. Port that
connects to PC3 (Ethernet 0/1) is administratively shut down. This is one reason why PC3 cannot use SSH
to connect to the internal server!
The interesting part is that even though ASW2's Ethernet 0/1 is shut down, on the other end of the link, PC3
has Ethernet 0/0 declared "up/up". If client was connected directly to the switch and the switchport was shut
down, then the client's port would not be declared operational at first two Layers of the OSI model. This
behavior appears when there is an intermediary device between the client and the switch - like a VoIP phone
or a hub.
Note In this lab, having client's interfaces "up/up" is a feature of the IOL virtual system setup.
Bring up the shutdown interface on ASW2.

This is the only issue that prevented PC3 from connecting to the internal server via SSH. Your connection
test should now be successful.
Note Save the changes on ASW2 and inform the PC3 user that the problem has been fixed.
Documenting the changes that were made in the network is a crucial last step of every
troubleshooting.
Troubleshooting Interfaces
Interface is operational only when it has both, line and data-link protocol status in "up" state.
To verify the interface status, use the show ip interface interface slot/number or show ip interface brief
command.
The status refers to the hardware layer and reflects whether the interface is receiving the detected signal
from the other end. The line protocol refers to the data link layer and reflects whether the data link layer
protocol keepalives are being received.
Possible problems for interface not being operational:

 If the interface is up and the line protocol is down, some possible causes are no keepalives, mismatch in
encapsulation type, or a clock-rate issue.
 If the line protocol and the interface are both down, a cable might not be attached to a switch, or the
other end of the connection may be administratively down.
 If the interface is administratively down, it has been manually disabled.
Access Internet Through IPv6
This topic offers an example troubleshooting flow for solving the issue of PC4 not being able to access the
Internet through IPv6.
Mithun, the user on PC4 tells you that he is trying to access an Internet site at 2001:DB8:D1:A5:C8::2. He
also tells you that the user on PC2 has connectivity on this IPv6 site.
The ping from PC4 to 2001:DB8:D1:A5:C8 should not work.

The ping from PC2 to 2001:DB8:D1:A5:C8 should work.
The behavior was exactly the same as PC4's user confirmed it.
problem.
Mithun, the user on PC4, already gave you a lead-in on how to approach troubleshooting the issue. Since
ping works from PC2 but not from PC4, you could start by comparing configurations of these two client
devices.
If you compare configurations on PC2 and PC4, you can see that PC2 has an ipv6 address autoconfig
command, but PC4 does not.
At this point you might want to ask yourself if PC4 has an IPv6 address assigned. Even though PC4 does
not have IPv6 autoconfiguration enabled, this does not mean that it does not have IPv6 address assigned. It
might be that IPv6 address is configured statically.
In order for PC4 to have connectivity to Internet site at 2001:DB8:D1:A5:C8, it needs to have an IPv6
address configured. Right now it does not have one.
To fix PC4's IPv6 Internet connectivity issues, you could configure it with an IPv6 address manually or set
up a DHCPv6 server. However since PC2 is configured with stateless autoconfiguration it makes sense to
do the same for PC4. You want consistent configuration on devices in your network.
After PC4 acquires an IPv6 address, the connectivity test to IPv6 Internet site should be successful.
Note Inform the user that the problem has been fixed. Document the changes and close the ticket.
Troubleshooting IPv6 Address Assignment on
Clients
For communicating with other IPv6 devices, client must have IPv6 enabled and configured.
Stateless autoconfiguration allows a host to generate its own address using a combination of locally
available information and information that is advertised by routers. The host creates a global unicast IPv6
address by combining its interface's EUI-64 address, that is based on MAC address, or random ID and link
prefix, that is obtained via router advertisement.
As with IPv4 address, DHCP can be used to obtain IPv6 address. The DHCPv6 client can obtain
configuration parameters from a server either through a rapid two-message exchange (SOLICIT, REPLY)
or through a normal four-message exchange (SOLICIT, ADVERTISE, REQUEST, REPLY).
DHCPv6 can also run in the stateless mode. This mode is also known as DHCPv6 Lite. In that mode,
DHCPv6 does not assign addresses to clients. It only provides configuration information such as NTP
servers, domain names, DNS server, etc. If stateless DHCPv6 is used, clients must obtain routable IPv6
addresses through some other means, for example via stateless autoconfiguration.
Using the show ipv6 interface command, you can verify that the IPv6 address is configured on the device.
You can also see if it was configured with stateless autoconfiguration. The other way to check if the client
has obtained the IPv6 address, is to use the show ipv6 interface brief command.
If you have a client configured to obtain IPv6 address via DHCPv6, you can verify that the client received
the IPv6 address using the show ipv6 dhcp interface command. If you are using DHCPv6 Lite, the output
is slightly different. There is only the domain name and the DNS server information, but no IPv6 address.
Summary
Lesson 2: Debrief of the
Second Troubleshooting at
Overview
This lesson serves as a debrief for the second troubleshooting lab at SECHNIK Networking Ltd.

• Describe how you solved PC1's connectivity problems to the Internet
• Troubleshoot routing loops
• Describe how you solved PC2's SSH connectivity issues to the internal server
• Explain the TCP three-way handshake
• Describe how you fixed port security configuration
• Troubleshoot error-disabled port state
The text introducing trouble tickets was the following:
Life at SECHNIK Networking Ltd. is never dull. Just yesterday, your colleague Peter tried to do some
upgrades on the network.
 Kimberly, the user on PC1, cannot access the Internet. Peter is unable to offer further explanation on
this issue.
 Peter explained to you that he tried to configure a security feature where users could only establish an
SSH session to the server (172.16.200.10), but nobody would be able to establish a session from the
server. However the feature is not functioning properly. Andrew, the user on PC2, now cannot use SSH
to connect to the server.
 Peter also performed port security configuration on access layer switches. However, now Mithun, the
user on PC4, does not get an IP address assigned via DHCP.
Peter is kindly asking you to help him implement both security features correctly and to fix Internet
connectivity.
Access the Internet Host
Kimberly, the user on PC1, cannot access the Internet. She is trying to connect to one specific host on the
Internet.
Before starting the troubleshooting process, you should confirm that the problem really exists. Try to
reproduce the original issue: non working ping from PC1 to the Internet host.
You can see that the ping is not successful, therefore the problem is real and still present.
Note This is one possible plan. Different approaches may yield faster resolution of the ticket.
The first step of troubleshooting is to define the problem. Facts of the matter are these:
 Kimberly cannot access a host on the Internet with an address 209.165.201.225.
 Kimberly complained that the Internet is not working for her. Given only one problematic address, the
rest of the Internet may work just fine.
The problem definition is therefore: there is no end-to-end connectivity between PC1 and Internet host
209.165.201.225.
If the fault is in your network, it can be anywhere between (or on) PC1 and the edge of the network. You
should use the “follow the path” method to reduce troubleshooting to the devices in the path of the traffic.
When the problem is found, it must be fixed and all changes documented.
As mentioned, you are using the “follow the path” method. The first tool that you should use is traceroute.
The output of traceroute to 209.165.201.225 indicates that packets are being sent back and forth between
the two devices with IP addresses 172.16.100.1 and 172.16.100.2. These two devices are two and three hops
away.
From this output the following conclusions can be made:
1 This is not a Layer 2 issue. You reach at least one hop away from PC1, therefore any ARP or Ethernet
problem at PC1 can be excluded. Layer 3 is working fine on PC1.
2 Traffic is looping between hop 2 and hop 3. This is a Layer 3 loop, which indicates some sort of routing
loop.
Looking at the network schematic, you can guess that hops are:
1 ASW1
2 DSW1
3 R1
Note This is a guess, as that ASW1 could be a Layer 2 device. Therefore the conclusion is that
best candidates for hops 2 and 3 are DSW1 and R1.
To confirm the guess, check IP addresses on devices R1 and DSW1.
You can see that:
 R1 has address 172.16.100.1.
 DSW1 has addresses 172.16.100.2 and 192.168.20.1.
You can conclude that:

 ASW1 is indeed Layer 2 device.
 First hop is DSW1.
 Second hop is R1.
 Third hop is DSW1.
 Traffic loops between DSW1 and R1.
Next, check the routing information on R1 and DSW1. First check R1, because when traffic reaches R1 it
goes back to DSW1 whence it came from instead of going out towards the Internet.
You see that:

 R1 has a default route, probably pointing towards the ISP.
 R1 has a more specific static route: 209.165.201.0/24, pointing back to DSW1.
If you want to see exactly which route the router uses when sending traffic to 209.165.201.225, you can use
the following command:
R1# show ip route 209.165.201.225
Routing entry for 209.165.201.0/24
Known via "static", distance 1, metric 0
Routing Descriptor Blocks:
* 172.16.100.2
Route metric is 0, traffic share count is 1
This command shows you, which routing entry is used for the destination specified as the command
parameter. This is especially useful when routing table contains hundreds or thousands of entries and
manual analysis cannot be relied upon.
Then check DSW1 where you see only default route. DSW1 therefore correctly sends traffic to
209.165.201.225 towards the Internet.
Again, to undoubtedly confirm that you can use the following command:
DSW1# show ip route 209.165.201.225
% Network not in table
Note that default route always matches, if there are no other more specific routes. The output above
indicates that there are in fact no specific routes, other than possible default. In this case, above command
alone is not enough and also requires a check if there is a default route:
DSW1# show ip route | include \*
ia - IS-IS inter area, * - candidate default, U - per-user static route
O*E2 0.0.0.0/0 [110/1] via 172.16.100.1, 3d04h, Ethernet0/0
Proposing the Hypothesis
Hypothesis is that R1 has an incorrect static route which sends all traffic destined to 209.165.201.0/24 back
to DSW1. As DSW1 files this subnet under “everything else” it sends it back to R1 resulting in a Layer 3
loop.
The solution to the problem is to remove the offending static route from configuration.
To test the hypothesis, locate the static route in the configuration and then remove it.
Note In case this process does not solve the issue, you should make a backup of the
configuration, either on the device you are changing or offline.
To check if your change has resolved the issue, you can either recheck every step or try to reproduce the
original problem. Notice that the ping now works and you can conclude that this was in fact the problem.
Note To make your changes permanent, you must save the configuration. You only modified the
configuration on R1 so that is where you must save it. You should also call Kimberly and let
her know that the issue is resolved and that she can now access the Internet host and
resume her work. As the last step, you must document your changes so that documentation
reflects the actual state of the network.
Troubleshooting Network Layer Connectivity
To route packets, a router combines information from various control plane data structures. The most
important of these data structures is the routing table. Unlike switches, which flood frames on all ports if
they cannot find an entry in their MAC address table, routers will drop any packet for which they cannot
find a matching route in the routing table.
When a packet needs to be routed, the routing table is searched for the longest possible prefix that matches
the destination IP address of the packet. Associated with this entry is an egress interface and, in most cases,
a next-hop IP address.
For point-to-point egress interfaces, such as a serial interface running PPP or HDLC, a next-hop IP address
is not mandatory, because all the information that is necessary to construct the frame and encapsulate the
packet can be derived from the egress interface itself.
For multipoint egress interfaces, such as Ethernet interfaces or multipoint Frame Relay or ATM
subinterfaces, the next-hop IP address is a mandatory element, because this next hop is necessary to find the
correct Layer 2 destination address or other Layer 2 identifier to construct the frame and encapsulate the
packet. The mapping between the next-hop IP address and the Layer 2 address or identifier is stored in a
data structure that is specific for that Layer 2 protocol. For example, in the case of Ethernet, this information
is stored in the ARP cache and for Frame Relay multipoint interfaces, the information is stored in the Frame
Relay map table.
So, a routing table lookup might need to be followed up by a lookup in a Layer 3 to Layer 2 mapping table
to gather all the necessary information that is required to construct a frame, encapsulate the packet, and
transmit it.
Executing all these different table lookups and combining the information to construct a frame every time a
packet needs to be routed is an inefficient approach to forwarding IP packets. To improve this process and
increase the performance of IP packet switching operations on routers, Cisco has developed Cisco Express
Forwarding. This advanced Layer 3 IP switching mechanism can be used on all routers and is at the core of
the Layer 3 switching technology that is used in Cisco Catalyst multilayer switches. On most platforms, the
Cisco Express Forwarding switching method is enabled by default.
In essence, Cisco Express Forwarding combines the information from the routing table and various Layer 3
to Layer 2 tables into two new data structures, the FIB and the adjacency table. The FIB mostly reflects the
routing table, supplemented with entries for directly connected hosts and multicast entries. A lookup in the
FIB results in a pointer to an adjacency entry in the adjacency table. Similar to what you have seen in the
routing table, an adjacency entry can consist of an egress interface only for point-to-point interfaces or an
egress interface and next-hop IP address for a multipoint interface. The adjacency entry contains the exact
frame header that is necessary to encapsulate packets destined for that adjacency.
To verify the information that is used to route packets, you can verify the availability of specific routes in
the routing table or the Cisco Express Forwarding FIB. The choice whether to check the routing table or FIB
is dependent on what you are exactly trying to diagnose. To diagnose control plane problems, such as the
exchange of routing information by routing protocols using the show ip route command is a clear choice,
because it contains all the control plane details for a route, such as the advertising routing protocol, routing
source, administrative distance, and routing protocol metrics. To diagnose problems that are more closely
related to the data plane, for example by tracking the exact traffic flow between two hosts through the
network, the FIB is often the best choice, because it contains all the details that are necessary to make
packet switching decisions.
To display the content of the routing table, you can use the following commands:
 show ip route ip-address: Supplying the destination IP address as an option to the show ip route
command will cause the router to perform a routing table lookup for that IP address and display the best
route that matches the address and all associated control plane details.
 show ip route network mask: By supplying the network and mask as an option to the show ip route
command, the routing table will be searched for an exact match for that network and mask, and if it is
found, this entry is displayed with all of its associated control plane details.
 show ip route network mask longer-prefixes: Using the longer-prefixes option will cause the router to
display all prefixes in the routing table that fall within the prefix specified by the network and mask
parameters. This command can be very useful to diagnose problems that are related to route
summarization.
To display the content of the Cisco Express Forwarding FIB, you can use the following commands:
 show ip cef ip-address: This command is very similar to the show ip route ip-address command, but it
searches the FIB instead of the routing table. Therefore, the displayed results do not include any routing
protocol-related information, but only the information that is necessary to forward packets. (Note that
this command will display the default route if it is the best match for a particular IP address.)
 show ip cef network mask: This is similar to the show ip route network mask command, but displays
information from the FIB instead of the routing table.
 show ip cef exact-route source destination: This command will display the exact adjacency that will be
used to forward a packet with source and destination IP addresses, as specified by the source and
destination parameters. The main reason to use this command is in a situation when you are tracking a
packet flow across the routed network, but the routing table and FIB contain two or more equal routes
for a particular prefix. In this case, the Cisco Express Forwarding mechanisms will balance the traffic
load across the multiple adjacencies associated with that prefix. By use of this command, you can
determine which of the possible adjacencies is used to forward packets for a specific source and
destination IP address pair.
After an egress interface and, in case of multipoint interfaces, a next-hop IP address for the destination of a
packet has been determined by the routing table or FIB, the router needs to construct a frame for the data
link layer protocol associated with the egress interface in order to be able to encapsulate and transmit the
packet. Depending on the data link layer that is used on the interface, the header of this frame will require
some connection specific parameters, such as source and destination MAC addresses for Ethernet.
These data link layer parameters are stored in various different places. For point-to-point interfaces and
subinterfaces, the relation between the interface and data link identifier or address is usually statically
configured. For multipoint interfaces and subinterfaces, the relation between the next-hop IP address and the
data link identifier and address can be manually configured or dynamically resolved through some form of
an address resolution protocol. The commands to display the statically configured or dynamically obtained
mappings are unique for each data link layer technology. Research the command references for the data link
layer protocol that you are troubleshooting to find the appropriate commands for that protocol.
The show ip arp command can be used to verify the dynamic IP address to Ethernet MAC address
mappings that were resolved by the Address Resolution Protocol. Routers cache this information for 4 hours
by default, so if a change is made to a host or other router that would require this information to be
refreshed, you may need to issue the clear ip arp command to clear a particular entry from the cache in
order to allow it to be refreshed.
When Cisco Express Forwarding is used as the switching method, the information from the various Layer 2
data structures is used to construct a frame header for each adjacency that is listed in the adjacency table.
The full frame header that will be used to encapsulate the packet can be displayed by use of the show
adjacency detail command. In addition, this command displays packet and byte counters for all traffic that
was forwarded via this adjacency.
When do you typically resort to verifying the Layer 3 to Layer 2 mappings during a troubleshooting
process? When you have verified that the routing table, or even better the FIB, lists the correct next-hop IP
address and egress interface for a particular destination, but packets do not arrive at that next hop, you
should verify the Layer 3 to Layer 2 mappings for the data link protocol that is used on the egress interface.
Specifically, you should verify that a correct frame header is constructed to encapsulate the packets and
forward them to the next hop.
Example Troubleshooting Flow: PC2 Cannot Use
SSH to Connect to Internal Server
Peter explained to you that he tried to configure a security feature where users could only establish an SSH
session to the server (172.16.200.10), but nobody would be able to establish a session from the server.
However the feature is not functioning properly. Andrew, user on PC2, now cannot use SSH to connect to
the server.
You must make sure that the feature is implemented correctly and this must allow Andrew to connect to the
SERVER.
Before starting the troubleshooting process, you should confirm that the problem really exists. Try to
reproduce the original issue: non working SSH from PC2 to SERVER.
You can see that the SSH attempt is not successful, therefore the problem is real and still present.
Not only is the reported problem present – Peter did not accomplish his original goal of implementing a
security policy properly. Trying to establish a telnet connection through port 22 to 1.1.1.1 is not successful.
1.1.1.1 is OSPF router ID of R1. Performing a telnet to port 22 is a great way to test SSH connection
without needing to specify SSH credentials.
Known facts is that SSH does not work from PC2 to Server.
You also know from what Peter explained that:
 SSH must work from PC2 to Server.
 SSH must not work from Server to anywhere. For testing purposes use destination address 1.1.1.1, an
address of a device on the Internet which has a working SSH server configured on it.
Planned troubleshooting procedure is the following:

 Try to gather as much information about the problem.
 Locate the problem.
 Identify the problem.
 Fix the problem.
 Document the changes and inform the user.
Very important step of any troubleshooting process is to gather more information, if it is not immediately
obvious what and where the problem is.
Your first step in this plan is to gather more information about the location of the problem.
To do that, try to check where SSH works and where it does not.
Telneting to port 22 is a way to see if the service is active without specifying username. Also, this type of
test shows you SSH version, if that is something you wish to know. In certain cases platform name is
included in the version string.
When testing SSH connectivity from R1 and DSW1 (devices in the traffic’s path) you can see the following
results:
1. SSH works from R1 to the server.
2. SSH does not work from DSW1 to the server.
3. Where SSH does not work, the error message response is immediate (you do not have to wait several
seconds for the result).
From facts 1 and 2 you can deduce that the problem is probably on the R1 or DSW1.
From fact 3 you can deduce that one of the following cases are likely causes for the error:
 DSW1 locally cannot reach the destination (no route).
 DSW1 receives a message saying that the destination is unreachable.
From resolving the first problem, you already know that there is a default route on DSW1 so you can
assume higher probability for the second case. There are two ways DSW1 can be told that it cannot connect
to port 22 on 172.16.200.10: via ICMP unreachable message or via a TCP RST packet.
To see the ICMP messages, you can use debug ip icmp command. This command will show all IPv4 ICMP
packets which are handeled by router’s or switch’s main CPU. Usually this means only control and
management traffic, which is low, but caution is advised whenever issuing such commands on devices
under load.
To see inbound RST packets, there are several ways. One of them is to use debug ip tcp packet in port 22
in your case. This will show all incoming TCP packets from or to port 22. Be advised that such debugs will
show all TCP packets, including those of existing sessions. Therefore using this command on an active
device with active SSH sessions will produce a lot of debug output.
Note When using debug commands over vty lines, make sure that you also use terminal
monitor command.
You can try to see ICMP packets first, because this is usually easier, with less output to check.
When retrying to connect to Server’s SSH port you can see received ICMP destination unreachable packet
with code “communication administratively prohibited”. This most often indicates ACL dropping packets.
In the debug message, you can also identify destination address, this is the address of the PC2, and source
address. The source address tells you who is sending you the message and it must be the device with the
ACL configured.
Note From the error message it is actually evident that an ICMP packet is received, because an
ICMP packet or no route produces this message:
% Destination unreachable; gateway or host down
Whereas a TCP RST packet would produce the following message:

% Connection refused by remote host
Hypothesis Proposal
Hypothesis is that a device with an address 172.16.100.1 has an ACL configured and this ACL drops SSH
traffic.
This also does not contradict the original information which states that a security feature should have been
implemented. In fact the basic security features are implemented using ACLs. An incorrect implementation
of an ACL could certainly produce observed behaviour.
Your task now is to locate the device with the address in question and the ACL on it.
After you must correct the configuration to implement the required security feature properly.
Locating the Device
In your first steps you narrowed the problem to two devices: DSW1 and R1.
You should check addresses first on these two devices.
By using show ip interface brief | exclude unass you can see only those interfaces and ports which have
addresses on them. Especially useful on switches.
After looking for the address you find it on R1, on Ethernet 0/2. Interface ethernet 0/2 must be the interface
towards R1 but this does not indicate where the ACL is.
Locating the ACL
The next step is to locate the ACL on R1.

On R1 you can issue the following command show running-config | include ^interface|access-group to
see all configuration lines containing access-group and all configuration lines beginnig with interface. The
output of such show command is easy to parse and will reveal which interfaces have ACLs on them.
You can see that interface ethernet 0/0 has an ACL 111 configured in the outbound direction. Since this is
the only access-group command, there are no other ACLs attached on any other interface.
Analysis of the ACL
Your next step is to analyse the ACL.

Command show interface description shows configured interface descriptions, but they must be there.
Fortunately in your case someone has configured them. You should use show cdp neighbors command to
verify if interface descriptions are configured correctly.
The ACL is attached on ethernet 0/0 which goes towards the server. Ethernet 0/1 is not relevant because it
goes towards the Internet. Ethernet 0/2 goes towards PC2.
Based on your knowledge of TCP handshake (SYN, SYN/ACK, ACK) you can see that SYN packets going
from PC2 to the server will be dropped by the ACL because they will not match the only entry. Address
parameters, namely source and destination addresses, do match the traffic flow.
Designing a Solution
Now when you are fully aware of the situation it is time to design a working solution.
You already know the way ACL works and why it does not produce the desired result. There are several
solutions.
The one you pick is the following:

 Keep the old ACL, to allow quick fallback.
 Put the new ACL on the same interface.
 Alter parameters to achieve desired result.
Since we do not have “match SYN packets” keyword you must match absence of them. Presence of SYN
flagged TCP packets is determined by the traffic flow therefore the only solution is to turn around the ACL.
Which also means you must swap source and destination addresses.
Deploying the Solution
To write the new access list you will take the old ACL, 111, and swap destination and source.
“From 192.168.0.0 0.0.255.255 to host 172.16.200.10” becomes from “from host 172.16.200.10 to
192.168.0.0 0.0.255.255”.
You can pick another unused ACL number, for example 112.
You remove the old ACL and attach the new one in the opposite direction on the same interface.
Testing the hypothesis is easy. Repeat the checks that you made at the beginning when you verified the
problem.
You can see that SSH now works as it should:

 It is possible to connect to the server from PC2.
 It is not possible to connect anywhere from the server (in the example connection is attempted to
1.1.1.1, which is OSPF router ID of R1).
Note Final steps include deleting the old and now redundant ACL and saving the configuration.
You should also inform Andrew that he can now access the server. Nothing changes in the
network documentation.
TCP Handshake
The TCP is one of the core protocols of the IP suite. TCP provides a reliable means of data transport.
TCP establishes the connection by using a process that is called the three-way handshake. Control bits in the
TCP header indicate the progress and status of the connection. The three-way handshake processing is as
follows:
 Establishes that the destination device is present on the network.
 Verifies that the destination device has an active service and is accepting requests on the destination
port number that the initiating client intends to use for the session.
 Informs the destination device that the source client intends to establish a communication session on
that port number.
This process involves setting the SYN bit and ACK bit in the segment header between the two devices.
Another important function that is performed during connection establishment is that the first device
informs the second device of the ISN, which is used to track data bytes on this connection. This table
includes a simplified explanation of the three-way handshake process.
FTP, HTTP, HTTPS, SMTP, IMAP, POP3, Telnet, SSH, or any other protocol that uses TCP for transport,
has three-way handshake performed as connection is opened.
TCP Connection Setup Procedure
Step Action Notes
1 The connection requestor sends a The synchronization segment specifies the number of the
synchronization segment to the receiving port to which the sender wants to connect. The
device (SYN bit set), which starts the synchronization segment also contains the ISN value to be
handshake process. used by the acknowledgment process.
2 The receiving device responds with a segment The receiving device responds by indicating the sequence
with the SYN bits and ACK bits set to number of the next byte of data that is expected from the
negotiate the connection and acknowledge sender. The next sequence number is the ISN of the
receipt of the synchronization segment of the sender, incremented by one.
sender.
3 The initiating device acknowledges the The SYN bit is unset in the TCP header, which confirms
synchronization segment of the receiver. that the three-way handshake is completed.
TCP performs sequencing of segments with a forward reference acknowledgment. The forward reference
acknowledgment comes from the receiving device and tells the sending device which segment it is
expecting to receive next.
One of the functions of TCP is to make sure that each segment reaches its destination. The TCP services on
the destination host acknowledge the data that it has received from the source application.
For this lesson, the complex operation of TCP is simplified in a number of ways. Simple incremental
numbers are used as the sequence numbers and acknowledgments, although in reality the sequence numbers
track the number of bytes that are received. In a TCP simple acknowledgment, the sending computer
transmits a segment, starts a timer, and waits for acknowledgment before transmitting the next segment. If
the timer expires before receipt of the segment is acknowledged, the sending computer retransmits the
segment and starts the timer again.
Imagine that each segment is numbered before transmission. At the receiving station, TCP reassembles the
segments into a complete message. If a sequence number is missing in the series, that segment and all
subsequent segments may be retransmitted.
Example Troubleshooting Flow: PC4 Does Not
Acquire IP Address Via DHCP After Port Security
Is Implemented
Peter also performed port security configuration on access layer switches. However, now Mithun, user on
PC4, does not get an IP address assigned via DHCP anymore.
With this trouble ticket you could have skipped verifying that PC4 does not have an IPv4 address assigned,
straight to investigating access switch's port security configuration. However, it is a good habit to first verify
reported trouble ticket before diving into troubleshooting.
Since Peter already provided you with a clue of where might the trouble lie, you can dive straight into
checking port security configuration on access layer switches.
Since PC4 connects to ASW2, it makes sense to check port security configuration on this switch. ASW2 is
also one of two access layer switches that Peter configured with port security.
You can see that ASW2 has Ethernet 0/0 configured with port security and it's violation counter is
increased. By doing further investigation you can see that Ethernet 0/0 is indeed in error-disabled mode.
There are multiple possibilities of why port is in error-disabled mode, one of them being port security. Since
Peter told you issues appeared after he configured port security, it is a reasonable assumption that port is
error disabled due to misconfigured port-security.
On ASW2 you should discover that it's Ethernet 0/0 connects to DSW1. You can also see that interface is
not operational, which correlates with interface being in error-disabled state. You can also investigate
connections using show cdp neighbors command.
Peter incorrectly configured port security on ASW2. Port security should be configured on access ports-
ports that connect to end-devices. In ASW2's case, port security should be configured on ports that connect
to PC3 (Ethernet 0/1) and PC4 (Ethernet 0/2).
At this point you should remove port security configuration from ASW2's trunk link. In order to enable
error-disabled port, you need to first shut it down and the bring it back up.
Now PC4 should acquire IPv4 address via DHCP. However this can take a while and you can speed-up this
process by restarting PC4's Ethernet 0/0.
Even though now PC4 has an IPv4 address, you still need to perform port security configuration on ASW2.
Also ASW1 needs to be checked if it's port security configuration is correct.
Implement Solution
To discover MAC addresses of PC3 and PC4, you can use show mac address-table. To bind port to MAC
address use switchport port-security mac-address mac-address command. Do not forget to enable port
security. You do that by issuing switchport port-security command.
If you issue show port-security on ASW1 you will see there is no port security configured.
You can configure port security in a similar fashion like you did on ASW1.
Verify the Solution
By issuing show port-security you should be able to confirm that Ethernet 0/1 (connects to PC1) and
Ethernet 0/2 (connects to PC2) are configured with port security.
By issuing show port-security you should be able to confirm that Ethernet 0/1 (connects to PC3) and
Ethernet 0/2 (connects to PC4) are configured with port security.
You should also check if PC1, PC2, PC3, and PC4 have connectivity to the Internet.
Note Save configurations on ASW1 and ASW2. Do not forget to inform the user and document
the changes.
Troubleshooting Error-Disabled Port
Error-disabled interface status is a feature of Cisco Catalyst switches that protects the network from
misconfiguration and equipment failure.
After an error is detected on the interface it will be shut down by the switch and it declared to be in "error-
disabled" state. After interface is put into error-disabled state administrator must remove the offending
configuration or replace failed part. Then to enable the port the administrator must first shutdown the port
and then bring it back up.
These are some common reasons of why a port is put into error-disabled state:
 Port security: Using port security administrator can specify which hosts can connect to a specific
interface. This is done by interface-to-MAC mappings. If a frame is received with unauthorized MAC
address that interface is put into error-disabled state. However port is shut into error-disabled state only
if violation is set to "shutdown". Violation modes "restrict" and "protect" will not put port into error-
disabled state.
 Spanning-tree BPDUGuard: BPDUGuard is used to protect PortFast-enabled ports. PortFast-enabled
port will go into forwarding state immediately after it's enabled by the administrator. Only interfaces
that connect to end-user devices should be configured with PortFast. If somebody connects a switch to
two PortFast-enabled ports that can cause a Layer 2 loop. BPDUGuard is used on PortFast. If BPDU is
received on that interface BPDUGuard will shut it down, thus preventing a Layer 2 loop.
 UDLD: Commonly used on fiber connections where physically separated fiber strands are used for
transmitting and receiving data. If transmitting or sending of data fails that can result into a Layer 2
loop. UDLD mechanism will prevent a Layer 2 loop by shutting down unidirectional interfaces.
 EtherChannel misconfiguration: When you are configuring EtherChannel it is important that all
parameters are identical between interfaces that are bundled. Misconfigured EtherChannel can cause
interfaces to be put into error-disabled state. A common misconfiguration problem is speed and duplex
mismatch.
 Other issues: DHCP snooping rate-limiting, non-Cisco GBIC inserted, excessive collisions (broadcast
storms), duplex mismacth, falpping link, and PAgP flapping.
You can use show errdisable detect command to discover all possible reasons why a port can become
error-disabled.
When port becomes error-disabled, there are multiple ways how this can be detected. On a switch the LED
light goes from green to orange and network is not functioning as expected. If you have Syslog or SNMP
notifications set-up, you will also receive a notification about the error.
To check interface status use the show interfaces interface slot/number command:
ASW1# show interface ethernet 0/0
Ethernet0/0 is down, line protocol is down (err-disabled)
The next step of troubleshooting an error-disabled port is to figure out what is the reason behind that. If you
just re-enable the port it is most likely problem will reoccur.
The switch operating system will notify you that port got error-disabled and the reason behind it. This is an
example where interface Ethernet 0/0 was put into error-disabled state due to BPDUGuard activation.
Somebody plugged in a switch into an access port that is configured with BPDUGuard.
*Dec 17 11:08:54.917: %PM-4-ERR_DISABLE: bpduguard error detected on Et0/0, putting
Et0/0 in err-disable state
You can also use show interfaces status err-disabled command to verify what is the reason behind port
being shut down.
ASW1# show interfaces status err-disabled
Port Name Status Reason Err-disabled Vlans

Et0/0 link to DW1 err-disabled bpduguard
The next step is to resolve the issue that is behind port being error-disabled. In this example you would
disconnect the switch that connects to the access port.
The last step is to get port back to operational state. To do that you need to first shut down and then bring up
the port:
Switch(config-if)# shutdown
Switch(config-if)# no shutdown
Note You can also configure error-disabled state with automatic recovery. This is where the
system will attempt to get port back to operational state after a specified time interval.
Summary
Third Troubleshooting at
Overview
This lesson serves as a debrief for the second troubleshooting lab at SECHNIK Networking Ltd

• Describe how you solved PC1's and PC2's connectivity problems to the Internet
• Describe how to troubleshoot DHCP
• Describe how passive interfaces behave with different routing protocols
• Describe how you solved the issue of PC3 not being able to connect to IPv6 Internet
• Describe basic IPv6 addressing
At SECHNIK Networking Ltd. headquarters there are again problems. Peter is asking you to help him with
two problems.
 Users on PCs 1 and 2 are reporting that they don't have Internet access. They are both trying to access
site at 209.165.201.225.
 Your ISP informed you that they are changing your first hop IPv6 address from 2001:DB8:D1:A5:C8::2
to 2001:DB8:D1:A5:C8::33. Peter reconfigured R1 and informed the ISP that they can go ahead and
change the IPv6 address on their end. Carol, the user on PC3, quickly reported back to Peter that she
cannot access an Internet site at 2001:DB8:AA::B.
Peter is also telling you that the internal server was acting strangely yesterday. He is asking you not to bring
up the interface Ethernet0/0 on R1, that connects to the internal server.
Example Troubleshooting Flow: PC1 and PC2
Cannot Ping Internet Host
This topic offers an example troubleshooting flow for solving the issue of PCs 1 and 2 not being able to
access the Internet.
Users on PC1 and PC2 cannot access Internet. More specifically, they are trying to connect to one specific
host on the Internet. Peter, your coworker, could not tell you anything further about this issue.
Before you start the troubleshooting process, you should confirm that the problem really exists. Try to
reproduce the original issue: an unsuccessful ping from PC1 and PC2 to the Internet host.
You can see that the ping is not successful, so the problem is real and still present.
These outputs also provide you with a hint. The "% Unrecognized host or address, or protocol not running."
tells you that PC1 and PC2 do not have an IP address configured.
Note This is one possible troubleshooting plan. There can be many different approaches to this
problem.
Facts of the matter are these:

 Users on PC1 and PC2 cannot access the Internet host with an address 209.165.201.225.
 Users complained that the Internet is not working for them. Given only one problematic address, the
rest of the Internet may work just fine.
The problem definition is therefore: there is no end-to-end connectivity between PC1, PC2, and the Internet
host 209.165.201.225.
If the fault is in your network, it can be anywhere between (or on) PC1, PC2, and the edge of the network.
The verification step gave you a clue where to start troubleshooting—why PC1 and PC2 do not have an
IPv4 address assigned.
According to the outputs, the configured IP address assignment method for PC1 and PC2 is DHCP but the
IP address is not obtained successfully.
Next, you should verify how DHCP is set up in your network.
Execute the show running-config | section dhcp command to verify if the DHCP server is configured on
R1. The only two reasonable guesses as in which device is the DHCP server are DSW1 and the internal
server. However eventually you will figure out that R1 is the designated DHCP server for your network.
On ASW1 you should be able to verify, using show vlan command, that PC1 is a member of VLAN 10 and
PC2 is a member of VLAN 20.
PC1 should be assigned IP address in the 192.168.10.0/24 subnet and default gateway 192.168.10.1. PC2
should be assigned IP address in the 192.168.20.0/24 subnet and default gateway 192.168.20.1.
By investigating the running configuration on either ASW1 or DSW1, you can verify that ASW1-DSW1
link is a Layer 2, trunk link. Similarly you should be able to deduce that DSW1-R1 is a Layer 3 link.
The output on DSW1 verifies that the ip helper address configuration exists. However, that does not
necessarily mean that the configuration is correct.
At this point you can use the debug tool on R1 to investigate DHCP communication.
In the outputs you can notice that the DHCPDISCOVER message is received through the relay ip address of
192.168.10.1. You have confirmed in previous steps that this is the IP address on DSW1's VLAN10
interface. R1, as the DHCP server, sends a DHCPOFFER message back. However there is no
DHCPREQUEST message. In other words, there is no response to the DHCPOFFER message. Because
there is a DHCPDISCOVER message you can safely assume this message came from PC1. Since there was
no reply to the DHCPOFFER message it seems that it never got delivered.
At this point you should ask yourself, does R1 has a path to the 192.168.10.0/24 subnet?
When you check the routing table for R1, you can see that there is no specific routing entry for
192.168.10.0/24 and 192.168.20.0/24 subnets. Packets destined for VLAN10 and VLAN20 networks match
the R1's default route entry. The default route points to the Internet.
After discovering that R1 does not have OSPF neighbors, you can investigate further, issuing show ip
protocols. You will see that Ethernet0/2 is activated for OSPF Area0 routing, however you can also notice
that this same interface is configured as passive. This means Ethernet 0/2 is disabled from "talking OSPF".
You can discover that R1 connects to DSW1 by issuing show cdp neighbors or show interfaces
description command.
So R1 does not have route to VLAN 10 and VLAN 20 subnets because it did not receive routes via OSPF.
R1 did not receive routes via OSPF because it is not an OSPF neighbor to DSW1. R1 is not DSW1's
neighbor because it has interface that connects to DSW1 (Ethernet 0/0), configured as passive.
After passive-interface default command is issued, all other interfaces must be explicitly configured as
non-passive.
After you fixed the OSPF adjacency problem R1 should be able to get VLAN10 and VLAN20 information
from DSW1.
To speed-up the process of address acquiring via DHCP you can shutdown and then bring back up PC1's
Ethernet 0/0.
Since PC1 can acquire an IP address from R1 and PC2 cannot it makes sense to investigate R1's and
DSW1's configuration and compare configurations for VLAN10 and VLAN20.
If you investigate DHCP configuration on R1, you should deduce that there are no suspicious differences
between VLAN10 and VLAN20 configurations.
When you compare DHCP relay configurations on DSW1, you will see that IP helper addresses are
different. By itself there is nothing wrong with that. R1 can have multiple interfaces with different IP
addresses configured. However since DHCP works for PC1, but not for PC2, it makes sense to investigate if
you can ping 172.16.200.1.
At this point you could test your hypothesis and change VLAN 20 helper address on DSW1 to the one that
VLAN 10 uses (172.16.100.1). However it is smart to investigate R1 for interface configured with
172.16.200.1 IP. Before applying changes to the configuration you should investigate the network and try to
understand why it is set up the way it is.
The IP address of 172.16.200.1 belongs to Ethernet 0/0 on R1. This is the interface Peter told you to leave
shut down.
So DSW1 has IP address helper for VLAN 20 pointed to a shutdown interface. This is why VLAN 20
clients are unable to acquire IP address via DHCP.
It is a bad practice to configure ip helper address to point to IP address of an interface that connects to an
end device. That interface can go down and as a consequence parts of your network are without a DHCP
service.
One way to solve the issue you have found is to change the IP helper address for VLAN 20 on DSW1 to the
same as it is for VLAN 10. The IP helper address used for VLAN 10 is on the link that connects R1 to
DSW1. This is not a bad solution, however it would make more sense to have IP helper addresses point to
one of R1's loopback interfaces.
After changing the IP helper address for VLAN 20 on DSW1, PC2 should acquire an IP address and have
connectivity to the Internet.
Note Your work is not finished. To make your changes permanent, you must save the
configuration on R1 and DSW1. As the last step, you must document your changes so that
documentation reflects the actual state of the network.
Troubleshooting DHCP
When you are troubleshooting a network issue and you suspect it is DHCP-related, consider the following
potential issues:
 Server Misconfiguration: Confirm that the following is correctly configured: DHCP pools, default
gateways, DNS server addresses, and excluded IP addresses.
 Duplicate IP addresses: There might be a client in the network that has an IP statically configured, but
the DHCP server hands out that same IP address to another client. Duplicate IP addresses can cause
connectivity issues for devices with same IP address.
 Redundant services not working: You can set up redundant DHCP server for the sake of reliability.
These redundant servers need to communicate between each other in order for the DHCP service to
function properly. If inter-server communication fails, server can end up handing out overlapping IP
addresses.
 DHCP pool runs out of addresses: A DHCP pool has finite number of IP addresses. If this pool gets
depleted, requests for new IP addresses will be rejected.
 A router not forwarding addresses: If your DHCP server is not in the same subnet as the DHCP
server then the intermediary Layer 3 devices need to be configured with an ip helper address. A router
does not by default forward broadcast messages-that includes DHCPDISCOVER broadcast message.
 Client is not requesting an IP address: Client must be configured to acquire an IP address through
DHCP. Also remember DHCP is a "pull" system. A client will only acquire an IP after it requests it.
Server cannot push the IP to the client.
In the first example output you can see that IP address of 192.168.10.2 was assigned to DHCP client with
client-id of 0063.6973.636f.2d61.6162.622e.6363.3030.2e31.6230.302d.4574.302f.30. To clear all active
DHCP leases use the clear ip dhcp binding * command.
The second example output indicates that there is a duplicate IP address of 172.16.1.32 on the network.
Router1 discovered IP conflict via ping. You can resolve this conflict by issuing the clear ip dhcp conflict
* command.
To display information about the DHCP address pools, use the show ip dhcp pool command in user EXEC
or privileged EXEC mode.
Passive Interfaces with Different Routing
Protocols
The passive-interface command is used to disable sending of routing protocol information through a
specific interface. However, this command does not behave the same way with all protocols.
With OSPF, the passive-interface command suppresses hello packets and consequently routers will not be
able to build a neighbor relationship, and therefore both incoming and outgoing routing updates are stopped.
With EIGRP the passive-interface command has similar effect as with OSPF. The command suppresses
hello packets and with that the adjacency.
With RIP, the passive-interface command will disable sending multicast update via a specific interface but
will allow listening to incoming updates from other routers that "speak" RIP. So the router will still be able
to receive updates and do updates to the routing table.
BGP does not support passive-interface command. With BGP you can use filters access-lists, or distribute
lists to control communication.
You can also use the passive-interface default command and then configure individual interfaces where
adjacencies are desired, using the no passive-interface command.
You can investigate if passive interfaces are configured by issuing show running-config or show ip
protocols commands.
Note To configure passive interface in OSPF you can also use ip ospf passive-interface
interface-level configuration command.
Example Troubleshooting Flow: PC3 Cannot
Connect to the Internet
This topic offers an example troubleshooting flow for solving the issue of IPv6 Internet connectivity
problem.
Your ISP informed you that they are changing your first hop IPv6 address from 2001:DB8:D1:A5:C8::2 to
2001:DB8:D1:A5:C8::33. Peter, your coworker, reconfigured R1 and informed the ISP that they can go
ahead and change the IPv6 address on their end. Carol, the user on PC3, quickly reported back to Peter that
she cannot access an Internet site at 2001:DB8:AA::B.
Note This is one possible troubleshooting plan. There can be many different approaches to this
problem.
Since the ISP requested that you change the first hop IPv6 address and problems started to appear right after
that, it makes sense to start on R1, verifying Peter's configuration.
Peter was supposed to change IPv6 address to 2001:DB8:D1:A5:C8::33. However the IPv6 routing table
still has the old next-hop address of 2001:DB8:D1:A5:C8::2.
Your hypothesis is that Peter did not configure the new next-hop address. Or maybe he configured it, did
not save the configuration, and then the router reloaded.
It seems that your hypothesis was not entirely correct. Peter did configure the new IPv6 default route,
however for some reason he configured it with administrative distance of 2. At the same time he did not
remove the old entry. Since the old entry has the default administrative distance of 1, it is the route that gets
installed into the IPv6 routing table.
There are a few ways that you can solve the issue. You could remove the old IPv6 route. But a better way
would be to remove both configured routes and configure a new one, pointing to 2001:DB8:D1:A5:C8::33
2, leaving administrative distance to default.
Test from R1 and from PC3 to the user-specified Internet site should now be successful. You resolved the
problem.
R1's default route gets redistributed into the RIPng routing process on R1. R1 than exchanges default route
information with DSW1.
Note Save the configuration on R1, inform the user and document the changes made.
IPv6 Review
IPv6 offers great increase in address space, simplified header, and can coexist with IPv4.
With IPv6 there is no broadcast. IPv6 has three types of addresses:

 Unicast: One-to-one communication flow.
 Multicast: One-to-many communication flow.
 Anycast: With anycast a single IPv6 address is assigned to multiple devices. The communication flow
is one-to-nearest from the perspective of a router's routing table.
IPv6 address has 128 bits. Each 4 bits can be represented using a hexadecimal. Thus an IPv6 address has
eight fields with 4 hexadecimal digits each. IPv6 addresses can be abbreviated by eliminating leading zeros
and by representing contiguous fields containing all zeros with a double colon. You can only do this once in
an IPv6 address.
IPv6 maintains a separate routing table from IPv4.
These are methods of routing in IPv6 world:

 Static routes: Very similar to IPv4 static routes.
 OSPFv3: In comparison with OSPFv2 it does not only have IPv4 support, but also IPv6.
 EIGRP for IPv6: Configured in a similar fashion like OSPFv3.
 RIPng: RIP next generation is a distance vector protocol with 15 count maximum. It is very simple to
configure.
 IS-IS for IPv6: Similar to IS-IS for IPv4.
 Multiprotocol BGP: Allows BGP to route IPv6 and other protocols other than IPv4.
Summary
Overview
Check
1. There is no connectivity between devices in your network. You suspect that the issue is trunk link
configuration between switches. What commands could you use to verify the configuration of the trunk
ports? (Choose two.) (Source: Debrief of the First Troubleshooting At SECHNIK Networking Ltd.)
A. show interfaces trunk
B. show running-config
C. show trunking
D. show vlan trunk
2. Is the following a NAT NVI configuration? (Source: Debrief of the First Troubleshooting At SECHNIK
Networking Ltd.)
ip nat source static 10.0.0.1 209.165.200.211
A. Yes.
B. No. This is a legacy NAT configuration.
C. There is not enough information to deduce the answer.
D. Both. NAT NVI and regular NAT configuration would both work with this statement.
3. How did Router1 acquire an IPv6 address on Ethernet0/0? (Source: Debrief of the First Troubleshooting
At SECHNIK Networking Ltd.)
Router1# show ipv6 interface
Ethernet0/0 is up, line protocol is up
IPv6 is enabled, link-local address is FE80::A8BB:CCFF:FE00:500
No Virtual link-local address(es):
Stateless address autoconfig enabled
Global unicast address(es):
2001:DB8:A:0:A8BB:CCFF:FE00:500, subnet is 2001:DB8:A::/64 [EUI/CAL/PRE]
valid lifetime 2591961 preferred lifetime 604761
A. Via DHCPv6.
B. Via DHCPv6 Lite.
C. Via Stateless Autoconfiguration.
D. It was manually configured.
4. Which of the following static routes will send only traffic destined for the 172.16.14.0/24 network to the
next-hop IP address of 192.168.5.5? (Source: Debrief of the Second Troubleshooting At SECHNIK
Networking Ltd.)
A. ip route 172.16.14.0 255.0.0.0 192.168.5.5
B. ip route 172.16.14.0 255.255.255.0 192.168.5.5
C. ip route 192.168.5.5 172.16.14.0 0.0.0.255
D. ip route 192.168.5.5 172.16.14.0
5. Which command sequence will allow only traffic from the 172.16.16.0/24 network to enter the
Ethernet0/0 interface? (Source: Debrief of the Second Troubleshooting At SECHNIK Networking Ltd.)
A. access-list 25 permit 172.16.16.0 0.0.255.255
!
interface ethernet 0/0
ip access-list 25 out
B. access-list 25 permit 172.16.16.0 0.0.0.255
!
ip access-list 25 in
C. access-list 25 permit 172.16.16.0 0.0.0.255
!
ip access-group 25 in
D. access-list 25 permit 172.16.16.0 0.0.0.255
!
ip access-group 25 out
6. Which of the following port security violation modes will disable the port if violation occurs? (Source:
Debrief of the Second Troubleshooting At SECHNIK Networking Ltd.)
A. restrict
B. protect
C. shutdown
D. all of the above
7. Which statements correctly describe OSPF passive interface? (Select two.) (Source: Debrief of the
Third Troubleshooting at SECHNIK Networking Ltd.)
A. Passive interfaces do not send routing updates.
B. Passive interfaces do not accept routing updates.
C. It is configured using the ip ospf passive-interface interface configuration command.
D. When you configure OSPF, every interface is passive by default.
8. Which command is used to enable IPv6 routing on a router? (Source: Debrief of the Third
Troubleshooting at SECHNIK Networking Ltd.)
A. ipv6 unicast-routing
B. ipv6 routing
C. ip routing ipv6
D. ip routing unicast-ipv6
Answer Key
1 A, B
2 A
3 C
4 B
5 C
6 C
7 A, C
8 A
TINC Garbage Disposal Ltd.
Introduction
You work for SECHNIK Network Ltd. as a network engineer. TINC Garbage Disposal Ltd. is a customer
company. You are the engineer responsible that the customer's network is running smoothly.
In this module you will be faced with five challenge labs. Each lab has multiple troubleshooting tickets that
you need to investigate, analyze, and finally resolve
• Solve troubleshooting tasks for the first challenge lab at TINC Garbage Disposal Ltd.
• Solve troubleshooting tasks for the second challenge lab at TINC Garbage Disposal Ltd.
• Solve troubleshooting tasks for the third challenge lab at TINC Garbage Disposal Ltd.
• Solve troubleshooting tasks for the fourth challenge lab at TINC Garbage Disposal Ltd.
• Describe how you solved fourth challenge lab
Troubleshooting Lab at TINC
Garbage Disposal Ltd.
Overview
This lesson serves as a debrief for the first troubleshooting lab at TINC Garbage Disposal Ltd.
Example troubleshooting flows are provided, but keep in mind that there are multiple ways to
troubleshooting the problems.

• Describe the issues that you had to solve in the challenge lab
• Describe how you solved PC1 and PC2 Internet connectivity issues
• Describe the possible port security issues
• Describe how you solved the VLAN issue
• Describe the possible issues with the VLAN configuration
• Describe how you solved the misconfigured BGP AS number issue
• Describe the possible BGP neighborship issues

You work for SECHNIK Networking Ltd. and TINC Garbage Disposal Ltd. is your company's customer.
One day customer engineer Donovan calls.
There are a few network issues that he wants for you to help him with:
 During the maintenance of GW1, the customer realized that GW2 does not serve as the backup to the
Internet. Now, GW1 is connected back and there is Internet connectivity through GW1. However,
Donovan has strong suspicion that if GW1 fails, the whole network will be without Internet
connectivity again!
 Classroom PCs—PC1 and PC2—do not have Internet connectivity.
Note Network documentation is provided in the Job Aids.
Example Troubleshooting Flow: GW2 Does Not
Serve as the Backup to the Internet
Customer engineer Donovan stated that when GW1 was down during maintenance, GW2 did not serve as a
gateway to the Internet.
Verify the Problem
The first step in troubleshooting should always be the verification of the reported problem. Doing that, you
must not disturb the network flow and break Internet connectivity. Therefore instead of recreating the
situation where GW1 fails, you must find another way to verify the reported problem.
For overall Internet connectivity both redundant routers R1 and R2 must have the default route in their
routing tables. According to the Job Aids, the default routes are being redistributed from the BGP routing
process at OSPF border routers GW1 and GW2. The output of the show ip ospf database command shows
that the default route is being advertised only from the GW1 router, which has the IP address 209.165.200.1.
Analyzing the Information
GW2 cannot serve as a backup getaway to the Internet, because for some reason the OSPF process does not
propagate the alternative default route.
Since it is obvious that the problem is related to IP routing, you should start investigating why GW2 is not
being able to redistribute its default route into the OSPF routing process.
With a full OSPF Adjacency formed between routers R2 and GW2, there is no reason to doubt that OSPF is
not running on GW2. Search for the reason why GW2 is not redistributing the default route in its OSPF
routing process.
After the logon to the router GW2 you are greeted with a sequence of log notifications, which are self
explanatory. Someone has already configured BGP to log any neighbor changes. You notice that
notifications are indicating a BGP neighbor adjacency problem, while also pointing out that the peer is in
the wrong AS. Find more information by checking the overall BGP status on router GW2.
IP address 209.165.201.6 in the CLI output belongs to ISP2 as described in the Job Aids section of the lab.
Use the show ip bgp summary command to gather information about the BGP status. You can see that the
router has 2 BGP neighbors. One of them is GW1 and has formed an adjacency. The other is located at the
Internet provider and is experiencing problems and cannot form an adjacency. Because you have a clear hint
that there is something wrong with the AS numbers, you should double check the network design in the Job
Aids and compare the AS numbers from the design with the configured ones.
GW2 has a neighbor configured in ASN 65335. You can see in the Job Aids that the AS number should be
65535.
You can see that indeed a wrong AS number was configured for the neighbor 209.165.201.6. Since that
default route information from neighbor 209.165.201.6 cannot reach router GW2 without first forming a
proper neighbor adjacency, your hypothesis was correct and you can conclude that this is the route cause for
the lack of the alternative default route.
Implement Solution
Under BGP autonomous system 65000, neighbor 209.165.201.6 is defined with remote autonomous system
number 65335. Remove this statement and define a new one that defines the same neighbor but specifies
remote autonomous system number 65535.
Verify the Solution
A good practice is to always check if all the configuration changes have actually taken effect. By looking at
the output of the show ip bgp summary command, you can confirm that now both neighbors have formed
proper adjacencies. In the routing table, you can also check if the router GW2 has a default route
information originating from its BGP neighbor.
The output of the show ip ospf database command confirms that now both routers R1 and R2 have 2
alternative default routes in thier OSPF database.
Note At this point you could inform the support engineer that the problem is partly solved. Internet
connectivity for client PC1 is restored. Save the configuration on SW1. Document the
changes. Continue troubleshooting PC2‘s Internet connectivity problem.
Troubleshooting BGP Neighborship
BGP routers start exchanging routing information only after the BGP neighborship is successfully
established. The establishment begins with the creation of a TCP connection between the devices.
Afterwards, a BGP session is created by the exchange of BGP Open messages, where peers exchange BGP
version, AS number, hold time, and BGP identifier.
BGP neighborship cannot be formed if the TCP connection between peers cannot be established. That
happens if there is no IP connectivity between peers or TCP port 179 is blocked in some ACLs.
Neighborship is also not possible if the BGP neighbor configuration is incorrect. You have to be careful that
if you are using loopback addresses, you should also include the update source command. The proper AS
number and passwords must be set and you should check that the timers have the same values on both
neighbors.
Additionally, BGP requires that in case neighbor is not directly connected, router has to have the route to
the neighbor installed in its routing table. The default route, that would otherwise be sufficient for IP
connectivity, is not enough to form a BGP neighborship.
The BGP state that you want the routers to be in is called ESTABLISHED. In process of forming BGP
neighborship, routers pass through several other BGP states:
 IDLE: The router is searching routing table to see whether a route exists to reach the neighbor.
 CONNECT: The router found a route to the neighbor and has completed the three-way TCP
handshake.
 OPEN SENT: Open message sent, with parameters for the BGP session.
 OPEN CONFIRM: The router received agreement on the parameters for establishing session.
 ACTIVE: The router didn't receive agreement on parameters of establishment.
 ESTABLISHED: Peering is established; routing begins.
To troubleshoot this type of error, verify the following issues:

 Verify IP connectivity with extended ping / traceroute tools.
 Check route to the neighbor with the show ip route command. This command enables you to verify
whether any non-default route to the neighbor exists.
 Check all neighbors‘ states with the show ip bgp summary command. Commands display the state of
all peers.
 Check additional information with the show ip bgp neighbor command.
 Enable debug BGP related outputs with the debug ip bgp command, which will give you a good hint as
to why a peer is not establishing. Command debug ip bgp events displays the state transitions for peers.
Example Troubleshooting Flow: PC1 and PC2 Do
Not Have Internet Connectivity
When talking on the telephone with customer engineer Donovan, you should have spent more time asking
him additional and relevant questions about the case. As the result your trouble ticket now has a very plain
description. It only states that the classroom PCs—PC1 and PC2—do not have Internet connectivity and
does not provide you with any other useful information. In the conversation about a redundant gateway
problem, you have a hint that the Internet connectivity was successfully restored in other parts of the
network. You should include that in the verification of the reported problem.
Make notes that next time, if possible, you should ask some additional questions like:
 Do other parts of the network also experience internet connectivity failure?
 When did this problem first appear?
 Was there any recent change in the network?
Those answers could give you a head start in your troubleshooting process. Since you do not have them, it is
even more important to have a structured troubleshooting approach.
Verify the Problem
The first step in troubleshooting should always be the verification of the reported problem.
Unsuccessful pinging from PC1 and PC2 to a public server is an obvious verification of the reported
problem. Additionally you already make the first troubleshooting step as you narrow down the possibillites
of what could be wrong. At least for start you can leave upper layers and focus on Layer 3 and lower layers
of OSI model.
When a part of the network is working as expected and the other is failing, it is a good choice to use the
“spot-the-differences” approach, which you could do by comparing clients PC1 and PC2 with the
unproblematic PC3. But since all clients are in different VLANs this approach will only verify what you
already suspect, that root issue is localized within VLAN 11 and 22.
Due to the lack of proper network documentation, you have little information to check possible
interconnection errors. With the „Follow-the-path“ approach you can avoid relying only on potentially
wrong configuration of interfaces and use tools like the MAC table and the CDP database inquiries. With
this approach, you start at clients PC1 and PC2 and make your way towards R1.
There could be several reasons why PC1 and PC2 could not get the IP settings assigned via DHCP.
According to the Job Aids there are actually 2 redundant DHCP servers, routers R1 and R2, which are
serving all 4 VLANs. Clients from VLAN 33 and VLAN 44 have no problems with network connectivity,
therefore, at first glance, misconfiguration of both DHCP servers for VLAN 11 and VLAN 22 seems less
likely.
A common cause of DHCP problems is the lack of Layer 2 connectivity between the hosts and the DHCP
server.
Use the show interfaces command to gather information about the interface‘s Layer 2 operational status
and the MAC address and write them down. You will search for those 2 addresses all the way along the
path.
You can conclude that both connections are up and operational on Layer 2, but not on Layer 3.
As you can see, switch SW1 has no entry for either PC1‘s or PC2‘s Layer 2 address in its MAC address
table. This information is far from expected and should stir your attention. You have found the root problem
for the Layer 2 connectivity failure, now search for the cause.
When an Ethernet frame passes into the switch, the MAC address table of that switch is updated with the
frame‘s source MAC address, incoming interface, and VLAN. Both affected PC‘s have an operational Layer
2 connection towards the switch SW1 and therefore it is strange that DHCP broadcasts did not make any
new entry in the MAC address table.
You should investigate further what is blocking client PC‘s Ethernet frames. It could be some Layer 2
filtering or port security that is configured on the interface. You can check it straightforwardly with the
show running-config command.
Note Note that the MAC addresses of PC1 and PC2 were probably different in your lab.
As suspected, you find port security enabled on interface Ethernet0/2, which is the only access interface that
is configured for VLAN 11. You can also notice that there is one static MAC address configured and it is
different from the MAC address of PC1.
You can check the detailed port security configuration and status for this specific interface by issuing the
show port-security interface Ethernet 0/2 command.
By doing additional enquiries you can find out that Ethernet 0/3 is configured as an access interface for
VLAN 22 and has almost identical port security configuration as interface Ethernet 0/2.
Analyzing Gathered Information
You can see that on both interfaces the default value is set to maximum 1 MAC address. With the only
„seat“ already taken by statically configured address, you can conclude that there is no more space left for
new dynamic MAC addresses. If the interface is configured with port security using the »protect« mode,
then it will only drop the traffic from the unknown MAC addresses. As the one solving the network issue,
you should suggest to change the port security to the »restrict« mode. The difference is that the »restrict«
mode not only drops the traffic from unknown sources but also increases the violation counter each time
when such event occurs.
The proof that you are checking the right interfaces can be seen in the „last logged source MAC address“,
which is the MAC address of PC1 or PC2 respectively.
Port security on switch SW1 is configured with a wrong static MAC addresses on the access interfaces and
therefore effectively breaks Layer 2 connectivity between PCs and DHCP servers.
With almost identical configuration mistake on both access interfaces, you can assume that classrooms‘ PCs
were recently replaced with new ones and the network engineer just forgot to update the port security on the
access switch. After the configuration change, the PCs should obtain the IP addresses from the DHCP
servers and regain the Internet connectivity.
From the CLI output you can see that the MAC addresses are statically associated with Ethernet 0/2 and
Ethernet 0/3. These MAC addresses are not those of PC1 and PC2. Statically defined entries will never age.
Your hypothesis was correct.
Implement Solution
Configure the correct static MAC addresses on both interfaces. You should also change the port security
violation mode. The restrict mode has the same basic functionality as the protect mode, except that it also
increases the violation counter. It will be a bit easier to troubleshoot problems with future computer
replacements.
Verify the Solution
A good practice is to always check if all the configuration changes have actually taken effect. By looking at
the MAC address table you can confirm that both relevant addresses are now present. You can now proceed
to check if the change also eliminated all the obstacles for Internet connectivity.
It is probably nice to start with the good news first and continue with the bad one later. The good news is
that you made a progress, since client PC1 has regained Internet connectivity. The bad one is that
troubleshooting is not finished, since PC2 still does not have Internet connectivity.
Note At this point you could inform the support engineer that the problem is partly solved. Internet
connectivity for client PC1 is restored. Save the configuration on SW1. Document the
changes. Continue troubleshooting PC2‘s Internet connectivity problem.
Troubleshooting Port Security
With static MAC address configuration on the interface, the switch expects to receive the frames that are
sourced from the device with that MAC address. When the configured maximum number of MAC addresses
per port is exceeded, the switch will start blocking all frames from unknown sources and make further
actions according to the configured port security violation mode. If only one static MAC address is allowed,
the limit is reached whenever a frame from a new device reaches the interface. Connecting anew device on
that port requires the change in the port security configuration.
In the most rigorous shutdown mode, the switch places the affected interface into the Error-Disabled state
and starts blocking all frames, including frames from valid sources. In this case the recovery procedure must
be done to place the interface back into the operational mode. The recovery procedure can be done
automatically with the errdisable recovery cause psecure-violation command or you can disable and re-
enable the interface.
Other port security issues are related to the restore process and are usually noticed after the reboot of the
switch. If the port security configuration has never been saved, the switch will not apply it at bootup. Same
also go for sticky MAC addresses. After the dynamically learned sticky MAC addresses have been learned,
you must enter the copy running-config startup-config or write memory command to save them in the
startup-config file.
If the show mac address-table command does not reveal the specific MAC address entry in the MAC
address table, the switch is effectively blocking all the frames from that source.

 Verify the interface status with the show interface command. The access interface connecting the host
to the network must be in up/up state.
 Check the MAC table for dynamic and static entries with the show mac address-table command. You
can distinguish entries according to the interface, VLAN, and type.
 Check the port security overall status with the show port-security command. You can see on which
interfaces port security is actually applied, what kind of violation mode is set for each interface, and
check all the relevant counters.
 Check the port security status on the interface with the show port-security interface ethernet 0/2
command. The command reveals some additional information about the port security status of the
specific interface.
 Check the port security configuration with the show running-config command.
 Check the port security configuration with the show startup-config command.
Example Troubleshooting Flow: Classroom PC2
Does Not Have Internet Connectivity
While all the other computers in the network have Internet connectivity, PC2 in VLAN 22 is the only one
that cannot access Internet site at 209.165.201.225.
Verify the Problem
Verify that the reported problem for computer PC2 is still present.
From the output, it is obvious that PC2 does not have Layer 2 connectivity with any DHCP server.
Additionally you confirmed that the link between switch SW1 and router R1 is indeed configured as a trunk
and therefore in compliance with network design.
To verify that interface Ethernet 0/0 on SW1 is indeed trunking you could also use show interface trunk
command.
You should now regroup, analyze all the known facts, and prepare an updated troubleshooting plan.
At the beginning of troubleshooting process you have dismissed the possibility of a DHCP misconfiguration
as less likely. The assumption has proven more or less correct, since you have found port security error,
which effectively broke Layer 2 connectivity of both PCs. It is now obvious that PC2 lost its Layer 2
connectivity due to more than one reason and you should not ignore the possibility of DCHP
misconfiguration anymore.
When part of the network is working as expected and the other is failing, it is a good choice to use the
“Spot-the-differences” approach. The VLAN 11 client that now has connectivity and the VLANN 22 client
that still does not have connectivity have several common elements. They share the same access switch and
other components all the way to the DHCP servers. Since you have checked all but the last hop using the
„Follow the path“ approach, you can now focus on 2 potential problems:
 Trunk link to DHCP server misconfiguration.
 DHCP misconfiguration.
PC1 needs to communicate with at least one DHCP server, therefore it will be enough to check only one of
the two redundant servers. You should compare the DHCP configuration for VLAN 11 and VLAN 22 on
router R1.
You cannot spot any significant difference in the DHCP configuration between VLANs. For the DHCP pool
to be operational, at least one interface must have a configured IP address from that pool. Check if that is so
by looking at the trunk interface configuration.
Eliminating Possible Problem Causes
You can see that both DCHP and subinterface configuration are almost identical for both VLANs. Because
one of them is experiencing Layer 2 connectivity and the other is not, you can assume that the problem can
be found on switch SW1.
From the output of show interfaces trunk command on switch SW1, you can conclude that both trunks are
operational and are not filtering any VLANs. The strange thing is that actually active VLANs are only
VLAN 1 and VLAN 11. VLAN 22 is missing, probably because it is not even configured.
You can conclude that VLAN 22 is not configured on switch SW1. First test your hypothesis and, if true,
configure VLAN 22.
You could also issue show interfaces ethernet 0/3 switchport on SW1 and see that port indeed belongs to
VLAN 22, but is inactive. This means that VLAN does not exist on the switch.
Further investigation in VLAN database confirms your hypothesis. VLAN 22 is indeed not configured. You
should do it yourself.
Implement Solution
Configure the missing VLAN 22 on switch SW1.
Verify the Solution
Now, you should again check the Internet connectivity for both classroom PCs that are connected to the
access switch SW1. You must be sure that the last configuration change did not have any side effect on
already previously resolved PC1 issue.
Everything is now working as expected. You can proceed with wrapping up the whole troubleshooting
process.
Note At this point, you should inform the support engineer that the problem is solved. Internet
connectivity for clients PC1 and PC2 is now restored. Save the configuration on SW1.
Document the changes. Close the ticket.
Troubleshooting VLANs
In order to support network virtualization, there must be 2 things configured on each switch. First, it has to
have all new VLANs defined in its database, and second, each interface must belong to one VLAN or be
configured as a trunk.
First of the possible mistakes is wrong VLAN information. Each switch has its VLAN database, which can
be filled manually or dynamically with VTP. If, for some reason, VLAN is not in the database or due to the
typo error new VLAN is defined with wrong number, devices in that specific VLAN will not be able to
communicate.
Second group of possible issues is related to the wrong VLAN configuration on access or trunk interfaces.
Access interfaces must be configured according to the network design. A misconfigured VLAN number will
isolate the interface or assign it to the wrong VLAN. In case no VLAN is configured, interface becomes part
of a default VLAN 1. Interfaces that are connecting the switch to other switches or routers are usually
configured as a trunk links. That way they can carry multiple VLANs across one link using tagging
mechanism. Otherwise, the interconnecting links will not forward frames from any other VLAN besides the
one that the interface belongs to.
If the switch does not forward the frame as expected in a particular VLAN, it is likely that you have a
VLAN configuration error.

 Verify the status of all the relevant interfaces with the show interface command. The interface must be
in "up/up" state.
 Check the VLAN database with the show vlan command. This command enables you to verify the
VLAN existence and port-to-VLAN mapping. Trunks are not listed because they do not belong to any
particular VLAN.
 Check trunk interfaces with the show interfaces trunk and the show interfaces switchport command.
Commands display all interfaces that are configured as trunks and include per trunk information about
the configured trunk mode, encapsulation type, native VLAN, and allowed VLANs.
 Check the MAC table for dynamic and static entries in a particular VLAN with the show mac-address-
table command. This is the main command to verify Layer 2 forwarding. It shows you the MAC
addresses that are learned by the switch and their corresponding port and VLAN associations.
Troubleshooting Native VLAN
The native VLAN is used to carry untagged traffic across an 802.1Q trunk. The default native VLAN is set
to VLAN 1, but can be changed on individual switchport. Values can be different on each trunk across the
switch, but on one trunk link, the same native VLAN must be used on both sides.
The common error is the native VLAN mismatch, where configured values are not the same on both sides of
the trunk. Native VLAN misconfiguration generates traffic leak between 2 VLANs, which results in strange
forwarding behavior and possible fatal errors in protocols such as STP.
Cisco CDP monitors the native VLANs and displays a notification if a mismatch is detected. Mismatch
messages are displayed by CLI every minute: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN
mismatch discovered on GigabitEthernet1/0/25 (100), with sw1 GigabitEthernet1/0/25 (300).
STP is also able to detect native VLAN mismatches and can block the affected VLAN on the interface. This
will result in notifications such as:
 %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 300 on
GigabitEthernet1/0/25 VLAN100.
 %SPANTREE-2-BLOCK_PVID_PEER: Blocking GigabitEthernet1/0/25 on VLAN0300. Inconsistent
peer vlan.
 %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/25 on VLAN0100. Inconsistent
local vlan.
In Cisco LAN switch environments, the native VLAN is typically untagged on trunk ports. This can lead to
a security vulnerability, since it is possible to create frames that are encapsulated with two 802.1Q tags. If
the attacker uses native VLAN for the outer tag and victim‘s VLAN for the inner tag, the switch strips the
outer tag and forwards the remaining single-tagged frame toward the destination VLAN across a trunk port.
This is called VLAN hopping attack and can be prevented with the vlan dot1q tag native command, which
explicitly tags also the native VLAN. For older switches, which do not support this feature, you can remove
the native VLAN from the allowed VLAN on that trunk.
You can use the show interface trunk and the show interfaces interface slot/number switchport
command to verify the configured native VLAN and other trunk parameters.
Summary
Second Troubleshooting Lab
at TINC Garbage Disposal
Ltd.
Overview
This lesson serves as a debrief for the second troubleshooting lab at TINC Garbage Disposal Ltd.
Example troubleshooting flows are provided, but keep in mind that there are multiple ways to
troubleshooting the problems.
• Describe how you solved OSPF neighbor adjacency issue
• Describe the possible OSPF adjacency issues
• Describe how you solved SSH version 2 connectivity to the router R2
• Describe possible device management issues via SSH or telnet
• Describe how you solved duplicate IP address issue
• Describe possible HSRP issues
You work for SECHNIK Networking Ltd. and TINC Garbage Disposal Ltd. is your company's customer.
Donovan, the customer engineer, sent you an e-mail, asking you for help.
 Senior customer engineer noticed that GW1 only has one OSPF neighbor—GW2. This engineer asked
Donovan why GW1 does not have an adjacency with R1. Donovan would love for you to help him with
this question and solve it for him. He does not have the expertise to tackle this issue.
 Donovan configured all the routers in the network with remote access via SSH version 2. Now, when he
tests the connectivity from his PC—PC4—he can access GW1, GW2, and R1, but not R2. The
connection is refused and Donovan is begging you to help him investigate why.
 Routers R1 and R2 keep complaining about a duplicate IP address. Donovan has no idea what is the
cause. He insists that this is an IOS bug since all PCs have proper Internet connectivity.
Note Credentials to access the routers in this lab are provided in the Job Aids.
Example Troubleshooting Flow: GW1 Only Has
OSPF Adjacency With GW2
GW1 only has one OSPF adjacency with the GW2 router. There is no adjacency with the R1 router.
Verify the Problem
Always verify the reported problem.
Note This is one possible troubleshooting plan. There are many different approaches to
addressing this problem.
Router GW1 has only one OSPF neighbor, which is the router GW2. It should establish adjacency with R1
as well. If adjacency is not established between the routers GW1 and R1, there is no routing information
exchange, high availability cannot be achieved, and problems with suboptimal routing could appear.
When you are trying to solve OSPF adjacency problem, it is a good choice to start with the “bottom-up”
approach. With this approach you can verify that network connectivity between both OSPF routers exists. A
common cause of OSPF adjacency problems is the lack of network connectivity.
Checking the interface status and IP addresses on the routers is the first step in the „bottom-up“ approach.
As you can see from the outputs, both interfaces are up and operational. You can also find the IP addresses,
which are a great step to Layer 3 connectivity.
Check the interface details on both routers with the show interfaces command. Based on the IP addresses
information you can confirm that those interfaces are used for the interconnectivity.
You can use the ping tool to verify that there is connectivity between R1 and GW1, but traffic can flow
from R1 to GW1 via R2 and GW2. On R1 use the traceroute IOS tool and specify GW1 as the target. This
way you can confirm that the traffic goes directly from R1 to GW1 and not through R2 and GW2. If you
would use ping, a successful ping will only tell you if there is connectivity between R1 and GW1 but that
ping can go through R2 and GW2—it will not tell you if you have Layer 3 connectivity between the two
devices.
With the “bottom-up” approach, first check the status of the interfaces that connect devices, using
commands show ip interface brief and show interface. As you can see, both Ethernet 0/3 interfaces are up
and operational.
Using the “bottom-up” approach you continue by checking higher layers. Since routing protocols work on
top of IP the next step in the troubleshooting process is to check those. According to the company's
documentation or the device configuration investigation, OSPF is used as the routing protocol. You should
check if GW1 and R1 have correctly configured OSPF.
Check the OSPF configuration with the show running-config command and the OSPF status with the show
ip ospf command. As you can find out from the outputs, GW1 is working as OSPF backbone router with all
the interfaces in Area 0. You also learn the GW1 OSPF router ID, which will help you identify the router
further down in the troubleshooting process.
Check the OSPF configuration and the status on the router R1 as well. As you can learn from the outputs,
the R1 router is working as OSPF backbone router since it has all the interfaces in Area 0. Remember the
OSPF router ID for further troubleshooting.
You could use the show ip ospf interfaces command to verify which interfaces are enabled for OSPF.
When troubleshooting OSPF, it is always a good practice to check the configuration with the show
running-config command to verify that the basic OSPF configuration is correct. The next step in the
troubleshooting process is to gather information about the router ID, areas, and the operational status with
the show ip ospf command. From the outputs, you can conclude that OSPF is operational on GW1 and R1.
You can find out that all the interfaces on GW1 and R1 are running in OSPF Area 0.
You confirmed that the basic OSPF configuration is correct and in accordance with the documentation. In
the problem verification step, you found out that there is no OSPF neighbor state present in the OSPF
neighbor table on GW1 and R1. Based on this information and information that was gathered during the
troubleshooting process, you can conclude that there is an issue with OSPF neighbor establishment process.
OSPF uses Hello messages to establish neighbor adjacency. To observe the neighbor establishment process,
you can use the debug commands.
Eliminating Possible Problem Causes
Run the OSPF Hello debug to observe the Hello messages exchange. As you can see from the output, the
GW1 router is sending Hello messages from interface Ethernet 0/3 to multicast address 224.0.0.5.
To observe the OSPF Hello messages, you can use the debug ip ospf hello command. Based on the
information in the debug output on R1, you notice that there is a Hello and Dead interval mismatch.
You found out from debugging that there is a Hello and Dead interval mismatch. To check the intervals, use
the show ip ospf interface command. You can notice that the Hello interval is 10 seconds, while the Dead
interval is 40 seconds. These are the default values for the Broadcast network type.
The show ip ospf interface command can be used to gather information about OSPF parameters, including
timers, such as Hello, Dead, Wait, and others. Default timers’ values are related to the network type
configured for the OSPF process. As you can notice, different network types are configured on GW1 and
R1. GW1 uses the BROADCAST network type, which is used for Ethernet networks, while R1 uses the
NON_BROADCAST network type, which is used in NBMA networks. the default Hello and Dead intervals
on a broadcast network type are 10 and 40 seconds, respectively, while default Hello and Dead intervals on
a nonbroadcast network type are 40 and 120 seconds.
The behavior of OSPF is determined by the OSPF network types. The network type is specified per
interface and it is generally tied to layer 2 technology that is used on that interface. Usually, it is a good
practice to leave the network type as is, but sometimes you need to change it. DMVPN in a hub-and-spoke
topology could be one example where it is a good practice to change the default network types.
The main cause for missing adjacency between R1 and GW1 is the hello and dead interval mismatch, which
is caused by the misconfigured network type on one of the neighboring devices. The issue could be solved
by tuning the Hello and the Dead timers on one of the routers, but this would leave different network types
configured on neighboring OSPF interfaces and such practice is not recommended. Broadcast networks use
multicast to send Hello messages, while nonbroadcast networks use unicast. Therefore the proposed solution
is to configure the same network type on neighboring interfaces. Since Ethernet is used as the layer 2
technology on the link, the OSPF network type should be broadcast.
First check the configuration on the interface Ethernet 0/3 to confirm the hypothesis, before implementing
the change. The result of the show running config interface ethernet 0/3 command confirms that network
type is set to nonbroadcast.
Using the no ip ospf network non-broadcast command, you can reset the network type on the interface to
its default value. In this case, the default value is the correct value, which is broadcast for the Ethernet
interface.
Verify the Solution
After implementing the change, verify that the change was properly configured on the interface.
To test the hypothesis, configure the network type to broadcast on Ethernet 0/3 interface of R1. You can
accomplish this by using the no ip ospf network non-broadcast command. Check whether the correct
network type is applied to the interface using the show ip ospf interface ethernet 0/3 command.
To confirm that the issue is solved, you should check neighbor table on GW1 and R1. As you can notice
from the command output, routers GW1 and R1 established the adjacency, which confirms that the issue no
longer exists.
Note At this point you should inform the support engineer that the problem is solved. Router GW1
has established adjacency with both GW2 and R1. Save the configuration on R1. Document
the changes. Close the ticket.
Troubleshooting OSPF Adjacency
OSPF neighbors must form adjacency to be able to exchange routing information. OSPF neighbors go
through several states during the adjacency establishment phase. The neighbors must transition to Full state
to form the adjacency. To check the neighbor state, you could use the show ip ospf neighbor command. If
neighbors are stuck in any other state, the adjacency will not be established.
If the show ip ospf neighbor command does not reveal the neighbor, the router has not seen any „valid“
OSPF Hello message from that neighbor or the Hello packet failed the basic check.

 Verify the interface status with the show interface command. The interface to the neighbor must be in
up/up state.
 Check the IP connectivity to the neighbor with the ping command. Ping the IP address of the neighbor
and multicast IP address 224.0.0.5, which is the address for Hello messages.
 Check the interface access list with the show ip interface command.
 Check the OSPF status on the interface with the show ip ospf interface command. The command
reveals all the interfaces with OSPF enabled.
 Check if the interface is configured as passive. You can use show ip ospf interface as well. Active
OSPF interface should display a remaining time to the next Hello message.
 Verify that the router ID of the routers is not the same. If the router receives a Hello packet with the
same router ID, the router will ignore this Hello packet. Use the show ip ospf command.
 Verify that the Hello parameters match on the neighboring interfaces. There are several parameters that
must match to establish the Full adjacency:
− OSPF area number: You can check it with the show ip ospf interface command.
− OSPF area type (such as stub or NSSA): To check it, use the show ip ospf command.
− Subnet and subnet mask: Check the IP address and the mask with the show interface command.
− OSPF Hello and Dead timers: Check with the show ip ospf interface command.
Router can be stuck in several others states:

 Down state
 Init state
 Exstart/Exchange state
Dynamically discovered neighbor could be in Down state when the OSPF process does not receive the
Hello packet from the neighbor for a time period longer than the Dead timer interval. The Down state is
transient, which means that the neighbor will either advance to other states or it will be deleted from the
OSPF neighbor table.
the neighbor could also be seen as Down when it is manually configured with the neighbor command on
the router. Manually configured neighbor is always kept in the OSPF neighbor table. When it is seen as
down, it usually means that no Hello message was received or that the Dead timer expired. A manually
configured neighbor can only be configured in Non-Broadcast Multi Access networks or Non-Broadcast
Point-to-Multipoint networks.
To troubleshoot the Down state, follow the same procedure as in case of no neighbor in the OSPF neighbor
table.
When a router is stuck in Init, it indicates that the router has seen the Hello packets from the neighbor, but
the two-way communication was not established, because the router did not see its own Router ID in the
Hello packets from the neighbor.
The most likely reason is that neighbor has not received the Hello packet from the router. There can be
several issues:
 Check the connectivity between the neighbors with the ping and the traceroute command.
 Check if the multicast IP address 224.0.0.5 is permitted in the input access list.
 Check if the OSPF authentication is enabled on both sides.
Most commonly, the router is stuck in Exstart/Exchange state when there is an MTU mismatch between
routers. This happens if the router on one side send message larger than the MTU configured on the other
router. Larger message is ignored by the router, which stays in Exstart/Exchange state. To check MTU on
the router use the show ip interface command.
Example Troubleshooting Flow: R2 Is Not
Accessible Via SSH Version 2
Donovan cannot access router R2 via SSH version 2 from PC4. But on the other hand he is able to access
routers GW1, GW2, and R1 via SSH version 2.
First step in the troubleshooting process is to verify the problem. Connection via SSH version 2 from PC4 to
the router R2 is not successful, therefore the problem is confirmed and you can start the troubleshooting
process.
problem.
When you are troubleshooting the specific protocol version, it is always a good practice to check whether
the problem exist with the other version as well. This method approach is called „move-the-problem“. In the
current troubleshooting ticket, PC4 cannot access the router R2 via SSH version 2. Try SSH version 1 and
verify if the problem is still there.
Eliminate Possible Causes
You were able to access the router R2 via SSH version 1. Based on this information you can conclude that
there is network connectivity between PC4 and the router R2 on TCP port 22, which is the default SSH port.
You can also conclude that SSH process is operational on router R2. The problem is most likely related to
the SSH version misconfiguration.
Since you were able to eliminate the possible causes of the problem, you can swap the troubleshooting
method to „spot-the-difference“. You have network devices that are manageable via SSH version 2 and one
device that is not. This is an appropriate environment to use the „spot-the-difference“ troubleshooting
method, since you can compare the configuration on the working device with the configuration on the faulty
device.
Check the SSH status on routers R1 and R2 with the show ip ssh command. You can notice that SSH is
enabled on both devices, but only version 1 is enabled on router R2. The SSH version 1.99 is enabled on
router R1. This only indicates that the server supports both SSH versions 1 and 2. It is not really a version,
as such, but an indication of the backward compatibility.
Using the show running-config command with the include ssh filter will output all the SSH-related
configuration on the router R2. Based on that information, you can conclude that version 2 is not enabled on
the R2 router. The further action is to enable the SSH version 2 on R2.
With the no ip ssh version 1 command, you enabled both SSH versions on the router R2. You can verify
that version 2 is enabled on the router R2.
Now that the SSH version 2 is enabled on the router you try to access router R2 again. Although version 2
was enabled on the router R2, PC4 is still unable to access the router R2 via SSH version 2. The error
message indicates that something is wrong with the key length. The SSH version 2 mandates that the
minimum key size must be 768 bits. Check the key status on router R2.
You can check the keys on the router with the show crypto key mypubkey rsa command. Since you
cannot check the key length, you can compare the private key length with that of router R1.
You notice that the private key on router R1 is longer than the private key on router R2. This leads you to
the conclusion that the key size on router R2 is too short for the SSH version 2 and must be increased.
To generate the new key with bigger key size, use the crypto key generate rsa modulus 1024 command.
As you can notice, it takes a couple of seconds to generate the key. The required time depends on the key
length. Generally, it is recommended to use keys of at least 2048 bits. Cisco IOS software does not support
key sizes greater than 4096 bits.
Verify the Solution
After generating the new key, you should test the connectivity to the router R2 via SSH version 2. As you
can see, you are now able to connect, which confirms that the solution was properly implemented.
Note At this point you should inform the support engineer that the problem is solved. You are now
able to connect to the router R2 from PC4 via SSH version 2. Save the configuration on R2.
Troubleshooting Management Access
The Telnet protocol is much simpler protocol than SSH, but its use is not recommended since it sends
commands in plain text, including passwords. Therefore telnet should only be used in the lab or test
environments.
 Access list is blocking the access:
− It is a good practice to limit the access to the network device to only authorized IP addresses. To
limit the IP addresses, you could use access lists in the line configuration. If the IP address is not
permitted in the access list, this IP address cannot access the device. If connections to the device are
refused, you should check whether the access-list is blocking the IP address. Check which access
list is applied with the show running-config | begin line command.
 Telnet/ssh not enabled on the line:
− By default, all protocols are enabled on the line. You could limit the protocols using the transport
input command. To check the enabled protocols, you can use the show line command.
 Authentication problems:
− To authenticate the access to a device, several authentication methods could be configured such as
Local authentication, RADIUS, TACACS+, LDAP. When you are trying to access the device, you
should use the username and the password that are configured for the authentication method in use.
You should check the running configuration to find out the AAA mechanisms that are used. The
debug radius and the debug tacacs command can be used to troubleshoot RADIUS and
TACACS+ protocols.
 All vty lines are busy:
− There are multiple vty lines to manage network devices. Vty lines are identified by numbers,
starting with 0. Though it happens rarely, eventually all the vty lines can become busy if a client
does not properly close the connection. Vty lines will be cleared when the connection timeout
expires. To check all the connections to the vty lines, use the show line command. To clear some of
the vty lines, use the clear line command.
With SSH you need to make sure that correct SSH version and key size must be configured.
Example Troubleshooting Flow: Duplicate IP
Addresses on Routers R1 and R2
Routers R1 and R2 keep complaining about a duplicate IP address.
Verify the Problem
Connect to the console of the routers R1 and R2. You observe the console messages. They indicate that a
duplicate IP address problem exists. You also see what the duplicate IP address is and which MAC address
it is sourced from. You already have many pieces of information from the console messages, so you can
immediately start finding devices and interfaces on those devices.
problem.
Based on the information that you gathered from the console messages, you can confirm that the duplicate
IP address problem exists in the network. You already found out the duplicate IP address and the MAC
addresses corresponding with that IP. HSRP uses MAC addresses 0000.0c07.acXX, where XX is the HSRP
group number.
If you didn't figure out right away that duplicate address messages were referencing MAC addresses
belonging to HSRP, you could have used show ip interface brief on R1 and R2. This would give you a
confirmation that duplicate IP addresses are indeed present on these two devices. If you figured out that
MAC addresses belong to HSRP, you are able to skip one step in information gathering phase.
Check the interface configuration on both routers. From the output you can notice that there are different
HSRP groups with the same IP address configured on the routers. Therefore you can conclude that there is
an HSRP misconfiguration.
HSRP protocol is used for the first-hop redundancy. Devices within the same HSRP group, share the same
virtual IP address. This IP address is used as the default gateway or the next-hop IP address for the devices
in the network.
To confirm the hypothesis, you can check the HSRP details with the show standby command. As you can
notice from the output, the virtual MAC address for group 33 on the router R1 is 0000.0c07.ac21. This is
the MAC address that was displayed in the console messages that were announcing the IP conflict.
HSRP virtual MAC address is derived from the HSRP group number. The virtual MAC address is
0000.0c07.acXX, where XX is the group ID in HEX. The router R1 is using group ID 33, which is 21 in
hexadecimal notation.
You can also check the HSRP details on the router R2 to confirm the hypothesis. The virtual MAC address
that is related to R2’s virtual interface also appeared in the console messages that were indicating the IP
conflict.
From the information that was gathered during troubleshooting process you can conclude that there is an
HSRP group misconfiguration on routers R1 and R2. To test your hypothesis, your next step is to
reconfigure HSRP with the correct group ID. For the HSRP group, you can choose any number between 0
to 255 or even 0 to 4095 if HSRP version 2 is used instead of (default) HSRP version 1. It is common to
choose the same number as the VLAN number. In your case, you should choose number 33 for the HSRP
group configuration.
Implement the Solution
You noticed in the previous troubleshooting steps that HSRP group 3 parameters are configured on R1
router. Since you will use group number 33, the configuration is not needed so it should be deleted.
Since you decided to use HSRP group 33, you should delete all the configuration for HSRP group 3 from
router R2. You should reapply the same configuration with the correct HSRP group number. To verify that
the IP conflict disappeared from the router, you should monitor the console for IP duplicate address
messages. If you do not see any of those messages, you can conclude that the problem is solved.
Note At this point you should inform the support engineer that the problem is solved. The IP
conflict disappears from routers R1 and R2. Save the configuration on routers R1 and R2.
Troubleshooting HSRP
There are several possible misconfigurations of HSRP:

 If a wrong HSRP group is configured, this leads to duplicate IP address problem, as seen in the
troubleshooting ticket.
 There could be different HSRP virtual IP addresses configured. Console messages will notify you about
this situation. With such configuration, when the active router fails, the standby router takes over with
virtual IP address, which is different to the one used previously, and different to the one configured as
the default-gateway address for end-devices.
 HSRP authentication can ensure that there is no rogue HSRP router in the network. Incorrect
configuration of the HSRP authentication is notified through console messages.
 HSRP comes in two versions—1 and 2. If there is a version mismatch, both routers will become active.
This results in duplicate IP addresses.
 If a wrong HSRP group is configured on the peers, this leads to both peers becoming active. This will
manifest as a duplicate IP address problem.
 HSRP versions 1 and 2 are not compatible.
Most of the HSRP misconfiguration problems can be solved by checking the output of the show standby
command. In the output, you can notice the active IP and the MAC address, timers, the active router, and
several others parameters.
HSRP messages are sent to multicast IP address 224.0.0.2 and UDP port 1985 in version 1 and multicast IP
address 224.0.0.10 and UDP port 1985 in version 2. These IP addresses and ports need to be permitted in
the inbound access lists. If the packets are blocked, the peers will not see each other and there will be no
HSRP redundancy. To check the interface access list use the show ip interface command.
Besides the HSRP misconfiguration, the duplicate IP address problem can be the result of STP loops,
EtherChannel configuration errors, or duplicated frames.
Constant HSRP state changes could be the result of network performance problems, application timeouts,
connectivity disruption, link flapping, and hardware issues. In this case, the Hello and the Hold timers need
to be appropriately set to address these issues.
Summary
Third Troubleshooting Lab at
TINC Garbage Disposal Ltd.
Overview
This lesson serves as a debrief for the third troubleshooting lab at the TINC Garbage Disposal Ltd.
One troubleshooting approach is used and described in the debrief. Keep in mind that there are many
approaches that you can apply in order to solve the problem.
Upon completion of this lesson you will be able to meet these objectives:
• Describe how you solved PC1’s and PC2's problems with Internet connectivity
• Describe the possible issues with illegitimate routing sources
• Describe how you solved the issue with multiple routers acting as Masters in a VRRP group
• Describe common issues related to VRRP protocol
• Describe how you solved the problem with non functional EtherChannel
• Describe the possible EtherChannel issues
Trouble Ticket Overview
You work for SECHNIK Networking Ltd. and TINC Garbage Disposal Ltd. is your company’s customer.
The customer did some changes to the network recently and now they need your help:
 PC1 and PC2 clients can only access the Internet sporadically! Most of the pings fail. This issue needs
to be resolved as soon as possible.
 Donovan, the customer engineer, got an assignment to migrate the network’s first hop redundancy from
the proprietary HSRP to the industry-standard VRRP. He reports that he was mostly successful,
however, he is asking you to help him investigate why the “Master” state is assigned on both R1 and R2
for the VLAN 33 clients.
 Donovan connected another switch, SW4, to SW3. Since a lot of traffic will go between the two
switches, Donovan wanted to increase the throughput by connecting them with two links and bundling
the links into an EtherChannel. However, the EtherChannel is not functioning.
To test the Internet connectivity, use IP address 209.165.201.225. The credentials to access network devices
can be found in the Job Aids. You will not be able to access SW4.
Example Troubleshooting Flow: Sporadic Access
To Internet
The users from two LAN segments that are connected to switch SW1 claim to have only sporadic access to
the Internet.
Using the ping tool, you will check whether the connectivity issue exists. From both PCs you issue a ping to
the Internet address ping 209.165.201.225.
The ping is not successful. You confirm that the users cannot access Internet. Also, you see that the
response is not always the same, because it contains two different characters, namely “.” and “U”s. These
confirmations mean that you can start troubleshooting.
Note This is one of the possible troubleshooting plans. There are many different approaches to
When a client has problems accessing the Internet, it is always a good idea to check other clients. This is the
“swapping components” approach. If other PCs are able o access the Internet site, than the problem is
probably local to PC1—disconnected cable, wrong address configuration, misconfigured port on the local
switch, and so on. If other PCs are unable to access the Internet, then the issue is probably closer to the
network edge or on the ISP side. You determine what is the extent of the problem, whether it is confined to
one segment or spreads across entire network. This helps you to narrow the possible causes, thus
accelerating the troubleshooting process.
Note that you do not have to swap the components necessarily. If you have access to the other network
segment, for instance, if you can use the PC in the segment, it is the same as if you were attaching your
device to that segment and results would be the same. There are problems though in which swap will be
necessary.
Information Gathering and Analyzing
The results of the ping 209.165.201.225 command for PC3 and PC4 are the same as for the PC1 and PC2.
This means that the problem is not local to the access network segments, but is placed somewhere farther in
the network.
You noticed that the ping response in all cases contains dots and Us (“.” and “U” characters). The presence
of different characters in the ping response indicates that the packets are not treated in the same way along
their path. For some of them the destination is unreachable (indicated by a “U”) and others time out
(indicated by a “.”). The presence of the “U” character in particular, indicates that PCs have received an
ICMP echo reply – destination unreachable from a device along the path. This tells you that the problem is
unlikely to be related to the lower layers’ connectivity. Therefore, you assume that there are network or
higher level issues and you decide to start the troubleshooting from there. You select the “follow-the-path”
method.
You now proceed and gather some information at the next stop on the path: router R1. The next stop might
as well be the R2 router.
You are moving from the end devices further into the network. You decide to check Internet connectivity
from the router and issue the ping 209.165.201.225 command. The response is the same as in case of pings
from PCs.
In order to understand how R1 handles packets, you issue the show ip route command. You see that OSPF
is configured on the router. Also, you notice 2 default routes – both external to OSPF and redistributed
within the area. Both are used to forward the packets, and the traffic is load balanced between the two
routes.
What you notice about the default routes is that the next-hop of the first default route is on GW1 router. The
second default route points back to one of the access network segments. Its next hop is 10.0.11.111,
belonging to VLAN 11 network segment. This segment is already listed in the routing table as the
connected network. the next-hop IP is further resolved to egress interface Ethernet 1/0.11. This egress
interface is the same as the ingress interface for the packets originating in the 10.0.11.0/24 segment.
You now decide to examine the routing table of the R2 router.
Other information in the routing table looks ordinary. You note down the network addresses and interface
information, which you will use later to update the network documentation.
You decide to take a look at the R2 routing table. You expect to see two default routes here as well. Your
expectations are based on the fact, that OSPF is running as the only routing protocol on R1 and R2. So, the
problematic information must have been propagated and reached R2 as well.
You issue the same show ip route command on R2. The output verifies that R2 routing table contains two
default routes, one of them pointing to the same VLAN 11 IP address 10.0.11.111 as in the case of R1’s
routing table.
Based on the topology, you do not expect two default routes. This prompts you to gather more information
in order to better understand how the traffic travels in the network.
The existence of two default results in packets taking different paths. You trace the routes towards the
Internet from both R1 and R2. You issue the traceroute 209.165.201.225 command from R1 and R2 and
observe the output.
You notice different IP addresses in hops 1 and 2. The addresses correspond to the next-hop IP addresses of
the two default routes. The same is true of the R2’s output. “!H” indicates that, for some of the PDUs sent,
the host, network or protocol is unreachable. You notice these characters in relation to the 10.0.11.111 hop.
You have now gathered enough information to construct the hypothesis. The analysis shows that:
 There are two default routes in the routing tables of both R1 and R2.
 One of the routes points towards the next-hop, which is in the access segment of the network.
 As both default routes are used to forward packets, some of the packets are routed correctly, others not.
Your hypothesis states: There is an incorrect default route information in the routing tables of R1 and R2,
which hampers the normal functioning of the network.
Before you delve into where the incorrect information originates from, you want to support your hypothesis
with more information. You go on and check whether the same problem exists on remaining routers in the
OSPF area.
Issuing the show ip route command on GW1 shows only one default route. Its source is the BGP routing
protocol. The same is true of the GW2’s routing table entries.
To check what OSPF routing information were received about the default route, you issue the show ip ospf
database command. Under the ADV Router column, among the advertisement sources, you recognize
OSPF identifications 10.0.1.10 and 10.0.1.11, belonging to GW1 and GW2 respectively. However, you do
not recognize the 172.16.0.1 OSPF id. You now suspect that the entity with OSPF id 172.16.0.1 has sourced
problematic link state advertisements for the default route. This is evident from the output section that is
dedicated to the type 5 link state advertisements, i.e. LSAs about the routes external to the OSPF.
The same 172.16.0.1 device is sourcing other types of LSAs as well.
Analyzing the information that you gathered in the previous step, you have noticed that in the network there
is a device that is participating in the OSPF routing process and is injecting the erroneous information on the
default route. The OSPF id of the device is 172.16.0.1 and its IP address is 10.0.11.111. You would like to
understand the OSPF neighboring relations of this device.
First, you check the list of R1 and R2 OSPF neighbors, by issuing the show ip ospf neighbors command
from routers. Based on the topology, you expect two neighbors for each router.
However, you are seeing three different ones. The device with OSPF id 172.16.0.1 appears to be the
neighbor of both R1 and R2. Based on the output you see that the adjacency is current. The neighbors table
confirms that the device is part of VLAN11 subnet, it has the IP address of 10.0.11.111 (the same address as
the next-hop address in the unexpected default route), and is reachable via Ethernet1/0 interface of R1. This
is the connection to switch SW1.
Besides, you are able to ping the device from both R1 and R2, confirmed by issuing the ping 10.0.11.111
command. It is another proof that the device is operational and part of the network. The topology drawing
you received is probably incorrect.
If you decide to check the LSA database by issuing the show ip ospf database on routers R1 and R2, you
will again confirm, that the source of the incorrect information is the entity with OSPF id 172.16.0.1.
Suggesting the Hypothesis
Your findings so far have reinforced your hypothesis. To summarize:

 There is an incorrect default route in the routing table of routers running only OSPF.
 Incorrect default route information is propagated within the OSPF process.
 The source of the information is the device with id=172.16.0.1 and an IP address=10.0.11.111.
 There are current adjacencies with the 172.16.0.1 device on R1 and R2.
 You can ping the problematic device at 10.0.11.111, showing that the device is operational.
This provides a solid base to propose the solution: the device with id 172.16.0.1, along with any other
illegitimate device, should be prevented from accessing the network and from participating in the routing
process.
To pursue your solution, you first have to understand where the 172.16.0.1 device connects to the
customer’s network. You have to locate the device.
You noticed earlier that you can ping the device’s IP address from routers R1 and R2. You decide to repeat
the ping and immediately afterwards issue the show arp statement. From R1 you issue the ping 10.0.11.111
command. The ping is successful. You now proceed with the show arp command. The ping traffic refreshes
the ARP entry and from the output you see that the device is using the MAC address of aabb.cc00.7100.
Besides, you see that the MAC address was learned through the Ethernet 1/0.11 interface, connecting to
SW1.
Next, you decide to find the exact SW1 port, which the device is connected to. On SW1, you issue the
show-mac-address table command.
You find the problematic device’s MAC address of aabb.cc00.7100 in the entry for switch port Ethernet 2/0.
You found the device.
Now that the device is found, you decide to shut down the switch-port, which the device connects to. The
incorrect routing information originated from a device which was allowed access network 10.0.11.0/24.
Also, this device was allowed to participate in the routing process. Therefore, you decide to shutdown all
the interfaces which might enable the connection of illegitimate devices. This will also effectively prevent
them from participating in the routing process.
After these actions are taken, you expect the connectivity to restore.
You must be careful not to shut down the ports, which the legitimate devices connect to. Once again you
analyze the output of the show mac-address-table command, issued on SW1. The MAC addresses
0000.5e00.01## belong to the VRRP virtual interfaces. On PC1 and PC2 you issue the show interface
command in order to note their MAC addresses, which are aabb.cc00.6300 and aabb.cc00.6400
respectively. You do the same for router interfaces R1 Ethernet1/0 and R2 Ethernet1/0, by issuing the show
interface ethernet1/0 command. Their MAC addresses are aabb.cc00.6a10 and aabb.cc00.6b10
respectively.
The interfaces in the range Ethernet 0/0-3 must not be shut down. Interface Ethernet 2/0 must be shutdown.
All other interfaces are not used and should be shut down.
Taking Action
You now shut down all the ports on SW1 except for those in range ethernet0/0-3. You apply the shutdown
command to the range of interfaces. You start by changing to the interface-range configuration mode using
interface range ethernet 2/0-3 command. Once in the if-range configuration mode, you issue the
shutdown command. You proceed in the same manner for other interface ranges.
Verifying the Solution
Now that device 172.16.0.1 is denied access to the network, you have to check whether the problem is
solved.
You first check the routing tables of R1 and R2. You issue the show ip route command on R1 and on R2.
The output has only one default route pointing to the correct next-hop IP address of the GW1 router.
Now you check the connectivity from PCs. You ping the Internet address using the ping 209.165.201.225
command at PC1, PC2, PC3, and PC4. All the pings are successful. The connectivity is restored.
In order to prevent similar problems in future, the following security elements could be configured:
 Authentication of OSPF messages in area 0
 Passive interfaces on R1 and R2 where no networking devices are expected
 Shutting down interfaces on SW2 and SW3
 Removing unused interfaces from VLAN 1 on all switches.
Note You have solved the problem and are now ready to conclude. You want to make sure that
the changes are saved. Make sure to copy the new configuration to the startup-config by
issuing the copy running-config startup-config command on all devices where
the configuration was modified. You also document the changes and update the
documentation. You then inform the support engineer that the problem was solved and close
the ticket.
Troubleshooting Problems With Routing Sources
Routing is an essential part of the networking. Its functions and protocols are used between network devices
to learn and propagate route information. They build data structures that are used to forward packets.
Devices that participate in routing processes affect the forwarding behavior of the network. Routing relies
on the exchange of routing information, sourced by routing devices in the network. If those devices are
compromised or misconfigured, they can negatively affect or disrupt the network operation. Any
unauthorized participation in the routing process should be prevented to ensure the stable functioning of the
network. It is essential to know and control the sources of routing information.
Common issues with routing sources are:

 Illegitimate devices participating in the process
 Legitimate devices sourcing incorrect information
To troubleshoot these issues, you will need to use the commands to gather information from the routing
protocol data structures. Detailed information, including data origination, about specific routes that are
installed in the routing table can be gathered by use of the show ip route network mask command. The
debug ip routing command displays the routes being installed or removed from the routing table in real
time.
Every routing protocol has its own set of commands that could be used for troubleshooting:
 OSPF:
− To check the OSPF database, use the show ip ospf database command.
− To check the OSPF neighbors, use the show ip ospf neighbor command.
 EIGRP:
− To check the EIGRP topology table, use the show ip eigrp topology command.
− To check the EIGRP neighbors, use the show ip eigrp neighbors command.
 BGP:
− To check the IPv4 BGP neighbors, use the show bgp ipv4 unicast summary command.
 RIP:
− To check the RIP database, use the show ip rip database command.
In order to secure the networking infrastructure, all layers and devices should be considered. Physical access
limitations should be in place. On access devices such as switches, all unused ports should be shut down.
All ports should be removed from the default VLAN1 VLAN. Additional security configuration should be
in place on the switch ports connecting legitimate routing devices, such as port-security features. Access to
management functions of all devices should be restricted. After these measures are taken, the routing
functions and protocols should be hardened.
Most routing protocols require the establishment of specific relationships or sessions in order to exchange
routing information. Peer discovery mechanisms are automatic by default and operate under assumption that
the peers are legitimate, trusted devices.
It is possible to restrict the peer relations only to trusted sources, to control the extent of the message
exchange and to verify the information exchanged. In order to do so, the following mechanisms are
available:
1. Explicit configuration of routing peers
If you want to have a strict control over peering relations, disable the automatic peer discovery
mechanisms. All the neighbors of the device should be explicitly configured. In case of OSPF and
EIGRP, the protocols behave differently. When EIGRP adds the first static neighbor, the messages are
exchanged via unicast transmissions and only packets from statically configured neighbors are received,
others are discarded. This does not hold true for OSPF. Static configuration of a neighbor does not
allow message exchange from other neighbors. For BGP, the neighbors must be statically defined.
Dynamic neighbors support for BGP must be configured.
Once the message exchange entities are defined, the exchange should be further secured with message
authentication.
2. Neighbor authentication
Neighbor authentication is supported for BGP, IS-IS, OSPF, RIPv2, and EIGRP. It provides for source
authentication and message integrity checks. It is applied to the peer establishment and routing update
messages. Most routing protocol support two types of authentication: plain text and MD5. As MD5 is
more secure, it is recommended to use this authentication.
− In OSPF, the authentication is enabled and method is defined per area, in either router or interface
configuration mode with the area number authentication auth_method command. Authentication
parameters are defined in the interface configuration mode, using commands depending on the
authentication method selected. To verify the configuration use show ip route ospf. To observe the
ospf packet exchange in real-time use debug ip ospf packet.
− In EIGRP, the authentication is enabled in interface configuration mode, by using the ip
authentication mode eigrp as_nubmer auth_method command. Authentication parameters are
defined with the ip authentication key-chain eigrp as_number key_chain_id command. EIGRP
uses key chains. To verify the configuration use show ip eigrp neighbors. To follow the packet
exchange in real time use debug ip eigrp packets.
− To enable authentication in RIPv, issue the rip authentication mode auth_method command in
interface configuration mode. You define the authentication parameters in the same mode. To
verify, issue show ip route rip. To follow the exchange of messages in real-time issue the debug ip
rip command.
− In BGP, neighbor authentication is configured in the routing configuration mode as part of the
neighbor parameters. Only the password parameter needs to be set for the neighbor. Use the
neighbor ip_address password password command.
3. Limiting the extent of the routing information exchange
Routing information exchange should be limited only to the network segments where peering
adjacencies are expected. To exclude segments from routing information exchange:
− In the router configuration mode, issue the passive-interface interface slot/number command for all
the interfaces where peering relationships are not expected.
− Issue the passive-interface default command. This command treats all the interfaces that are
configured to participate in the routing process as passive. Therefore, interfaces supposed to
exchange information should be explicitly configured to do so using the no passive-interface
command. Both commands are used in the router configuration mode..
To avoid the routing-related problems in the network, you can follow these recommendations:
 Limit the access to the network by:
− Shutting down unused ports on all access devices
− Apply additional access control on used ports by using port-security interface features and/or ACLs
− Restrict the access to device’s management functions
 Harden routing functions and protocols:
− Limit peering to trusted devices: configure the peers explicitly.
− Limit the extent of routing message exchange only to interfaces where adjacencies are expected, by
configuring passive interfaces.
− Verify routing message exchange by using the authentication.
 Control routing messages’ content and distribution (advertising and reception)
− Use infrastructure and receive ACLs, QoS policies, route filtering.
Know which of the features have been implemented in your network and where they have been
implemented, because they restrict which devices can participate in routing. If they are misconfigured, they
can cause the operations of a routing process to fail.
Example Troubleshooting Flow: Multiple Masters
In A VRRP Group
This lesson suggests a troubleshooting flow for solving the issue of multiple Masters present in a VRRP
group.
Here, the VRRP configuration for VLAN 33 segment (10.0.33.0/24) results in two routers having the
Master state.
In order to verify the problem you issue the show vrrp brief command on routers R1 and R2. The
command output on R1 shows that R1 is the Master for VRRP group 3. Similarly, the command output on
R2 shows that R2 is also the Master for VRRP group 3. The problem is confirmed and you can begin
troubleshooting.
When there is a multiple-master issue in a VRRP group, it is usually due to different VRRP parameters
configuration on routers. Therefore, “Spotting the differences” approach seams adequate in this situation.
Also, during the problem verification you have noticed that VRRP is configured on other segments as well
and that there are Masters and Backups for those VRRP groups. So, you assume that these are correct
configurations. You can use them as a reference in the configuration comparison.
With the “Spot the differences” approach, you would like to perform two comparisons:
 The comparison of two configurations for the same VRRP group on two routers,
 The comparison of different VRRP group configurations on one router
The differences should give you the clue on what is causing the problem.
At first, you decide to compare R1 and R2 VRRP configurations for VRRP group 3. As you noticed from
the show vrrp brief output, this group is configured on interfaces Ethernet 1/1 on both routers. Hence you
issue the show vrrp interface ethernet 1/1 command on R1 and then on R2. You look carefully at both
configurations and notice immediately that different authentication methods are configured. R1 is using
clear-text authentication, while R2 is configured to use stronger MD5 based authentication. With different
authentication methods configured, there is no election between those routers and any misconfigured router
in the group changes its state to Master.
Based on the information you gathered and the analysis you made, you know that:
 VRRP group 3 is configured with different authentication methods on routers R1 and R2.
 VRRP group 3 is configured with the same primary IP address and the same advertisement intervals on
both routers.
Since authentication method difference is the only mismatch between R1 and R2 configurations, you claim
that the mismatch in the VRRP authentication method is preventing a normal VRRP PDUs exchange. This
results in both routers setting their state to Master.
Before you propose what authentication should be used, you decide to look at the VRRP configuration of
other VRRP groups, namely groups 1, 2, and 4. You would like to make the configurations consistent across
all the groups.
You issue the show vrrp command on R1. You notice that VRRP groups 1, 2 and 4 on R1 have MD5
authentication configured. Group 3 though has the clear-text authentication.
To summarize:
 There is a mismatch in VRRP authentication method for VRRP group 3.
 The only authentication method that is not MD5 and therefore differs from all others, is the one
configured for VRRP group 3 on R1.
 There are no mismatches in primary IP address and advertisement timers for VRRP group 3
configurations on routers R1 and R2.
You propose the following solution: the authentication method for the VRRP group 3 on router R1 should
be changed to MD5 and the same authentication string should be configured on both routers R1 and R2.
You could have decided to change the configuration of R2 to text-based authentication. But, this
authentication method is less secure and is not consistent with the configurations of the other VRRP groups.
In order to implement your solution, you follow these steps:

1. Enter the Ethernet 1/1 configuration mode.
2. Disable authentication on routers R1 and R2 for VRRP group 3 by issuing the no vrrp 3
authentication command.
3. Choose a key to be used for the authentication. You decide to use the Cisc0Brde3 string.
4. You configure the MD5 authentication method for VRRP group 3 on both routers by issuing the vrrp 3
authentication md5 key-string Cisc0Brde3 command.
In order to verify your solution, you issue the show vrrp brief command on R1 and R2. On R1, you see that
R1 is the Master for all groups, including VRRP group 3. R2 is the Backup router for all groups, including
VRRP group 3.
Note You have solved the problem and are now ready to conclude. You make sure that the
changed configurations are available after reboot. You copy the new configuration to the
startup-config by issuing the copy running-config startup-config command on all
devices where the configuration was modified, that is. on R1 and R2. You also document the
changes and update the documentation. You then inform the support engineer that the
problem was solved and close the ticket.
Troubleshooting VRRP
VRRP provides redundancy for the first-hop layer 3 device in the network, similar to HSRP.
There are several possible issues that could affect normal VRRP operations:
 VRRP parameters mismatch
 Virtual router IP address
− All devices in one VRRP group must have the same IP address. Misconfigured devices assign
themselves the Master status. the hosts are able to reach the remote networks, but have only
sporadic connectivity to virtual router’s address or addresses.
− In HSRP, only one router becomes Active. Only the hosts configured with active router’s IP address
can reach the remote networks.
 VRRP group number
− All the devices sharing a virtual router’s IP address must be in the same VRRP group. If the same
virtual IP is assigned to different groups, duplicate addresses appear in the network. Misconfigured
devices become Masters. This is similar to HSRP behavior. The hosts can reach the remote
networks. The traffic splits among routers in different groups.
 Authentication method
− All the devices in a VRRP group must have the same authentication method and a corresponding
key. Otherwise, the misconfigured routers become the Masters. HSRP behaves the same. The hosts
are able to reach the remote networks with traffic split among the routers in the group.
 Advertisement timer mismatch
− Advertisement timers must match across the VRRP group. This timer determines how often the
Master sends out the advertisement message to other group members. It is also used in the
calculation of the master-down timer.
− Routers can be configured to learn the advertisement timers from the Master. Sub-second timers
cannot be learned. The timers learned from the Master override the manually configured timers. The
show vrrp command displays the originally configured timer values. Learning is not the default
behavior of VRRP.
− Note that the default advertisement timer value for VRRP is 1 second and the master-down interval
is calculated from it. Default advertisement timer for HSRP is 3 seconds and hold timer is
configurable. There is only one VRRP version.
Preemption issues:
 Unexpected device elected as Master
− When you enable a VRRP group on an interface, the protocol is fully operational. Role election
begins immediately. If you disable the preemption, the first operational device is elected as a Master
and will stay so until its configuration or the interface status changes.
− Preemption is a configuration default in VRRP. In HSRP you have to configure the preemption
explicitly.
 Disruption of packet forwarding
− Master routers can be configured to track the state of the objects of interest and to decrease their
priorities accordingly. When the priority value drops below the priority of the Backup routers, their
default preemption mechanism triggers the overtake of the Master role.
− The changes of the tracked object can also trigger other network protocols. You should use the
preemption delay to allow these protocols to converge in order to avoid flapping between Master
and Backup state, which can disrupt the packet forwarding.
Improper exclusion of an interface from the VRRP group

 In VRRP, the virtual IP address can be equal to the address of one of the physical interfaces of one of
the routers. When you exclude this router, you can create duplication of IP addresses. To avoid this, you
can change the virtual IP address and the corresponding hosts configurations or you can either remove
the VRRP group or shutdown the physical interface.
In HSRP, the virtual IP address must be different from the interface IP addresses in the group.
Blocked VRRP messages
 VRRP messages are sent to multicast IP address 224.0.0.18 and UDP port 112. This IP address and
port must be permitted in the inbound access lists. If the packets are blocked, the peers will not see each
other and there will be no VRRP redundancy. To check the interface access list, use the show ip
interface command.
Diagnostic tools specific to the VRRP protocol:

 To get a concise overview of the VRRP groups, their basic parameters, and router statuses, issue the
show vrrp brief command.
 To view VRRP groups on a specific interface, issue show vrrp interface type number [brief].
 To display debugging messages for VRRP errors, events, and state transitions, use debug vrrp all.
 To limit the debug output only to the MD5 authentication messages, messages about error conditions or
status transition, use the debug vrrp authentication, the debug vrrp error and the debug vrrp state
command.
 To view the summary about the sent and the received packets, use debug vrrp packets and for
following the VRRP events use the debug vrrp events command.
Example Troubleshooting Flow: Non-Functional
EtherChannel
This lesson suggests a troubleshooting flow for solving the issues of a configured but inactive EtherChannel
aggregate link.
Here, the customer configured the EtherChannel between switches SW3 and SW4. However, the
EtherChannel does not function.
When troubleshooting the EtherChannel link between SW3 and SW4 you only have access to SW3. On
SW3 you issue the show etherchannel summary command. You confirm that the link aggregation was
configured and that it does not function as Ethernet 2/0 port is in the suspended state.
The output also shows that there is one channel-group with port-channel Po1 logical interface. The
EtherChannel was configured statically, that is, the channel establishment protocols, such as LACP or
PAgP, were not used.
You have confirmed the problem. Because you cannot check the configuration at SW4 switch, you assume,
that its configuration is correct and troubleshoot the SW3 switch. The information that you gathered in the
verification step shows that there are no lower layer connectivity issues. Output of the show etherchannel
summary command shows Ethernet 2/0 interface that is marked with “s” for suspended. This means that its
configuration does not correspond with the configuration of other interfaces in the channel.
You decide to continue by gathering more information on EtherChannel and interface configuration
parameters.
Gathering and Analyzing Information
In order to see the details of EtherChannel configuration you use the show etherchannel 1 detail command.
The output reveals that the misconfigured trunking mode might be putting interface Ethernet 2/0 in the
suspended mode. Next you go on to verify the trunking configurations of Ethernet 2/0 and Ethernet 2/1
interfaces.
In order to check the trunking modes of interfaces Ethernet 2/0 and Ethernet 2/1 you use show interface
ethernet 2/0 and show interface ethernet 2/1. What you see is that Ethernet 2/0 has trunking mode set to
access, while Ethernet 2/1 is configured as trunking interface. Native VLAN of both interfaces is VLAN1
and all VLANs are allowed on the trunk. In order for interfaces to aggregate into one channel they must
have the same switchport mode set. You are ready to make your hypothesis.
One of the preconditions for an operational port-channel is the compatibility of port parameters for
participating interfaces, such as:
 speed
 duplex
 trunking mode
 trunking encapsulation
 native VLANs
 VLANs allowed on trunks, and so on
If this condition is not met, the port-channel is not operational.
In your case, the channel establishment process is static, that is, the channel is manually configured. The
switch forces all the compatible ports to become active in the EtherChannel. Since Ethernet 2/0 and Ethernet
2/1 interfaces have different trunking modes, access and trunk respectively, only one of them is actively
included in the channel.
If the hypothesis is true, configuring the same trunking mode on interfaces should solve the problem. You
decide to remove the channel-group 1 and to reconfigure the interfaces. You perform the following steps for
the interface range Ethernet 2/0-1:
1. Shut down the interfaces with the shutdown command.
2. Set the trunking mode to trunk, by issuing switchport mode trunk.
3. Bring up the interfaces with the no shutdown command.
The console output gives you a positive feedback: interfaces Ethernet 2/0 and Ethernet 2/1 are up with line
protocols up. Interface port-channel changes to up with the line protocol up. You proceed with the solution
verification.
Verifying the Hypothesis
In order to verify the solution, you issue the show etherchannel summary command again. In the output,
you see both Ethernet2/0 and Ethernet2/1 in the passive state, denoted by the “P” flag. This indicates the
normal operation. Your hypothesis proved correct. You have solved the problem.
Note You can now finalize the troubleshooting. What remains to be done is to save the changes,
document them, inform the customer that the issue was resolved and close the ticket. You
make sure that the changed configuration is available after reboots. You copy the new
configuration to the startup-config on SW3. No other configurations were changed.
Troubleshooting EtherChannel
EtherChannel is a technology that bundles multiple physical Ethernet links (100 Mb/s, 1 Gb/s, 10 Gb/s) into
a single logical link and distributes the traffic across these links. This logical link is represented in Cisco
IOS syntax as a port-channel interface. Control protocols such as spanning tree or routing protocols interact
only with this single port-channel interface and not with the associated physical interfaces. Packets and
frames are routed or switched to the port-channel interface, and then a hashing mechanism determines
which physical link will be used to transmit them.
Check the following operational characteristics. these must match for all interfaces in the EtherChannel:
 Interface speed and duplex
Use the show interface type number command to verify the port speed and duplex. To correct
misconfigured values, use the speed {10 | 100 | 1000 | auto | nonegotiate} interface configuration
command and/or duplex {auto | full | half}.
 Interface trunking mode and related VLANs
Use the show interface type number switchport or show running-config interface type number
command to verify trunking mode of an interface. The following must match: - mode (access or trunk),
- VLAN number (for access interfaces), - native VLAN, VLANs allowed and encapsulation (for trunk
interfaces).
 Layer, on which an interface operates
All of the physical interfaces must be acting in the same layer, otherwise the channel will not form; if
you are creating an L3 EtherChannel make sure to issue no switchport command for all participating
interfaces.
To achieve the interface configuration consistency it can be simpler to use the interface range configuration
mode. Always verify the operational i.e. the negotiated characteristics of the interfaces. Be aware that there
are two sides to the link. The operational parameters of an interface will depend on the configurations of
both sides.
For instance, it is possible for a link to be connected (up, up) even if there are duplex mismatches (only one
side uses autonegotiation). Duplex mismatches on an operational link can be noted in the output of show
interface type number and is indicated by a high number of packet errors. Once the EtherChannel is
established, logical interface port-channel is automatically created. Member interfaces can be further
configured by using the logical interface configuration mode – configuration will apply to all physical
interfaces in the channel. So, using the port-channel configuration mode it is simpler to achieve VLAN
consistency and configuration over the members of the aggregated link.
To check the status of the member interfaces and the logical channel interface use: show etherchannel
summary and show etherchannel groupnumber detail. In the output of the show etherchannel summary
command operational links are marked with the capital P letter, which stands for passive and signifies that
the link is bundled to the port-channel. Suspended links are marked with a small letter s in the output of the
show etherchannel summary command or with the suspended port state in the output of the show
etherchannel groupnumber detail command.
If one of the physical links changes its operational status in such a way that a mismatch with other physical
links is created, this link will be suspended and removed from the EtherChannel bundle until consistency is
restored.
To check which distribution algorithm is configured for the channels, use the show etherchannel load-
balance command and for traffic statistics use the show etherchannel traffic command.
Channel establishment protocol must be the same for all the interfaces in the EtherChannel. You cannot run
two EtherChannel protocols in one EtherChannel.
Switch interface can be configured to one of three channel-establishment protocol:
 Manual: no protocol, statically configured channel
 PAgP: Cisco proprietary link aggregation protocol
 LACP
Make sure that the channel establishment protocol is the same for all the interfaces in the EtherChannel.
Use show etherchannel summary or show etherchannel group_number detail to verify the status of the
channel and protocol of choice. In the output, search for the keyword Protocol. Dash “-“ means manually
set, other valuse “LACP” or “PAgP” are self-explanatory.
Check the protocol mode of operation on all interfaces and on both devices.
Only the following combinations of modes on a link form a portchannel:

 active-active and active-passive,
 desirable-desirable and desirable-passive,
 on-on.
To check for misconfigurations use show etherchannel summary command. Look under Port-channel. If it is
marked with D letter, meaning Down – channel has not formed.
Look under Ports. Member interfaces are marked with:
 I: Means individual. When there is a protocol mismatch, a link is treated as an individual link and it
does not join the channel group.
 D: Interfaces are disabled manually or by software.
The “on” mode means that EtherChannel is established immediately, provided that all the physical
interfaces have same characteristics. There is no message exchange with the other side of the link. That
means, the channel forms even if the other side is not configured to aggregate ports. In such situation the
other nonconfigured side considers physical interfaces as individual interfaces and STP protocol detects a
loop, shuts the interface down placing it in err-disabled state. The same happens if both sides are configured
to aggregate, but one ahead of the other with more than a minute between activating configurations.
It is a common misconception that the load balancing among the links in the channel is equal. This depends
on the traffic distribution algorithm and the number of interfaces in the channel. The EtherChannel
distribution algorithm determines which header fields are used to calculate the hash value. If you could
assume those fields in the traffic to be entirely random, it would not matter what hashing mechanism is
used. Typically, header fields are not random and the choice of header fields to be hashed affects the
distribution. The simplest example of this is where only the destination MAC address is used as the input
for the hash calculation. If 90 percent of all frames would be destined for a single MAC address (for
example, the MAC address of the default gateway), all of that traffic would end up on the same physical
link. So if you see a very uneven distribution of traffic over the links in the channel, you should examine the
hashing method and the traffic mix to determine the cause.
To check the distribution algorithm configured for the channels, use the show etherchannel load-balance
command and for traffic statistics use the show etherchannel traffic command.
Summary
Fourth Troubleshooting Lab
at TINC Garbage Disposal
Ltd.
Overview
This lesson serves as a debrief for the fourth troubleshooting lab at the TINC Garbage Disposal Ltd.

• Describe how you solved occasional lack of connectivity to PCs 1 and 2
• Describe how to troubleshoot GLBP
• Describe how to troubleshoot FHRPs
• Describe how you solved sporadic loss of connectivity on PC4
• Describe usage of DHCP snooping
• Describe how you solved SSH connectivity issues from PC4 to GW2
• Describe Cisco Technical Assistant Center
You work for SECHNIK Networking Ltd. TINC Garbage Disposal Ltd. is your company's customer.
Donovan, the customer engineer, needs your help troubleshooting some network issues.
 Donovan attended a networking workshop and learned that if he replaces VRRP with GLBP, the
uplinks in his network would be more evenly utilized. After the implementation, GLBP seems to be
working just fine for VLANs 33 and 44. However, in VLAN 11, Donovan noticed some strange
behavior. VLAN11 clients sometimes have Internet connectivity, other times they do not. Since you are
troubleshooting this problem in non-working hours Donovan is giving you the permission to use
invasive test that would cut-off PC1's and PC2's connectivity, if needed.
 Donovan reports that PC4 sometimes looses the IP connectivity. He tells you that renewing the IP
address usually helps solve the issue.
 Donovan's PC, PC4 (and only PC4), should be able to use SSH to connect to Layer 3 devices in the
network. For some reason Donovan cannot connect to GW2.
To test the Internet connectivity use the IP address of 209.165.201.225.
Example Troubleshooting Flow: Occasional Lack
of Network Connectivity For PCs 1 And 2
VLAN 11 clients sometimes have Internet connectivity, other times they do not.
To check whether VLAN 11 clients have connectivity to Internet, you issue the ping 209.165.201.225
command from PC1 and PC2. You conclude that at the moment there is no connectivity problem.
In an effort to verify the problem, you decide to check the network connectivity to the default gateway.
Note Not all issues are present all the time. It's your job, as the troubleshooter, to try to reproduce
them or to dig further.
From the output of the ping 10.0.11.1 command you conclude that PC1 can reach the default gateway and
PC2 cannot.
The virtual MAC addresses used in GLBP have the form 0007.b40#.##XX, where ### indicates the group
number and XX indicates the forwarder.
The ARP table, that is, the output of the show arp command, on PC1 shows that the default gateway’s IP
address 10.0.11.1 is mapped to the virtual MAC address 0007.b400.0101, which is assigned to the
Forwarder 1 for the GLBP group 1. On PC2, the same IP address is mapped to 0007.b400.0102, which is
the virtual MAC address of the Forwarder 2 for the GLBP group 1.
PC2 sends ICMP echo requests to the R2 router, which does not respond to them.
As you can see, the R2 router does not associate itself with the IP address 10.0.11.1, which also means that
it will not answer to those specific ARP requests. So, in case of the R1 failure, Internet connectivity would
be broken since no router would be left to reply to ARP requests for the default gateway IP address. You
decide to simulate the failure of one of the routers to verify that the problem exists under such conditions.
The simulation you are planning will probably disrupt the normal network operation. That is why you
coordinate the activities with the customer and your team, so that you perform the simulation when the
negative impact is minimal.
When the GLBP redundancy mechanism is deployed on the network, the default gateway IP address
received via DHCP should be a GLBP virtual IP address. The most probable cause for unreachable virtual
address is GLBP misconfiguration.
The load-balancing mechanism of the GLBP protocol relies on the distribution of Forwarders’ virtual MAC
addresses in the ARP replies served by the Active router. In your case, the first ARP requests were sent by
PC1 and PC2 when you checked the Internet connectivity trying to verify the problem. In ARP reply from
the Active router, PC1 must have received the virtual MAC address of Forwarder 1, and PC2 must have
received the virtual MAC address of the Forwarder 2. To verify, you check the ARP table entries on both
PCs.
GLBP supports three different load-balancing mechanisms: „round-robin“, „weighted“, and „host-based“. If
either round-robin or weighted were configured, the resulting virtual MAC address distribution would be the
same. In the case of host-based distribution, both hosts could receive the same MAC address in the ARP
response.
You begin by shutting down the Ethernet 1/0.11 sub-interface on the R1 router. This initiates the fail-over
procedure.
First, you make sure to erase the IP to MAC address mappings from the ARP table. Use the clear ip arp
10.0.11.1 command to erase the ARP table entry for the default gateway. You can also decide to wait for the
entry to age out, but this would render the troubleshooting lengthier than necessary.
Now you repeat the network connectivity check from both PCs. It is not successful. You look into the ARP
table and you notice that there is an entry for the default gateway, but without concrete data. The ARP
request was sent, but was not answered.
You have verified that the problem is present when R1 leaves the GLBP group 1 and R2 takes over. You
can begin troubleshooting.
Note This is only one of the possible troubleshooting plans. There are many different approaches
to addressing this problem.
Before you start troubleshooting, you should restore the initial network conditions. When a network is
configured with the first-hop redundancy protocol, lack of connectivity can be the result of the redundancy
protocol misconfiguration. You decide to start troubleshooting by verifying the GLBP operational status.
To restore the initial network conditions, you enable the Ethernet 1/0.11 sub-interface on R1.
On both routers, issue the show glbp brief command. Confronting the two configurations means that you
will use the “spot the difference” troubleshooting approach.
From the outputs, you see that on both routers GLBP group 1 is configured for VLAN11 on the sub-
interface Ethernet 1/0.11. R1 has the role of the Active router with the priority of 100 and R2 is the Standby
router for the group, with the priority configured to 90. There are two Forwarders—R1 is acting as
Forwarder 1 and R2 as Forwarder 2. On R1, the virtual IP address of the group is configured to 10.0.11.1.
On R2 it is different and is 10.0.11.11.
Since there are Active and Standby routers in the group, you assume that the other GLBP parameters in R1
and R2 configurations match.
Analyzing the Information and Proposing a Hypothesis
If for any reason the fail-over process would be initiated in VLAN 11, R2 would take the role of the Active
router. It would assume its role as the default gateway available at the 10.0.11.11 IP address. Since the
default gateway of the client’s in VLAN 11 is different, the clients would soon loose the Internet
connectivity. That would happen when the ARP entries for IP 10.0.11.1 age out.
You can make the first hypothesis: mismatch in virtual IP address in configurations for the GLBP group 1 is
effectively limiting the GLBP redundancy in the VLAN 11 segment.
Choosing to simulate the failure at this point in troubleshooting will cause disruption in network operation.
Another way to gather supportive information is to check the connectivity of the clients to the default
gateway itself.
In order to configure the new IP address as the virtual IP for the GLBP group 1 you first issue the no glbp 1
ip 10.0.11.11 command in the sub-interface configuration mode at R2 to remove the incorrect configuration.
At this point, GLBP group 1 on R2 does not have configured virtual IP address. If you check the new
configuration by issuing the show glbp ethernet 1/0.11 | include IP address command, you can see that
Active router’s virtual IP is learnt via Active routers hello messages. To configure the virtual IP, to avoid
relying on the information sent in hello packets, you issue the glbp 1 ip 10.0.11.1 command. To make sure
that the change is applied, you check the configuration with show glbp ethernet 1/0.11 | include IP
address. The output verifies that the new configuration is in place.
You want to make sure that this change has solved the problem.
You decide to check connectivity to the default gateway and to the Internet.
Before you issue the ping commands on PC 1, you would like to resend the ARP requests. You clear the
existing mapping for the default gateway IP address by issuing the clear ip arp 10.0.11.1 command and
verify the result by issuing the show arp command. The show arp command does not contain the entry for
the default gateway IP address.
PC1 has connectivity to the default router and the Internet. From the output of the show arp command, you
see that it is using Forwarder 1 via the virtual MAC addresses 0007.b400.0101
You repeat the same check on PC2. It has connectivity to the default router and the Internet. It is using
Forwarder 2, seen from the ARP entry, which shows the MAC addresses 0007.b400.0102.
Since both VLAN11 clients can reach Internet and their default gateway, you presume that the problem is
solved.
The final check would be to simulate the Active router’s failure and observe whether it impacts the
connectivity. If it does not, you have solved the problem. If, however, it creates connectivity issues, you will
have to troubleshoot further.
Since there is a possibility of interruption to network operation, you coordinate the simulation with the
customer so that its negative impact is reduced to minimum.
In order to check connectivity, you should not clear the ARP tables on PC1 and PC2. You want to make
sure that PCs will have connectivity while relying on previously received information. Based on this
information, PC1 and PC2 will use the virtual MAC addresses of Forwarder 1 and Forwarder 2,
respectively.
On both clients, you issue the ping 10.0.11.1 and the ping 209.165.201.225 commands. Both clients can
reach the default gateway and the Internet.
To verify that the traffic is passed to R2, you check the MAC address table on switch SW1.
On switch SW1, you issue the show mac address-table command. It shows that virtual MACs of
Forwarder 1 and Forwarder 2 map to the same port Ethernet 1/0.
By issuing the show cdp neighbors command you confirm that Ethernet 1/0 port is connecting SW1 with
R2 router.
The simulation has shown that the GLBP configuration is providing the required redundancy and will not be
the cause of Internet connectivity problems in case of a real failure of the R1 router.
You now bring the R1’s Ethernet 1/0.11 interface up.
You have verified that in case of failure of the router R1, R2 router takes Active role in group 1 with the
same IP address as was configured for the other router in the group. It is also active as Forwarder 1 and
Forwarder 2. Clients do not notice this change.
At the beginning of the troubleshooting this was not the case. R2 has taken the Active role but with the
incorrect IP address. This has cut off VLAN 11 clients from the remote networks.
Bring interface Ethernet 1/0.11 up. After 30 seconds verify that GLBP is working as expected. As you can
see R1 has reassumed its role as the Active gateway for the GLBP group 1. Next step is to check
connectivity from PC1 and PC2.
Clients in VLAN 11 have the Internet connectivity. The simulation of R1’s failure demonstrated that the
fail-over mechanism is in place and will ensure an alternate Internet connectivity path.
The problem is solved.
Note You are now ready to conclude. You want to make sure that the changes are saved. Make
sure to copy the new configuration to the startup-config file by issuing the copy running-
config startup-config command on all the devices where the configuration was
modified. You also document the changes and update the documentation. You then inform
the support engineer that the problem was solved and close the ticket.
Troubleshooting GLBP
GLBP provides the first-hop redundancy, as do VRRP and HSRP. In addition, GLBP facilitates the load-
balancing across the redundant routers.
There are several possible issues that could affect normal GLBP operations.
GLBP misconfigurations
 Virtual IP address
− Devices can reach remote networks. Some of these devices will not be able to reach the default
gateway address.
− The behavior is similar to HSRP. In VRRP hosts have connectivity to remote networks and only
sporadic connectivity to the default gateway address.
 Group number
− All the misconfigured devices become Active routers. There is no interruption in connectivity to the
remote networks, but there is no redundancy either.
− This is similar to HSRP and VRRP behavior.
 Authentication method
− If authentication method or key do not match in a group, the misconfigured devices become Active.
Remote networks are reachable, but the redundancy is lessened.
− This is similar to HSRP and VRRP behavior.
Preemption issues:
 Unexpected device becomes Active or Forwarder
− Check the preemption configuration
 The virtual gateway preemption is not the default configuration. If priority is not
configured, the first configured device becomes the Active router.
− Preemption is a configuration default in VRRP. HSRP and GLBP do not preempt by default.
 Disruption of packet forwarding
− Check the preemption related delays.
 The GLBP fail-over mechanism, in itself, does not cause packet loss. At all times, there
will be an active router and at least one forwarder on the segment. GLBP is only involved
in the convergence of the outbound packets that the host sends via the default gateway.
Convergence for the packets returning to the host is not governed by GLBP but by routing
convergence. The overall convergence is as fast as the protocol that is the slowest to
converge. Until the convergence is reached, packet loss is possible.
Blocked GLBP messages

 GLBP messages are sent to multicast IP address 224.0.0.102 and UDP port 3222. This IP address and
port must be permitted in the inbound access lists. If the packets are blocked, the peers will not see each
other and there will be no GLBP redundancy. To check the interface access list use the show ip
interface command.
Troubleshooting FHRPs
Diagnostic tools for GLBP protocol:

 To get a concise overview of the GLBP groups, their basic parameters and router statuses, issue the
show glbp brief command.
 To view GLBP groups on a specific interface, issue the show glbp interface type number [brief]
command.
 To view the summary about the sent and the received packets, use debug glbp packets, and for
following the GLBP events, use the debug glbp events command.
 To view the relevant messages but exclude the GLBP hellos, use debug glbp terse.
 To display debugging messages for GLBP errors, events, and state transitions, use debug glbp all.
 To limit the debug output only to messages about error conditions or status transition, use the debug
glbp error and the debug glbp state commands.
The structure of the command is similar to all the first-hop redundancy protocols. The equivalent
troubleshooting commands are given in the table.
The following are the major differences between these protocols:
 HSRP and GLBP always require an additional IP address to function as the virtual IP address. With
VRRP, you can use one of the router’s assigned IP addresses as the VRRP IP address. Consequently,
this router will then always be the master router for that IP address when it is up, even if another router
has a higher priority.
 VRRP is an IETF standard (RFC 5798), which makes it suitable for multivendor environments.
 HSRP and GLBP do not preempt by default. If you want a higher-priority router to take over when it
comes up on the segment, you must configure the preemption option. In VRRP, it is the opposite.
Higher priority routers will preempt by default. If you do not want this, you should disable the preempt
option.
 GLBP can have multiple routers forwarding traffic for a single virtual IP address. It achieves this by
using multiple virtual MAC addresses for a single virtual address. There is still a single router that is in
control of the virtual IP address and answers the ARP requests for that IP address. This router
effectively balances the load over the different forwarding routers.
 Default hello timers are different (3 seconds for HSRP, 1 second for VRRP, and 3 seconds for GLBP).
Example Troubleshooting Flow: Sporadic Loss of
Connectivity on PC4
The customer reports that PC4 sometimes loses the IP connectivity. He tells you that renewing the IP
address usually helps to solve the issue.
Verify the Problem
To verify the problem, check the documentation to find out which VLAN and which subnet PC4 belongs to.
You see that PC4 should be located in VLAN 44 within the subnet 10.0.44.0/24.
To check the IP connectivity from PC4, ping the default gateway in VLAN 44. You can see from the output,
that the ping is not sucessful. You have verified that the problem is present and you can start
troubleshooting.
Note If you notice that the ping is successful, you should release and renew the IP address on the
interface to recreate the problem. Use the release dhcp ethernet 0/0 and the renew
dhcp ethernet 0/0 command.
Note This is one possible troubleshooting plan. There are many different approaches to
PC4 should be located in VLAN 44 with the subnet 10.0.44.0/24. You have verified that there is no
connectivity to the default gateway.
When there is no connectivity to the default gateway, you should use „bottom-up“ method. Therefore you
should first check the status of the interfaces on the PC4.
When there is no connectivity to the default gateway, it is a good approach to start by checking the status of
the interfaces on the host. You should use the show ip interface brief command. You notice that the current
IP address is 10.0.0.3 and it is assigned by DHCP. Based on the documentation, the IP address should be
from the10.0.44.0/24 range. You can conclude that there is a wrong IP address assigned to PC4. Therefore
you should check if PC4 is connected to the correct VLAN, which should be 44.
To verify if PC4 is connected to VLAN 44, first check the MAC address of the interface Ethernet 0/0. The
next step is to check the MAC address table on the switch SW3 using show mac address-table. As you
can see from the output, PC4 is connected to VLAN 44 on switch interface Ethernet 0/2.
Consult the documentation for more information on VLAN 44 DHCP servers. It states that routers R1 and
R2 should be DHCP servers for VLAN44. Next, check the relevant DHCP configuration on both routers to
verify that the DHCP servers are correctly configured.
Check the DHCP configuration on the R1 router. Based on the information gathered, you conclude that the
DHCP configuration on R1 is correct.
You should check the DHCP configuration on the R2 router as well. As you can see from the output, the
DHCP configuration for the VLAN 44 is correct.
With the „bottom-up“ approach, you first checked the status of the interface on PC4, using the show ip
interface brief command. You discovered that interface Ethernet 0/0 was up and operational but its IP
address was 10.0.0.3. Based on the documentation, you found out that the IP address for VLAN 44 devices
should be from subnet 10.0.44.0/24. The next step was to check if PC4 is connected to the correct VLAN.
You also verified that the DHCP configuration on the DHCP servers for VLAN 44 was correct. You can
conclude that all configurations are correct.
Since a wrong IP address was assigned to PC4 you can assume that there is a rogue DHCP server in the
network, assigning IP addresses from the wrong subnet. Therefore your next step is to find that server and
prevent this server from assigning IP addresses to the hosts in VLAN 44.
Find the Cause of the Problem
To find the IP address of the server, turn on debugging with the debug dhcp command. To force DHCP
process, release the current IP address and renew it on the interface Ethernet 0/0. The commands are release
dhcp ethernet 0/0 and renew dhcp ethernet 0/0.
Note If PC4 gets the correct IP address, you should repeat release and renew until you find out
the IP address of the server that offers a wrong IP address..
Verify the debug output of the debug dhcp command. As you can notice from the output, the DHCP offer
was received from the IP address 10.0.0.1, which is most likely the rogue DHCP server that you are looking
for.
Check the connectivity to the rogue DHCP server. You can find out the MAC address of the rogue DHCP
server in the ARP table of PC4, using the show arp command.
To locate where the rogue DHCP server accesses the network, you use the MAC address table of switch
SW3. As you can see from the output of the show mac address-table | include aabb.cc00.5200 command,
the rogue DHCP server is connected to switch port Ethernet 2/0, which belong to VLAN 44.
To prevent rogue DHCP servers from offering IP addresses, you should implement DHCP snooping on the
switches. DHCP snooping allows only the interfaces that are configured as trusted to offer IP addresses to
the hosts. By default, all the interfaces are configured as untrusted interfaces, therefore you must explicitly
configure the trusted interfaces.
You must find out, which interfaces the legitimate DHCP servers connect to. This information is necessary
in order to correctly configure interfaces as trusted DHCP snooping interfaces. Use the show cdp neighbors
command. As you can see from the output, routers R1 and R2 are connected to interfaces Ethernet 0/0 and
Ethernet 0/1, respectively.
Configure the DHCP snooping on VLAN 44 on switch SW3. You first need to enable the DHCP snooping
with the ip dhcp snooping command and then enable the DHCP snooping for VLAN 44 using the ip dhcp
snooping vlan 44 command. You should also enable trusted interfaces to specify the location of the
legitimate DHCP servers. You should use the ip dhcp snooping trust command .
Use the show ip dhcp snooping command to verify the configuration of the DHCP snooping on the switch.
As you can see from the output, the DHCP snooping is enabled on VLAN 44 and interfaces Ethernet 0/0
and Ethernet 0/1 were configured as trusted DHCP snooping interfaces.
Verify the Solution
After enabling the DHCP snooping on switch SW3, you should force the DHCP release and the DHCP
renewal on PC4. PC4 should stop receiving the DHCP offers from the rogue DHCP server 10.0.0.1. As you
can see from the output, the IP address assigned to PC4 was 10.0.44.146, offered by DHCP server
10.0.44.3.
Note The implementation of the DHCP snooping in this IOS version does not work as expected
and it is still allowing DHCP offers although everything is configured correctly. This is a bug
and, if you are using real hardware equipment, the DHCP offers received on the untrusted
interfaces are dropped. However, bugs can be present on software of real devices. In that
case, you would contact Cisco Technical Assistance Center to report the problem on your
version of IOS.
To prevent similar situations in other VLANs, you should implement the DHCP snooping on switches SW1
and SW2 as well.
Note What you would need to do in this situation is upgrade the IOS image on your access layer
switches and then implement and verify DHCP snooping solution.
Note At this point, you should inform the support engineer that the problem is solved. There
should be no more IP connectivity loss. Save the configuration on switches SW1, SW2 and
SW3. Document the changes. Close the ticket.
DHCP Snooping
DHCP snooping is a L2 security feature that acts like a firewall between untrusted hosts and trusted DHCP
servers. The primary function of the DHCP snooping is to prevent rogue DHCP servers in the network.
DHCP snooping is enabled on switches on per VLAN basis. Interfaces on the switches are configured as
trusted or untrusted. Trusted interfaces allow all the types of DHCP messages, while only requests are
allowed on untrusted interfaces. Trusted interfaces are interfaces that connect to a DHCP server or are an
uplink towards the DHCP server.
With DHCP snooping enabled, switch also builds a DHCP snooping binding database. Each entry in the
database includes the MAC address of the host, the leased IP address, the lease time, the binding type, the
VLAN number, and interface information associated with the host. The DHCP snooping binding database is
also used by other security features, such as dynamic ARP inspection.
DHCP snooping can also be used to limit the rate of DHCP messages. This option is configured per switch
interface.
To configure the DHCP snooping on the interface follow the steps:

 Enable the DHCP snooping globally with the command ip dhcp snooping.
 Enable the DHCP snooping on the VLANs using command ip dhcp snooping vlan number.
 Configure interfaces that connect to the DHCP server or uplink to the DHCP server as trusted with the
ip dhcp snooping trust command.
 Optionally enable rate limit on the untrusted interface. Rate is configured as DHCP packets per second.
Use the ip dhcp snooping limit rate rate interface command.
 To verify the DHCP snooping configuration, use the show ip dhcp snooping command. You can check
the DHCP snooping binding database using the show ip dhcp snooping binding command.
Cisco TAC
When you have a technical issue associated with the device configuration process or you have found failure
in the software or in the network that you are unable to fix and you did not find the answer to you problem
on the Cisco Support Website, you can contact Cisco Technical Assistance Center.
Cisco TAC provides 24-hours-a-day technical assistance, for all who hold valid Cisco service contracts.
Depending on the priority of your problem, you have two options for opening a case with Cisco TAC:
 Using the Support Case Manager website or email for lower priority incidents:
− Priority 4: Additional information or assistance with Cisco product capabilities, installation or
configuration are required. There is little or no effect on business operations.
− Priority 3: Operational performance of network is impaired while most business operations remain
functional.
 By telephone for higher priority incidents:
− Priority 2: Operation of an existing network is severely degraded, or significant aspects of business
operation are negatively affected by inadequate performance of Cisco products.
− Priority 1: The network is “down” or there is a critical impact on business operations
When using the website, the TAC Service Request Tool automatically provides the recommended solutions.
If this does not solve your problem, it is assigned to Cisco TAC engineer.
Note Before opening a case with Cisco TAC, you should test if the failure is present in the latest
software maintenance version, try to resolve the problem on your own, and try to obtain a
minimum configuration that causes the problem.
When you open a case with the Cisco TAC, you must provide preliminary information to better identify the
issue.
Following information should always be provided:

 Cisco service contract number
 Network Layout: Provide a detailed description of the physical and logical setup, as well as all the
network elements involved and their software versions.
 Problem Description: Provide step-by-step detail of actions that the user performed when the issue
occurs. Include the information about expected behavior and detailed observed behavior.
 General Information: Is this a new installation, what changes were recently made to the system, is the
issue reproducible, what are the affected devices, how did you try to troubleshoot, relevant syslog/tac
logs before the issue occurred, and so on.
You have an option to check the status of your opened case at any time, based on the case reference number.
There is also an applet that allows you and your Cisco TAC engineer to work together more effectively by
using Collaborative Web Browsing, whiteboard, Telnet and clipboard tools.
Note Cisco TAC is also available as a mobile application.
Example Troubleshooting Flow: No SSH
Connectivity to GW2 From PC4
The customer is not able to access the router GW2 via SSH from PC4.
Note This issue is also connected to issues in other challenge lab tickets.
Verify the Problem
First verify if the problem exists. Try to access the router GW2 from PC4. The connection is refused,
therefore you are not able to access the router GW2 via SSH and the problem is confirmed.
problem.
When you are checking the connectivity to a specific device it is a good practice to first check if there is
network connectivity. This method is called „divide and conquer“, because you first test the network
connectivity before advancing to lower or higher layers in the TCP/IP stack.
Eliminate Possible Causes
You decided to go with the “divide and conquer” troubleshooting approach. With this approach the first step
is to test the network connectivity. As you can see from the output, the ping to the GW2 from PC4 is
successful. You can conclude that there is network connectivity between PC4 and GW2.
Since you found out that there is network connectivity between the PC4 and the router GW2, you can
conclude that the problem is most likely related to SSH configuration on the router GW2. Check the SSH
status on the router GW2 with the show ip ssh command. You can see from the output that SSH is enabled
and operational. Only SSH version 2 is enabled.
Since only SSH version 2 is enabled on the router GW2, you should try to connect with SSH version 2 to
eliminate the SSH client problem. Although you used the SSH version 2, connection is still refused.
Next step in the troubleshooting process is to check the configuration on the router GW2. You should check
the vty line configuration with the show running-config | section line command. As you can notice from
the output, there is an access list applied to line vty 0 4. You should check if the access list is permitting the
access from PC4.
Check the access list that is applied to the vty line with the show access-lists 22 command. You can notice
that network 10.0.44.0/24 is not permitted to access lines vty 0 4. Therefore you can conclude that the
access list 22 applied to the vty line is blocking the packets from subnet 10.0.44.0, to which PC4 is
connected. Before you make any changes to the access list, you should confirm with the customer that he is
allowed to access the router GW2. You should also verify the documentation if there is any note on who can
access the GW2 router. If you find out that the access list was not configured correctly, you should make
changes to the access list.
After gathering information you can conclude that the most likely issue is related to the access list that is
applied to vty line. The access list does not permit management access from subnet 10.0.44.0/24. You
should add this subnet to the access list and check if the problem is still present.
Check the vty line configuration and access list 22 on the GW1 router since PC4 can access it via SSH.
You should check the devices with similar configuration for consistency. You should always try to have
consistent configuration across the network. This way the troubleshooting will be much easier.
As you find out, only subnet 10.0.44.x/24 is permitted in access list 22 on GW1. You should have similar
configuration on GW2.
To test the hypothesis, delete the incorrect access list and configure a new standard access list with the same
access list number. The access list should permit connections from subnet 10.0.44.0/24. To verify that the
access list is correct, use the show access-list 22 command. Now you can try the SSH connectivity from the
PC4.
After correcting the access list, you are ready to test the SSH connectivity again. As you can see, you are
now able to connect to the GW2 router from the PC4. This confirms that the solution was correctly applied
and you can close the ticket.
Note At this point you should inform the support engineer that the problem is solved. You are now
able to connect to the GW2 router from PC4 via SSH. Save the configuration on GW2.
Summary
Overview
Check
1. Which statement correctly describes the "protect" violation mode? (Source: Debrief of the First
Troubleshooting Lab At TINC Garbage Disposal Ltd.)
A. The interface is "error-disabled" when a security violation occurs.
B. A security violation sends a trap to the network management station.
C. Drops packets with unknown source addresses until you remove a sufficient number of secure MAC
addresses to drop below the maximum value.
D. The Interface clears all dynamic MAC-addresses when a security violation occurs.
2. Where is VLAN information stored? (Source: Debrief of the First Troubleshooting Lab At TINC
Garbage Disposal Ltd.)
A. Only in vlan.dat file.
B. In vlan.dat file or in the case of VTP transparent mode in the running and startup configuration.
C. Only in RAM and is not restored after reboot.
D. Only in running and startup configuration.
3. Match the adjacency state in which a BGP router can be stuck with the description of the state's cause.
(Source: Debrief of the First Troubleshooting Lab At TINC Garbage Disposal Ltd.)
connect
A. Router found a route to the neighbor and has completed the
three-way TCP handshake.
open
B. Router didn't receive agreement on parameters of
confirm
establishment.
idle
C. Router received agreement on the parameters for
establishing session.
active
D. Router is searching routing table to see whether a route
exists to reach the neighbor.
4. Match the issues with the adjacency states in which neighbors could be stuck. (Source: Debrief of the
Second Troubleshooting Lab At TINC Garbage Disposal Ltd.)
No neighbor
A. MTU mismatch between the routers.
Exstart/Exchange
B. OSPF authentication is not enabled on both sides.
state
Down state
C. Hello parameters mismatch.
Init state
D. Neighbor is manually configured and access list is
blocking OSPF packets.
5. What does version 1.99 indicate in the show ip ssh output? (Source: Debrief of the Second
A. only the SSH version 1 is enabled
B. only the SSH version 2 is enabled, but the key size is 512 bits
C. the version 2 is enabled, but the server also supports version 1 for backward compatibility
D. Cisco proprietary SSH version is enabled
6. Which multicast address must be allowed in the inbound access list when using HSRP? (Choose two.)
(Source: Debrief of the Second Troubleshooting Lab At TINC Garbage Disposal Ltd.)
A. 224.0.0.2 when using HSRP version 1
B. 224.0.0.10 when using HSRP version 1
C. 224.0.0.2 when using HSRP version 2
D. 224.0.0.10 when using HSRP version 2
7. Connect the commands with the appropriate description. (Source: Debrief of the Third Troubleshooting
Lab At TINC Garbage Disposal Ltd.)
show ip ospf
A. Displays more information on a specific route,
database
including the routing source, metric, last update, and
other.
show ip eigrp
B. Displays the detail link state advertisement
topology
information of a router.
show ip route
C. Displays alternative routes and DUAL algorithm
states.
show ip route
D. Sets the load-distribution method among the ports in
network mask
the bundle.
8. Two routers, R1 and R2, are configured as part of VRRP group 1. They were incorrectly configured
with different virtual addresses. R1’s virtual address is configured to 10.0.1.1. R2’s virtual address is
configured to 10.0.1.2. What are the MAC addresses the routers are sending in gratuitous ARP
messages? (Source: Debrief of the Third Troubleshooting Lab At TINC Garbage Disposal Ltd.)
A. R1 – 0000.5e00.0101, R2 – 0000.5e00.0102
B. R1 – 0000.5e00.0101, R2 – 0000.5e00.0101
C. each router sends its own MAC address
D. R1 sends its own MAC address, R2 – 0000.5e00.0101
9. Match the commands with the appropriate description. (Source: Debrief of the Third Troubleshooting
Lab At TINC Garbage Disposal Ltd.)
port-channel load-
A. sets the load-distribution method among the
balance
ports in the bundle
no channel-protocol
B. remove an interface from the EtherChannel
LACP
group 5
no interface port-
C. removes the port channel 5
channel 5
no channel-group 5
D. disables LACP on an interface
10. In the output of the show glbp detail command, in the Forwarder 2 section, what information is given
in the field Client selection count? (Source: Debrief of the Fourth Troubleshooting Lab At TINC
Garbage Disposal Ltd.)
R2# show glbp detail
Ethernet1/0.11 - Group 1
Forwarder 2
State is Active
3 state changes, last state change 00:14:48
MAC address is 0007.b400.0102 (default)
Owner ID is aabb.cc01.b801
Redirection enabled
Preemption enabled, min delay 30 sec
Active is local, weighting 100
Client selection count: 1
A. The number of ARP replies sent with virtual MAC of the Forwarder 2 in response to ARP requests.
B. The number of packets that were passed to the virtual MAC address of the Forwarder 2.
C. The number of clients in the segment that are using the virtual MAC of the Forwarder 2.
11. Bellow you have a partial output of the DHCP snooping configuration. Which of the DHCP messages
are allowed on the interface FastEthernet 0/2. (Choose two answers.) (Source: Debrief of the Fourth
!
ip dhcp snooping
ip dhcp snooping vlan 10
!
interface FastEthernet0/1
switchport mode access
switchport access vlan 10
ip dhcp snooping trust
!
interface FastEthernet0/2
switchport mode access
switchport access vlan 10
!
A. DHCP DISCOVER
B. DHCP OFFER
C. DHCP REQUEST
D. DHCP ACK
12. What is the correct syntax used to apply an access list to the line vty? (Source: Debrief of the Fourth
A. ip access-group
B. access-group
C. ip access-class
D. access-class
Answer Key
1 C
2 B
3
A. Router found a route to the neighbor and has completed the three-way TCP handshake. — connect
B. Router received agreement on the parameters for establishing session. — open confirm
C. Router is searching routing table to see whether a route exists to reach the neighbor. — idle
D. Router didn't receive agreement on parameters of establishment. — active
4
A. Neighbor is manually configured and access list is blocking OSPF packets. — No neighbor
B. MTU mismatch between the routers. — Exstart/Exchange state
C. Hello parameters mismatch. — Down state
D. OSPF authentication is not enabled on both sides. — Init state
5 C
6 A, D
7
A. Displays the detail link state advertisement information of a router. — show ip ospf database
B. Displays alternative routes and DUAL algorithm states. — show ip eigrp topology
C. Displays more information on a specific route, including the routing source, metric, last update, and
other. — show ip route
D. Sets the load-distribution method among the ports in the bundle. — show ip route network mask
8 B
9
A. sets the load-distribution method among the ports in the bundle — port-channel load-balance
B. disables LACP on an interface — no channel-protocol LACP
C. removes the port channel 5 — no interface port-channel 5
D. remove an interface from the EtherChannel group 5 — no channel-group 5
10 A
11 A, C
12 D
PILE Forensic Accounting
Ltd.
Introduction
You work for SECHNIK Network Ltd. as a network engineer. PILE Forensic Accounting Ltd. is a customer
In this module you will be faced with five challenge labs. Each lab has multiple troubleshooting tickets that
• Solve troubleshooting tasks for the first challenge lab at PILE Forensic Accounting Ltd.
• Solve troubleshooting tasks for the second challenge lab at PILE Forensic Accounting Ltd.
• Solve troubleshooting tasks for the third challenge lab at PILE Forensic Accounting Ltd.
• Solve troubleshooting tasks for the fourth challenge lab at PILE Forensic Accounting Ltd.
• Solve troubleshooting tasks for the fifth challenge lab at PILE Forensic Accounting Ltd.
• Describe how you solved fifth challenge lab
Troubleshooting at PILE
Forensic Accounting Ltd.
Overview
This lesson serves as a debrief for the fourth troubleshooting lab at the PILE Forensic Accounting Ltd.
A possible troubleshooting approach is used and described in the debrief. Keep in mind that there are many

• Describe how you solved the problem with the lost connectivity for remote Branch office.
• Describe common issues with EIGRP adjacencies
• Describe how you fixed the network so ISP2 serves as a functional backup again.
You work for SECHNIK Networking Ltd. and PILE Forensic Accounting Ltd. is your company's customer.
Your colleague Peter did some improvements on the customer network over the weekend.
On Monday, Carrie (uses PC1), a customer engineer calls. She's furious and reports the following problems
that need to be resolved immediately:
 Carrie got a call from the Branch office that they are cut away from the headquarters and thus from the
Internet.
 Carrie noticed that if ISP1 fails, the company is cut from the Internet. This is not acceptable—ISP2
should serve as a backup.
Note You can find customer's network documentation in the Job Aids. Be careful,
company's documentation might not be accurate or complete.
Note If you need to test Internet connectivity, use IP address 209.165.200.129.
Note Since you are troubleshooting during customer's maintenance window you have
permission to do failovers and similar network-disruptive test.
Note Since you are located in the headquarters, you cannot directly access the console of
a branch device. To access remote devices, you can use telnet as described in the Job
Aids section.
Example Troubleshooting Flow: Branch Without
Internet Connectivity
Clients from the Branch office are cut off from the headquarters and thus from the Internet.
First you should try to reproduce the reported problem yourself. Since you are not at the Branch office, you
do not have the luxury of a direct access to their network equipment and computers, but you can still verify
Internet connectivity problem using remote access. Telnet to the branch router BR and try pinging a public
IP from there.
Verifying the telnet command output you can see that connection cannot be established from the computer
PC1 to the branch router BR. This is strange, since you know that Carrie, a customer engineer, uses PC1 to
remote access network devices. Until your colleague Peter changed something on the network, internal
routing worked just fine.
You should try reaching the router BR from a directly connected router HQ2.
Because you were able to telnet to BR, you can safely conclude that there is no problem with the WAN
connection. Since router BR can be accessed only from the directly connected devices you should first
consider checking for any routing problems.
Verify Internet connectivity problem with an extended ping tool, where you can configure the source IP
address of sent ICMP packets. Using 10.0.30.7 source IP address you confirm that no host from network
10.0.30.0/24 can reach Layer 3 switch DSW in central office or the given public IP address.
Hosts from the central office have not reported any Internet connectivity problems.
to address this problem.
When one node in the IP network is reachable only from its layer 3 neighbors and not from any other part of
the network, the most probable reason is a routing problem. You should start verifying the internal routing.
Since no one from the central office complained about broken Internet connectivity your troubleshooting
should include a comparison of PC1‘s and PC4‘s routing towards HQ1. This troubleshooting method is
called „spot-the-difference“.
DSW is a layer 3 switch and is a central point of the company‘s network. Therefore it is an excellent choice
for the starting point in troubleshooting the routing operation. You can see that DSW has no routing
information for the destination 10.0.30.0/24 network that resides in the branch unit. On the contrary there is
an entry for the 10.0.10.0/24 network, where nonproblematic PC1 resides. You have found the root problem
of the reported issue, since it is obvious that no returning IP packets will ever be routed back to the
originating branch‘s 10.0.30.0/24 network.
Now you should investigate why DSW has no information about location of the 10.0.30.0/24 network.
Check routing information one hop closer to the remote branch, which is router HQ2.
HQ2 is a border router located in the central office and is an entry point for remote branch, which is
connected via WAN link. According to Job Aids the routing information about 10.0.30.0/24 network should
come from BR, the HQ2‘s EIGRP neighbor. There are several possibilities why this is not so, but you
should start checking if BR is indeed HQ2‘s EIGRP neighbor.
You should use the powerful show ip protocols command to verify as much information as possible at one
place. After checking EIGRP status on router HQ2 you can conclude that the EIGRP configuration correctly
includes all relevant interfaces in the EIGRP process. Router HQ2 therefore sends EIGRP packets towards
router BR, but nevertheless relationship with router BR is not established.
Information Gathering and Proposing a Hypothesis
You should issue the debug eigrp packets command which displays all EIGRP messages sent and received
on the router HQ2. The debug output shows you the reason why EIGRP adjacency is not formed. Router
BR is not sending EIGRP hello packets and since there is no problem with the connection between routers,
the most probable reason is EIGRP misconfiguration on router BR.
Router BR is not sending EIGRP packets towards router HQ2.
As you can see the EIGRP configuration has a mistake in one of its network statement. Regarding HQ2-BR
link, the defined network should include IP address of the related interface. Because the network mask is
defined with all its 32 bits, the correct network statement is network 10.3.0.8/32. Current configuration
excludes interface from the EIGRP routing process, therefore the BR router will not try to establish or
accept any EIGRP relationship on that interface.
You should first remove the incorrect network statement from EIGRP configuration and then add the correct
one. The evidence that you were right comes quickly in a form of a notification that a new EIGRP
adjacency was formed.
The output of the show ip route 10.0.30.0 command tells you that even the central router DSW now has the
correct routing information for the 10.30.0.0/24 network.
Verify Internet connectivity problem with an extended ping tool, where you can configure source IP of the
sent ICMP packets. Using 10.0.30.7 source IP address you confirm that hosts from 10.0.30.0/24 network
can now reach DSW Layer 3 switch in central office and given public IP address.
Note You are now ready to conclude. Make sure to copy the new configuration to the startup-
config file on router BR where the configuration was modified. You also need to document
the changes and update the documentation. You then inform the support engineer that the
Troubleshooting EIGRP Adjacency
EIGRP neighbor relationships can go wrong due to many different reasons. Most of the problems are related
directly to EIGRP misconfiguration, others to less obvious link and addressing problems.
For example EIGRP will not form any neighbor relationships with neighbors in different autonomous
systems. Same goes for K constant values which must be the same on all routers. In EIGRP's metric
calculation, the default K values are set so that only the bandwidth and delay metrics of the interface are
used. Network administrators might want other interface factors to determine the EIGRP metric, such as
load and reliability. When the K values are changed all routers must be updated with the same values.
EIGRP process takes into account all router‘s interfaces that are defined by the network network-number
[network-mask] command. Misconfiguration can result in unwanted passive interfaces and the lack of
prefixes in EIGRP updates. Explicit passive-interface command, placed on the wrong interface by mistake,
has a similar effect as it suppresses all incoming and outgoing routing updates and hello messages.
To configure EIGRP authentication, the keys used in the authentication process have to be configured and
attached to an interface along with MD5 as the mode of authentication. Any interface that has authentication
configured on it will not form neighbor relationships out that interface unless the neighbor passes the
authentication process.
In order to become EIGRP neighbors, routers must ensure an undisturbed exchange of EIGRP messages.
That is not possible if Layer 2 link becomes unidirectional, router does not support multicast or security
filtering is dropping EIGRP packets.
EIGRP will not establish neighbor relationships if the neighbors are not in the same subnet. Sometimes the
reason can be as trivial as misconfiguration of the IP address on the interfaces. For example, 192.168.0.1
255.255.255.252 subnet might be mistyped as 192.168.0.11 255.255.255.252, which causes EIGRP to
complain about the neighbor not being on a common subnet.
You can use a powerful show ip protocols command that allows you to verify configured AS number, K
values, networks, and neighbors in just one set of command output.
To ensure EIGRP adjacencies are in place use the show ip eigrp neighbors command. The show ip eigrp
interfaces command is useful for a quick view of which local interfaces on the device are actually running
EIGRP. In case relevant interfaces are missing from the output, check if affected interface is enabled and
properly referenced in an EIGRP network command.
The debug eigrp packets command can be used to see EIGRP packets as they are sent or received. You can
control the debug with keywords to limit the packet types that are displayed. The debug ip eigrp command
is similar to the debug eigrp packets command, but it focuses more on events that occur with EIGRP, with
an emphasis on the contents of routing updates and events that occur as a result of these updates.
Example Troubleshooting Flow: ISP2 Not Serving
as a Backup
When ISP1 failed to provide Internet connection, failover to the ISP2 was not successful.
You can verify the reported problem with the simulation of ISP1 failure. You should do this only with
customer approval and only at times when impact of loosing Internet connectivity is low. To initiate failover
procedure you can shutdown the interface Ethernet 0/1 on the HQ1 router using the shutdown command.
to address this problem.
Once again you have a situation where part of the network is working as expected and the other, which is
almost the same, does not. Using the „spot-the-difference“ troubleshooting method you should be able to
find the problem in the exterior routing of company‘s network. Compare the configurations and statuses of
BGP routing for connections to both ISPs.
Your next step should be investigating the differences of BGP routing and both BGP peers relationships on
HQ1.
The output of the show ip protocols command shows you that although two BGP neighbors are defined,
only BGP peer from ISP1 has successfully formed the adjacency. This explains why Internet connectivity
failed when router HQ1 was cut off from ISP1. You have found the root cause of the problem, now you
should explore why neighbor relationship with 209.165.201.5 BGP peer is not formed.
When router HQ1 discovers the failure of the ISP1, it removes all routes learned from the BGP neighbor
located at ISP1. Since nothing comes from the BGP neighbor located at ISP2, Internet traffic is forwarded to
the Null0 interface and dropped.
You can see that both BGP peers are configured correctly. IP addresses and AS numbers match those found
in the documentation. 209.165.201.5 BGP peer is in „Active“ state, which means that for some reason the
TCP connection between peers cannot be established. You should check if 209.165.201.5 BGP peer is
reachable at all.
Although the 209.165.201.5 BGP peer is directly connected, it does not respond to a ping. You can be sure
that the Ethernet 0/2 interface is up and operational as otherwise this routing entry would not be present in
the routing table. You have a suspicion that something is filtering the traffic. You should check if any access
list is applied on interfaces towards ISPs.
209.165.201.5 BGP peer is correctly configured according to the documentation, is in the „Active“ state and
is directly connected to the router HQ1. With these facts an access list applied on the interface seems the
most probable source of the problem. Your hypothesis is that 209.165.201.5 BGP peer cannot form
adjacency due to misconfigured access-list 100.
You can find 4 rules in the access-list 100 that define traffic which is allowed to pass into the router. This
access-list has no permit any rule and therefore the default explicit deny any rule comes into effect at the
end of the list.
Since none of the seen rules allows ICMP traffic you now understand that pinging BGP peer was not
successful; access-list blocked all returning ICMP packets.
You can see numerous matches for the rule number 10. That proves that access list 100 is correctly
configured for 209.165.200.5 BGP peer since each occurrence (match) signifies the traffic was permitted
into the router. On the contrary, no match for the rule number 30 means that all incoming BGP traffic from
the 209.165.201.5 BGP peer matched only the last deny all rule and was therefore dropped. Your
hypothesis is confirmed as you can spot that in rules 30 and 40 source and destination IP addresses are
switched by mistake.
You should correct rules 30 and 40.
You should first remove the incorrect rules 30 and 40 and add the correct ones. The evidence that you were
right comes quickly in a form of a notification that a new BGP adjacency was formed.
Verify Internet connectivity problem with an extended ping tool, where you can configure source IP address
of the sent ICMP packets. Using 10.0.30.7 source IP address you confirm that hosts from 10.0.30.0/24
network can now reach DSW Layer 3 switch in central office and given public IP address.
You can verify this solution with the simulation of ISP1 failure. You should not do this without customer
approval and only during maintenance period. To initiate failover procedure, you can shutdown the Ethernet
0/1 interface on the HQ1 router using the shutdown command.
As you can see the failover to ISP2 was successful. Do not forget to revert the change on HQ1 router.
Note You are now ready to conclude. Make sure to copy the new configuration to the
startup-config file on HQ1 router where the configuration was modified. You also have
to document the changes and update the documentation. You then inform the support
engineer that the problem was solved and close the ticket.
Summary
Ltd.
Overview
This lesson serves as a debrief for the second troubleshooting lab at PILE Forensic Accounting Ltd..
Example troubleshooting flows are provided, however keep in mind that there are multiple ways to
approach troubleshooting problems.

• Describe how you solve Internet connectivity
• Troubleshooting BGP filtering
• Describe what is a BGP transit area and what challenges it brings
• Troubleshooting BGP
• Describe how you solved NTP synchronization issue
• Describe NTP troubleshooting issues
• Describe how you solved of PC3 not being able to access the branch router remotely
A guy called that introduced himself as an employee at PILE Forensic Accounting Ltd. He is reporting that
he is unable to use telnet to access branch router via his PC—PC3. He is kindly asking you to fix this issue.
Right after you hang up the phone, Carrie, customer engineer at PILE Forensic Accounting Ltd. calls in
panic:
 None of the company users have Internet connectivity. However she can ping Internet server at
209.165.201.129 from HQ1 router. This is obviously the most pressing issue you need to resolve.
 Carrie also noticed that HQ1 is not synchronized to the NTP server that she has configured as
preferred—209.165.201.193. 209.165.201.193 should be the primary NTP server, 209.165.201.225 and
209.165.201.129 should serve as backups.
Note If you need to test the Internet connectivity, use the 209.165.200.129 IP address.
the branch device. To access remote devices, you can use telnet as described in the
Job Aids section.
Example Troubleshooting Flow: PC3 Unable To
Remotely Access The Branch Router
A guy called you and introduced himself as the employee at PILE Forensic Accounting Ltd. He told you
that he does not have telnet connectivity to the branch router from PC3. You have decided to check the
configuration on the branch router BR.
You checked the telnet connectivity from PC1 to see if there is a telnet issue on the BR branch router. As
you can see, you were able to connect to the branch router. You should check why PC3 is not able to
connect.
Since you were able to connect you assume that something is blocking the access to the BR branch router
from PC3. Therefore you checked the configuration on the vty line. You can notice that access list is applied
on the vty line. Your next step is to check access list 10.
You have checked the access list 10. You can notice that access list 10 is blocking the 10.0.20.0/24 subnet
in which PC3 is located. You can also see the remark that tells you that users in the VLAN 20 are not
allowed to connect to the router. Since PC3 is located in the VLAN 20, you should reject the request.
Wrapping up the Troubleshooting Process
You have seen that VLAN 20 is not allowed to have management access to the branch router. Networks that
are allowed to connect to network devices are usually specified in the security policy. Every organization
should have security policy which is the core security document. The security policy is used in the
organization to define the behaviour of the employees from the security point of view. The part of the
security policy should also define the procedure to grant management access to administrators. The official
channel and people that are responsible for granting access must be defined in the procedure. The person
who is responsible to implement new privileges to the administrator should get the approval through official
channel, otherwise the request must be rejected.
You should always stick to defined procedures since attackers often use social engineering to gain
unauthorized access. Attackers use social engineering to somehow trick people to give away the access
information even if that is now allowed by the security policy. Any decision against security policy could
lead to security breach, which could have major impact on the organization.
Since request for management access to the branch router of the PILE Forensic Accounting Ltd. was done
through a phone call, which is not an official channel to authorize the access, you should reject the request.
You should also inform the guy who calls you that the request was rejected and that he needs official
approval that should come through official authorization channel. You should report to customer engineer
about the incident.
When implementing security for the management access you should follow some of the best practices:
 Allow access to network devices only from the management networks.
 Use centralized AAA servers with RADIUS or TACACS+ protocols.
 Every administrator should have its own account.
 Define different access authorization levels.
 Periodically audit administrators.
 Implement role separations.
 Define password policy.
Example Troubleshooting Flow: No Internet
Connectivity
Carrie, customer engineer at PILE Forensic Accounting Ltd. informs you that the company does not have
the Internet connectivity. You were also called by some guy that introduced himself as an employee of the
PILE Forensic Accounting Ltd. and he asked you to fix the telnet access to the branch router. Since the lack
of Internet connectivity is a much bigger issue, you start troubleshooting that problem first.
First verify if the problem exists. You can test Internet connectivity from the PC1 with the ping
209.165.200.129 command. As you can notice from the output the ping is not successful. When you try the
connectivity test from the border router HQ1, the ping is successful. You can confirm the problem as it was
described by Carrie. You can begin troubleshooting the problem.
Note This is one of the possible troubleshooting plans. There are many different
approaches to this problem.
You were able to reproduce the problem and you found out that the issue is present.
When you are dealing with the lack of Internet connectivity a good approach to use is "follow-the-path".
You can start checking connectivity from PC1 to default gateway.
First check which IP address is assigned to PC1 and what is the default gateway of the PC1. Use show ip
interface brief and show ip route commands. As you can see from the outputs the IP address is 10.0.10.2
and the default gateway is 10.0.10.1.
Next step in the "follow-the-path" approach is to check connectivity to default gateway.
Test the connectivity from PC1 to default gateway. As you can see from the output, ping is successful. You
have connectivity to default gateway from PC1. The next step is to check routing on the distribution
network switch.
At this point you can conclude that there is connectivity from PC1 to the DSW distribution switch, but there
is no connectivity from PC1 to the Internet.
Next step is to check connectivity to the Internet from the distribution switch. As you can see there is no
connectivity from the distribution switch either.
You also verify the routing table on the DSW with the show ip route command. You can see from the
output that there is a default route pointing to 10.1.0.7 IP address. Default route usually points out of the
organization, therefore you can assume that IP address belongs to the edge router. You can also see that the
default route was received by the EIGRP protocol. You should move your troubleshooting process to the
edge router.
You should check interfaces on the HQ1 edge router with the show ip interface brief command. You can
notice that Ethernet 0/0 interface has the 10.1.0.7 IP address assigned. This is the IP address that was the
next hop destination for default route on the DSW distribution switch. You also notice that there are two
interfaces with public IP addresses, Ethernet 0/1 and Ethernet 0/2.
You have tested connectivity from PC1 to the HQ1 edge router. As you can see the ping is successful.
Therefore you can conclude that internal routing is working as expected.
During the troubleshooting you found out that PC1 has no connectivity to the Internet, but it has
connectivity to the HQ1 router. You have also tested the connectivity from the DSW distribution switch and
you have discovered that there is no connectivity to the Internet either. The switch has received the default
route through EIGRP routing protocol from the HQ1. Based on that information you can conclude that
internal routing is working. You also tested the connectivity from the HQ1 edge router and you found out
that HQ1 can access the hosts on the Internet.
The issue is most probably located on the HQ1 edge router. Therefore your next step in the troubleshooting
process is to investigate configuration on the HQ1 edge router.
Since you already found out that HQ1 edge router has Internet connectivity you should check the route
details on the edge router using the show ip route 209.165.200.129 command. As you can see from the
output the route was received by the BGP routing protocol. The BGP routing protocol is used to exchange
routing information between autonomous systems. It is typically used by organizations to provide high
availability through multihoming and to advertise its provider independent address space to the Internet.
Since the source of the route you use for connectivity tests is the BGP protocol your further action is to
check the status of the BGP protocol on the edge router.
You have checked the documentation and you can see that the HQ1 edge router has BGP peering
established with two ISPs. Next step is to verify the status of the BGP peering on the HQ1 router.
To check the status of the BGP protocol on the router use the show ip bgp summary command. As you
can see from the output, peering with both neighbors is established. Peering status is shown in the last
column. If the last column is a number, peering with the neighbor is established, otherwise the peering is not
established. The next step is to check BGP table on the HQ1 router.
Check the BGP table on the HQ1 router with the show ip bgp command. BGP table shows that all prefixes
received by the neighbors with next hops, metric, local preference, weight, and path. You can also see all
prefixes advertised by the local peer to its neighbors. The next hop of these prefixes is 0.0.0.0. As you can
see from the output, the HQ1 router advertises the prefix 209.165.200.248/29 to its neighbors.
You check details of the 209.165.200.248/29 prefix, which is the provider independent prefix. You use
show ip bgp 209.165.200.248/29 command. You can notice from the output that although the prefix is in
the BGP table it is not advertised to any peer. The reason for the lack of connectivity is most probably that
hosts on the Internet do not have route to the public address space of the organization. You should
investigate why prefix is not being advertised to the BGP neighbors.
Finding the Cause of the Problem
To find out why the prefix is not being advertised you should check BGP configuration on the HQ1 router,
using the show running-config | section bgp command. As you can see there is a "RouteOut" distribute list
implemented in the configuration of both neighbors. To verify if this is the cause of the problem you should
check the "RouteOut" access list.
Analyzing Information and Proposing a Hypothesis
You should check the "RouteOut" access list that is applied as the distribute list for both BGP peers. As you
can see from the output, the only line in the access list is preventing the prefix from being advertised.
Eventually everything is being filtered out because implicit deny is located at the end of the access list.
Therefore you should change the access list to allow 209.165.200.248/29 prefix.
To test the hypothesis you should first reconfigure the "RouteOut" access list. This access list should permit
the 209.165.200.248/29 prefix.
Second, BGP will not apply the configuration until neighbor peering is reestablished. Therefore you should
reset BGP peering with the clear ip bgp * command. This command will reset all BGP sessions and delete
all BGP routes from the routing table. Connectivity will be lost until BGP session is reestablished. BGP is
slow protocol so this will not happen instantly. You should use this command during a maintenance
window.
After BGP session is reestablished you should check if prefix is being advertised to the neighbors.
After peering is reestablished, you check the status of the prefix. As you can see the prefix is being
advertised to two peers. You can assume that Internet connectivity is being restored. You should verify the
connectivity from the HQ1 router and PC1.
Verify the Solution
As you can see from the output the Internet connectivity is restored. You are now able to access the Internet
from PC1.
Note At this point you should save the configuration on the HQ1 router and inform support
engineer that problem was solved. Internet connectivity was restored. Document the
changes and close the ticket.
BGP Filtering
BGP router can receive a large number of routing updates. You may need to apply filtering to optimize BGP
and routing tables. Filtering may be applied in inbound or outbound direction.
There are several methods to apply filtering on Cisco routers.
The simplest way to filter BGP routes is to use a distribute list. Distribute list can be used with standard or
extended access lists. With standard access list you can only control network portion, with extended access
list you can also control network mask length. To apply distribute list to the neighbor use the neighbor ip-
address distribute-list access-list-number {in | out} command.
Similarly to distribute lists, prefix lists could be used to filter BGP routing updates. Use the ip prefix-list
list-name {deny | permit} network/length command to specify the prefix list. Several entries can be added
to the same prefix list. To apply prefix list to the neighbor use the neighbor ip-address prefix-list prefix-
list-name {in | out} command. You cannot use distribute list and prefix list at the same time.
Route maps are a much powerful tool to manipulate the BGP routing updates. Besides filtering you can also
set several BGP attributes at the same time. To control updates using route maps, you must first configure a
route map. Networks in the route maps can be referenced with access or prefix list. To apply route map on
the neighbor use the neighbor ip-address route-map name {in | out} command.
Filter list is a special type of BGP filtering. It is based on the AS path attributes in BGP updates. To
configure filtering with filter lists, you must configure AS path access list and apply this list to the neighbor.
Use the ip as-path access-list access-list-number {permit | deny} as-regexp command to configure the AS
path access list and the neighbor ip-address filter-list access-list-number {in | out} command to apply the
AS path list to the neighbor.
You can use several methods at the same time. For the inbound updates the order in which the methods are
applied is:
 Route-map
 Filter-list
 Prefix-list/distribute-list
For the outbound updates the order is:

 Prefix-list/distribute-list
 Filter-list
 Route-map
BGP Transit Area
Typical enterprise environment is using BGP to advertise its own prefixes to the BGP neighbors. By default
eBGP advertises all prefixes received by specific BGP peer to all other BGP peers. If no filtering is
configured all prefixes received by one ISP will be advertised to the other ISP. Therefore there is a danger
that the enterprise AS will become a transit area for traffic between ISP1 and ISP2. It is strongly
recommended to apply outbound filtering to prevent such situation. Filtering can be applied using distribute-
lists/prefix-lists, filter-lists, route-maps. The example shows typical configuration on the enterprise BGP
routers to announce only own prefixes to the ISPs using filter lists.
It is also recommended that you configure outbound filter on your devices that have BGP enabled. You do
not want to advertise internal networks. In most cases, even you do not have an outbound filter, ISP and
networks beyond will not have connectivity to your internal network since ISPs normally filter these.
However, to be safe, it makes sense for you to also configure outbound filtering of BGP updates.
Troubleshooting BGP
To display BGP table use the show ip bgp command. You can observe all prefixes received and advertised
by BGP router.
The show ip bgp summary command can be used to display the status of the BGP connections. All
configured neighbors are displayed in the output. The last column displays the status of the BGP
connection. If a number is displayed, you know that connection is up, any other state indicates failure or
connection establishment phase.
To see received and advertised routes use the show ip bgp neighbors ip-address command with the routes
or advertised-routes keywords.
The debug ip bgp updates command can be used to observe updates on the BGP router. To see significant
events use the debug ip bgp events command.
Example Troubleshooting Flow: HQ1 Does Not
Synchronize With the Primary NTP Server
Carrie also informed you that she configured three NTP servers on the router. The NTP server
209.165.201.193 should be the primary, 209.165.201.225 and 209.165.201.129 should serve as backups.
She told you that the router does not synchronize with the primary NTP server.
To verify the problem, you use the show ntp status command. You can notice that the clock is
synchronized but the reference is NTP server with the 209.165.201.225 IP address. Carrie told you that she
configured this server as a backup. Therefore you can conclude that the problem is confirmed and you can
start troubleshooting the issue.
Note This is one of the possible troubleshooting plans. There are many different
approaches to this problem.
Based on the information that is provided by the support engineer you can conclude that there is one
primary NTP server with the 209.165.201.193 IP address and two backup NTP servers with
209.165.201.225 and 209.165.201.129 IP addresses. You have also checked the NTP synchronization status
and you can confirm that the router is now synchronized with a backup NTP server.
You should find the solution for the issue and inform the support engineer about it.
First step in the troubleshooting process is to check the NTP configuration on the HQ1 router. There are
three NTP servers configured and the one with 209.165.201.193 IP address is configured as primary with
the ntp server 209.165.201.193 prefer command. You can confirm that the router is configured as you
were told by the support engineer.
To check the status of the NTP associations use the show ntp associations command. First two characters
of the line in the command's output indicate the status of the association with the particular NTP server. As
you can see from the output, the asterisk indicates the server that is currently used to synchronize the clock,
while plus symbol indicates the candidate server. In the case of the 209.165.201.193 NTP server there is no
symbol at the beginning of the line. This means that the server is configured, but there is no association. As
there is no association with the NTP server you should check the network connectivity to it.
After you discovered that there is no association with the 209.165.201.193 NTP server, you decide to check
the network connectivity to the server. You can see that there is no problem with the connectivity.
After confirming that there is an issue with the primary NTP server association you have started to
troubleshooting the NTP problem. You have discovered that the NTP configuration on the HQ1 router is
correct. The router has associations with the backup NTP servers. You have also tested the connectivity to
the primary NTP server and found out that routing is working as expected.
Verify the NTP packets with the debug ntp packet command. As you can see from the output NTP packets
are sent to the backup NTP servers and there are also responses to these packets. On the other hand, there
are packets sent to the primary NTP server, but there is no response. You can assume that the NTP packets
are blocked somewhere in the communication between the HQ1 router and primary NTP server. As you
already know that NTP is using UDP port 123.
Since you are assuming that something is blocking the NTP packets to the primary NTP server, you should
check if there are any access lists configured on the interfaces. You already know from the previous
troubleshooting ticket that HQ1 router has two uplinks to two different service providers. You check the
access list configuration on the interfaces connected to the IPSs. As you can see there is the access list 100
configured on both interfaces. The next step is to check the access list 100.
As you can see from the output, the UDP communication is allowed from the backup NTP servers, but the
rule is missing to permit UDP for the primary NTP server. You can conclude that support engineer has
configured preferred NTP server, but she forgot to allow NTP communication from this server to the edge
router. Therefore your next step is to reconfigure access list 100 to permit UDP from the primary NTP
server on the Ethernet 0/1 and Ethernet 0/2 interfaces.
You have discovered that access list that is configured on the Ethernet 0/1 and Ethernet 0/2 interfaces does
not permit NTP packets from the primary NTP server. Therefore you should reconfigure the access list 100
to permit UDP packets from the 209.165.201.193 IP address. You should insert the new entry with the other
two NTP server entries, for easier troubleshooting in the future.
In order to force clock re-synchronization you can use clock read command. NTP is a slow protocol so it
can take quite a while to re-synchronize.
NTP is a slow protocol, so you should wait for a while to see the response packets. You can debug the NTP
packets again to see if there is a response from the primary NTP server.
To verify if the solution is appropriate you check the NTP status and NTP associations on the HQ1 router.
As you can see from the output the clock on the router is synchronized with the 209.165.201.193 NTP
server. The router also has association with the primary NTP server as indicated in the output of the show
ntp associations command. The association with the primary NTP server is shown with the asterisk at the
beginning of the line.
You can conclude that the problem is solved and you are ready to close the trouble ticket.
Note Since you have concluded that the problem was solved you can close the trouble
ticket. You should save the configuration on HQ1 router, inform the support engineer
that the problem was solved and document any changes that were made.
Troubleshooting NTP
NTP is widely used to synchronize clock on the network devices. NTP is a client-server protocol where
NTP servers use an accurate time source. Clients use NTP protocol to communicate with servers in order to
obtain correct time information.
There are several issues that could affect clock accuracy on network devices. Some of these are:
 NTP uses UDP port 123. For NTP to work correctly this port should be permited in the inbound access
lists.
 It is recommended to use NTP packet authentication. NTP uses MD5 authentication. Both server and
client must know the authentication key, otherwise synchronization will fail. If you suspect
authentication mismatch you should check running configuration for the correct keys.
 NTP uses UTC time to synchronize clock. To have the correct local time on the Cisco network device
you should configure time zone with the clock timezone zone hours-offset [minutes-offset] command.
To configure summer time, you should use the clock summer-time zone recurring command.
 If NTP server is not accessible, device will fail to synchronize clock. You should configure primary
NTP server with the ntp server ip-address prefer command and several backup servers without the
prefer keyword.
 If NTP server is misconfigured, the clock information could be wrong or synchronization will fail. You
should check regularly if clock is accurate on network devices.
 If NTP server loses clock synchronization this will consequently affect the network device. You should
check regularly for the clock accuracy on NTP servers.
 If network device is working under high CPU utilization it may fail to process NTP packets and
consequently, the clock will fail to synchronize.
 If offset is high between the client and the server, the time it takes to synchronize the clock may be very
long or synchronization may fail. You should manually set the clock on the router close to the correct
local time with the clock set hh:mm:ss day month year command.
 NTP uses the concept of a stratum to describe how far away a device is from an authoritative time
source. The number is between 1 and 15. If NTP client is synchronizing on NTP server with stratum 15,
it will fail because the stratum 16 is invalid.
To display the status of the NTP on a Cisco device, you should use show ntp status. The command output
will display the synchronization status, the stratum, the reference NTP server and some other NTP values.
In order to check the status of the configured NTP servers you should use show ntp associations. Output
will show all configured servers, which server is used for synchronization and candidate servers. You can
also see some of the other detail values used for clock synchronization.
Two important debug commands could be used to troubleshoot NTP-related issues. These are debug ntp
packets and debug ntp events. The first one will display the messages that are received or sent by the
router and the other will display all the events related to NTP.
Summary
Ltd.
Overview
This lesson serves as a debrief for the third troubleshooting lab at the PILE Forensic Accounting Ltd.
• Describe how you have solved Internet connectivity issue for PC3 segment
• Describe recommended practices to prepare for a disaster recovery in your network
• Describe inter-VLAN routing issues and how to troubleshoot them
• Describe how you solved the Internet connectivity issue on PC4
• Describe DNS-related issues and how to troubleshoot them
• Describe the challenges of remote management access
You work for SECHNIK Networking Ltd. and PILE Forensic Accounting Ltd. is your company’s customer.
Over at PILE Forensic Accounting Ltd. they had a flood that destroyed DSW, ASW1, and ASW2. Carrie,
customer engineer, replaced devices, and restored connectivity for critical PCs – users in VLAN 10.
However there are some issues that she just cannot get her head around and she is asking for your help:
 PC3 does not have Internet access. For example Carrie can ping 209.165.201.129 from PC1, but not
from PC3.
 PC4, over at the branch office, cannot access the Internet. Specifically, user is trying to access
http://www.cisco.com.
Note You can find customer's network documentation in Job Aids. Be careful, company's
documentation might not be accurate or complete.
Note If you need to test the Internet connectivity, use the 209.165.200.129 IP address.
a branch device. To access remote devices, you can use telnet as described in the Job
Aids section.
Example Troubleshooting Flow: Connectivity
Issue After Disaster
PC1 is able to connect to the Internet. PC3, on the other hand, does not have Internet access.
First, you verify the problem. You ping the Internet address from PC3. There is no response, the problem is
confirmed and you start troubleshooting.
When connectivity problem exists in a segment within a network, it is a good practice to check whether the
problem is also present in other segments. The troubleshooting technique is called „swapping the
components“. There is no need for a physical swap if you can access the same sort of devices in the segment
under verification. This helps in estimating the scope of the problem – if it is present in other segments also,
the cause is most probably somewhere upstream in the network, i.e. it is not local to the segment.
Using ping tool you check the Internet connectivity from PC1 and PC2. Ping is successful. You conclude
that connectivity problem is local to the PC3 segment. To continue you choose the bottom-up approach.
You will rely on network documentation in this troubleshooting exercise, since this is the only source of
information on how the network should be configured.
Sometimes the lines between stages of troubleshooting are blurry. In this case we are calling this step
verification, but it could indeed be information gathering after you formed your troubleshooting plan.
Your troubleshooting will aim at solving the problem customer has stated in the ticket. However, since there
was a disaster, your other goal is to restore network functionality as it was before the disaster.
Examining the interface operational status, by using the show ip interface brief command, you see that
lower layer’s operational statuses are „up-up“ which means there are no physical problems (cabling,
network card, etc) and no media access related issues.
You do notice that the interface has an IP address from the VLAN 20 segment, as indicated in the
documentation.
However, the IP address information was configured manually. NVRAM is the value of the method field in
the output. It means that the source of configuration is a file in NVRAM. According to the documentation
IP address should be assigned by a DHCP server. That prompts you to check the default gateway
configuration, which moves your troubleshooting path from Layer 2 upwards. The output of the show ip
route command shows no default gateway is configured.
Having no default gateway configuration can explain the lack of Internet connectivity, and lack of
connectivity to any remote network.
Your first hypothesis is that the lack of gateway configuration is the cause of the issue.
You configure PC3 to acquire IP address via DHCP, as documentation suggests it should.
From the log output you notice that PC3 successfully acquired the address via DHCP. You confirm this by
looking at the interface operational status. But, the IP address assigned is not in the VLAN 20 segment as it
should be, according to the documentation.
You check Internet connectivity and it is successful. Nevertheless, this is not the correct network
configuration and you continue troubleshooting.
The DHCP process completed correctly, therefore you dismiss the network and upper layer connectivity
issues as cause for the ticket problem.
You conclude with the "bottom-up" troubleshooting approach and decide to use the "follow-the-path"
method and check the configuration of the ASW2 switch.
Connecting to ASW2 switch, immediately discloses an awkward configuration. The switch name is ASW1,
equal to another access switch in the network. Also, there is a message indicating that there is a duplicate IP
address for VLAN 99.
When you check which VLANs are configured, using show vlan brief, the output shows VLANs 1, 10, and
99 with two interfaces in VLAN 10. There is no VLAN 20.
You understand that DHCP messages coming from PC3 were understood by DSW as coming from VLAN
10.
Your assumption is that the wrong configuration was applied to the ASW2 switch - the configuration of
ASW1.
Your new hypothesis is that wrong interface VLAN assignment is causing the wrong DHCP address
assignment.
You decide to add the VLAN 20 and assign the proper interface to it. In order to check which interface PC3
connects to, you search for PC3’s MAC address in the MAC address-table. The PC3 connects to Ethernet
0/1 port.
Besides adding VLAN 20 and assigning Ethernet 0/1 interface to it, you need to remove VLAN 10, rename
the switch, and reconfigure the IP address for management interface VLAN 99. You can look-up in the Job
Aids that VLAN99 ip address on ASW2 should be 10.0.99.7/24.
Since you do not have backup configuration files of ASW2, the only thing you can do is to look at network
documentation and configure the device in accordance to the network documentation. However, network
documentation might not be correct and complete! You may end up in a situation, where even though you
configured device accordingly to network documentation, you still have no connectivity!
To verify that changes were applied you check the VLAN configuration on ASW2, with show vlan brief.
You check the upstream ASW2 Ethernet 0/0 interface. It should be configured as a trunk and should allow
VLAN 20 on it as well. Therefore you issue the show interface trunk command and verify that indeed it is
a trunk interface, and both VLAN 20 and VLAN 99 are allowed on it.
If your hypothesis was correct, the new DHCP assignment should now work correctly. You proceed by
initiating a new DHCP address assignment procedure on PC3.
After forcing PC3 to reacquire IP address via DHowever, there is no IP address assigned to PC3.
The "follow the path" troubleshooting continues. Next on the path is DSW distribution switch.
You look into the DHCP configuration on DSW by issuing the show running-config | section dhcp
command.
It shows that DSW is configured as DHCP server for both VLAN 10 and VLAN 20. This is in accordance
with the documentation. The configuration is correct.
You note down that DHCP server on DSW is also configured to supply the DNS server information. This is
of interest because the other problem in the ticket concerns the lack of connectivity to
http://www.cisco.com. You note down that the DNS server for the network is available at 209.165.201.209.
Next, you start debugging DHCP messages on DSW.
DHCP assignment procedure is initiated by the client. It sends DHCP discover message to the broadcast
address to reach DHCP servers in the network. In order to verify the message exchange between PC3 and
DSW you issue the debug ip dhcp server packet command. Output confirms that messages are arriving
from PC3 but there are no response messages sent by DSW.
Next you check the IP interface configuration on DSW. In order for DHCP to operate, at least one of the
devices interfaces must be configured with an IP address from the DHCP pool. Next you will check the
interface configuration on DSW.
To get an overview of IP interfaces configured on the switch you issue the show ip interface brief
command. There is a VLAN 20 interface. Its IP address 10.20.0.1 is incorrect; it does not belong to
10.0.20.1/24 subnet defined for the DHCP pool.
Your hypothesis is, that wrongly configured VLAN 20 interface is preventing the normal operation of
DHCP. Additionally, this means that Inter-VLAN routing is misconfigured.
To verify your hypothesis you change the IP address of VLAN 20 interface to 10.0.20.1, as stated in the
documentation. First you remove the wrong address and then reconfigure it with the correct IP.
Since debugging is still on, the debug messages confirm that DHCP messages are leaving DSW now.
To verify that PC3 is assigned the correct IP address and default gateway, you continue by inspecting PC3’s
configuration.
On PC3 there should be a log message confirming that DHCP assignment completed successfully. You
could verify it by checking the interface configuration and operational status. The Ethernet 0/0 interface has
an IP from the 10.0.20.0/24 subnet and default gateway set to 10.0.20.1.
To complete the verification process you check Internet connectivity. Ping 209.165.201.129 is successful.
Internet connectivity is restored and devices are configured in accordance with documentation.
Note You are now ready to conclude. You want to make sure that changes are saved. Make
sure to copy the new configuration to the startup-config on all devices where the
configuration was modified – ASW2, DSW, and PC3. You also document the changes
and update the documentation. You then inform the support engineer that the
Disaster Recovery
Disasters are inevitable but mostly unpredictable and they vary in type and magnitude. Effects of disasters
range from small interruptions to total operations shutdown for days or months. In case of such incidents,
the operations in the disaster recovery plan should be assumed. The goal is to restore full network
functionality in an efficient way.
Efficient recovery from disaster relies on the quality (or existence) of your documented disaster recovery
plan and availability of all elements required by the recovery plan.
Disaster recovery happens in the following sequential phases:
1 Activation Phase—the disaster effects are assessed and announced.
2 Execution Phase—the actual procedures to recover each of the disaster affected entities are executed.
3 Reconstitution Phase—the original system is restored and execution phase procedures are stopped.
In case of device failures you will need to replace devices as part of the execution phase. In order to
complete the replacement you will need the following items:
 Replacement hardware.
 The current software version for a device.
 The current configuration for a device.
 The tools to transfer the software and configuration to the new device, even if the network is
unavailable.
 Licenses (if applicable).
 Knowledge of the procedures to install software, configurations, and licenses.
Missing any of these items will severely affect the time it takes to restore normal network conditions.
To ensure that you have all required elements available when you need them, follow these guidelines:
 During network design, build redundancy into the network at critical points and ensure that a single
device or link failure can never cause your whole network to go down.
 Crete disaster recovery plan document.
 Document your network:
− Network drawings: Diagrams of the physical and logical structure of the network.
− Connection documentation: A document, spreadsheet, or database listing all relevant physical
connections, such as patches, connections to service providers, and power circuits.
− Equipment lists: A document, spreadsheet, or database listing all devices, part numbers, serial
numbers, installed software versions, and (if applicable) licenses for the software.
− IP address administration: A document, spreadsheet, or database that lists the subnetting scheme
and all IP addresses that are in use.
− Configurations: A set of all current device configurations or even an archive that contains all
previous configurations.
− Design documentation: A document describing the motivation behind certain implementation
choices.
 Incorporate actions into your regular maintenance cycle which ensure that you always have the
necessary elements available when disaster strikes.
− Ensure the availability of the tools to access new devices.
− Have a Backup infrastructure in place.
− The backup infrastructure should be placed locally and in disaster recovery facilities in a location
away from the area that is affected by the disaster.
 Create backups of configurations.
− After any change, you should create backups, copying the configuration file to NVRAM on the
device as well as to a network server.
− Configuration archiving is a helpful feature, enabling you to create archive of configurations.
 You set up the archive by entering the archive command in global configuration mode.
 You can archive copies of the configuration:
− Manually: by issuing the archive config command, or
− automatically: by adding the write-memory option or by specifying the
time-period option.
 Verify the presence of the archived configuration files with the show archive command.
− Use configuration replace feature when restoring the device to its last archived configuration.
Troubleshooting Inter-VLAN Routing
Inter-VLAN routing is the capability to route traffic between VLANs.

If a switch supports multiple VLANs but has no Layer 3 capability to route packets between those VLANs,
the switch must be connected to a device that has this capability. That device could be a multilayer switch or
an external router.
Inter-VLAN routing can be ensured using:

 SVIs on a multilayer switch.
 The router-on-a-stick configuration on an external router.
When using a router-on-a-stick solution you create trunk connection between a switch and a router. Once
the trunk is created, you use subinterfaces on the router to perform inter-VLAN routing.
Use the following verification commands to troubleshoot the routing issues on both devices: show ip route,
show ip protocols and show ip routing-protocol interface.
To provide Layer 3 switching between VLANs connected to a switch, SVIs need to be configured. A SVI is
a Layer 3 virtual interface for a specific VLAN and SVIs must be created and configured with IP addresses
for each VLAN. By default, a SVI is created for the default VLAN (VLAN1) to permit remote switch
administration. Additional SVIs must be explicitly created. When you are using an SVI, the SVI interface
has to be operational. An SVI is considered to be up as long as at least one of the associated VLAN ports is
up.
To create SVI you use the interface vlan vlan-id command. To perform routing between VLANs, you must
enable routing globally with the ip routing command.
To connect a multilayer switch to other L3 devices you can use two methods:
 SVI method: you add another VLAN and use it between the multilayer switch and the port connected to
the external router. The physical port connected to the router is then assigned to that VLAN.
 “Routed port” method: you change the physical interface connected to the outward router into a Layer 3
interface by using the no switchport command at the interface level. The IP address and subnet mask
can be configured directly on the physical interface itself.
In order to verify inter-VLAN routing on multilayer switches:

 Verify that VLANs are configured on the switch and multilayer switch and that the ports are assigned to
them. Check both access ports and trunk ports. VLANs have to be in the active state. Use show vlan
[brief] to examine VLAN database or show vlan vlan-id to verify information about a particular
VLAN.
 Verify that trunking is configured properly and that the native VLAN matches on both sides of the
trunk. Use the show interfaces trunk command.
 Verify that the host's default gateway points to the corresponding SVI interface IP address and that the
subnet masks match.
 Verify SVI interface status on a multilayer switch by issuing the show ip interface brief command.
 Verify that IP routing is enabled by checking the running configuration for the ip routing statement.
To provide inter-VLAN routing using the “router-on-a-stick” configuration you configure a router interface
to operate as a trunk link and connect it to a switch port configured in the trunk mode. The router performs
the inter-VLAN routing by accepting VLAN tagged traffic on the trunk interface. It routes internally
between VLANs using subinterfaces. A router interface must be configured with a subinterface for each
VLAN for which it will perform routing. Each subinterface on the physical link must then be configured
with the same trunk encapsulation protocol using the encapsulation dot1q vlan vlan-id command.
In order to verify inter-VLAN routing on routers:
 Verify that VLANs are configured on the switch and router and that the ports are assigned to them.
Check both access ports and trunk ports. VLANs have to be in the active state. Use show vlan [brief] to
examine VLAN database or show vlan vlan-id to verify information about particular VLAN.
 Verify that trunking is configured properly on the switch. Use the show interfaces trunk command.
 Verify that the host's default gateway points to the corresponding subinterface’s IP address and that the
subnet masks match.
 Verify subinterfaces with the show ip interface interface.subinterface command.
 Verify the subinterfaces status and configuration by issuing the show ip interface brief command on
the router:
− If the interface status is administratively down, issue the no shutdown command in the subinterface
mode. If it is not possible to turn on subinterfaces issue the no shutdown command on the interface.
Example Troubleshooting Flow: Connectivity
Issue When Using Domain Names
PC4 cannot access the Internet. Specifically, user is trying to access http://www.cisco.com.
BR router and PC4 are part of the branch office. You can access both devices only remotely. The available
management option is telnet connection from PC1. Note to remember: PC4 is router, simulating a PC. At
best, you would have remote desktop connection to it, if any. Usually, you will not have a direct access to
the remote device, but you will be able to access it‘s first hop. Then you can use the extended ping feature
from the next-hop device – sourcing the ping on first hop from the LAN interface that the end-device
connects to.
To verify the problem you need to access PC4 and you need to know its IP address in order to establish the
telnet connection. According to the documentation PC4’s IP address is assigned from the range
10.0.30.0/24, by the DHCP server on BR router.
In order to learn the IP address assigned to PC4 you first use telnet to access BR router. You issue telnet
10.0.30.7 command on PC1.
Once you have access to BR, in the privileged mode you issue the show ip dhcp binding command. It
shows you all the IP addresses assigned by the DHCP server. From the output you assume that PC4 is
assigned the 10.0.30.2 address, since according to the network documentation PC4 is the only end-device
connected to BR router.
Using telnet 10.0.30.2 you access PC4. Once the connection is established, you issue ping cisco.com. The
attempt is unsuccessful. This confirms that the problem exists and you begin troubleshooting.
If you cannot access PC4, but can access BR, you could alternatively issue ping cisco.com source 10.0.30.7
command from BR router.
As you are able to establish the telnet connection from VLAN 10 you rule out the possibilities of physical
layer or first-hop IP connectivity issue on PC4. You decide to start the troubleshooting by verifying Internet
connectivity from PC4 using an IP address and not a domain name.
From the PC4 console output you noticed an error message indicating that the name could not be resolved.
This creates reasonable doubt over the existence of Internet connectivity problem. Instead it points to the
problems with domain name resolution. To test your assumption you enter ping 209.165.201.129. The ping
is successful.
Since you can reach Internet destination host by its IP address you rule out any IP connectivity issue along
the path from PC4 to the Internet. The problem lies in name resolution. PC4 acquires its network
configuration via DHCP, so you check DHCP server configuration on BR router for DNS server
information.
The only DHCP pool on BR, named VLAN30POOL, is configured to provide network address along with
the default gateway address. It does not provide DNS server IP addresses.
Analyzing Information
Based on the gathered facts you are ready to make your hypothesis.
PC4 does not know DNS server’s address and consequently cannot resolve domain names to IP addresses.
This creates connectivity problems when destination hosts are specified with their domain names.
In order to provide DNS configuration to PC4 you can choose either to modify the DHCP server
configuration on BR router or you can configure the name server manually on PC4.
It is more efficient to complement DHCP configuration by adding DNS server information. It is not a
problem to add DNS information manually on a couple of devices. However, in networks with large number
of hosts, such task would be inefficient and prone to mistakes. This is why you decide to make changes to
DHCP configuration on BR.
From DHCP server configurations on DSW switch you know that DNS server IP address is
209.165.201.209.
Before you make changes, make sure that DNS server is actually reachable from PC4. Issue ping
209.165.201.209 command. It is successful and you can proceed.
To add DNS server’s IP address to DHCP configuration use dns-server 209.165.201.209 on BR router.
After configuring DHCP server you need to update the configuration on PC4.
You choose to renew the DHCP lease. You need to know the type and number of the interface that PC4 is
using to connect to BR router. You look for the information using show ip interface command on PC4.
From the output you see that there is only Ethernet 0/0 interface is operational.
For the manual configuration, you would have used ip name-server 209.165.201.209 on PC4.
Now you issue the renew dhcp ethernet 0/0 command. This command ensures that any changes to the
DHCP server configuration are supplied to the client. After performing renewing, you will need to
disconnect from the telnet session using "Ctrl+Shitf+6" keyboard key combination and then pressing "X" on
your keyboard. Alternatively, you can use ip dhcp client request dns- -nameserver command to request
only the DNS server IP address from the DHCP server. PC4 now has DNS server’s IP address and can
contact it in order to resolve domain names.
Note There is also another way to force PC4 to acquire IP address, restarting the device.
You can also update the configuration by reloading the device, but this would disrupt the network operation.
You try to avoid it and apply it only when necessary, coordinating to minimize the negative impacts of the
disruption.
In order to verify that the connectivity exists when the host address is given in the form of a domain name,
you issue ping cisco.com command. It is successful.
Note You are now ready to conclude. You want to make sure that changes are saved. Make sure
to copy the new configuration to the startup-config file command on all devices where the
configuration was modified. You also document the changes and update the documentation.
You then inform the support engineer that the problem was solved and close the ticket.
Troubleshooting DNS
DNS is used to map domain names to IP addresses. Configure DNS server on the router with the ip name-
server ip-address command.
The common DNS issues in the network are:

 DNS server is not configured.
− Typical error message says: % Unrecognized host or address, or protocol not running.
− Check configured DNS servers with show running-config | include name-server.
 The name server is configured on the device, but parameters are not correct.
− Verify connectivity to the DNS server.
− Check the name-server IP address configured and verify it is the correct one.
− Check whether DNS-based hostname-to-address translation is enabled, by using show running-
config | include ip domain-lookup. If no ip domain-lookup is present, the translation is disabled.
This functionality is enabled by default.
− Verify that correct default domain name is configured. Use show hosts to display the default
domain-name.
− Verify also if there is domain list configured. Look for ip domain list in the output of the show
running-config command. The list defines a list of domains, each to be tried in turn to complete
unqualified host names. If there is a domain list, the default domain name is not used. Check
whether the list is overriding the domain name configured, thus excluding it from queries.
 DNS resolution not working, all configuration parameters are correct.
− You cannot resolve names specific to one domain, other names resolve correctly.
 There might be a problem with the DNS for the destination host. Contact the administrator
of the destination device.
− You cannot resolve names within a large number of external domains.
 There may be a problem with the local DNS.
 Confirm that your host is using the right domain server.
− You cannot resolve any names.
 Access lists are blocking DNS messages.
− Check outbound and inbound access lists. DNS messages are sent to UDP or
TCP port 53. This port must be permitted in access-lists.
Remote Device Management Challenges
You can configure the network device via telnet of SSH by manually issuing CLI commands or using more
sophisticated tools like copy or configure replace commands. Both mentioned tools require additional
source file with the changed configuration, which is later applied to the running configuration. Those files
can be located on the device or transferred to the target device via TFTP or preferably FTP, that offers
secure transfer.
Copy command merges the files and preserves all the commands from both the source file and the current
running configuration. The source file therefore does not need to be the complete device‘s configuration. On
the contrary the configure replace command uses a smarter file comparison method and applies the whole
difference to the running configuration. The source file in this case must be the complete device‘s
configuration. Configure replace method uses the Contextual Configuration Diff utility, which works great
for most of the configuration changes, but can fail in certain areas like order-dependent route maps. This
configuration method should therefore be used with caution.
In situations where your configuration change results in a badly misconfigured device, you must roll back
the configuration to a previous known and working state. Assuming that you did not already save the new
configuration as soon as you changed it, the working configuration is still present in the NVRAM. In this
case you have an option to revert the configuration commands to bring the device back to the previous state
or reload the device and thus extend the downtime. Obviously, you will not be able to do anything if the
miscnfiguration also effectively locked you out of the device.
It is a good practice to always set an automatic fallback to the last working state. Automation of the fallback
methods is usually based on a timer or event/condition detector, which must be set before any change is
made. The timeout or some other event triggers an automatic configuration change or a reload of the device.
Until then you can apply changes to the configuration and verify the results. If everything is working as
expected you can cancel the automatic fallback.
 Fallback with the reload command: Before any configuration change is made, issue a reload in
[hh:]mm [text] command or reload at hh:mm [month day | day month] [text] command. Proceed to
configure the device as needed. As long as no configuration changes are saved, the device will revert to
its previous configuration when it reloads. If configuration changes are successful, reload cancel
command will stop the pending reload. If configuration changes cause a loss of connectivity, it will be
restored, when the device automatically reloads. You should use this command with caution and double
check for typos. For example reload in 30 will reload the device in 30 minutes, but reload in 30 will
reload the device immediately with "in 30" as the comment. The same advice applies to "reload
cancel„. In all cases you will be prompted for confirmation, but it's all too easy to hurry past the
confirmation without thinking.
 Fallback with the configure replace command:
Before any configuration change is made, issue a configure replace url time seconds command. The
fallback configuration file is not limited only to startup configuration file, but can be set to any that is
available. Proceed with the configuration change. After a successful verification you must stop the
pending revert action by using configure confirm command. You can update or speed up the timer with
the configure revert {now | timer {minutes | idle minutes}} command. This fallback method saves you
from remote access lockout in a way similar to the reload in command.
Note Fallback action can also be triggered with the use of EEM applets, which can be custom
structured to suit any scenario.
Summary
Fourth Troubleshooting at
Ltd.
Overview
This lesson serves as a debrief for the fourth troubleshooting lab at the PILE Forensic Accounting Ltd.

• Describe how you have solved Internet connectivity issue caused by EIGRP misconfiguration
• Troubleshoot EIGRP named configuration
• Troubleshoot EIGRP stub behavior
• Describe how you have solved management connectivity issues
• Describe different ways to configure default route information on devices
Good news from PILE Forensic Accounting Ltd.! They just acquired 47 branch offices worldwide! Carrie,
customer engineer read online about how to make EIGRP more scalable and ran into the concept of EIGRP
stub router. She decided to reconfigure the single branch router to announce only connected and summary
networks via EIGRP. This way she will have a proof of concept and when other 47 branches need to be
connected, Carrie will have everything in place to basically just copy and paste configurations. At least that
was the plan. Carrie calls you Saturday afternoon asking for your help:
 After EIGRP reconfiguration, branch router lost connectivity to the Internet.
 For some reason Carrie cannot access ASW2 via telnet. She can access all network devices except
ASW2.
Note If you need to test the Internet connectivity, use IP address 209.165.200.129.
branch device. To access remote devices, you can use telnet as described in the Job
Aids section.
Example Troubleshooting Flow: EIGRP
Reconfiguration Issue
From the branch router BR it was possible to access Internet before the EIGRP was reconfigured. Now BR
does not have Internet connectivity.
In order to verify the problem, you have to access the BR router. According to the documentation you can
telnet to it from either PC1 or HQ2 router. You first try from PC1. Telnet connection cannot be established.
Next you try connecting from HQ2. This attempt is successful.
Once on BR, you can check Internet connectivity with ping 209.165.200.129. Ping is not successful. This
confirms there is a connectivity issue. You start troubleshooting.
When there is an Internet connectivity problem it is a good practice to check the connectivity from other
devices in the network. In this way you determine the scope of the problem, i.e. is it local to the device or it
spreads larger network portion.
Let‘s say you decide to verify Internet connectivity from HQ2 router and PC1 computer. Both checks
should be successful.
At this point you are gathering information before laying out a troubleshooting plan! But that is OK.
Sometimes you‘ll need to dig a bit before stepping back and forming a plan of further actions.
Note This is one possible troubleshooting plan. There are many different approaches to solving
this problem.
You decide to start by checking directly the routing configuration on the branch router. You are choosing a
„divide-and-conquer“ approach, starting with the network layer, i.e. routing. Your decision is based on the
fact that there is no lower-layer connectivity issue, since telnet connection was established. Also, the
customer has indicated that they have made changes to EIGRP protocol, after which the connectivity was
lost. Internet connectivity is not an issue at the headquarters site, meaning that routing is operating normally.
You continue the troubleshooting process by checking the EIGRP adjacency between HQ2 and BR routers.
Using the show ip eigrp neighbors command on routers HQ2 and BR you check the established EIGRP
adjacencies. BR does not have any EIGRP adjacencies and HQ2’s only adjacency is to the DSW switch.
From the output you should notice that autonomous system numbers are not the same. According to the
documentation there is only one instance of EIGRP protocol in the network. Its autonomous system number
is 100.
Your hypothesis is that misconfigured EIGRP AS number is causing the issue. In order to confirm this as a
cause, you decide to take a look at EIGRP configuration on BR router.
The command show running-config | section router eigrp gives you an overview of the routing protocol
and its configuration. You notice that customer’s engineer has used the newer way to configure EIGRP,
using named configuration.
The name selected for the EIGRP configuration is PILE_BRANCH. The customer has planned to copy this
configuration to all new 47 branches. The address-family section of the configuration starts up an EIGRP
instance. There is only one EIGRP instance on BR and its autonomous system number is 1 as evident from
the address-family ipv4 unicast autonomous-system 1 statement. HQ2 is configured in the conventional
way, using EIGRP AS number configuration, router eigrp 100. The output confirms that EIGRP AS
numbers are different. AS number configured on BR router is 1, whereas on HQ2 it is 100. With EIGRP AS
numbers must match in order for devices to build an adjacency.
From HQ2 configuration you notice that the customer intended to use the stub routing feature, because there
is the statement eigrp stub connected summary. Stub routing feature should be used on stub routers only,
i.e. on routers connected to the core or a hub, through which core transit traffic should not flow. However,
you presume that new 47 branch routers will all connect via WAN to the headquarter. In this case HQ2 will
not be a stub router.
On the other hand, BR router is a stub router. You also presume that other branches will connect directly to
the headquarters and they will be stub routers also. Stub feature should be configured on BR router and
removed from the HQ2 configuration.
Another interesting point in BR configuration is that it uses network 0.0.0.0 command. This statement
includes all interfaces into EIGRP process. You are now ready to extend your initial hypothesis.
According to the documentation there is only one instance of EIGRP protocol in the network. Its
autonomous system number is 100. Therefore, you decide first to change the autonomous system number
on BR router to 100. Afterwards, you are going to remove stub feature configuration from HQ2 and
configure it on BR router.
To annul the existing EIGRP AS number configuration you enter no address-family ipv4 unicast
autonomous-system 1. This removes all related EIGRP configuration. You have to configure it anew,
basically copying the initial configuration. First you enter command, followed by topology base and
network 0.0.0.0 command. The notification immediately verifies that there was a new adjacency
established. Nevertheless, you verify it by issuing show ip eigrp neighbors on BR. BR now has HQ2 as an
EIGRP neighbor. Next you decide to check Internet connectivity and routing table on BR.
It is very convenient to use network 0.0.0.0 command to include all interfaces in the routing process.
However, be careful. In case static default route is configured using ip route 0.0.0.0 0.0.0.0 interface, where
exit-interface and not the next-hop IP is given, the network 0.0.0.0 command would include the default
route in the EIGRP process. Since BR is a stub router, it should not propagate the default route. Rather, the
default route should be sourced by the headquarters router. Therefore, the more specific network statement
should be used.
You can understand why the customer has chosen this network statement. It would simplify the
configuration on new branch routers as it would allow the configuration to be copied without alterations.
But, better practice would be to include concrete networks under network command, i.e. network 10.3.0.0
0.0.0.255 and network 10.0.30.0 0.0.0.255. This would require that customer alters the configuration before
copying it to new branch routers. You make a note of this, so that you can make recommendation to the
customer in a debrief to the ticket.
After the adjacency is established, you check Internet connectivity from branch router. As you assumed, the
Internet connectivity is not restored. When you look at the BR’s routing table you notice that there is a route
for only one remote network—10.2.0.0/24. This is one of the connected networks on HQ2. There is no
default route, as you see from the “Gateway of last resort is not set” statement at the beginning of the table.
Next you will check whether HQ2 router has information about the default route.
When you inspect the HQ2’s routing table you notice that there is a gateway of last resort via DSW at
10.2.0.8. This route is learned via EIGRP. Another route that is now present in the routing table is for
remote office network 10.0.30.0/24. But, the stub feature configured on HQ2 prevents it from advertising
either of the two routes. As a stub HQ2 advertises only connected and summarized routes to its neighbors.
That is why other routers in the headquarters do not have route information for 10.0.30.0/24 . You confirm
this by checking the DSW’s routing table.
EIGRP AS number change resulted in new adjacency between HQ2 and BR. However, the routing
information is still not exchanged as it should be. Stub feature, which is configured on HQ2, is limiting the
routing information it shares with its peers. It can send information only about connected and summarized
routes. That means it is sending information about 10.2.0.0/24 and 10.3.0.0/24 networks. BR is part of the
letter, so it already knows about it.
It seams that stub feature configuration was misunderstood by the network engineer. Stub feature should be
configured on a router that directly connects a stub network, i.e. the network accessible by only one path –
in the case of your network this is the BR router.
Next you remove the stub feature from HQ2.
You remove the stub feature by issuing the no version of the configuration command eigrp stub. As soon
as you remove it several notifications appear. They inform you that the adjacencies with peers 10.2.0.8 and
10.3.0.8. are reset.
Using do show running-config | section router eigrp command you immediately verify your
configuration. There is no stub feature configured any more. Next you will configure stub feature on BR
router.
Using the stub eigrp command without additional parameter you configure BR as a stub that shares
information about connected and summary routes with all neighbor routers. This means that in case of the
link changes in the headquarters part of the network, BR will not be queried. Again, you are notified about
adjacency reset.
You should now verify the configuration by issuing do show running-config | section router eigrp. Note
that stub feature appears in the configuration with connected and summary keywords.
Note Notice network 0.0.0.0 command on BR router. This command enables EIGRP on all
interfaces. This command is not recommended without first marking all interfaces EIGRP-
passive using passive-interface default command, then enabling individual
interfaces using the no passive-interface interface slot/number command. However
due to customers specifications that configuration on BR router should be universal enough
to be just copied and pasted to other 47 branches, current configuration is adequate.
In BR’s routing table there is a default route as well as other remote destinations, that were not present
before. In the routing table of DSW there is a route to the branch office network 10.0.30.0/24 learned via
EIGRP. You now make the final check by verifying that branch is accessible from the headquarters hosts
and that there is Internet connectivity from the branch.
You issue ping 10.0.30.7, which is the remote office interface on BR router. Ping is successful. This shows
that there is connectivity from headquarters to the branch. From the branch router you check the Internet
connectivity from the remote location by specifying the source of the ICMP messages, using ping
209.165.200.129 source 10.0.30.7.
Internet connectivity is restored. You have solved the problem.
to copy the new configuration to the startup-config file on all devices where the configuration
was modified. You also document the changes and update the documentation. You then
inform the support engineer that the problem was solved and close the ticket.
EIGRP Named Configuration
There are two methods to configure EIGRP on the router:

 EIGRP autonomous system configuration.
 EIGRP named configuration.
When using the AS method you must change several configuration modes to complete the configuration.
Commands are scattered among them. For every address family (IPv4/IPv6) you have to configure EIGRP
separately.
Named configuration unifies all configuration commands and steps. It provides one place to configure all of
EIGRP in the same way. It uses the concept of address families, which can be IPv4, IPv6 or virtual router.
EIGRP parameters are configured per address-family and all address-family configurations are unified
under one named configuration. Within the EIGRP configuration of each address family there are 3
submodes available:
1 Address-family configuration mode–used to configure general EIGRP parameters such as networks
included in the process, default metric, statically define neighbors, etc and also used to access other
submodes.
2 Address-family interface configuration mode–used to configure interface specific EIGRP parameters,
such as bandwidth, intervals, split-horizon, authentication, etc. Interface parameters can be configured
for:
− All interfaces–by configuring a dedicated default interface, whose configuration is applied to all
EIGRP enabled interfaces of that address family.
− A specific interface–by configuring a specified interface.
When using the named configuration method, there is configuration inheritance hierarchy that follows the
rule:
 address-family interface specific configuration applies, which overrides the
 address-family interface default configuration, which overrides the
 address-family factory configuration.
Therefore you must be careful when configuring EIGRP interface default configuration. This configuration
will be applied to all interfaces. If you want different configuration for specific interface you should inherit
this configuration under EIGRP-specific interface subcommand.
3 Address-family topology configuration mode is used to provide options which operate on EIGRP
topology table, such as administrative distance, redistribution, distance, load balancing. The main
routing table is populated from the base topology.
If you do not see a configuration parameter in the address-family configuration you should:
 Make sure you have checked all the address-family submodes looking for the parameter.
 Verify the default values–either those customized by yourself or defaults. For instance, autosummary is
disabled as a default in named configuration.
One of the advantages of the named configuration mode is that verification commands mirror configuration
commands–only show keyword is appended at the beginning.
Troubleshooting EIGRP Stub
Stub feature is one of the convergence improvement mechanisms. Typical topology in which it is used is
hub and spoke, which can be implemented with more than one hub when redundancy is provided. In hub-
and-spoke topology, spokes are usually the remote sites and hubs central sites. At spoke locations one or
more routers can be used to connect to hub(s). Spokes communicate between each other using the hub.
Traffic from hub routers should not use remote routers as transit paths.
When configured as stub EIGRP routers, the remote spokes are configured to share only a subset of routing
information with their neighbors. With default eigrp stub configuration a remote router advertises only its
connected and static routes.
There are several stub types that determine what routing information is advertised. They are:
1 eigrp stub connected
− Remote router advertises only its connected routes.
2 eigrp stub static
− Remote router advertises only its static routes.
3 eigrp stub redistribute
− Remote router advertises only other protocols and autonomous systems routes.
4 eigrp stub summary
− Remote router advertises only summary routes.
5 eigrp stub receive-only
− Remote router does not advertise routes.
When troubleshooting, use the following tools:
 When no routes or incorrect routes are advertised to the neighbors:
− Use: show running-config | section router eigrp to check what stub type is configured on the
device.
 No routes are learned from neighbors:
− Check whether the neighbor is configured as stub and what type of routes it advertises.
− Use show ip eigrp neighbors detail – to display neighbors’ stub status and what routes are
advertised.
 For a more terse output you can include the filter:
− show ip eigrp neighbors detail | include ^[0-
9]|process|Interface|(sec)|Stub
 To monitor EIGRP activities related to stub feature use:
− debug eigrp packet terse - displays all EIGRP packets except hellos.
− debug eigrp packet stub - to display information about the stub status of the peer routers.
Example Troubleshooting Flow: Lack of
Management Access
For some reason Carrie cannot access ASW2 via telnet. She can access all network devices except ASW2.
The customer complained only about establishing connection to ASW2.You try to connect to ASW1,
ASW2, and DSW. Attempts to ASW1 and DSW should be successful. Connecting from PC1 to ASW2
should not be successful.
You have confirmed that the issue exists and start troubleshooting.
You confirmed that there is telnet connectivity issue to ASW2. In order to verify whether it is only telnet or
maybe there is a network connectivity issue, you ping ASW2 from PC1 and ASW1. PC1 is part of a
different network, while ASW1 is in the same network as ASW2. Ping from the local network is successful,
while ping from other network is not.
this problem.
You have ruled out the lower layer connectivity issues along the path, because ASW1 switch was able to
establish network connection with ASW2 and because telnet attempts from PC1 to other devices were
successful.
Since network connectivity problem exists when trying to verify it from networks other than ASW2’s
network, you suspect there is a routing problem along the way. In order to systematically check all the
devices along the path, you choose follow-the-path method.
You start by looking into the routing configuration of the first device along the path, namely switch ASW1.
The first thing you want to verify on ASW1 is whether IP routing is enabled on the device. You choose
show ip route command, which will also give you information about available routes, if IP routing is not
enabled. ASW1 is a Layer 2 device and it has IP default gateway of 10.0.99.1 configured.
To verify that ASW1 is correctly configured you use ping 10.0.99.6 from the HQ2 router. Ping is successful
and confirms that the configuration is correct.
You proceed to DSW switch.
On DSW you examine the routing table with show ip route. It contains routes to 10.0.99.0/24 network, as
well as routes to 10.0.10.0/24 network of PC1. You conclude that DSW is correctly configured.
Next, you check the configuration of ASW2 switch.
ASW2 is a layer 2 device but does not have IP default gateway configured. So traffic going to ASW2 has no
return path. Since ASW1 and ASW2 are devices with very similar roles in customer's network you could
have quickly identified the problem using "spot-the-differences" troubleshooting approach.
To test the hypothesis, you will add default gateway configuration on ASW2 as it is on ASW1.
First you check for the network connectivity. You issue ping 10.0.99.7 from PC1. The ping is successful.
Next you try to establish telnet connection using telnet 10.0.99.7. You are prompted to enter the credentials,
which is the proof that telnet traffic can flow. You enter the credentials and establish the connection.
The problem is solved. You can finalize the trouble ticket.
Providing Default Route On Layer 2 And Multi-
layer Devices
There are three ways you can add default route to layer 2 and multilayer devices. These are:
 using ip route 0.0.0.0 0.0.0.0 {ip-address | interface-type interface-number [ip-address]} command, or
 using ip default-network network-number command, or
 using ip default-gateway network-number command.
The effect of the command would depend on whether routing capability is enabled on the device.
The ip default-network network-number command is used to specify a network which will serve as a
default route. It is used when routing is enabled on the device. For the command to have an effect, the
network-number must be present in the routing table. When this condition is satisfied, a static route entry is
automatically created for the major classful network of the network-number used in the command..
The command ip default-network network-number is a classful command. If you use a classless network
mask for the network-number, the gateway of last resort will not be set and no default route would be added
to the routing table.
The gateway of last resort is set only when the network-number used is a classful network address. Because
of its classful nature, the command is a legacy command. Instead, you should use the ip route 0.0.0.0
0.0.0.0 {ip-address | interface-type interface-number [ip-address]} command to specify the default route.
This command will set the gateway of last resort. If you specify a next-hop IP, the network this IP belongs
to must be in the routing table.
Another way to specify a default path on a layer 2 device is to use the ip default-gateway network-number
command.
It is used to provide the default path on pure layer 2 devices that do not support IP routing. As soon as the
device has IP routing enabled this command has no effect. It would have no effect neither on a router with
IP routing disabled using the no ip routing command.
Summary
Lesson 5: Debrief of the Fifth
Troubleshooting at PILE
Forensic Accounting Ltd.
Overview
This lesson serves as a debrief for the fifth troubleshooting lab at PILE Forensic Accounting Ltd..
Example troubleshooting flows are provided, however keep in mind that there are multiple ways to
approach troubleshooting problems.

• Describe how you solved Internet backup issue
• Troubleshoot BGP route selection
• Describe how you solved telnet access issue
• Secure the management plane in your network
Carrie, customer engineer, was worried about lack of reliability in the company's network. PILE Forensic
Accounting Ltd. bought another router to connect their headquarters to the Internet. They now have two
routers connecting their headquarters to two different service providers. However, right now traffic goes
through HQ1 and if it fails, all Internet connectivity for network's end-devices is lost. Carrie wants you to
help her resolve this issue so the company has a fully-functional Internet redundancy in place. Carrie must
have access to all network devices via telnet, she uses PC1.
Also Carrie noticed that a user from PC3 seems to be accessing the BR router via telnet. This is not
acceptable. Fix this issue in such a way PC3 still has connectivity to the Internet and can ping Branch end
devices, but cannot access any of the routers or switches via telnet or SSH.
Note If you need to test Internet connectivity, use IP address 209.165.200.129.
branch device. To access remote devices, you can use telnet as described in the Job
Aids section.
Example Troubleshooting Flow: Internet Access
Via Router HQ0 Does Not Work
Customer engineer has installed second router HQ0 for high availability. She told you that Internet
connectivity does not work when router HQ1 is not operational.
Customer engineer informed you that Internet connectivity is not working when router HQ1 is not
operational. To verify the problem, you need to disable interface on the router HQ1 to simulate
nonoperational router. Since this would interrupt production traffic, you should only do this in the off-hours.
You should test connectivity to the Internet from the PC1 by issuing the ping to the 209.165.200.129. Ping
is successful. Therefore you should shutdown the interface on the router HQ1 and test connectivity again to
verify if the problem exists.
Disable interface Ethernet 0/2 on the router HQ1 to simulate nonoperational primary Internet router.
Check connectivity from the PC1. As you can see from the output the ping is not successful. The problem is
confirmed and you can start troubleshooting.
problem.
Customer engineer has informed you that the second edge router HQ0 was installed for high availability.
However she discovered that the new router does not provide internet when router HQ1 fails.
Before you begin troubleshooting the issue, you should verify the reported problem. You tested the Internet
connectivity from the PC1. You have discovered that PC1 has Internet connectivity during normal
operation. To simulate non operational router, you have disabled interface on the router HQ1. You have
issued connectivity test from the PC1, and you have discovered that there is no connectivity to the Internet
when router HQ1 fails.
You should find the solution for the issue and inform customer engineer about it. Since you are dealing with
Internet connectivity issue you can use „follow-the-path“ approach. Therefore you begin with testing
connectivity to the default gateway from the PC1.
With the „follow-the-path“ approach you should start with testing connectivity to the default gateway from
the PC1. As you can see, there is connectivity, therefore you should move your troubleshooting process to
the distribution switch.
You check the routing table on the distribution switch and you have discovered that there is a default route
pointing to the IP address 10.1.0.6.
IP address of 10.1.0.6 belongs to router HQ0. To exclude internal routing issue, you should check
connectivity from the PC1 to this IP address.
During the troubleshooting process you have discovered, that PC1 has a default route pointing to the DSW
and DSW has a default route, pointing to HQ0. The default route on DSW is learned via EIGRP protocol
(denoted by „D*EX“ in the routing table). You should test if there is a connectivity from the PC1 to the
HQ0.
As you can see, the ping is successful to both edge routers. Therefore you can conclude that internal routing
is working as expected. In the next step you should check if routers have connectivity to the Internet.
You issue ping command to a host on the Internet. As you can see from the output ping is successful from
both routers.
During the troubleshooting process so far you were using „follow-the-path“ approach. You were able to
discover, that the PC1 has connectivity to the edge router HQ0 via distribution switch DSW.
Based on the information gathered you can conclude that the issue is most probably located on the edge
routers. Therefore you should move your troubleshooting focus there. You decide to check if NAT
configuration is correct.
Since you have two devices that should have almost identical configuration (HQ0 and HQ1), you should
change your troubleshooting method to „spot-the-difference“.
You checked NAT configuration on the edge router HQ0. As you can see from the output all hosts that
match access list 1 are translated to the public IP address 209.165.200.250. The access list allows hosts from
the subnet 10.0.0.0/8, which means that those hosts are translated to the public IP address, when accessing
the Internet.
You repeat the same steps on the edge router HQ1. You can see that hosts from the 10.0.0.0/8 are translated
to the IP address 209.165.200.249. You can conclude that there is an almost identical configuration on both
edge routers.
To check if NAT is performed correctly you shutdown the interface Ethernet 0/0 on the router HQ1. Then
you should ping the IP address 209.165.200.129 and check NAT translations on the router HQ0 using
command show ip nat translations. As you can see from the output the local IP address 10.0.10.3 is
translated to 209.165.200.250.
You can conclude that NAT is working correctly.
You checked NAT configuration on both routers and you discovered that both edge routers have almost
identical NAT configuration. Both are translating the internal hosts from the subnet 10.0.0.0/8 to public IP
address. The router HQ0 is translating to IP address 209.165.200.250 and the router HQ1 is translating to
the IP address 209.165.200.249.
Check if something is blocking packet out of the router HQ0.
Check external interface Ethernet 0/1 on the router HQ0. You can see that access-list 100 and CBAC
firewall is applied to the interface. In the next step check if traffic is permitted out of the interface Ethernet
0/1.
Check access-list 100 and CBAC configuration. As you can conclude from the outputs ICMP traffic and is
permitted and you you should be able to ping the host on the Internet.
During troubleshooting process you have discovered that NAT is working and ICMP traffic should be
permitted. In the next step check if public address space is correctly advertised to the Internet.
You check the BGP configuration on the HQ1. You have found out that the prefix 209.165.200.249/32 is
being configured with network command to be advertised to the BGP peers. This IP address is used as
public IP address in the NAT configuration on the router HQ1. Check BGP configuration on the router
HQ0. You expect similar configuration.
Check BGP configuration on the HQ0. As you can see from the output the command network
209.165.200.250 mask 255.255.255.255 is used to advertise network 209.165.200.250/32 to the BGP peers.
You can conclude that this is the correct configuration.
You check BGP status with show ip bgp summary command on both routers. You can see that edge
routers have each one external BGP peering.
You already discovered that edge routers should advertise IP addresses used for NAT to the BGP peers.
Therefore you check if routes are correctly advertised to the neighbors.
You issue the show ip bgp neighbors 209.165.201.5 advertised-routes to check which routes are
advertised to ISP2. As you can see from the output the route 209.165.200.249/32 is advertised to ISP2. This
is an address that is used to translate hosts from subnet 10.0.0.0/8 to the public IP address space.
Use command show ip bgp neighbors 209.165.200.5 advertised-routes to check which routes are
advertised via external BGP. HQ0 is not advertising routes via BGP.
During current troubleshooting process you have discovered that router HQ0 has a BGP peering with the
ISP1 and HQ1. You also checked the advertised routes and you have discovered that only
209.165.200.249/32 is advertised from the internal AS. Therefore you can assume that connectivity problem
is related to the prefix 209.165.200.250/32 is not being advertised to the Internet. This is an IP address used
to translate hosts from subnet 10.0.0.0/8 to the public address on the router HQ0.
You should check BGP configuration on both routers to discover if your hypothesis is correct.
When routes is configured to be advertised by the network command, the exact same routes must be
installed in the routing table. This is the common reason for routes not being advertised. Therefore you
should check routing table on the router HQ0 for the route 209.165.200.250. Use command show ip route
209.165.200.250. You can see that IP address 209.165.200.250 is matched to the route 209.165.200.248/29
which is the best match for this IP. You can conclude that route 209.165.200.250/32 is missing on the
router. This is the reason that route is not being advertised by BGP on the router HQ0. You also check the
running configuration to confirm your conclusion.
Since the router HQ0 and HQ1 should be configured similarly, you decided to check the route
209.165.200.249/32 in the route table and running configuration on the HQ1 router. As you can see from the
outputs, the best match in the routing table for the IP address 209.165.200.249 is the route
209.165.200.249/32. This is what you have expected. You also see that route is present in the running
configuration. The route is pointing to the Null0 interface. This is virtual interface that discards all packets.
This route does not have any effect on the router and was added to the configuration just for BGP.
Your should add similar route to the router HQ0.
You should add route 209.165.200.250/32 to the router HQ0 and remove incorrect entry. After adding the
route to the configuration you should check if this route is being advertised to the neighbor. As you can see
from the output, the route is now present in the BGP table. You are ready to test the connectivity to confirm
the solution.
BGP is a slow protocol, therefore you cannot expect instant effect. You should wait for couple of minutes to
see the results.
After implementing the route to the HQ0 you should test the connectivity to the Internet from the PC1. As
you can see from the output, PC1 is now able to ping the host on the Internet. To be completely sure that
your solution is proper, you should check if PC1 has Internet connectivity in all scenarios - if both edge
routers are online or if one of them fails.
Verify the Solution
During the testing of hypothesis you already verified that PC1 had connectivity if HQ1 was down. Right
now you confirmed that PC1 has Internet connectivity if both, HQ0 and HQ1, are up. Last step would be to
check if PC1 has Internet connectivity if HQ0 is down.
Note At this point you should save the configuration on the router HQ0 and inform the support
engineer that problem was solved. There is Internet connectivity when one of the router is
not available. Document the changes and close the ticket.
Troubleshooting BGP Route Selection
When several paths for the same destination are received by the BGP router it uses specific set of rules to
choose the best path. Loop-free, synchronized routes (when synchronization is enabled) paths with
reachable next-hop are considered as candidates for the best path. Attributes are evaluated in the following
order:
 Prefer the path with the highest weight.
− Weight is Cisco-specific attribute, local to the router on which it is configured.
 Prefer the path with the highest local preference.
− Local preference is similar attribute to weight but it is used in whole AS. The default value is 100.
 Prefer the local router originated path.
− A locally originated route has a next hop of 0.0.0.0 in the BGP table.
 Prefer the path with the shortest AS path.
− This step can be ignored with the command bgp bestpath as-path ignore.
 Prefer the path with the lowest origin code.
− IGP is lower than EGP, which has lower origin type than INCOMPLETE.
 Prefer the path with the lowest MED.
− By default MED is compared only when first AS in the AS path is the same for both paths. This
behavior can be disabled with the command bgp always-compare-med.
 Prefer eBGP over iBGP paths.
 Prefer the path with the lowest IGP metric to the BGP next hop.
 Prefer oldest route.
 Prefer the path that comes from the BGP router with the lowest BGP router id.
− You can use bgp router-id command to change the BGP router id value.
 Prefer the path that comes from the lowest neighbor address.
− IP address that is used in the BGP neighbor configuration.
The command show ip bgp is used to display the BGP table. In the output you can see networks with next-
hop addresses, MED, local preference, weight, AS path, and origin code. If route is valid it is marked with
asterisk at the beginning of the line. The best path is marked with the symbol >. Multi-Exit Discriminator or
MED is a hint to external neighbors about the preferred path into an AS that has multiple entry points.
To display the details on the specific BGP route you can use command show ip bgp prefix. You are able to
see more details on specific route.
The command show ip bgp neighbors ip-address received-routes will display all routes received from
specific neighbor. In order to display the output of the command the inbound soft reconfiguration must be
enabled. When soft reconfiguration is enabled, router stores all routes received by the neighbor before
policy is applied to the routes. When you change the BGP policy, you can apply it to those stored routes.
This is helpful since BGP session does not need to be restarted.
To see the updates and routes received by the neighbor you should use debug ip bgp updates command.
To restart BGP session with the neighbor, you should use clear ip bgp * command. When this command is
issued all BGP sessions are terminated. When issuing this command, you can temporally remain without the
connection, therefore you should be careful when using this command. To restart only one neighbor, you
can use clear ip bgp ip-address.
Example Troubleshooting Flow: PC3 Is Able to
Telnet to the Router BR
Carrie, the customer engineer, informed you that PC3 is able to telnet to the router BR. Since this is not in
accordance to the security policy she asked you to check and fix the issue.
First you investigate if you can connect from PC3 to BR directly via telnet. Since you cannot the next step is
to investigate configuration of BR router. Maybe PC3 can connect to BR through some other device.
Verification continues since at this moment you are not sure if PC3 can or cannot telnet to BR.
You should check the line vty configuration on the BR router. As you can see from the output, access list 10
is configured on the line vty. This access list permits subnets 10.0.10.0/24 and 10.3.0.0/24, while subnet
10.0.20.0/24 is explicitly denied. Based on the information gathered you can conclude that PC3 is not able
to connect directly, while there is to connect indirectly through some other device. You also found out in
the documentation that the subnet 10.0.10.0/24 belongs to VLAN 10 in which PC1 is. You should check
which devices have the IP address from the subnet 10.3.0.0/24.
You should check the route 10.3.0.0/24. You can see that the network 10.3.0.0/24 is directly connected.
Therefore you should check the ARP table to get IP addresses of the devices that are connected to this
interface. You can see that besides the local IP address 10.3.0.8 one more device is connected and has IP
address 10.3.0.7. Based on the information in the documentation, you can safely assume that this device is
router HQ2. However, you should check the configuration on the HQ2.
You should check the interfaces on the router HQ2 to confirm your assumption. As you can see from the
output, the router HQ2 is connected to the router BR.
Check if there is a telnet connectivity to the router BR, by issuing telnet 10.3.0.8 command. As you can see
you are able to access the router BR2. In the next step you should check the line vty configuration on the
router HQ2.
You should check the line vty configuration on the router HQ2. You can see from the output that access list
10 is implemented on the line vty, which permits subnet 10.0.0.0/16. This subnet also includes subnet
10.0.20.0/24, which is the subnet of the PC3. You can conclude that PC3 is able to access the router HQ2.
Hypothesis Proposal
Based on the information gathered, you can propose the hypothesis. You assume that PC3 is not able to
directly connect to the router BR, but it is able to connect to the router HQ2 and from there to the router BR.
The access list on the router HQ2 should be more strict.
Implementing the Solution
Delete line 10 from the access list 10 and add new line which permits only the subnet 10.0.10.0/24. You
should check if configuration was correctly applied to the router.
Test the telnet connectivity form the PC3. You can see that connection is refused.
Based on this information you can inform the customer engineer that the problem is solved.
Note At this point you should save the configuration on the router HQ2 and inform the support
engineer that the problem was solved. Document the changes and close the ticket.
Securing the Management Plane
Management plane handles the traffic that is sent to the Cisco IOS device and is used for device
management.
Some of the best practices to secure management plane include:
 You should use complex password. It is recommended to have minimum of eight characters for a
password. The minimum password length can be enforced with the command security password min-
length. You should use any alphanumeric character, uppercase and lowercase characters and symbols.
 Users should be authenticated when accessing the device. To provide authentication you should use
AAA services, which offer credentials to be stored locally on the device or on the remote server. AAA
services are enabled with the aaa new-model command. The best practice is to use TACACS+ or
RADIUS remote server (tacacs or radius server command), then configure authentication to use the
local database as a fallback method when remote server is not accessible (aaa authentication login
default group radius local command).
 The goal of RBAC is to provide set of permissions and assign that set to users or group. For example if
there is junior administrator you might limit the permissions. The RBAC can be implemented with
custom privilege levels or parser views. Use command privilege mode {level level | reset} command-
string to set privilege levels for the specific commands. Use parser views to implement even more
sophisticated RBAC. Command to create views is parser view view-name.
 When devices are configured remotely you should always use encrypted management protocols. This
includes SSH or HTTPS. You should use SSH version 2. This could be configured with the command
ip ssh version 2. To enable only SSH use transport input ssh under line vty configuration. When
using GUI to manage a device, use ip http secure-server command to enable HTTPS.
 Event logging provides you visibility into the operation of a Cisco IOS device. Log outputs can be
directed to variety of destinations, including console, vty lines, buffer, SNMP server or syslog server.
Use service timestamps log datetime to include date and time in the log messages.
 Use NTP to synchronize the clocks of network devices with the ntp server command. In this way you
can colerate logs from different devices for troubleshooting.
 It is recommended to use SNMP v3 for monitoring and management of the device. SNMP v3 uses the
concept of a security model and levels. When using SNMP version 2c, you should use a complex
community string and you should also limit the SNMP access to hosts that need that access. This can be
implemented with the access list. Do not use community public for the read access or community
private for read-write access, which are common default communities.
 To help protect a router from accidental or malicious tampering of the IOS or startup configuration,
Cisco offers a resilient configuration feature. This feature maintains a secure copy of the router IOS
image and running configuration. When feature is enabled, it cannot be disabled remotely. To protect
IOS image use secure boot-image command and to protect configuration use secure boot-config.
Summary
Overview
Check
1. How can you prevent the forming of an EIGRP adjacency on a specific segment while still including the
interface’s address in the EIGRP routing updates? (Source: Debrief of the First Troubleshooting At
PILE Forensic Accounting Ltd.)
A. By issuing the proper no network network [mask] command.
B. By issuing the no auto-summary command.
C. By issuing the passive-interface ethernet slot/number command.
D. By issuing the passive-interface default command.
2. What can go wrong with BGP session on Layer 4? (Source: Debrief of the First Troubleshooting At
PILE Forensic Accounting Ltd.)
A. Access lists or firewalls are dropping relevant TCP packets.
B. BGP authentication is failing.
C. Clock is not synchronized between BGP routers.
D. BGP neighbors do not agree on session parameters.
3. What does “Active state” mean in the following command output? (Source: Debrief of the Second
Troubleshooting At PILE Forensic Accounting Ltd.)
Router# show ip bgp summary
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down

State/PfxRcd
192.168.1.1 4 65000 192 184 10 0 0 02:44:11
Active
A. The neighbor is up and BGP is working.

B. The router is trying to establish a BGP session.
C. The router is exchanging updates.
D. Router does not have BGP enabled.
4. Match the filtering technique with the order of processing of the inbound BGP update. (Source: Debrief
of the Second Troubleshooting At PILE Forensic Accounting Ltd.)
route-map
A. Second
filter-list
B. First
prefix-list / distribute-list
C. Third
5. Where is the outbound distribute list applied in the configuration? (Source: Debrief of the Second
A. On the outbound interface.
B. In the global configuration.
C. Under BGP configuration with the neighbor command.
D. None of the above.
6. Which NTP server is used for clock synchronization in the output below? (Source: Debrief of the
Second Troubleshooting At PILE Forensic Accounting Ltd.)
Router# show ntp associations
address ref clock st when poll reach delay offset

disp
~192.165.100.101 .INIT. 16 - 1024 0 0.000 0.000
15937.
*~192.165.100.102 .LOCL. 1 615 1024 377 0.000 0.000
2.036
+~192.165.100.103 .LOCL. 1 509 1024 377 0.000 0.000
2.016
A. 192.165.100.101
B. 192.165.100.102
C. 192.165.100.103
D. 127.127.0.1
7. A DNS query was sent by the 10.0.3.33 host to a DNS server at 8.8.8.8. Which access list line will be
matched when the response arrives? (Source: Debrief of the Third Troubleshooting At PILE Forensic
Accounting Ltd.)
access-list 100 permit udp host 8.8.8.8 eq 53 10.0.3.33 0.0.0.255 eq 53
access-list 100 permit udp any 10.0.3.33 0.0.0.31 eq 53
access-list 100 permit udp any eq 53 10.0.3.3 0.0.0.31
access-list 100 permit udp any 10.0.3.32 0.0.0.31
A. Line 1.
B. Line 2.
C. Line 3.
D. Line 4.
8. You have set up a configuration archive. Its configuration is shown in the figure. What is the purpose of
the write memory option? (Source: Debrief of the Third Troubleshooting At PILE Forensic Accounting
Ltd.)
R1# show running-config | section archive
archive
path tftp://10.1.152.1/R1-config
write-memory
time-period 10080
A. It specifies that the archive should be created in the non-volatile memory at the remote location.
B. It triggers an archive copy of the running configuration to be created any time the running
configuration is copied to NVRAM.
C. It specifies that the new file added to the archive should overwrite the old file added previously.
9. You have set up the configuration archive without options for automatic archiving of the configuration
file. How do you add the file to the archive? (Source: Debrief of the Third Troubleshooting At PILE
Forensic Accounting Ltd.)
A. You enter copy startup-config archive command.
B. You enter archive config command.
C. You enter copy running-config archive command.
D. This is done by default.
10. You access a remote router via its Serial 0/0 interface address using SSH. The first thing you do when
you access the device is to check whether there is a backup of the currently running configuration. You
find an archive, created automatically when issuing the write command. The archive is current. Two
minutes after issuing a reload in 120 command you are cut-off and cannot restore SSH session. When
you think back, you remember that you have changed the IP address of the Serial 0/0 interface just a
moment before the lockout. How can you restore the remote access? (Source: Debrief of the Third
A. You will not be able to access the device until someone with physical access to the device helps you
out.
B. The router will reboot in 118 minutes. Then you will be able to access the device.
C. The router will reboot in 120 minutes.Then you will be able to access the device.
D. Timeout of 120 seconds expired and router rebooted. When it comes up, you will be able to access it
again.
11. Match the EIGRP verification commands with their appropriate description. (Source: Debrief of the
Fourth Troubleshooting At PILE Forensic Accounting Ltd.)
show eigrp plugins
A. Displays a summary of information about
EIGRP address families.
show ip eigrp protocols
B. Reveals information for all address-
families configured.
show eigrp address-family
C. Display information on passive interfaces
ipv6 100 accounting
and a list of network prefixes.
show running-config |
D. Displays a list of network prefixes and
section router eigrp
the related information.
12. You are configuring a router. After you have disabled routing with the no ip routing command, you
enter the ip default-gateway 10.55.47.88 command. What will you see if you do the show ip route?
(Source: Debrief of the Fourth Troubleshooting At PILE Forensic Accounting Ltd.)
A. There will be a gateway of last resort set to 10.55.47.88 and the routing table will be empty.
B. A new static route marked as a candidate for the default route would appear.
C. There would be no change to the routing table, because the command used is not the appropriate one.
D. There would be no change to the routing table, because there is none on the router.
13. What is the default value for the local preference attribute on the Cisco router? (Source: Debrief of the
Fifth Troubleshooting At PILE Forensic Accounting Ltd.)
A. 0
B. 50
C. 100
D. 200
14. The route is configured as shown below. Which network command will import the route to BGP
process? (Source: Debrief of the Fifth Troubleshooting At PILE Forensic Accounting Ltd.)
ip route 10.10.0.0 255.255.0.0 Null 0
A. network 10.0.0.0
B. network 10.10.0.0
C. network 10.0.0.0 mask 255.0.0.0
D. network 10.10.0.0 mask 255.255.0.0
15. Match attribute with the description. (Source: Debrief of the Fifth Troubleshooting At PILE Forensic
Accounting Ltd.)
Local
A. Cisco proprietary attribute.
preference
AS path
B. Attribute to indicate best exit point out of AS.
MED
C. Hint to external neighbors about preferred path into an
AS.
Weight
D. List of AS numbers
Answer Key
1 C
2 A
3 B
4
A. First — route-map
B. Second — filter-list
C. Third — prefix-list / distribute-list
5 C
6 B
7 D
8 B
9 B
10 B
11
A. Displays a summary of information about EIGRP address families. — show eigrp plugins
B. Reveals information for all address-families configured. — show ip eigrp protocols
C. Displays a list of network prefixes and the related information. — show eigrp address-family ipv6
100 accounting
D. Display information on passive interfaces and a list of network prefixes. — show running-config |
section router eigrp
12 A
13 C
14 D
15
A. Attribute to indicate best exit point out of AS. — Local preference
B. List of AS numbers — AS path
C. Hint to external neighbors about preferred path into an AS. — MED
D. Cisco proprietary attribute. — Weight
Bank of POLONA Ltd.
Introduction
You work for SECHNIK Network Ltd. as a network engineer. Bank of POLONA Ltd. is a customer
In this module you will be faced with four challenge labs. Each lab has multiple troubleshooting tickets that
• Solve troubleshooting tasks for the first challenge lab at Bank of POLONA Ltd.
• Solve troubleshooting tasks for the second challenge lab at Bank of POLONA Ltd.
• Solve troubleshooting tasks for the third challenge lab at Bank of POLONA Ltd.
• Solve troubleshooting tasks for the fourth challenge lab at Bank of POLONA Ltd.
Troubleshooting at Bank of
POLONA Ltd.
Overview
This lesson serves as a debrief for the first troubleshooting lab at the Bank of POLONA Ltd.
• Describe how you solved the problem with the lost connectivity for remote Branch office
• Troubleshoot redistribution in IPv4 environment
• Describe how you improved VRRP solution with tracking interface state
• Troubleshooting first hop redundancy protocols
• Describe how you solved the problem with IP SLA
• Troubleshoot IP SLAs

You work for SECHNIK Networking Ltd. and Bank of POLONA Ltd. is your company's customer.
Customer has a headquarters network that connects to three branch offices. They just bought another branch
- Branch 3. Newly acquired branch is configured with EIGRP, the headquarters, and other two branch
offices are configured with OSPF. They are planning to reconfigure EIGRP part of the network sometime
soon.
Customer engineer Tina calls, she has a number of network issues that she needs to resolve but just doesn't
know how. You will need to help her.
 User from the newly acquired branch office cannot access server SRV2. User sits behind PC3.
 Tina noticed that if R1's uplink fails all the traffic from PC0 goes to R1 and then to R2 instead of going
directly to R2. She would like to implement interface tracking, but is not allowed to configure HSRP
per company policy. She would like for you to find and implement a solution using VRRP that
eliminates this situation of suboptimal traffic path.
 Tina is getting reports from users in headquarters, that SRV2 is sometimes inaccessible. To confirm
this, she configured an IP SLA test on HQ router that tests reachability of SRV2 around the clock.
However, Tina complains that IP SLA test will not start.
Note Since you are troubleshooting during the company's maintenance window you have
customer engineer's permission to perform failovers and other intrusive tests that shouldn't
be performed during working hours.
Note You can find customer's network documentation in the Job Aids. Be careful, company's
Note Since you are located in the headquarters, you cannot directly access the console of branch
device. To access remote devices, you can use telnet.
Note If you need to test Internet connectivity, use IP address of 209.165.201.45.
Connectivity
User from the newly acquired branch office cannot access server SRV2. User sits behind PC3.
You were able to telnet to remote router BR3 from central router HQ, which means that Internet connection
between them works just fine. Additionally with the use of BR3‘s private IP address for the remote access,
you also successfully tested the IPsec GRE tunnel between central and branch office.
Unsuccessful ping from BR3 using an IP address from the IP subnet 192.168.3.0/24 confirms that any host
from that subnet cannot reach remote server SRV2, which verifies the issue.
Successful ping from HQ confirms that server SRV2 is operational and reachable from the central office.
If router HQ can reach a private IP address of the router BR3, then IPsec GRE tunnel must be up and
operational. Consequently it is also clear that BR3 has an Internet connectivity. IPsec GRE tunnel, which is
operating over public IP addresses, would otherwise not be established. Server SRV2 can be reached from
the Central office.
Since the Internet connection and IPsec GRE tunnel between offices work as expected, this is obviously an
internal network problem. Router HQ can ping server SRV2, therefore you can conclude that the problem
most probably lies in internal routing or some security filtering. First you should verify that the routing
works using „follow the path“ troubleshooting method. You can check access lists later.
According to Job Aids the internal routing in this Branch office should be done by EIGRP, which would
learn all routes for remote networks via redistribution from OSPF. Therefore it is strange that there are no
EIGRP routing entries in the BR3‘s routing table. When you double check the routing table you notice that
you can see only directly connected networks and a static default route, which is enough for the Internet
connectivity.
You can conclude that the remote Branch office is almost completely cut off from the rest of the company‘s
network, except for the working IPsec GRE tunnel. Having the IPsec GRE tunnel operational at least router
HQ can be reached.
You should now proceed to investigate what is wrong with the EIGRP routing process. Start by checking its
status at router BR3.
You can see that EIGRP process is correctly configured according to Job Aids, but there are no routing
information sources. Router HQ should be BR3‘s EIGRP neighbor, providing BR3 all OSPF routes from
the Central office and all other Branch offices. You can see that the relevant IPsec GRE tunnel is also
included in the EIGRP process and therefore BR3‘s hello packets are sent through the tunnel. You can think
of several reasons why adjacency is not forming, but the most probable are that HQ‘s Hello packets are not
coming through the tunnel or Hello packets have wrong parameters.
You can check EIGRP event logs on BR3 to see if any EIGRP communication has successfully passed
through the tunnel.
That is a surprise. You can see numerous event logs indicating that neighbor relationship between routers is
actually operational. You can quickly double check it with the show ip eigrp neighbors command.
The output proves that indeed you jumped to the conclusion too quickly. The missing routing information
source does not necessary mean that there are no neighbor relationships formed. It just means that the router
BR3 has not received any useful information from any neighbor.
You should move your focus to the router HQ in check the EIGRP status there.
EIGRP neighbor BR3 is sending useful routing information and is therefore listed under sources. You can
also see that routing information from OSPF is set to be redistributed into EIGRP.
You can see that HQ has routing information for all relevant destination networks, including remote
networks learned via EIGRP and OSPF respectively. Connected and OSPF entries found in the routing table
are candidates for a redistribution into EIGRP routing process. You now know that those routes were never
redistributed into EIGRP. If they would, you could see those entries also in BR3‘s routing table.
You should verify the hypothesis with the configuration check.
Testing the Hypothesis and Implementing Solution
Looking at the EIGRP configuration you can notice that although redistribution of OSPF routing
information is set, the redistribution metric was not configured. Without redistribution metric EIGRP cannot
start including external routes into its process. When configuring EIGRP seed metric go with realistic
values. After you add a redistribution metric, verify the results.
The output of the show ip route eigrp tells you that now router BR3 has received additional routing
information via EIGRP. This proves that the routing redistribution from OSPF is now fixed, and with that
also the reachability towards server SRV2. That can be verified using ping tool from router BR3, explicitly
setting the source IP address to 192.168.3.101.
config file on router HQ where the configuration was modified. You also document the
changes and update the documentation. You then inform the support engineer that the
Troubleshooting Redistribution
Misconfigured route redistribution can lead to lost connectivity, suboptimal routing, and even routing loops.
The final results of redistribution are stored in routing tables throughout the network. There you can check
for strange routes and missing prefixes. The wholesome behavior of your network can be verified using the
traceroute command.
When routers are not redistributing external routes as expected, you should first check if redistribute
command is referencing the correct routing process with the accompanying process number and furthermore
that routes are not filtered by any misconfigured distribute list or route map. They can be checked for errors
using show access list and show route-map commands.
Routes redistributed from the injecting protocol must be updated with a metric that is understood by the
receiving protocol. In some cases this seed metric is not globally predefined. Redistribution will not take
place until this metric is in any way set. The default metric can be checked with the show ip protocols
command.
When configuring the route redistribution you should be aware that only routes that actively exist within the
routing table will be redistributed. For example, if a router is being configured to redistribute OSPF into
EIGRP, only those OSPF routes that are actively in the redistributing router’s routing table will be
redistributed into EIGRP. This behavior is the same for all dynamic routing protocols. You can check the
content of the routing table using show ip route command.
Redistribution limits and hides the original information, therefore no IGP can guarantee a loop-free
topology and optimal routing. Loops usually occur when routes that originated in a routing domain are
reinjected back into that domain.
There are 2 major rules that redistribution policy design should follow:
 Router should always prefer internal prefix information over any external information.
 Never redistribute a prefix injected from domain A into B, back to domain A.
Distribution lists, route maps and, the distance command can be used to implement these rules.
Tagging routes provides a robust mechanism to both identify and filter those routes further along in the
routing domain. A route retains its tag as it passes from router to router. Route tags are applied using route
maps.
Router(config)# router eigrp 100
Router(config-router)# redistribute ospf 1 route-map OSPF2EIGRP
Router(config-router)# default-metric 1500 100 255 1 1500
Router(config)# route-map OSPF2EIGRP deny 5
Router(config-route-map)# match tag 33
Router(config)# route-map OSPF2EIGRP permit 15
Router(config-route-map)# set tag 44
Static and connected routes have different administrative distance assigned in cases of network and
redistribute commands. With the redistribute command routing protocols like EIGRP, that can distinguish
internal and external routes, will redistribute static and connected routes as external routes if the
redistribute command is used. If network command is used then routes will be redistributed as internal
routes.
EIGRP:
 Unlike most of the other dynamic routing protocols, EIGRP does not automatically have a default
metric for any learned routes. If the default-metric or a manual metric is not specified, EIGRP will
assume a metric of 0 , and will not advertise the redistributed routes.
 EIGRP will not auto-summarize external routes unless a connected or internal EIGRP route exists in the
routing table from the same major network of the external routes.
 If EIGRP stub router needs to redistribute routes, it has to be explicitly configured to do so using the
eigrp stub redistributed command.
OSPF:
 OSPF uses parameter „subnets“ to distinguish classful and classless behavior. When any protocol is
redistributed into OSPF, if the networks that are being redistributed are subnets, you must define the
subnets keyword under OSPF configuration. If the subnets keyword is not added, OSPF will ignore all
the subnetted routes when generating the external LSA. The situation could also arise when connected
or static routes are being redistributed into or out of OSPF. In that case, the same rule applies: The
subnets keyword must be entered to redistribute subnetted routes.
 Redistribute static subnets command is used to redistribute static routes in OSPF. However static
default route ip route 0.0.0.0 0.0.0.0 is not injected in to OSPF topology database. If you want to
redistribute static route in OSPF, default-information originate always command should be used
instead.
 OSPF stub areas do not redistribute external routes.
 One of the functions of a Type 4 summary LSA is to announce the reachability of an ASBR to the other
areas. This Type 4 LSA is not required if the ASBR exists in the same area. The ASBR doesn't generate
the Type 4 summary LSA if it's not connected to area 0. To generate a summary LSA of Type 3 or Type
4, a router must have a connection into area 0. As a result, the external routes will not be installed in the
network.
BGP:
 When redistributing static and connected routes into BGP, it’s more important than ever to carefully
filter announcements toward external ASes. If you don’t, your transit ISPs, peers, and customers will
receive all your internal routes. These routes are of no interest to them and unnecessary use up
resources.
 When BGP is redistributed into an IGP, only eBGP learned routes get redistributed. The iBGP learned
routes known on the router are not introduced into the IGP in order to prevent routing loops from being
formed. By default, iBGP redistribution into IGP is disabled. Issue the bgp redistribute-internal
command in order to enable redistribution of iBGP routes into IGP. Precautions need to be taken to
redistribute specific routes using route maps into IGP.
To check all relevant information (routing source, route type, AD, metric, tag) about each route installed in
the routing table use show ip route command.
To check relevant configuration of routing protocols, placement of distribution lists and route maps use
show running-config | section router command.
Details and statistics for route maps and distribution lists can be checked with show route-map and show
access list commands.
Show ip protocols command displays relevant information about routing protocols statuses, redistribution,
default metric and in case of OSPF also if subnets are included in the redistribution.
The behavior of your network can be verified using the traceroute command.
Example Troubleshooting Flow: Suboptimal
Routing
Troubleshooting ticket states that if link R1-HQ fails suboptimal routing occurs. A solution for optimal
routing with the interface state tracking is requested. The First hop redundancy protocol should remain
VRRP.
You should first verify that in network‘s normal state, the routing from PC0 goes through intended router
R1.
You can verify the reported problem with the simulation of R1-HQ link failure. Since you are
troubleshooting during the company's maintenance window, you can proceed with the shutdown command
on R1‘s the interface Ethernet 0/1.
Use the traceroute command to see the path towards Internet. Suboptimal routing is present and you should
find a solution to avoid it.
VRRP object tracking provides a way to ensure that the best VRRP router is the virtual router master for the
group by altering VRRP priorities to the status of tracked objects such as the interface or IP route states.
Check the current VRRP status and its configuration, whether any object tracking is already set.
You can see that VRRP process on R1 has the highest priority in the group 1. Since the VRRP process is not
registered to receive any notification from the object tracking process, this priority number will never
change. You can also conclude from the output of the show track command that most likely there was never
an attempt to track any object on router R1.
For R2 to become the virtual router master in case of R1 uplink failure, an automatic priority lowering
mechanism must be implemented in VRRP process of the router R1. VRRP does not have a native
mechanism to accomplish this, but can freely use the IOS object tracking process.
The output of the show vrrp command shows you most of configuration information and the current state
of the protocol. In cases where two or more devices have to work together in synchronization it is a good
practice to use the „spot the difference“ troubleshooting approach. You should note down the priority value
of the virtual backup router, which will be important later in the implementation of the automatic failover.
Proceed by checking the R1‘s VRRP status.
Implement Solution
Register VRRP with the tracking process to be informed of any changes to the line protocol state of
Ethernet interface 0/1. If the line protocol state on Ethernet interface 0/1 goes down, then the priority of the
VRRP group is reduced by 20. Router R2 should in that case have the highest priority and take the role of
the virtual router master in group 1.
Note Tracking object number does not need to match VRRP group number.
Simulate the R1‘s uplink failure and check the R1‘s updated VRRP status. You can see that R1 was now
degraded to the role of a virtual router backup with the priority of 90. Using traceroute command you can
verify that indeed route towards the Internet now avoids R1.
config file on routers HQ and R1 where the configuration was modified. You also document
the changes and update the documentation. You then inform the support engineer that the
Troubleshooting FHRP Tracking
HSRP has a built in ability to track interface, but it only checks the interface line-protocol state. It has
limited use in some situations where the link is UP but highly dysfunctional.
Enhanced Object Tracking provides a more complete alternative to the HSRP tracking mechanism.
You can track either the interface line protocol state or the interface IP routing state. When you track the IP
routing state, these three conditions are required for the object to be up:
 IP routing must be enabled and active on the interface.
 The interface line-protocol state must be up.
 The interface IP address must be known.
If all three of these conditions are not met, the IP routing state is down.
You can configure a tracked list of objects with a Boolean expression, a weight threshold, or a percentage
threshold. A tracked list contains one or more objects. An object must exist before it can be added to the
tracked list.
Cisco IOS IP Service Level Agreements is a network performance measurement and diagnostics tool that
uses active monitoring by generating traffic to measure network performance. Cisco IP SLAs operations
collect real-time metrics that you can use for network troubleshooting, design, and analysis.
IP SLA is not limited to the local information like standard interface line protocol and IP routing state. It
can incorporate several troubleshooting tools and even communication with the remote IP SLA responder.
Local information is very limited and in some cases does not reflects the actual state.
The standby track interface-priority interface configuration command specifies how much to decrement
the hot standby priority when a tracked interface goes down. When the interface comes back up, the priority
is incremented by the same amount.
 When multiple tracked interfaces are down and interface-priority values have been configured, the
configured priority decrements are cumulative. If tracked interfaces that were not configured with
priority values fail, the default decrement is 10, and it is noncumulative.
 When routing is first enabled for the interface, it does not have a complete routing table. If it is
configured to preempt, it becomes the active router, even though it is unable to provide adequate routing
services. To solve this problem, configure a delay time to allow the router to update its routing table.
Example Troubleshooting Flow: IP SLA Does Not
Start
Customer engineer Tina has configured an IP SLA test on HQ router that tests reachability of SRV2 around
the clock. She does not know why the test has not started.
You can see that there is one IP SLA test configured, but is not listed also as active. This confirms that the
reachability test that customer engineer Tina has configured is currently not running. The reported problem
is verified. You can start troubleshooting.
Since you do not have any similar working IP SLA test to at least use the „spot the difference“
troubleshooting approach, you have to dig into information gathered from the configuration and current
status of the IP SLA.
You can see that otherwise IP SLA for testing remote server SRV2 reachability is correctly configured. The
only thing missing is the scheduled start time. Proceed with checking the status of the IP SLA.
You can see that IP SLA 1 has not even started, since it has no statistics and an invalid time to live. You
conclude that that is probably due to missing schedule configuration. You should verify your hypothesis by
checking the running configuration.
Testing the Hypothesis and Implementing Solution
Looking at the running configuration you notice that the ip sla schedule command is missing in the IP SLA
configuration. Do that yourself and again check the IP SLA status.
Both outputs show that IP SLA is now active and is already gathering the required statistics.
config file by issuing the copy running-config startup-config command on router HQ where
the configuration was modified. You also document the changes and update the
documentation. You then inform the support engineer that the problem was solved and close
the ticket.
Troubleshooting IP SLA
To implement Cisco IOS IP Service Level Agreements you need to correctly perform these tasks, which can
be checked with the above listed common IP SLA show and debug commands.
 Enable the Cisco IOS IP SLAs Responder, if appropriate.
 Configure the required Cisco IOS IP SLAs operation type.
 Configure any options available for the specified Cisco IOS IP SLAs operation type.
 Configure threshold conditions, if required.
 Schedule the operation to run, then let the operation run for a period of time to gather statistics.
 Display and interpret the results of the operation using Cisco IOS CLI or an NMS system with SNMP.
Summary
Bank of POLONA Ltd.
Overview
This lesson serves as a debrief for the second troubleshooting lab at the POLONA Ltd.
• Describe how you have solved the issue of missing EIGRP summary-route
• Troubleshoot EIGRP summarization
• Describe how you solved IPv4 and IPv6 Internet connectivity
• Troubleshoot basic RIPng
• Describe how you solved the IPv6 Internet connectivity issue from branch BR3 devices
• Describe ACL troubleshooting
Customer has a headquarters' network that connects to three branch offices. They just recently bought
another branch - Branch 3.
Customer engineer Tina calls, after an upgrade to the network, she has a number of network issues that she
needs help with:
 Even though BR3 is configured to summarize BR3 network, HQ still sees individual entries, instead of
just one for all 172.16.x.x networks.
 PC0 does not have Internet connectivity through IPv6 or IPv4.
 BR3 devices lost IPv6 Internet access.
branch device. To access remote devices, you can use telnet.
Note If you need to test IPv6 Internet connectivity, use IPv6 address of
2001:DB8:D1A5:C92D::1.
Example Troubleshooting Flow: Incorrect EIGRP
Summarization
Although summarization is configured on BR3, the routing table of HQ router contains all individual entries
for BR3 networks.
In order to verify the problem, you check the routing table on the HQ router using the show ip route
command. There should be 20 entries, one for each of the 172.16.0.0/16 BR3 networks. If summarization
had been correctly configured, the routing table would have shown only the summary route.
The problem is confirmed and you can start troubleshooting.
Another way to verify the problem is to check the routing table on the BR3 router using show ip route
command. The first entries are the individual, connected BR3 networks. There is no entry for the summary
route. If the summarization was in effect, the routing table would contain the configured summary-route
with the Null0 as the exit interface.
this problem.
There are individual BR3 networks’ EIGRP entries in the routing table of HQ router, which confirms that
summarization is not in effect and that there are no issues with connectivity at layers above and below the
network layer. You are choosing a „divide-and-conquer“ approach, i.e. you begin troubleshooting at the
network layer, by examining the EIGRP routing.
You start by checking on which interface the summarization was configured.
EIGRP summarization is configured per interface, so you enter show running-config | include
interface|summary on BR3. The summarization is configured on the Ethernet 0/0 interface. You also
notice that the conventional EIGRP, not named EIGRP is used as the configuration method. If named
configuration were used, the summary address would appear as the parameter of address-family interface
configuration.
According to the documentation, the headquarters router is connected to the branch via a GRE tunnel
interface Tunnel3. From the output of the previous command you learned that Tunnel3 interface is
configured on BR3. To verify the configuration, you enter show ip interface brief Tunnel3. The tunnel is
operational and has the 192.168.13.1 IP address.
To check whether there is an EIGRP adjacency between the HQ and BR3 routers, you enter show ip eigrp
neighbors. You see that the adjacency with the HQ router is established via the Tunnel3 interface and not
via the Ethernet 0/0 interface, which has summarization configured.
You are ready to make a hypothesis.
So far you know that:

 HQ and BR3 routers are neighbors on the Tunnel 3 interface.
 Besides HQ, there are no other EIGRP adjacencies established by BR3.
 There is routing information exchange between the neighbors – HQ routing table contains EIGRP
entries for BR3 networks.
 There is a summary route configured on BR3 on the Ethernet 0/0 interface, which does not participate
in EIGRP.
Since no routing information is exchanged on the Ethernet 0/0 interface – there are no EIGRP neighbors on
that interface, you conclude that EIGRP summary route was configured on the wrong interface.
To verify your hypothesis, you should configure the summary route on the Tunnel3 interface. First you
delete the summary route configuration from the Ethernet 0/0 interface, using no ip summary-address
eigrp 100 172.16.0.0 255.255.0.0. Before you continue, you verify whether the EIGRP AS number used in
the command is correct. You issue show ip protocols | include AS command. EIGRP AS number is 100.
Next, you configure the summary route on the Tunnel3 interface. You enter the ip summary-address eigrp
100 172.16.0.0 255.255.0.0. The summary address and mask used in the command encompass all the
networks of BR3 and much more. In fact the entire range 172.16.0.0 – 172.16.255.255 is included. This
summary address is too broad and includes the networks that do not exist on BR3. It would be more precise
to use 172.16.0.0 255.255.224.0 as the summary address and mask. Such range includes all already
configured networks and allows another 12 networks with the /24 mask to be added to BR3, without the
need to change the summary address. Also note that configuring the summarization on the interface resets
the EIGRP adjacency.
To check the new configuration results, you take a look at the BR3 routing table.
The routing table of the BR3 now contains a summary-route entry. It has a Null0 exit interface, i.e. the
packets matching this route are dropped. Note, that the route appears in the routing table with the letter „D“.
If you use the show ip route 172.16.0.0 255.255.0.0 command, you will note that the administrative
distance of the summary-route is 5. The metric for the summary-route is equal to the minimum metric of the
specific component routes.
When you check the routing table on the HQ router using the show ip route eigrp 100 command, this time
the output shows only one summary-route entry for 172.16.x.0/24 networks of BR3, learned via EIGRP.
EIGRP network summarization is now correctly configured. You have solved the issue.
Troubleshooting EIGRP Summarization
Route summarization can be used to decrease the size of the routing table. Smaller routing tables make the
routing update process less bandwidth demanding. In EIGRP the summarization feature is available in the
form of automated summarization, limited to summary-routes with classful network boundaries or as a
manually configurable – allowing for the summary-routes with arbitrary network masks.
In the conventional, AS number configuration method, the summarization is active by default.
 The summarization is automatic at classful boundaries and the summary-route always have a major
network mask.
 EIGRP will not automatically summarize networks to which the router is not attached.
 To check whether auto-summarization is active and which networks are included in the EIGRP process
use show ip protocols | section eigrp.
 In case the addressing in the network is discontiguous auto-summarization might result in same
summary address being advertised from multiple routers. If the metrics advertised are the same, the
router receiving such updates might load-balance between several routes, effectively rendering
destinations unreachable for some packets.
 To disable auto-summarization use no auto-summary command in the router configuration mode – in
case of AS number EIGRP configuration method or in the router address-family topology sub-mode –
in case of the named EIGRP configuration method.
When configuring manual summarization the summary route is advertised only when there is a route in the
routing table for the more specific, summary component network:
 Summary route must be configured on the interface via which it would be advertised to other routers, i.e
the interface must be included in the EIGRP routing.
 Use show ip route network network-mask longer-prefixes command to display all networks in the
routing table to verify the existence of the more specific routes in the routing table.
− The command should be used carefully—the network and network-mask pair are ‘and’ed to become
the prefix, and any address that matches that prefix is displayed.
 If no summary route is present in the routing table check whether:
− The summary-route and its network mask include more specific networks present in the routing
table.
− If the summary-route does not include more specific networks, you can also add a loopback
interface and configure it with an IP address belonging to the summary-route range to ‘force’ its
propagation.
Manually configured summary-route should not be too broad.

 In designs that involve too broad summarization and default route propagation, the packets sent to non-
existent networks might end up in a loop.
 Configure the summary-route to cover only the existent networks or networks that are planned to be
added in near future.
Example Troubleshooting Flow: IPv4/IPv6 Internet
Access Issue
PC0 does not have Internet connectivity, neither using IPv4 nor using IPv6.
In order to verify the problem, you check the Internet connectivity from PC0. First you use an IPv4 address
and enter ping 209.165.201.45. Dots in the output indicate that there is no connectivity. Next you use the
IPv6 address and enter ping 2001:DB8:D1A5:C92D::1. The output confirms that there is no IPv6
connectivity to Internet either. Report is correct and you can start troubleshooting.
Note that ping tool is dual-stack. The user input to the tool is parsed from the command line and, depending
on whether the resulting IP address is IPv4 or IPv6, sends ICMP or ICMPv6 messages.
this problem.
You have confirmed that there is no Internet connectivity at PC0. You decide to troubleshoot the IPv4
connectivity first, and then IPv6. You use the bottom-up approach and begin by inspecting the interface
status on PC0.
You enter the show ip interface brief command on PC0. The output tells you that Ethernet 0/0 interface is
used to connect to the network. It is operational, the interface and line-protocol statuses are up. You now
check the network layer information. The IP address assigned to the interface is 192.186.0.100. To check
the network mask you issue show ip interface ethernet 0/0. The mask is 255.255.255.0.
According to the documentation, which might be incorrect, the IP address is assigned from the
192.168.0.0/24 and not 192.186.0.0/24 subnet. In order to verify which information is correct, next you
check the default gateway configuration on PC0.
In order to learn which subnet should be configured in the network you first check the default gateway
information on PC0 using show ip route command. The default gateway is 192.168.0.1 and is in the
different subnet than Ethernet 0/0 interface. This is in accordance with the documentation. To be sure, you
check the IP configuration on SW1 and you check whether router R1 is configured with 192.168.0.1
address.
On the SW switch you issue show ip route command. There are routes belonging to 192.168.0.0/24 range.
On router R1 you use show running-config | include 192.168.0.1. Virtual router group 1 is configured with
the 192.168.0.1 IP address.
In order to test your hypothesis, you first remove the incorrect IP address assignment using no ip address
192.186.0.100 255.255.255.0 and then configure the correct IP address 192.168.0.100/24.
Then you test the Internet connectivity entering ping 209.165.201.45. The ping is successful. You have
restored the IPv4 connectivity.
Now you proceed with troubleshooting the IPv6 connectivity.
While troubleshooting IPv4 connectivity issue, you have concluded that there are no lower layer issues.
This conclusion is still valid—lower layers’ operation does not depend on upper layer protocols.
Therefore, you check the network layer information on PC0 using show ipv6 interface brief. Ethernet 0/0
interface has IPv6 address assigned. The information is in accordance with documentation.
Next you check the default gateway information using show ipv6 route. There is a default route using
FE80::11 link-local address as the next-hop. According to the documentation, this is R1’s link-local address.
The information is learned via Neighbor Discovery protocol, as evident from the ND letters at the beginning
of routing table entry.
this problem.
You have confirmed that there is no IPv6 Internet connectivity at PC0. IPv6 Internet connectivity has to be
ensured on all devices along the way. That is why you will use the follow-the-path method in solving the
IPv6 connectivity issue. You start by checking IPv6 Internet connectivity from the next-hop device R1.
On R1 you issue ping 2001:DB8:D1A5:C92D::1 and it is not successful. This time the output indicates the
reason – there is no valid route towards the destination. To confirm that, you can also check the IPv6 routing
table. As you can see, there is no route for the destination prefix and there is no default route. There is only
one route learned via RIPng, from HQ and R2, towards the 2001:DB8:C0A8:200::/64 segment connecting
HQ and R2 routers. This means that RIPng routing is operational in the network.
According to the documentation, both R1 and R2 routers, should be advertising a default route via RIPng.
Next you check the configuration on R2.
You repeat the steps on R2. The ping is not successful with output indicating the same reason as at R1 –
there is no valid route towards the destination. You verify that the routing table has no route for the
destination prefix and there is no default route.
Next you check HQ configuration.
Before you delve into routing protocol configuration, you check the Internet connectivity from HQ. You
enter ping 2001:DB8:D1A5:C92D::1. It is successful. Next you check the RIPng routing protocol
information. You enter show ipv6 protocols | section rip. RIP protocol is configured as documentation
suggests. The instance tag is „ccnp“ and it includes Ethernet 0/1 and Ethernet 0/2 interfaces towards R1 and
R2, respectively. This was expected as there were RIPng entries in routing tables of R1 and R2.
Next you check the route towards the Internet destination. You enter show ipv6 route
2001:DB8:D1A5:C92D::. The output tells you that HQ is using a default route ::/0 to reach Internet. It is
statically configured.
You can now recap the information that is acquired so far and propose a hypothesis.
The default route information is present at HQ, but it is not advertised to other routers participating in the
same RIPng instance.
In order to test your hypothesis you first verify whether RIPng „ccnp“ process is configured to advertise the
default route in the first place. You enter show ipv6 rip ccnp. There is a line in the output stating Default
routes are not generated. This means that default route announcement was not at all configured.
RIPng announces a default route out of an interface participating in the RIPng process. This means, the
default route should be announced on Ethernet 0/1 and Ethernet 0/2 interfaces in order to reach R1 and R2.
In order to start sharing the default route information, you must configure the feature under interface
configuration mode. You configure it using ipv6 rip ccnp default-information originate command.
To verify the configuration use show ipv6 rip ccnp | include Default routes. The statement in the output
confirms that default route announcement is now configured.
Next you verify that R1 and R2 have the default route in their routing tables.
It remains to check whether PC0 has IPv6 Internet connectivity. The ping 2001:DB8:D1A5:C92D::1 is
successful. You have solved the problem.
Troubleshooting Basic RIPng
RIPng is a distance vector routing protocol, using hop count as a metric. It uses native IPv6 packets for
routing updates exchange and a well-known multicast address. UDP is the transport protocol using port
number 512.
Before starting to troubleshoot IPv6 routing issues make sure that IPv6 routing is enabled on the device and
that interfaces are configured with IPv6 addresses.
RIPng routes do not appear in the IPv6 routing table.

 Check that RIPng is enabled on the interface.
− RIPng must be explicitly enabled on each interface that participates in the process.
− RIPng process identificator must be the same for all the interfaces and routers in one routing
domain.
 Check that interface is operational, ie. its status must be up-up.
 Check whether the network missing the route is more than 15 hops away.
− RIPng has the maximal radius of 15 hops. Networks with more hops are considered unreachable.
 Check whether default-route is propagated via RIPng.
− If yes, routing updates for non default-route networks can be suppressed if the command ipv6 rip
name default-route only command was used to configure default-route announcement.
 Check whether IPv6 access-lists are blocking the RIPng traffic.
− FF02::9 IPv6 multicast address and UDP port 512 must be permitted in the ACL.
Default route is not announced.
 Check that default-route announcement is configured on the router.
− RIPng default-route announcement must be configured on the interface out of which it is to be
announced.
RIPng is not load balancing.

 Check RIPng configuration for the maximum-path parameter.
− Configuring maximum-path to 1 turns load-balancing off.
 Checkthat there are multiple routes to the destination received via RIPng and that they have the same
metric.
− RIPng load balances over equal cost paths.
Above, useful RIPng troubleshooting commands are shown.
Example Troubleshooting Flow: Internet
Connectivity Lost
BR3 devices cannot access Internet through IPv6.
To verify the problem you ping the Internet using IPv6 destination address, sourced from BR device
networks. To test IPv6 Internet connectivity, you choose one of the loopback interfaces, for instance
Loopback18. You use ping 2001:DB8:D1A5:C92D::1 source 2001:DB8:AC10:1300::1 and repeat the
ping sourcing it from Ethernet 0/1. There is no IPv6 connectivity.
this problem.
There is no IPv6 connectivity from BR3 networks. You decide to use the divide and conquer approach and
start from the network layer upwards.
Starting at the network layer, you will first check the IPv6 configuration of the device. Enter show ipv6
interface brief to check whether IPv6 addresses are configured. From the output you see that all interfaces
are configured with IPv6 addresses and are operational. The link-local address configured for the BR3 is
FE80::40. The configuration is in accordance with the documentation. Next you check the IPv6 routing
table.
You would like to check what IPv6 routing information is available for the Internet destination address.
Enter show ipv6 route 2001:DB8:D1A5:C92D::1. The route used to reach the Internet is actually the
default route. It is statically configured, as is written in the documentation. The default route points to
FE80::1 link-local address, reachable via Ethernet 0/0 interface. In order to verify that the default-route next
hop is accessible you enter ping FE80::1. Since you are using the link-local address in the command, IOS
prompts you to enter the source interface, which in our case is Ethernet 0/0. The next-hop of the default
route is not reachable. Next you turn on debugging and repeat the ping.
To display the debug messages for IPv6 packets, use the debug ipv6 packet command. You repeat the ping
2001:DB8:D1A5:C92D::1. In the debug output you notice IPv6-ACL entry. It suggests that the packets
incoming to BR3 are discarded by ACL named „from_Internet“. Using show ipv6 access-list you verify
that there is a IPv6 ACL named „from_Internet“ configured on the BR3. Next you check to which interface
is the ACL „from_Internet“ applied and in which direction.
To see where the ACL is applied use show running-config | include interface|traffic-filter. From_Internet
ACL is applied to Ethernet 0/0 interface in the inbound direction. You check the content of the access-list
using show ipv6 access-list from_Internet command. The list is permitting any return traffic destined only
to global IPv6 addresses belonging to Ethernet 0/1 and Loopback interfaces. It contains an explicit deny
ipv6 any any statement at the end, which has the sequence number of 220. You are ready to make your
hypothesis.
So far you know the following:

 Traffic coming to BR3 from the Internet is filtered by from_Internet IPv6 access list.
 The list permits IPv6 packets to global IPv6 addresses of BR3 networks. However,
 It has an implicit deny ipv6 any any statement which is effectively blocking all traffic destined to link-
local addresses, including Neighbor Discovery messages.
The exchange of ND messages between the router and the link-local node are necessary in order to provide
connectivity.
To test the hypothesis, you should modify the ACL from_Internet.
There are two modifications you can use:

 You can add an explicit permit statement allowing the ICMP traffic to link-local address of the Ethernet
0/0, or
 You can delete the explicit deny statement at the end.
You choose the latter and delete the explicit deny using the no sequence 220 in the ipv6-acl configuration
mode.
Deleting the explicit deny will add an implicit deny to the ACL. Implicit deny in IPv6 ACLs consists of
three statements:
permit icmp any any nd-na
permit icmp any any nd-ns
deny ipv6 any any
This allows for ICMPv6 neighbor discovery message exchange.

You verify the solution by checking the connectivity to Internet using ping 2001:DB8:D1A5:C92D::1
source 2001:DB8:AC10:1300::1. The ping is successful and IPv6 Internet connectivity resolved. You have
solved the problem.
Troubleshooting Access Lists
When troubleshooting access-lists you should keep in mind the following:

 Determine whether access-lists are configured on the device.
 Determine where the ACLs are applied and in which direction with reference to the device.
 Understand how the list statements affect traffic.
− Ensure that IP address and wildcard masks are correctly entered into the ACL.
− Access-lists are read top to bottom, if a first match is found it will stop reading. Make sure to put
more specific entries before broader ones. A deny statement included before a permit statement may
block a permit statement. Order of ACL is important.
− Statements must define the type of traffic they relate to. TCP, UDP, ICMP, GRE are specific traffic
types. IPv4 and IPv6 are more general traffic types. If the statement checks for port numbers TCP or
UDP traffic type must be used in the ACL statement.
− Use an explicit deny to show ACL counter increments.
− If no traffic is permitted, all traffic will be denied, there is an implicit deny added to the end of the
ACL.
 Implicit deny in IPv4 denies all traffic.
 Implicit deny in IPv6 permits Neighbor Discovery messages.
Use the log keyword for specific access list entries.
 This keyword instructs the router to log a message to the system log whenever specific access list entry
is matched. The logged event includes details of the packet that matched the access list entry.
To determine where the ACLs are applied and in which direction with reference to the device use:
 show running-config | include line|access-class
− Displays access lines (vty, console) and the access-lists configured to control traffic to the line.
 show running-config | include interface|access-group
 show ip interface interface-type interface-number
− Displays interface and IPv4 access-list applied to it.
 show running-config | include interface|traffic-filter
 show ipv6 interface interface-type interface-number
− Displays interface and IPv6 access-list applied to it.
 show running-config | include [ACL-number| ACL-name |]
− Displays other applications of access-list, such as in NAT traffic specification.
Summary
Bank of POLONA Ltd.
Overview
This lesson serves as a debrief for the third troubleshooting lab at Bank of POLONA Ltd.

• Describe how you solved connectivity problem over GRE tunnel
• Describe the possible issues with GRE tunnels
• Describe how you solve OSPF summarization issue
• Troubleshoot OSPF summarization
• Describe how you solve BR1 authentication issue
• Describe AAA services and commands for troubleshooting

another branch - Branch 3. Over the weekend the customer reconfigured IPv4 routing. They moved the
newly acquired Branch 3 from EIGRP to OSPF. They also fine-tuned OSPF a little bit.
However, upgrade was not a total success. Customer engineer Tina calls, and she needs your expertise:
 Branch 1 cannot reach the headquarters via IPv4! User on PC1 reports that he is unable to ping PC0 and
adds it worked fine before the upgrade. This issue needs to be fixed as soon as possible!
 Even though customer configured OSPF summarization of 172.16.0.0/16 networks of Branch 3 office,
there are still individual routing entries for all 172.16.0.0/24 networks that are seen on the R1 router.
 Tina reconfigured access-restriction on BR1. She wants to use username/password combination of
"admin/c1sc0" and use these credentials as local AAA database that is then used for login. However,
when trying to Telnet from HQ to BR1, BR1 is still asking just for the password.
branch device. To access remote devices, you can use Telnet.
2001:DB8:D1A5:C92D::1.
Example Troubleshooting Flow: Branch 1 Cannot
Reach Headquarter
After the upgrade branch 1 devices cannot reach the headquarters via IPv4. User on PC1 reports that he is
unable to ping PC0.
Since you are not able to connect to the PC1, you can try pinging PC1 from the PC0 to verify if the problem
exists. As you can see ping is not successful, therefore you can conclude that the issue is present and it is
time to start troubleshooting.
problem.
Before you start troubleshooting, first verify if the problem exists.

You know that there were no issues before the upgrade, in which the OSPF was fine-tuned. Therefore you
can conclude that problem is most likely related to the OSPF. You should start troubleshooting at the
network layer.
Start troubleshooting on the HQ router, which is the closest router to the branch router BR1. Check if route
to the PC1 is located in the routing table. Based on the output you can conclude that route is not present in
the routing table. The routers HQ and BR1 should be connected via Internet with IPsec/GRE tunnel and
OSPF is used as a routing protocol. Therefore your next step is to check if OSPF is working correctly.
OSPF is used as the routing protocol in the network. You check OSPF neighbors with the command show
ip ospf neighbor on the router HQ. You can see that the router HQ has four OSPF neighbors, two of them
are connected through tunnel interface. The names of the tunnel interfaces are Tunnel2 and Tunnel3, which
most likely means that those are the tunnels to branch 2 and branch 3. You expect that tunnel to the branch 1
should be Tunnel1. Therefore you should check interfaces on the HQ to find if interface Tunnel1 exists.
You already discovered that router HQ has 4 OSPF neighbors, two of them are connected through the tunnel
interfaces. Since there are three branch offices you expect that there are three tunnel interfaces. To check
that, you issue command show ip interface brief. You have discovered that three interfaces are configured
on the router HQ. Interface Tunnel 1, which is most probably the interface used to connect to the branch 1,
has no IP address assigned. Your next step is to check the configuration on the interface Tunnel 1 and check
if this interface is used to connect to the branch 1.
You check the configuration on the interface Tunnel 1. You can see that there is no IP address configured.
The tunnel destination is the IP address 209.165.200.6. You should check if this IP is the public IP address
of the branch 1. You can use telnet to connect to the branch router.
Telnet to the IP address 209.165.200.6. As you can see from the output, this is the public IP address of the
router BR1. Therefore you can confirm that interface Tunnel 1 is used to connect to the branch 1.
You have discovered that interface Tunnel 1 is used to connect to the branch 1. From the configuration of
the interface Tunnel 1 you can conclude that there is no IP address configured on the interface. Therefore
you should check tunnel interface on the router BR1 to find out which IP address should be configured on
the router HQ.
You check the interfaces on the router BR1. As you can see interface Tunnel 1 is configured on the router
BR1. To see the configuration details, you issue the command show running-config interface Tunnel 1.
You can see that IP address used on the interface Tunnel 1 is 192.168.11.1/24. Therefore you can use next
IP address 192.168.11.2/24 for the interface IP on the router HQ.
To test your hypothesis, you should configure IP address on the interface Tunnel 1. After IP address is
configured you check the connectivity to the IP address Tunnel 1 on the router BR1. As you can see there is
connectivity through tunnel interfaces. Therefore your next step is to check if OSPF adjacency between
routers HQ and BR1 is up.
After you configure IP address on the router HQ, you should check if OSPF adjacency is established
between the routers HQ and BR1. You issue the show ip ospf neighbor command. As you can see from the
output, there is still no adjacency on the interface Tunnel 1. You should investigate why adjacency did not
come up between the routers.
You should also check the status of the OSPF on the interface Tunnel 1 on the router HQ. You can see that
OSPF is not configured on the interface. Therefore you check the OSPF configuration. You can see that
network command is missing for the subnet of the Tunnel 1 interface. Your next step is to include the
Tunnel 1 interface to the OSPF using the network command.
Since you have discovered that network command is missing for the interface Tunnel 1 you should use
command network 192.168.11.0 0.0.0.255 area 1 to enable OSPF on that interface.
PC0# ping 192.168.1.101
!!!!!
To test if your solution was correct you should issue ping from the PC0 to PC1. As you can see the ping was
successful. Therefore you can conclude that connectivity to the branch 1 was established. You can inform
the customer engineer that problem was solved and you can close the ticket.
Note At this point you should inform the customer engineer that the problem was solved. Save the
configuration on the HQ. Document changes made. Close the ticket.
Troubleshoot GRE Tunnels
GRE encapsulates packets inside a transport protocol. Some advantages of GRE tunnels are:
 Ability to transfer multiple network protocols over a single-protocol backbone.
 Provides workarounds for networks that contain protocols with limited hop counts.
 Connect discontinuous sub-networks.
 Allows VPN across WANs.
GRE takes a network layer packet and encapsulates that packet into IP packets. Special GRE header is
inserted after transport IP header.
To configure GRE tunnel use command interface Tunnel tunnel-id. You also need to specify tunnel source
IP address or interface with the command tunnel source ip-address|source-interface and tunnel destination
with the command tunnel destination ip-address. To configure GRE tunnel mode use command tunnel
mode gre ip. GRE is the default tunnel mode.
Some common GRE issues are:

 GRE source IP address is not reachable by remote host.
− Check if correct source IP address or interface is applied to the tunnel. You can also check routing
in the backbone between the end point hosts.
 GRE destination IP address is not reachable by local host.
− Check if correct destination was configured and also check if hosts are reachable between them.
 Recursive routing.
− When the same routing instance is used in the transport layer and tunnel interface or when
redistribution is used, recursive routing can occur. This means that tunnel interface begins flapping.
This could happen if the best route to the tunnel destination is through the tunnel.
 GRE traffic denied by ACL.
− GRE use IP protocol number 47. When using GRE this must be allowed in the access lists.
 Fragmentation due to insufficient MTU:
− Usually MTU is 1500 bytes. GRE header is 24 bytes, which decreases MTU to 1476 bytes. Packets
bigger than 1476 bytes will get fragmented. In case that there are a lot of these big packets, this can
result in processing delays and high CPU usage. Ensure that consistent MTU is used end-to-end.
To check tunnel interface status use show interfaces Tunnel tunnel-id. You will be able to see if interface
is up, the tunnel IP address, the tunnel mode, which should be GRE/IP for GRE tunnels, tunnel source and
destination and some other tunnel parameters.
When you need to check ip parameters on the tunnel interface use command show ip interface Tunnel
tunnel-id.
Use debug tunnel command to get tunnel debugging information. You will be able to see events related to
the tunnel.
To test if the tunnel destination is up use ping command. If you want to test connectivity between tunnel
endpoints use ping ip-address source ip-address|interface command.
Example Troubleshooting Flow: Route
Summarization From Branch 3 Does Not Work
The summarization for the 172.16.0.0/16 networks of the Branch 3 was configured, but individual routes are
still present on the R1 router.
Verify the problem by issuing the show ip route command. You can see that there is many routes from the
subnet 172.16.0.0/16 in the routing table on the R1. The problem is confirmed and you can start
troubleshooting.
problem.
You were told that the routing in the Branch 3 was migrated from EIGRP to OSPF over the weekend. Route
summarization was configured but individual routes are still present in the routing table on the router R1.
You will use the troubleshooting approach called „follow-the-path“ to find configuration errors on the
network path to the Branch 3. Therefore you should start by checking the routing table on the router HQ.
Information gathering
When using the „follow-the-path“ technique you follow the network path to discover the source of the
problem. Therefore you should check routing table on the router HQ. As you can see from the output
individual routes are present in the routing table. All routes are accessible via the interface Tunnel 3.
Telnet to the Branch 3 router using IP address found in the Job Aids.
There are two areas configured on the router BR3. These are backbone area 0 and area 3. As you can see
from the output the area 0 is inactive, since no interfaces are configured in this area. To see the current
configuration check running-config.
Check the OSPF configuration on the BR3. As you can see from the output, all interfaces on the BR3 are
configured to be in the area 3. You also notice that summarization has been configured using the area area-
id range network mask command. BR3 is an internal router. The summarization should not be configured
on an internal router, but on the ABR router.Therefore this router is called internal router. Summarization in
OSPF is always done on the ABR.
You should check if there is summarization configured on the router HQ.
You check the OSPF configuration on the router HQ. As you can see from the output, there are several
network commands with different areas. This means that this router is Area Border Router and
summarization should be configured on this router. But, as you can see, summarization is not configured.
You should delete misconfigured summarization on the router BR3 and reconfigure summarization on the
router HQ.
You have discovered that summarization was configured on the router BR3, which is an internal router.
Summarization in OSPF is always done on ABR. You have checked the OSPF configuration on the router
HQ, which is an ABR, and no summarization is configured. Therefore you can conclude that summarization
was configured on the wrong router.
To test your hypothesis you should delete the wrong configuration from the router BR3, using command no
area 0 range 172.16.0.0 255.255.0.0. You configure summarization on the router HQ with the similar
command area 3 range 172.16.0.0 255.255.0.0.
Check routing table on the router R1 to verify if the problem was solved. As you can see from the output
there is only one summary route for the networks from the subnet 172.16.0.0/16. Therefore you can
conclude that the problem was solved.
If you want to check LSA in the OSPF database on the router issue show ip ospf database summary
172.16.0.0 command. This will display LSA type 3 with the ID 172.16.0.0.
Note At this point you should inform the customer engineer that the problem was solved and that
now only one summary route is advertised from the area 3. Save the configuration on the
HQ. Document changes made. Close the ticket.
Troubleshoot OSPF Summarization
Summarization is the consolidation of multiple routes into one single advertisement. There are two types of
OSPF route summarization:
 Inter-area route summarization.
− Inter-area route summarization is done on ABRs and it applies to routes from within the area. This
does not have effect on the external routes injected into OSPF via redistribution. Summarization
could be configured between any two areas, but it is better to summarize in the direction of the
backbone. This way the backbone receives all the aggregate addresses. To summarize on the ABR
use area area-id range ip-address mask command, where area-id is the area containing networks to
be summarized, the address and mask will specify the range of addresses to be summarized in on
range.
 External route summarization.
− External route summarization is specific to external routes that are injected into OSPF via
redistribution. This type of summarization is done on ASBR. Use command summary-address ip-
address mask to perform summarization.
Check the routing table with the command show ip route on the OSPF routers to check if there are
individual routes or summarized routes in the routing table. When checking routing table on the routers that
perform summarization you should see summary route pointing to the Null0 interface. This route is created
automatically in order to prevent possible routing loops.
You can check OSPF status with the show ip ospf command on the ABR router to verify which area ranges
are configured for summarization.
To check which external routes are summarized on the ASBR use command show ip ospf summary-
address. You will be able to see all summary addresses that are configured on the router.
To check Type 3 or summary LSA use command show ip ospf database summary. You will be able to see
all summary LSAs with summary network number, network mask, metric and some other parameters. You
should use this command when troubleshooting if summary LSAs was received by the router.
Similarly you can use show ip ospf database external to check Type 5 or external LSAs. You will see all
external LSAs, with its network number, mask, metric and some other parameters.
Example Troubleshooting Flow: AAA Does Not
Work on the Router BR1
The customer engineer reconfigured access-restriction on BR1. She wants to use username/password
combination of "admin/c1sc0" if the AAA server is not reachable. (Note that the AAA RADIUS server has
not yet been purchased.) However, when trying to telnet from HQ to BR1, BR1 is only asking for a
password.
To verify if the problem exists, you should telnet to the router BR1. As you can see from the output, only
password is required to access the router BR1. You can conclude that problem exists and you should start
troubleshooting.
Note This is one possible plan. There are many different approaches to this problem.
You were told by the customer engineer that she had reconfigured access restrictions on the router BR1. She
wants the combination of username/password to be used for authentication when accessing router BR1. But
the configuration is not correct, since only the password is required.
You should use „top-down“ approach when troubleshooting access policy. Therefore you should first check
the line vty configuration on the router BR1.
First telnet to the router BR1 from the router HQ and check the vty line configuration on the BR1 router. As
you can see the basic login command is used for authentication to the router. This means that router will
authenticate the access by checking the password configured with password command under vty line.
You also check if there is an AAA configuration on BR1. As you can see, AAA is not configured.
You can conclude that this is actually an issue.
Testing Hypothesis
Before you are doing configuration changes for access control on the remote devices it is a good practice to
use reload in [hh:]mm for fall-back purposes. In this way router will be reloaded and reverted to the
previous configuration in case you lock yourself out of the device. Do not forget to cancel reload if
configuration is successful. Use command reload cancel.
When configuring AAA services with aaa new-model command, local authentication is immediately
applied to all lines except console line. If new Telnet or SSH session is opened to the router after enabling
this command, the user has to be authenticated using the local database of the router. Therefore you should
check if any user is configured otherwise you will not be able to access the router. As you can see the admin
user exists.
You can enable AAA services and configure the local authentication with the command aaa authentication
login default local. With this command local database will be used for authentication by default.
After AAA services have been configured, you can initiate new telnet session to check if configuration has
been correctly assigned.
After enabling AAA services you should test access to the router BR1. Use telnet 209.165.200.6 to access
the router BR1 from the HQ. As you can see from the output router BR1 now ask for username and
password. After providing correct credentials you are connected to the router BR1.
Note Ideally you would leave the current session to BR1 open and initiate another one from a
different device. This would lessen the risk of locking yourself out of the device in case you
misconfigured remote access.
Therefore you can conclude that issue was solved and you can inform the customer engineer.
Note After correct solution was applied save the configuration, inform the customer engineer,
document changes and close the ticket.
Troubleshoot AAA
AAA services are used to control access to the management plane of the router or switch. You can use local
user database or database on the remote server with the protocols like TACACS+ or RADIUS. TACACS+
is Cisco proprietary protocol that works over TCP port 49, while RADIUS is IETF standard that uses UDP
port 1812 (or 1645) for authentication and UDP port 1813 (or 1646) for accounting.
It is the best practice to use centralized AAA server with a local method as a backup, in cases when AAA
server is not accessible.
To enable AAA services on the router use aaa new-model command. After AAA is enabled, you are able to
configure authentication, authorization and, accounting methods. AAA services uses the concept of a named
list, which could be used under different location to where AAA services are used. You can also use default
list which means that this list is used when no other list is specified. To configure authentication use
command aaa authentication, to configure authorization use aaa authorization and to configure
accounting use aaa accounting command.
Some of the common problems of TACACS+ and RADIUS servers are very similar:
 The server failure or the server is not accessible.
− To prevent locking yourself out of the device when AAA server is not accessible use local method
as backup for authentication. You can define up to four methods for authentication.
 Mismatch between shared key.
− Both protocols need a shared key to be used for authenticating a network device. If keys do not
match on remote server and network device, the network device will not be able to access the user‘s
database.
 User credentials are rejected by the server.
− Usually you can check in server logs to verify if user was correctly authenticated or authorized.
When users are rejected, you should check these logs to verify if provided credentials were correct.
In addition to those common problems there could be RADIUS ports mismatch. Cisco uses UDP port 1645
and 1646 by default. UDP ports 1812 and 1813 are ports specified in the standard. Usually servers listen on
both sets of ports for backward compatibility, but you should always check if correct ports are used.
To check AAA services use show running-config | include aaa command. To verify that AAA services are
enabled, the output should contain the statement aaa new-model.
The command debug aaa authentication can be very useful for troubleshooting. It will display events that
are related to authentication when accessing the router. You will be able to see users used for authentication,
which named list is used, where the user wants to access etc.
To get more information about authorization process use debug aaa authorization command, which will
display services required by the user, the method used for authorization, the status of the authorization etc.
To debug AAA server events use commands debug tacacs and debug radius, respectively. You will be
able to see events related to remote server.
Summary
Bank of POLONA Ltd.
Overview
This lesson serves as a debrief for the forth troubleshooting lab at Bank of POLONA Ltd.

• Describe how you solved IPv6 connectivity from the PC0 to the Internet
• Troubleshoot OSPF for IPv6 environment
• Describe how you solved the OSPF summarization issues
• Troubleshoot OSPF stubby areas
another branch - Branch 3.
Customer engineer Tina is keeping herself, and therefore you, busy. Over the weekend she reconfigured
IPv6 routing from RIPng to OSPFv3 in order to run OSPF on IPv4 an IPv6. She also attempted to do a little
bit of OSPF fine-tuning on the branch routers.
It has been a long night and at this point Tina needs your help to fix things before people come to work!
 PC0 does not have connectivity to IPv6 Internet sites.
 Even though Tina configured branch as totally stubby, all branch routers still have a lot of inter-area
routes in their routing tables.
branch device. To access remote devices, you can use telnet.
2001:DB8:D1A5:C92D::1.
Example Troubleshooting Flow: PC0 Does Not
Have Connectivity to IPv6 Internet Sites
Customer engineer told you that she reconfigured IPv6 routing from RIPng to OSPFv3 during the weekend.
Transition was not totally successful, since PC0 does not have connectivity to IPv6 Internet sites.
You should fix the problem.
Before starting with troubleshooting, check if the problem exists. To test connectivity to the IPv6 enable site
use ping 2001:DB8:D1A5:C92D::1 command. As you can see from the output the ping is not successful.
Therefore you can conclude that problem is confirmed and you can start troubleshooting.
You were told by the customer engineer that IPv6 routing was reconfigured from RIPng to OSPFv3. After
the implementation PC0 does not have connectivity to the IPv6 enabled sites.
You have verified the problem and you can confirm that there is no connectivity.
You should try to find the issue and fix the problem.
Check the routing table on the PC0 to verify if IPv6 default route is present. Use command show ipv6
route. As you can see from the output, the default route is present and it was learned via neighbor
discovery. The next hop for the default route is link-local address FE80::A8BB:CCFF:FE00:4900. You
should check which device owns this IP address.
You should check neighbor table on the PC0. You will find IPv6 MAC address mapping in the table, similar
to ARP table in IPv4 world. Use command show ipv6 neighbors. As you can see from the output the MAC
address for the next-hop IPv6 address is aabb.cc00.4900.
First check MAC address table to get information about interface used to connect MAC address
aabb.cc00.4900. As you can see from the output MAC address is connected to interface Ethernet 0/2, which
is used to connect router R2.
Verify if IPv6 address used as next-hop IPv6 address by the PC0 is presented on the router R2. Use
command show ipv6 interface brief. As you can see from the output interface Ethernet 0/0 connected to
the same LAN as PC0 and router R2 is used as default gateway by the PC0.
In the next step check connectivity to the Internet from the router R2.
Check connectivity with the ping command. As you can see from the output there is no valid route to the
destination.
You were able to discover that there is no valid route to the Internet on the router R2. Check the
documentation to see how default route should be received.
Check the documentation to discover how default route should be received on the router R2. As you can see
from the IPv6 routing diagram the default route should be received from the router HQ via OSPFv3. In your
next troubleshooting step you should check if there is an OSPFv3 adjacency between router HQ and router
R2.
Check the OSPFv3 neighbors on the R2. As you can see, there is one OSPFv3 neighbor with the OSPFv3
ID 192.168.10.1. Check OSPFv3 status on the router HQ.
Check the OSPFv3 status on the router HQ with the command show ipv6 ospf. As you can see the router ID
is 192.168.10.1. This ID was seen in the OSPFv3 adjacency table on the R2 and therefore you can conclude
that there is a valid adjacency between HQ and R2.
Check the OSPFv3 configuration on the router HQ. As you can see from the output no additional
configuration is specified under ipv6 router ospf 1, which means that default route is not injected into the
OSPFv3 process. You should configure the router HQ to send default route to its neighbors.
At this point you can also verify, using show ipv6 route, that a default route is indeed configured on HQ.
You have discovered that default route was not advertised to the OSPFv3 neighbors on the router HQ.
Therefore you should use default-information originate command to inject default route into OSPFv3
process.
You should check if IPv6 connectivity was restored with this configuration. Check the connectivity from the
router R2. Only if a default route is configured on HQ, OSPF will be able to inject it into OSPF.
To test if your hypothesis is correct, check connectivity from the R2 and PC0. As you can see, from the
outputs, there is connectivity to the Internet from R2 while connectivity test is not successful from the PC0.
Check connectivity to the router HQ from the PC0. As you can see from the output there is no connectivity.
You already know that there was a default route present on the PC0 and default route is available on the R2.
You should check if router HQ has a route to the PC0.
The route to 2001:DB8:C0AB::6 is matched against the default route, which points towards the IPv6
Internet. This means that HQ may be able to ping from PC0, but there is no response from HQ since no
route is available.
Your next step is to verify why route is not presented in the routing table.
You discovered that there is no route from the HQ to the PC0. You should check if OSPFv3 is enabled on
interface facing LAN segment.
Check interface configuration on the router R2. As you can see OSPFv3 is not enabled on the interface
Ethernet 0/0. To test your assumption, you should enable OSPFv3 on the interface Ethernet 0/0 and test
connectivity from the PC0 again.
After analyzing gathered information, you can propose the hypothesis that IPv6 is not enabled on the
interface Ethernet 0/0. To test if your hypothesis is correct enable OSPFv3 on the interface and test
connectivity from the PC0.
To test your hypothesis enable OSPFv3 on the Ethernet 0/0 interface, using the command ipv6 ospf 1 area
0. After enabling OSPFv3 test connectivity from the PC0. As you can see from the output there is
connectivity from the PC0 to the IPV6 enabled sites on the Internet.
You can conclude that the problem is solved. However you know that there are two routers connected to the
LAN segment to provide high availability. You should check OSPFv3 configuration on router R1 as well.
Check interface Ethernet 0/0 configuration on the R1. As you can see there is no OSPFv3 configuration on
the R1. Therefore you can conclude that OSPFv3 is not enabled on interface Ethernet 0/0.
Enable OSPFv3 on the interface Ethernet 0/0 on the router R1. As you can see OSPF adjacency is
established with the router R2 after enabling the OSPF on the interface Ethernet 0/0.
To confirm that the route is received from both routers on the HQ, check routing table on the HQ.
Check the routing table on the HQ router for the route 2001:DB8:C0A8::/64 using command show ipv6
route 2001:DB8:C0A8::/64. As you can see from the output route is received from both routers. Therefore
you can conclude that the problem is solved and you can close the ticket.
Note After the problem was fixed you should save the configuration on the routers HQ, R1 and
R2, inform the customer engineer that the problem has been fixed, document changes and
close the ticket.
Troubleshooting OSPF For IPv6
OSPFv3 operates in similar way to OSPFv2. Some of the important differences are:
 Protocol processing per-link, not per-subnet:
− Multiple IP subnet can be configured on single link between two routers. OSPFv3 neighbors can
establish adjacency even if they do not share a common IPv6 subnet.
 Use IPv4-format Router ID:
− OSPFv3 neighbors are identified by Router ID only. It remains 32-bits long, therefore IPv6 address
could not be assigned as router ID. If IPv6 is the only protocol enabled on the router, router ID must
be manually specified to start OSPFv3 instance.
− IPv6 addresses are only present in LSU packets.
 Support for multiple instances per link:
− Multiple instances can be used on single link. Instances are separated with Instance ID, which is
contained in OSPF packet header.
 Use of link-local addresses:
− OSPF neighbors use link-local addresses as source and destination of the OSPF packets when
sending directly to a neighbor. Link-local address is also used as a next-hop in the IPv6 routing
table for OSPFv3 routes.
 Different multicast addresses:
− Multicast address of FF00::5 is used to address all OSPFv3 routers and multicast FF00::6 is used to
address all DR routers.
 IPsec is used for authentication:
− OSPF-specific authentication was removed and IPsec is used to authenticate OSPF packets. In order
to ensure that OSPFv3 packets are not altered and re-sent to the device, causing the device to
behave in a way not desired by its system administrators, OSPFv3 packets must be authenticated.
OSPFv3 uses the IPsec secure socket API to add authentication to OSPFv3 packets. This API
supports IPv6. OSPFv3 requires the use of IPsec to enable authentication. Crypto images are
required to use authentication, because only crypto images include the IPsec API needed for use
with OSPFv3.
To enable OSPFv3 globally use command ipv6 router ospf process-id. If IPv6 is the only protocol on the
router you must configure Router ID, which is configured with the command router-id router-id. To enable
OSPFv3 on specific interface use interface command ipv6 ospf process-id area area.
Use show ipv6 ospf process-id to display some global OSPFv3 settings. You can see Router ID, timers,
areas configured on the router etc. Use this command to check if OSPFv3 is enabled on the router.
To display OSPFv3 neighbors use command show ipv6 ospf neighbor. The output is very similar to
neighbor table in OSPFv2. You are able to see neighbor ID, priority, state, dead time, interface ID, and
interface that is used to establish adjacency.
To display interfaces on which OSPFv3 is enabled use command show ipv6 ospf interface. The output will
show all interfaces with OSPFv3 enabled. You will also able to see which area is configured on the
interface, Router ID, network type, timers etc.
To display OSPFv3 database use command show ipv6 ospf database. In contrast to IPv4 database table all
LSAs are identified by the router ID. To display details on the specific LSA you can use show ipv6 ospf
database lsa-type adv-router router-id.
OSPFv3 Hello packets can be observed with the command debug ipv6 ospf hello, while all OSPF packets
can be observed with the command debug ipv6 ospf packet.
Example Troubleshooting Flow: Totally Stubby
Area on the Branch Is Not Working
The customer engineer told you that she configured branch as totally stubby, but she noticed that routers
still have a lot of inter-area routes in the routing table. She asked you to verify and fix the problem.
First verify the problem by telneting to the branch routers. You can telnet from the router HQ. The public IP
addresses can be find in the documentation.
Verify the routing table on the BR1, using command show ip route. You can filter the output with regular
expression ^O IA to display only inter-area OSPF routes. As you can see there are multiple inter-area routes
on the router. If totally stubby area was configured correctly there would be only the default route in the
routing table.
Repeat the same procedure on routers BR2 and BR3.
Check the routing table on the router BR2. Similar to router BR1, multiple inter-area OSPF routes are
present.
Check the routing table on the router BR3. You can also find out that multiple inter-area OSPF routes are
present on the router BR3.
You can conclude that all three branch routers have inter-area OSPF routes and therefore totally stubby area
is not configured correctly. The problem is confirmed and you can start troubleshooting.
problem.
You have been informed by customer engineer that totally stubby areas were configured on branch routers,
but inter-area routes are still present in the routing table on the branch routers. You have verified the
problem. Start by troubleshooting OSPF on the router HQ, which connects all branches and should be ABR
router. Find the solution for the issue and inform the support engineer about it.
Check the OSPF status on the router HQ with the command show ip ospf. As you can see from the output
three stub areas are configured on the HQ router. You assume that this is not correct. When totally stubby
area is configured on the ABR router, this router should not send summary LSAs. You should see this piece
of information in the output of the command.
To verify if your assumption is correct, check the running configuration on the router HQ.
To verify your assumptions check the running configuration on the router HQ. As you can see from the
output the router is only configured with area area-id stub command. To configure totally stubby area, you
should use the command area area-id stub no-summary.
Therefore your next step is to reconfigure stub configuration on the router HQ.
Analysing the Information
The HQ router was only configured as a stub. Since HQ is ABR the change in configuration is needed to
establish totally stubby area.
You should reconfigure OSPF areas with the command area area-id stub no-summary. This will enable
totally stubby area feature on the ABR. The branch routers should only be configured with the command
area area-id stub.
Now that new configuration was applied on the HQ, you should verify that totally stubby feature is correctly
applied to the branch routers. You should telnet to the branch routers and check the OSPF configuration and
the routing table.
After totally stubby feature was enabled check the OSPF status with the command show ip ospf. As you
can see from the output, Area 1, Area 2, and Area 3 are configured as stub areas with no summary LSAs in
that area. This means that those areas are totally stubby areas.
Telnet to the branch router BR1 and check OSPF configuration. As you can see the router is configured with
the command area 1 stub no-summary. Stub routers can be configured with or without no-summary
keyword. It is the ABR that dictates if stub area is totally stubby or not. So if you removed the keyword no-
summary from stubs it would make no difference.
Check the OSPF routes on the BR1 with the command show ip route ospf. As you can see from the output
only one OSPF route is present and that is the default route to the router HQ. This means that totally stubby
feature was correctly configured for the Area 1.
Repeat the same procedure on the routers BR2 and BR3. You will be able to see that totally stubby feature
is correctly configured on all three branch routers.
You were able to verify that only default route is present on the branch routers, therefore you can conclude
that totally stubby was correctly configured for Area 1, Area 2, and Area 3.
Note Since you have concluded that the problem was solved, you can close the trouble ticket.
You should save the configuration on HQ, BR1, BR2 and BR3, inform the customer engineer
that the problem was solved and document any changes that were made.
Troubleshooting OSPF Stubby Areas
OSPF allows certain areas to be configured as stub areas. When area is configured as a stub, external routes
are filtered on the ABR router. Instead, the default route is propagated into the area. To configure area as a
stub area, all routers in the area must have the area area-id stub configured under global OSPF
configuration mode.
Stub feature could be extended with the totally stubby area. In addition to external routes inter-area routes
are prevented as well on the ABR. To configure totally stubby area use area area-id stub no-summary
command on the ABR and area area-id stub on all other routers within the area.
When troubleshooting stub feature on the router first issue show ip ospf process-id command. If stub is
configured for the specific areas, you can see „It is a stub area“ output under area section. When totally
stubby area is configured you can see „It is a stub area, no summary LSA in this area“ output under area
section.
Use command show ip ospf database to check database on the router. If stub area is configured there
should be no LSA type 5 and type 7. You can also see additional LSA type 3 with the ID 0.0.0.0. This is a
default route advertised by ABR. Other summary LSAs can also be seen in the database. When area is
configured as totally stubby area only one summary LSA can be seen. This is LSA with the ID 0.0.0.0,
which is a default route advertised by ABR.
To check routing table use show ip route command. When router is located in the area that is configured as
stub, you should not see any O E1 or O E2 routes in the routing table. In addition when router is located in
the totally stubby area no O IA routes can be seen.
To observe hello message exchange use debug ip ospf hello. If there is any stub mismatch between the
routers, you can see the output similar to:
„OSPF: Hello from 192.168.23.2 with mismatched Stub/Transit area option bit“
If you see such message, you should check the stub configuration on both adjacent routers.
Summary
Overview
Check
1. How do link-state routing protocols inject external routes during redistribution process? (Source:
Debrief of the First Troubleshooting At Bank of POLONA Ltd.)
A. They always preserve topology information received from the external source.
B. They include all external routes as if they are directly connected to the redistributing router.
C. They do not redistribute routes from distant-vector routing protocols.
D. They do not redistribute routes from routing protocols with different AD.
2. Which CLI command should be used on an OSPF router to advertise the ip route 0.0.0.0 0.0.0.0 static
default route? (Source: Debrief of the First Troubleshooting At Bank of POLONA Ltd.)
A. redistribute static subnets
B. default-information originate
C. redistribute default
D. redistribute static default
3. What does this output of the traceroute command mean? (Select two.) (Source: Debrief of the First
Troubleshooting At Bank of POLONA Ltd.)
PC0> traceroute 209.165.201.45
Tracing the route to 209.165.201.45
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.0.253 0 msec 0 msec 1 msec
2 192.168.0.253 !H !H *
A. !H – Host is unreachable.
B. Router with an IP address of 192.168.0.253 has responded to the ICMP request.
C. * - Network unreachable.
D. * - The probe timed out.
E. !H – Host interrupted test.
4. Which show ip sla command displays number of succeed and failed tests? (Source: Debrief of the First
A. show ip sla statistics
B. show ip sla application
C. show ip sla configuration
D. show ip sla results
5. The exhibit shows the relevant outputs of the running configuration on the router A. EIGRP is correctly
configured and includes Ethernet 0/0 interface to the process. Summarization is manual for
172.16.x.0/24 networks from the routing table. It is configured on the correct interface. However, the
routing table does not contain the summary-route. What is the reason? (Select two.) (Source: Debrief of
the Second Troubleshooting At Bank of POLONA Ltd.)
A# show running-config
interface Ethernet 0/0
ip address 192.168.13.1 255.255.255.0
ip summary-address eigrp 100 172.16.80.0 255.255.240.0
A# show ip route
172.16.0.0/16 is variably subnetted, 41 subnets, 3 masks
C 172.16.97.0/24 is directly connected, Loopback0
A. For the summary route to appear in the local routing table, the manual summary should be
configured in the router configuration mode.
B. The routes to be summarized are all local, therefore manual summary should not be used. Auto-
summary should be enabled.
C. The network mask of the summary route does not have a classfull boundary. It should be changed to
class B network mask 255.255.0.0.
D. The summary-route does not include any of the networks in the routing table. The summary-route
and network mask should be changed to 172.16.96.0 255.255.248.0.
E. The summary-route does not include any of the networks in the routing table. The loopback interface
should be configured and assigned with the IP address from the summary-route range.
6. A router is configured with a link-local address using ipv6 address fe80::123 link-local command.
When you ping another link-local address, the router prompts you to provide the source interface. Why?
(Source: Debrief of the Second Troubleshooting At Bank of POLONA Ltd.)
A. This is the default behavior for ping when IPv6 addresses are used. It determines which IPv6 address
to use as the source address.
B. The configured link-local address belongs to the device and not particular interface. This is the only
way for the router to determine which interface and link-local address to ping from.
C. It uses interface information to permit the traffic returning from link-local address in the access list,
in case one is configured on the interface.
D. The router does not know which interface leads to the link-local address you want to ping, therefore
the source interface must be manually set.
7. Once you have ACL list statements entered, you have to apply this ACL in order to filter traffic. When
you do so on an interface or a line you have to use the correct syntax. Given in the list are examples of
statements used. Two of them have invalid syntax. Which two? (Source: Debrief of the Second
A. ipv6 traffic-filter list2 out
B. ipv6 access-class cisco in
C. ip access-class out
D. ipv6 access-class 12 in
E. ip access-group 101 out
F. ipv6 traffic-filter 100 in
8. Which protocol and port are used by TACACS+? (Source: Debrief of the Third Troubleshooting At
Bank of POLONA Ltd.)
A. TCP/47
B. TCP/49
C. UDP/1645
D. UDP/1812
9. Which OSPF router is used for inter-area route summarization? (Source: Debrief of the Third
A. ASBR
B. ABR
C. BACKBONE
D. STUB
10. Which best describes GRE protocol? (Source: Debrief of the Third Troubleshooting At Bank of
POLONA Ltd.)
A. GRE adds new IP header, encapsulates original IP packet and adds GRE header at the end of the IP
packet.
B. GRE adds new IP header, inserts GRE header and encapsulates original IP packet.
C. GRE uses original IP header and adds GRE header at the end of the packet.
D. GRE uses original IP header and inserts GRE header between IP header and payload.
11. Which IPv6 address type is used as the Next-hop address when OSPFv3 installs a route in the routing
table? (Source: Debrief of the Fourth Troubleshooting At Bank of POLONA Ltd.)
A. link-local
B. global
C. site-local
D. private
12. Which OSPF route types are filtered on ABR when totally stubby area is configured? (Select two.)
(Source: Debrief of the Fourth Troubleshooting At Bank of POLONA Ltd.)
A. default route
B. inter-area routes
C. intra-area routes
D. external routes
E. host-routes
Answer Key
1 B
2 B
3 A, D
4 A
5 D, E
6 D
7 C, D
8 B
9 B
10 B
11 A
12 B, D
RADULKO Transport Ltd.
Introduction
You work for SECHNIK Network Ltd. as a network engineer. RADULKO Transport Ltd. is a customer
In this module you will be faced with four challenge labs. Each lab has multiple troubleshooting tickets that
• Solve troubleshooting tasks for the first challenge lab at RADULKO Transport Ltd.
• Solve troubleshooting tasks for the second challenge lab at RADULKO Transport Ltd.
• Solve troubleshooting tasks for the third challenge lab at RADULKO Transport Ltd.
• Solve troubleshooting tasks for the fourth challenge lab at RADULKO Transport Ltd.
Troubleshooting at
Overview
This lesson serves as a debrief for the first troubleshooting lab at RADULKO Transport Ltd.
• Describe how you configured a switch to prevent Layer 2 loops from occurring
• Describe STP troubleshooting and STP stability mechanisms
• Troubleshoot policy based routing
• Describe how you solved the CDP neighbor information exchange issue between two switches
• Troubleshooting spanning-tree protocol
• Troubleshoot CDP and LLDP
You work for SECHNIK Networking Ltd. and RADULKO Transport Ltd. is your company's customer.
Marjorie, customer engineer calls you. She has a number of network issues on her hands and needs your
help with resolving them:
 The customer had a Layer 2 loop in its network yesterday. Marjorie isolated the problem to HQ's switch
SW3 and disconnected the offending cabling. Apparently one employee wanted to have more ports at
his desk and he hooked up a small switch to SW3. Marjorie is asking you if you can look into the matter
and assure her that this will not happen again.
 At the remote location of the distribution center there are special servers that regularly update their
databases through the Internet. After the company bought a firewall and installed it into the corporate
headquarters, the policy was to route all user traffic through the headquarters before going to the
Internet. However, it turned out that the only way to have functional updates for servers at the
distribution center, is to route server traffic directly to the Internet. As a result, Marjorie configured
policy-based routing on DST router. All PC traffic that is destined for the Internet, goes through the
headquarters. All server traffic, which is destined for the Internet, goes directly to the Internet. Well,
everything works, except now PCA cannot access local server—SRV! Fix this without breaking the
existing routing policy!
 Marjorie noticed that even though SW2 is connected to SW3 and that CDP is enabled on both devices,
they do not recognize each other as neighbors. Is this a bug? Fix it if you can.
device. To access remote devices, you can use SSH as described in the Job Aids section.
Example Troubleshooting Flow: A Layer 2 Loop in
the Network
The customer had a Layer 2 loop, caused by the attachment of a rogue device to the SW3 switch.
Since the problem itself is not present any more – the customer’s engineer has removed the offending device
– you can only investigate why the loop was created. You check what STP stability mechanisms are in
place. Enter show running-config | include interface|spanning-tree to verify per-interface spanning-tree
configuration.
The output shows you that:
 From the "edge" keyword you can assume that STP is running in rapid-PVST mode.
 Ranges Ethernet0/0-3 and Ethernet2/0-3 are configured as edge ports: user devices are expected at these
ports.
 Range Ethernet2/0-3 is configured with two STP stability mechanisms, using per-port configuration:
BPDU guard and BPDU filter.
When both BPDU guard and BPDU filter are applied to an interface, BPDU filtering takes precedence,
meaning that BPDU exchange is blocked on the link.
Next you check the documentation for information on access vlan ports. The documentation states that
VLAN 10 is to be used to connect headquarters PCs. To determine the ports included in the VLAN10 enter
show vlan brief | begin 10.
From the previous outputs you saw that all of the ports in VLAN10 are configured as STP edge ports. The
range Ethernet2/0-3 has BPDU guard and BPDU filter mechanisms configured. When both mechanisms are
configured, BPDU filter blocks all BPDU exchange, effectively disabling STP protocol on the link while
keeping the link in the forwarding state. In this way the link is prone to closing a forwarding loop.
Applying the Solution
Administratively shutting down the unused interfaces would certainly prevent the Layer 2 loops. However,
this would also prevent the new, legitimate user devices from participating in the network, so this is not a
proper solution.
Note The customer’s request is answered. You save the configuration on SW3 and inform the
support engineer that the ticket is answered. You document the changes and close the
ticket.
Troubleshooting Spanning-Tree Protocol
STP is a Layer 2 protocol that runs on bridges and switches. The specification for STP is IEEE 802.1D. The
main purpose of STP is to ensure that you do not create loops when you have redundant paths in your
network.
A switch supports these spanning-tree modes and protocols:

 PVST+: This spanning-tree mode is based on the IEEE 802.1D standard and Cisco proprietary
extensions. It is the default spanning-tree mode that is used on all Ethernet port-based VLANs. There is
a separate instance of STP for each VLAN configured.
 PVRST+ or rapid PVST+: This spanning-tree mode is based on RSTP protocol, which is specified in
the IEEE 802.1w standard. RSTP behaves differently to STP in the following ways:
− All switches send a BPDU with its current information every <hello-time> seconds (2 by default),
even if it does not receive any from the root bridge. BPDUs are used as a keep-alive mechanism
between bridges.
− After the forwarding topology has changed, the L2 forwarding table is immediately flushed, as
opposed to STP, which shortens the aging timers. The immediate flushing of the forwarding table
restores connectivity faster, but will cause more flooding.
− RSTP introduces the proposal and agreement mechanism, which does not rely on timers, to
propagate new information across the network. By contrast, STP dictates that a transition of a port
state undergoes several stages, each involving a timer.
 MSTP: This spanning-tree mode is based on the IEEE 802.1s standard. The MSTP runs on top of the
RSTP and allows you to map multiple VLANs to the same spanning-tree instance, thus reducing the
number of needed STP instances.
Troubleshooting each of the STP versions starts before any problems are detected by familiarizing with the
topology and understanding the resulting tree structure of the stable and operational network. The final tree
structure can be influenced at configuration by specifying the following STP parameters:
 Switch priority
− It is common to install a switch in a network without changing the default STP parameter values. In
such case, the bridge priority defaults to 32769. Switch ID, derived from the priority and switch
MAC address. The switch with the lowest ID is elected as a root. If all switches in a single spanning
tree have the same bridge priority, the switch with the lowest MAC address will become the root
bridge. Older switches typically have lower MAC addresses, but such a device may not have
enough memory or CPU power to handle the task of being the root bridge. Also, access switch
should not be elected the root as otherwise traffic from other parts of the network would
unnecessary traverse it. To prevent unpredictable root switch selection, do not leave the STP
priority at the default value.
 Port priority
− STP prevents loops by putting less desirable links into discarding state. With port priority value
kept at default, if there is more than one link between switches, all nonroot ports will be placed in
the discarding state. This would render these uplinks good only for failover, instead of using them
to provide bandwidth along the path. This is especially true when multiple instances of STP
protocol are run, like PVRST+ or MSTP. The port priority could be used to enable traffic sharing.
When changing port priority, you should be careful to apply the change at the relevant switch.
 Path cost
− Even though this feature does not allow for precise and optimal traffic engineering it improves
redundant link utilization in case STP version with multiple instances is used. Traditionally,
spanning tree has used a 16-bit value for the link cost used by bridges for calculating the shortest
path to the root. With these older 16-bit metrics, a 10Mbps link would have a cost of 100 and a
1Gbps link would have a cost of 4. However, link speeds have outgrown these metrics and there are
now a 32-bit "long path cost". With the newer 32-bit metrics, a 1Gbps link would have a cost of
20,000 a 10Gbps link would have a cost of 2,000 and a 100Gbps link would have a cost of 200. To
enable the long path cost on a Cisco switch, simply enter spanning-tree pathcost method long
global configuration command. Problems occur when networks have a mix of switches that use the
16-bit and 32-bit path cost values. Therefore, it is important to be consistent in your configuration
and strive to have all your network devices use the newer 32-bit long path cost metrics.
 Timers
− Convergence in STP relies on propagation of new information across the network. This is ensured
by the use of forward timers, which prevents a port from actively forwarding data until the
forwarding timer expires, allowing for the new information to reach all devices in the network. By
default, the forward timer is 15 seconds. The timer values are usually greater than it is required to
propagate information, rendering network unjustifiably unusable for longer than necessary.
− Large networking environments supporting applications that rely on layer-2 connectivity across the
entire network can experience problems if their topology exceeds STP's maximum dimensions. The
802.1D specifications recommends that a spanning tree have no more than seven bridge hops.
− In MSTP timers such as Hello, ForwardTime, MaxAge can only be tuned for the the IST instance 0.
All other instances (MSTIs) inherit the timers from IST MSTP does not use MaxAge timer to age
out old information, like RSTP/STP do. Instead of this, it uses a special field called MaxHops. IST
root sends BPDUs with hop count equal to MaxHops and every other downstream switch
decrements the hop count field on reception of IST BPDU. As soon as hop count becomes zero, the
information in BPDU is ignored.
Misconfiguration, hardware errors or unexpected topology changes can result in common STP-related
issues:
 Forwarding loops.
 Suboptimal traffic flow.
 Excessive flooding due to the high rate of topology changes.
 Convergence time-related issues.
To verify the spanning-tree mode running on a switch, check the running configuration.
If you want to check the status of the spanning tree elements and parameters’ values, use show spanning-
tree command. It will display the spanning-tree status for all VLANs or MST instances. To check specific
the status for a specific VLAN or MST instance, use show spanning-tree vlan vlan-id or show spanning-
tree mst instance-id.
To verify all the features that are enabled for spanning-tree protocol use show spanning-tree summary.
This will also display the number of blocked, listening, learning, and forwarding interfaces.
To display which VLANs are configured for specific MST instances use command show spanning-tree
mst configuration.
To display spanning-tree events debug spanning-tree events command can be used.
Troubleshooting STP helps to isolate and possibly find the cause for a particular failure, while the
implementation of stability mechanisms is the only way to secure the network against forwarding loops.
PortFast causes a switch port to enter the spanning tree forwarding state immediately, bypassing the
listening and learning states. PortFast can be enabled globally with the command spanning-tree portfast
default or per interface with the interface command spanning-tree portfast. Enabling PortFast globally
will enable PortFast on all nontrunking ports. To verify interface PortFast status use show spanning-tree
interface interface-id portfast.
PortFast BPDUGuard prevents loops by moving a nontrunking port into an errdisable state when a BPDU
is received on that port. You can enable BPDUGuard in global level with spanning-tree portfast
bpduguard default. This will enable BPDUGuard on all PortFast ports. You can enable BPDUGuard on
interface with the command spanning-tree bpduguard enable without also enabling PortFast feature. To
recover interface from errdisable state, you can shutdown and no shutdown or use command errdisable
recovery cause bpduguard to enable switch to automatically tries to recover errdisabled interfaces.
BPDUFilter feature can be enabled globally or per interface. Depending on the configuration method, there
are differences in its operation. When BPDUFilter is enabled globally, by using spanning-tree portfast
bpdufilter default it will enable BPDUFilter on PortFast interfaces. This command will prevent interfaces
from sending or receiving BPDUs. The interfaces still send a few BPDUs when link comes up before the
switch begins to filter outbound BPDUs. If BPDU is received on a PortFast enabled interface, the interface
loses its PortFast operational status and BPDUFilter is disabled.
You can also enable BPDUFilter per interface by using the spanning-tree bpdufilter enable command
without also enabling the PortFast feature. This command will prevent the interface from sending or
receiving BPDUs, which is functionally the same as disabling spanning tree protocol and can results in
spanning-tree loops.
Note BPDUFilter is a very specific tool, tailored for occasions, such as when you need to merge
two Layer 2 domains, using different types of STP, and you need to filter both protocols on
the connecting link.
You can use LoopGuard to prevent alternate or root ports from becoming designated ports because of a
failure that leads to a unidirectional link. You can enable this feature by using the spanning-tree loopguard
default global configuration command.
In switched networks, any switch with the lower bridge ID can become the root bridge. To prevent switches
connected to the specific ports from becoming the root bridge you can use the RootGuard feature. When
Root Guard is enabled on a port and spanning-tree calculation cause an interface to be selected as root port,
the root guard places the interface in the root inconsistent state, which means that the port is blocked. When
BPDUs with higher bridge ID is received again the port is recovered from the root inconsistent state. Enable
root guard on an interface using the spanning-tree guard root command.
Example Troubleshooting Flow: Configuring
Route-Map Causes Local Connectivity Issue
After applying the routing policy on the DST, PCA cannot access the local server SRV.
To verify the problem, you ping the server from the PCA computer. In order to do so, you have to access
PCA via SSH. According to the documentation PCA has an address assigned by the DHCP server running
on the DST router. To learn PCA’s IP address you first connect to DST and issue show ip dhcp binding.
There are two addresses assigned. Try establishing the SSH connection to both addresses until you get the
„PCA#“ prompt.
Then issue ping 10.1.2.10. The ping fails. This confirms the problem and you can start troubleshooting.
Start by collecting information about the routing-related configuration on DST. To check the routing table
on DST issue show ip route. The output shows that there is a default route via HQ2 router and a directly
connected route towards the 209.165.201.0/30 network connecting DST to ISP2. You also know that the
customer has configured policy-based routing on the DST. Therefore, to check how the routing policy is
applied you issue show ip policy command. You learn that route-map „SRV-INET-RM“ is configured and
used to apply the routing policy to Ethernet 0/0.2 sub-interface. Depending on the content of the route-map,
you will check whether this is the correct interface to apply policy routing. You continue by checking the
route-map statements.
Next you issue show route-map SRV-INET-RM to examine the statements of the route-map. The content
of the route map tells you that for all the IP addresses matched by the „SRV-INET“ access list the next-hop
IP is set to 209.165.201.2. According to the documentation, this IP belongs to the ISP, i.e. the traffic is
routed directly to the Internet. To reveal the access-list statements, you enter show ip access-list SRV-
INET. „SRV-INET“ is an extended access-list and contains only one statement with the sequence number
10. It matches all the traffic going from the 10.1.2.0/24 segment. Therefore, all the traffic, regardless its
destination – local or remote, is matched and consequently routed directly to the ISP. This includes ECHO
replies destined to 10.1.10.0/24 segment.
Analyzing Information and Proposing a Hypothesis
The DST’s routing table contains a default route using HQ2 as the next-hop. Besides the default route, there
is a connected route for the ISP2’s network segment. Only traffic inbound on the Ethernet 0/0.2 sub-
interface, which belongs to the 10.1.2.0/24 network segment, is subject to the routing policy. All other
traffic is routed using the routing table as it is.
Based on these facts you propose the hypothesis: the access-list used by the route-map to select the traffic to
be forwarded directly to the ISP, matches all of the traffic sourced from 10.1.2.0/24 segment, regardless of
its destination. Instead, network segments within the range 10.0.0.0/8 should be excluded from this routing
policy rule.
To test the hypothesis add a statement excluding the traffic from the server segment to 10.0.0.0/8 segments,
which include the local network and other headquarters segments. The statement is more specific than the
existing one, therefore it should come before. Enter 5 deny ip 10.1.2.0 0.0.0.255 10.0.0.0 0.255.255.255.
The denied traffic is routed using the entries in the routing table. Therefore you expect that the ECHO
replies from the server to the local PCs PCA or PCB would be routed to the next hop specified in the
routing table, i.e. the next hop is not going to be altered by the route-map statements. From PCA you issue
ping 10.1.2.10. The ping is successful. Since there is no routing policy applied to PCA segment, the traffic
to Internet is routed using the default route, i.e. via HQ2. Tracing the route from PCA to Internet address
confirms this. It remains to verify that traffic from the server to the Internet is routed directly to the ISP2.
First you confirm that there is Internet connectivity from SRV. Then you traceroute from SRV to Internet.
The output shows that after the DST first hop, there is no HQ2, as was the case when tracing the route from
PCA. The * * * mean that no response was received. To see such an output when the next-hop is an Internet
provider is not uncommon. ISPs may block the traceroute packets. The main conclusion here can be derived
from the fact, that the traceroute 209.165.201.133 outputs on PCA and SRV are different in that the SRV’s
output does not contain HQ2 in the list of hops.
Note The problem is solved. Save the configuration on DST and inform the support engineer that
the problem has been fixed. You also document the changes and close the ticket.
Troubleshoot Policy Based Routing
To troubleshoot policy based routing, you should:

 Check path control route-map statement:
− Route maps are executed in the order of the lowest sequence number to the highest. If a match is
found within a route map instance, execution of further route map instances stops.
− Deny route-map statement means that the traffic is not policy routed, but routed normally.
− Permit route-map statement – when the traffic matches the criteria, the set commands are applied.
 The default is a permit statement.
− At the end of route map statements, there is an implicit route map statement that denies all packets.
If the packet has not found a match it will be forwarded by the router following the normal route
table.
 Check traffic-matching configuration:
− When ACLs or prefix-lists are used to define policy routed traffic, verify ACLs in order to
understand what traffic is policy routed.
 If a packet matches an ACL’s or prefix-list deny statement, it is considered as
nonmatching by the route map; consequently, the packet is passed to the next route-map
statement.
− If more than one ACL or prefix-list is specified, a packet matching any of them, results in a route-
map match.
− If there is no match statement in the route map instance or if ACL referred by the match statement is
missing, all packets are matched. The set statement will apply to all packets.
 Check actions for the matched traffic:
− Understand the set statements applied.
− When there are multiple set statements, all of them are applied if in a predefined order: set ip
{precedence | tos}, set ip next-hop ip_address, set interface interface_name, set ip default next-
hop ip_address, set default interface interface_name.
 Check how the route-map is applied:
− Policy routing only works on inbound packets; therefore, it must be applied on the interface
receiving the traffic to be policy routed.
 Check the interface on which the policy routing is enabled.
 The correct syntax is ip policy route-map map-tag, configured under interface
configuration mode.
− To policy route local, router-generated traffic:
 Check that local policy routing is in place.
 The correct syntax is ip local policy route-map map-tag, configured under global
configuration mode.
 Verify path control results.
Example Troubleshooting Flow: CDP Neighboring
Issue
The customer has informed you that even though SW2 is connected to SW3 and CDP is enabled on both
devices, they do not recognize each other as neighbors.
In order to verify the problem issue show cdp neighbor on SW1 and SW2. Output on both devices
confirms that SW2 and SW3 do not recognize each other as neighbors. The problem is confirmed you can
start troubleshooting. The customer claims that CDP is enabled on both devices.
On switch SW2 issue show cdp interface | include Ethernet1/0. Repeat the command on SW3 for Ethernet
1/1. There is no output, meaning that SW3 does not have CDP protocol enabled. To vary the diagnostic
commands, you could issue show running-config | include Ethernet1/1|cdp on SW3. It would include the
explicit statement no cdp enable.
You have enough information to propose a hypothesis: CDP is not enabled on both interfaces used to
interconnect switches. To test the hypothesis, enable CDP and check CDP neighbors on both devices.
To test the hypothesis enable CDP on SW3 by entering cdp enable command in the interface Ethernet1/1
configuration mode. Now, you can control if this configuration change solves the issue. Check the CDP
neighbors table on both switches. SW2 and SW3 now know of each other.
Note The problem is solved. You save the configuration on SW3 and inform the support engineer
that the problem has been fixed. You also document the changes and close the ticket.
Troubleshooting CDP And LLDP
The most common issue with CDP or LLDP protocol is that a device has no CDP/LLDP neighbor.
To troubleshoot, check:
 Whether devices are Cisco devices or there are also other vendor devices in the network.
− If the network is multi-vendor, you should use LLDP protocol.
 Whether device is configured to exchange CDP/LLDP messages on the interfaces connecting it to other
devices:
− To enable CDP/LLDP on all interfaces use global configuration command cdp run, or lldp run,
depending on the protocol of choice.
 Use the no version of the command to disable CDP on all interfaces.
− To enable CDP/LLDP on a particular interface, in that interface configuration mode use cdp enable
or lldp enable.
 Use the no version of the command to disable CDP on an interface.
 Check timers values.
− It is possible to configure the CDP/LLDP holdtime to be less than the update timer, in which case
the device will lose CDP/LLDP adjacency.
 Useful commands:
− show cdp, show lldp - display global protocol information, including timer and hold-time
information, CDP version.
− show cdp entry, show lldp entry - display information about a specific neighboring device
discovered using Cisco Discovery Protocol, including device ID, protocols and addresses, platform,
interface, hold time, and version.
− show cdp interface, show lldp interface - display information about the interfaces on which the
protocol is enabled, including status information and information about timer and hold-time.
− show cdp neighbors, show lldp neighbors - display detailed information about neighboring
devices, including the type of the device, its name, and MAC address or serial number, local
interconnecting interface, remaining holdtime interval, product number, and neighbor’s
interconnecting interface and port number.
 show cdp neighbors detail, show lldp neighbors detail - display additional detail about
neighbors, including network addresses, enabled protocols, and software version.
− show cdp traffic, show lldp traffic - display information about traffic between devices, such as the
total number of packets sent and received, advertisements per version.
− debug cdp, debug lldp - display the protocol messages exchange in real time.
Summary
Overview
This lesson serves as a debrief for the second troubleshooting lab at RADULKO Transport Ltd.

• Describe how you solved connectivity to the Internet from PC
• Troubleshoot VTP
• Describe how you solved IPv6 connectivity issue
• Troubleshoot EIGRP for IPv6
• Describe how you solved MP-BGP issue
• Troubleshoot BGP for IPv6

It's Sunday and Marjorie, the customer's engineer, calls in panic. She wants you to look at the e-mail she has
just sent you.
Note From: marjorie

Subject: Network down
SW2 got stolen so I had to replace it with an old switch that I've got from our storage unit. I
copy-and-pasted configuration from backup. However, PC1 and PC2 now do not have
connectivity to the Internet. I have no clue what happened and I need it to work before
Monday! Please fix this!!
I've also just configured the network to run IPv6 as well. Everything seems to be fine except I
cannot ping the IPv6 Internet from the BR router and MP-BGP only seems to be working
through ISP1 while the session with ISP2 is not being fully established. Could you please
take a look?
Thank you.
Marjorie
branch device. To access remote devices, you can use SSH as described in the Job
Aids section.
Note If you need to test IPv6 Internet connectivity, use IPv6 address of 2001:DB8:0:D::100.
Example Troubleshooting Flow: PC1 and PC2 Do
Not Have Connectivity to the Internet
The customer engineer told you that SW2 was stolen. After switch was replaced with a new switch, PC1
and PC2 lost connectivity to the Internet. She asked you to fix the issue.
Verify connectivity to the Internet from PC1 and PC2. As you can see ping is not successful. The problem is
confirmed and you can start troubleshooting.
In the verification phase you saw in the ping output that no protocol is running. You check the status on the
interface on PC1 and PC2. As you can see from the output, interfaces should get the IP address from the
DHCP server, but no address was assigned.
Check the documentation about LAN topology. You can see that routers HQ1 and HQ2 should terminate the
VLANs. You assume that the router HQ1 is primary HSRP router and also DHCP server. Check the
interfaces and DHCP configuration on the routers.
You were able to see in the documentation that PCs should be connected to subnet 10.0.10.0/24. The
interface Ethernet 0/0.10 is connected to this subnet on the router HQ1. As you can see from the output the
interface is up and operational.
Check the configuration on the interface Ethernet 0/0.10. You can see that the subinterface is connected to
VLAN 10. Based on the output you can conclude that interface is correctly configured.
In the next step check the DHCP configuration.
As you can see from the output DHCP server is correctly configured for the LAN segment. You can
conclude that DHCP server settings and IP addresses are correctly configured on the router HQ1. Since PCs
are not able to get IP addresses from DHCP server, you assume that there is L2 connectivity issue from PCs
to DHCP server.
You should move your troubleshooting focus to switches.
Check the VLAN database. You can see that VLAN 10 is not present. You are also missing VLAN 2 and
VLAN 100, which is mentioned in the documentation. Check switch SW2 and SW3 to check which VLANs
are configured on those two switches. You will be able to see that the same VLAN database is present on
those two switches.
Since all databases are synchronized and wrong on SW1, SW2 and SW3, you assume that VTP is used to
configure VLANs in the LAN network. Most probably something went wrong with VTP when new switch
SW2 was installed in the network. Therefore check VTP status on the switches.
Check VTP status with the command show vtp status. You can see that VTP mode is server and VTP
domain is "cisco". Next step is to check the VTP status on the SW1.
You can see that switch SW1 is also configured as VTP server with the VTP domain "cisco". The revision
number is the same on both switches. This means that switches have VLAN information synchronized
through VTP.
You assume that when switch SW2 was connected to the network it pushed VLAN configuration with the
VTP protocol. There are two possible causes for this to happen. Either SW1 and SW3 had the same VTP
domain and password configured as SW2 and the revision number on SW2 was higher or SW1 and SW3
were configured with the default VTP domain name, so SW2 pushed VTP configuration and VLAN
database to both switches.
To fix the issue, delete unused VLANs from the database and add all needed VLANs. This can be done on
any switch since all information will be pushed to other two switches. Since it is best practice not to use
VTP, you should also change the VTP operation mode to transparent.
Deploying the Solution
Remove all VLANs that are not needed on LAN network and add VLANs that are needed. Since VTP is
configured on the network, you can do this on only one switch and all information will be propagated to
other two switches.
After VLAN database was repaired, you should also change the VTP mode to transparent. In this mode VTP
information will not be processed by the switch, but switch will forward all VTP information.
Testing the Solution
To verify if the problem is solved, check interface status on the PC1. As you can see the IP address was
received by the interface. You should also check connectivity to the Internet. Connectivity test is successful.
Therefore you can conclude that the problem was solved.
Note Save running configurations on SW1, SW2, and SW3.
Troubleshooting VTP
VTP reduces administration in the switched network. When new VLAN is configured on a VTP server, this
information is propagated through all switches in the VTP domain. This reduces the need to configure the
same VLAN on every switch. It is a Cisco proprietary protocol. VTP works only over trunk interfaces.
Switch can operate in following VTP modes:

 Server – VLANs can be created, modified and deleted on VTP server. VTP servers also advertise their
VLAN configuration to other switches in the same VTP domain and synchronize their VLAN
configuration with other switches. This is the default mode.
 Client – VTP clients behave in the same way as VTP servers, except that you cannot create, change or
delete VLANs.
 Transparent – VTP transparent switches do not participate in VTP, which means that they do not
synchronize their VLAN configuration. However VTP advertisements are forwarded to other switches.
 Off – It behaves in the same way as in VTP transparent mode, but VTP advertisements are not
forwarded. This mode is available with VTP version 3.
You must be very careful when installing a new switch in the network that uses VTP. If the VTP revision
number on the new switch is higher than current revision number and VTP domain is the same, the VLAN
information is propagated from the new switch to the whole network. If new switch’s VLAN configuration
is incorrect, all switches will receive the wrong configuration and switched network could become non-
operational. Also if network switches have the default VTP configuration and a new switch is installed with
VTP domain configured, VTP protocol will push VTP configuration and VLAN database to all other
switches. If VLAN configuration is not correct, the whole switched network could become non-operational.
Therefore it is best practice not to use VTP in the network. Every switch should be configured in VTP
transparent mode or VTP should be turned off.
If you decide to use VTP, double-check VTP revision number on new switch. Best practice is to change
VTP domain to something and back to your VTP domain before installing new switch to network. This will
reset revision number to zero.
Several issues are possible when using VTP:

 VTP version mismatch.
 Authentication mismatch.
 Non-operational trunk line.
Use command show vlan to check VLAN database on the switch. Use this command to check if correct
VLANs are configured in the database.
To check the status of the VTP use command show vtp status. You are able to see VTP version, revision
number, VTP operational mode, VTP domain etc. Use this command to begin troubleshooting VTP.
Example Troubleshooting Flow: BR Does Not
Have Connectivity to the Internet via IPv6
Customer engineer configured IPv6 in the network, but she then reported that there is no IPv6 connectivity
to the Internet from BR router.
First verify the problem. Based on the documentation you should use IP address 10.2.10.1 to SSH to BR
from HQ1. After trying SSH connectivity you discover that you are not able to connect. This is probably
related to non-operational routing. You check the documentation again and you discovered that BR is
directly connected with the IP address 10.255.0.2. You tried to SSH to this IP address instead and you are
now able to connect.
You should first check why you are not able to connect to the IP address 10.2.10.1. Check IPv4 EIGRP
neighbors on the router BR to discover if EIGRP is operational. You can see that there are no EIGRP
neighbors, therefore you verify the EIGRP configuration. Based on the output it seems that configuration is
correct. Investigate configuration on interface Ethernet 0/1.
Check interface Ethernet 0/1 configuration—EIGRP is enabled end EIGRP authentication is used. You
assume that there are issues with authentication. Check EIGRP keys.
Check the key chain that is used for EIGRP authentication with the command show key chain. You can see
that key 1 with password "c1sc0" is used to authenticate EIGRP neighbors. This key is only accepted as a
valid key from November 11 2013 11:11:11 and is valid for 1111 seconds.
Check the current time with the command show clock. As you can see the key is not valid any more. You
can conclude that this is probably the reason that EIGRP adjacency is not established. Remove the accept
lifetime from the configuration.
Implementing the Solution
Remove accept lifetime from the key chain on the router BR. You can see that as soon as accept lifetime
was removed the adjacency came up. You assume that connectivity was restored, therefore you test SSH
connectivity from the router HQ1. The connectivity is successful. You can now move you focus to IPv6
Internet connectivity issue.
After successful connection you should try IPv6 connectivity from the router BR. As you can see there is no
IPv6 connectivity to the Internet. The reason for the failed ping is that router do not have the valid route for
the destination.
Check the routing table on the BR router. As you can see only connected routes are present in the routing
table. Check the documentation to see which routing protocol is used in the internal network to propagate
the IPv6 routes.
Check the documentation. You can see that EIGRP should be used to route IPv6 traffic between branch and
headquarters. The interfaces that are used to connect BR and HQ1 are Ethernet 0/1 on BR and Ethernet 0/2
on HQ1. Your next step is to verify if EIGRP is running on the router BR and HQ1.
Use command show ipv6 protocols to check routing protocols that are running on the router BR. You can
see that EIGRP is enabled and it is running on interfaces Ethernet 0/0 and Ethernet 0/1, which is correct.
Use the same command on the router HQ1. You can see that EIGRP is enabled but it is not running on
interface Ethernet 0/2. You should enable EIGRP on the interface.
Enable EIGRP on the interface Ethernet 0/2 using command ipv6 eigrp 1. You can see that the adjacency
forms. You assume that IPv6 connectivity was restored. To test if your solution solved the problem SSH to
the BR and check IPv6 connectivity to the Internet.
Note As you can see there is connectivity. You can conclude that IPv6 connectivity was restored
and that problem is solved. Save the configuration on HQ1 and BR. Close the ticket.
Troubleshooting EIGRP for IPv6
Configuring EIGRP for IPv6 is very similar to configuring EIGRP for IPv4. The main difference is that
EIGRP is enabled on the interface for IPv6 with the command ipv6 eigrp as-number. Therefore
troubleshooting EIGRP for IPv6 is very similar to that for IPv4.
To check IPv6 routing protocols on the router use command show ipv6 protocols. The output will show
IPv6 routing protocols that are enabled on the router. The EIGRP section shows metric weights, router id,
EIGRP interfaces, redistribution information, etc.
To check the IPv6 EIGRP neighbor status use the command show ipv6 eigrp neighbors. All EIGRP
neighbors are listed, with some useful information like interface used to connect the neighbor, timers etc.
Use show ipv6 eigrp interfaces to display detailed information about interfaces participating in EIGRP for
IPv6.
To verify topology table use command show ipv6 eigrp topology. You can see all routing updates received
by the router, with AD and FD information, next-hop etc.
To observe EIGRP events in the console use debug ipv6 eigrp.
Example Troubleshooting Flow: IPv6 BGP Is Not
Established to the ISP2
Customer engineer told you that MP-BGP session is not established with ISP2. You should check the
problem.
Use command show bgp all summary to check BGP neighbors on the router HQ2. As you can see from the
output only one neighbor is up for IPv6 address family. You checked the documentation and found that
RADULKO Transport Ltd. is using the AS number 65000, therefore you conclude that this is the IBGP
session.
Check the BGP configuration on the router HQ2. As you can see the neighbor 2001:DB8:0:11C::1 is
specified but it is not activated under address-family IPv6. To test you hypothesis activate neighbor under
address family IPv6.
Testing Hypothesis
Enable neighbor under address-family IPv6 configuration. As you can see the BGP neighbor came up. This
can take some time.
After using commands show bgp all summary, you are able to see that neighboring relation is established
between HQ2 and ISP2, as the customer engineer requested. The problem is fixed.
Note Save the configuration on the router HQ2 and inform customer engineer that the problems
were solved.
Troubleshooting MP-BGP
Multiprotocol BGP extensions for IPv6 support the same features and functionality as IPv4 BGP. IPv6
enhancements to multiprotocol BGP include support for an IPv6 address family and NLRI and next hop (the
next router in the path to the destination) attributes that use IPv6 addresses.
To configure IPv6 specific tasks IPv6 address family was introduced in the router configuration mode. Use
address-family ipv6 to enter unicast IPv6 address family and exit to exit address family configuration.
By default, neighbors that are defined using the neighbor remote-as command in router configuration
mode exchange only IPv4 unicast address prefixes. To exchange IPv6 prefixes, neighbors must also be
activated using the neighbor activate command in address family configuration mode for IPv6 prefixes.
Configuring IPv6 multiprotocol BGP between two IPv6 routers (peers) using link-local addresses requires
that the interface for the neighbor be identified by using the update-source command and that a route map
be configured to set an IPv6 global next hop. If peering is not established by this task, it may be because of
a missing route-map set ipv6 next-hop command. Use the debug bgp ipv6 unicast update command to
display debugging information on the updates to help determine the state of the peering.
To inject a network into IPv6 BGP database, you must define the network using the network command in
address family configuration mode for the IPv6 BGP database.
By default, route maps that are applied in router configuration mode using the neighbor route-
map command are applied to only IPv4 unicast address prefixes. Route maps for IPv6 address family must
be applied in address family configuration mode using the neighbor route-map command. The route maps
are applied either as the inbound or outbound routing policy for neighbors under the IPv6 address family.
Redistribution is the process of redistributing, or injecting, prefixes from one routing protocol into another
routing protocol. Prefixes that are redistributed into IPv6 multiprotocol BGP using the redistribute router
configuration command are injected into the IPv6 unicast database.
Some useful commands to troubleshoot IPv6 BGP are:
 clear bgp ipv6 unicast * - Use this command to reset all IPv6 BGP sessions on the router. You can use
several other keywords like IP address of the neighbor, AS number etc. instead of asterisk to make some
more specific clearing.
 show bgp ipv6 unicast – Use this command to display IPv6 BGP table. You can see IPv6 prefixes with
next-hop address, local preference, metric, as-path etc.
 show bgp ipv6 unicast summary – Use this command to verify all IPv6 BGP connections.
 debug bgp ipv6 unicast updates – This command will enable debugging for all update packets
received and sent by the router.
Summary
Overview
This lesson serves as a debrief for the third troubleshooting lab at the RADULKO Transport Ltd.
• Describe how you solved the problem with the lost connectivity for host in the central office
• Troubleshoot address families in OSPFv3
• Describe how you solved the problem with OSPFv3 authentication
Company network policy at RADULKO Transport Ltd. has changed. Now, they must not use proprietary
protocols in their network. Over the weekend your colleague Peter reconfigured customer's routing. Instead
of EIGRP, the network is running OSPF now.
While at the customer's site, Peter calls you. It is not all going according to plan and he needs your help:
 PC1 does not have connectivity to the distribution center server—SRV (10.1.2.10) under normal
conditions. Connectivity is only restored if HQ1 fails! Peter has a strong suspicion that this has
something to do with the HSRP implementation. Help him solve this mystery.
 Peter reports that migration from EIGRP to OSPF is going smoothly, except the authentication part. He
tried to configure authentication between HQ1 and BR routers, however after implementing
authentication, the adjacency was lost. Fix OSPF authentication between HQ1 and BR. Peter will look
at this example and implement it on other OSPF peers.
branch device. To access remote devices, you can use SSH as described in the Job
Aids section.
Connectivity
PC1 does not have connectivity to the distribution center server—SRV (10.1.2.10) under normal conditions,
connectivity is only restored if HQ1 fails. Connectivity is only restored if HQ1 fails!
With the use of the ping tool you can confirm that the remote server SRV is not reachable if HQ1 is up and
fully operational. To verify the second part of the reported issue, you have to find a more practical way to
exclude the router HQ1 from the participation in the forwarding process then actually shutting it down
completely.
According to the Job Aids, routers HQ1 and HQ2 are part of the HSRP process. As you check the routers‘
configurations you can see that router could be degraded by shutting down its uplink. With that HQ1 would
have a lower priority and would become a standby router.
Successful ping from PC1 confirms that server SRV is operational and reachable from the Headquarters if
the route goes via router HQ2.
The reason that remote server is reachable from the Headquarters via router HQ2, but not router HQ1, most
probably lies in the internal routing. Since you can compare a working and a nonworking situation, first
revert the change and try to spot the difference between HQ1 and HQ2 routing information.
As you can see the route for 10.1.2.0/24 is missing in the router HQ1‘s routing table. On the other hand the
router HQ2 has that route installed in its routing table. HQ2 learned it via OSPFv3 which is in accordance to
the provided Job Aids.
If router HQ2 knows about network 10.1.2.0/24, this specific routing information should be propagated to
the directly connected OSPFv3 neighbor HQ1. Since that obviously did not happen, check the HQ2‘s
OSPFv3 neighbors using show ospfv3 neighbor command. As you can see HQ1 is not listed among them,
which is not in accordance to Job Aids.
You should now proceed to investigate why adjacency is not forming.
You can notice that HQ2 is periodically sending Hello messages through interfaces Ethernet 0/1 and
Ethernet 0/0.2, but from the opposite direction there are none received. Since HQ1 should be sending Hello
messages on that particular interface, you can conclude that OSPFv3 at HQ1 must be misconfigured.
Testing Hypothesis
Going with „spot-the-differences“ approach, you can see that HQ1 has interface configured for OSPFv3
only for IPv6 address family.
Standby router HQ2 has both IP address families enabled on the interface. That means, that it is listening
and sending Hello messages on that particular interface. Since HQ1 is not sending any, adjacency will not
be formed.
Implementing and Verifying Solution
After configuring OSPFv3 on interface Ethernet 0/1 for address family IPv4, connectivity to the remote
server SRV was restored.
Note Save the configuration on router HQ1. Inform the support engineer that the problem has
been fixed, document the changes and close the ticket.
Troubleshooting OSPFv3 Address Families
Feature
OSPFv3 is a link state routing protocol. Unlike OSPFv2, which supports IPv4, the original implementation
of OSPFv3 supports IPv6 (unicast only). With the introduction of OSPFv3 support for AFs, OSPFv3 now
can support IPv4 unicast address families.
The Address Families feature maps an address family to a separate OSPFv3 instance, using the Instance ID
field in the packet header. Each OSPFv3 instance maintains its own adjacencies, link state database, and
shortest path computation.
OSPFv3 runs on IPv6 and it uses IPv6 link-local addresses as the source of hello packets and next-hop
calculations. To use the IPv4 unicast AF in OSPFv3, you must enable IPv6 on a link, although the link may
not be participating in IPv6 unicast AF.
OSPFv3 makes use of IPsec and therefore has a broader range of supported authentication algorithms than
OSPFv2 and also supports encryption.
To check address family IPv4 OSPFv3 entries in the routing table use show ip route ospfv3 command. For
address family IPv6 use show ipv6 route ospf command instead.
Global OSPFv3 configuration can be retrieved with show running-config | section router ospfv3
command and specific interface configuration with show running-config | section interface.
To check general information about routing process use show ospfv3 command. For more specific
information regarding involved interfaces or neighbors‘ adjacency use show ospfv3 interface and show
ospfv3 neighbor commands respectively.
To start displaying information about OSPFv3 events use debug ospfv3 command.
Example Troubleshooting Flow: Authentication
Problem
HQ1 and BR routers' OSPFv3 adjacency was lost after authentication between them was set by Peter.
A periodic CLI event notification confirms Peter‘s suspicion that he misconfigured OSPFv3 authentication
on HQ1-BR segment. You additionally verify that OSPF adjacency between those two routers did not form.
Since misconfiguration is suspected, you should straightforwardly compare OSPFv3 authentication
configuration on both routers.
Analyzing the Information and Testing Hypothesis
As you can see remote accessing the BR router using given IP address from the Job Aids is not successful.
You have a feeling that given IP address is a part of one of the Branch‘s internal networks, which cannot be
known by router HQ1 without an established OSPFv3 adjacency. A quick search in the routing table
confirms your suspicion.
You can find BR‘s directly connected IP address in the CDP database, which hopefully you can use for
remote access.
Testing Hypothesis
From the output you can conclude that there are different hashing algorithms configured at peering OSPFv3
interfaces. You decide to use stronger SHA1 for peer authentication, therefore you need to change
configuration on router BR.
Implementing and Verifying Solution
After configuring SHA1 based OSPFv3 authentication on router BR, neighbor adjacency was restored.
Note Save the configuration on router BR. Inform the support engineer that the problem has been
fixed, document the changes and close the ticket.
When choosing between MD5 and SHA-1 hashing algorithm, use SHA-1, since it is stronger.
Summary
Overview
This lesson servers as debrief for the forth troubleshooting lab at RADULKO Transport Ltd.
• Describe how you solved external OSPF routes issue
• Describe how you solved IPv6 connectivity from the PCs
Marjorie, the customer's engineer has two issues that she is unable to resolve and needs your help with:
 Marjorie noticed that router DST has learned a bunch of external routes through OSPF. These IPs are
that of public address space and, to her knowledge, do not belong to RADULKO Transport Ltd. address
space. These routes should not be seen on DST.
 PC1 and PC2 are unable to access IPv6 Internet. Marjorie has a strong suspicion that the problem lies in
neighbor discovery protocol. As the solution she suggests implementation of HSRP for IPv6!
device. To access remote devices, you can use SSH as described in the Job Aids section.
Example Troubleshooting Flow: External OSPF
Routes on Router DST
The customer engineer told you that she noticed a bunch of external OSPF routes that should not be in the
routing table. She asked you to fix the problem.
Check the routing table on the DST to verify if the problem really exists. You can see that there are many
external OSPF routes in the routing table. Problem is confirmed and you can start troubleshooting the issue.
First check the OSPF configuration on the DST. As you can see, OSPFv3 is configured on the router and it
is used to transport IPv4 routes as well. Check OSPFv3 neighbors to discover all possible sources of the
external routes. As you can see the only OSPFv3 neighbor is the router with OSPF ID 209.165.201.5. Based
on the documentation, this is the HQ2 router.
The next step is to check OSPFv3 on the router HQ2.
First check OSPFv3 status on the router HQ2. Among other things you can see that OSPFv3 is
redistributing routes from BGP protocol. This is not a good practice since scalability issues can arise,
because BGP tables can be huge and OSPF is not designed to perform SPF algorithm calculations on such
amount of routes. Redistribution must be removed from the configuration. You should also check if the
similar redistribution exists on the router HQ1, which also has BGP peering with the ISP.
Check the OSPFv3 configuration on the router HQ1. You can see that HQ1 is doing redistribution from
BGP to OSPF as well. This must be removed from both routers. Therefore your next step is to remove
redistribution and check the routing table on the router DST.
Remove the redistribution from the OSPFv3 configuration on both routers HQ1 and HQ2. Check the routing
table on the router DST.
After implementing the solution, check the routing table on the router DST. You can see that there are no
OSPFv3 external routes, that were present before. Therefore you can conclude that the problem was solved.
It is also important that by removing the redistribution command you did not destroy Internet connectivity
for distribution center devices. You should also verify Internet connectivity from end-devices of the
distribution center.
Note Save the configuration and close the ticket.
Example Troubleshooting Flow: PC1 and PC2
Cannot Access the Internet via IPv6
Customer engineer told you that PC1 and PC2 are unable to access IPv6 Internet. She thinks that the
problem lies in neighbor discovery and that HSRP for IPv6 will solve the problem.
First verify the problem by issuing the ping command. You can see that Internet is not reachable from PC1
and PC2. The output of the ping is letter „N“, which means that something is wrong with the neighbor
discovery mechanisms.
Check interface status on both PCs. You can see that interface Ethernet 0/0 has only link-local address, but
no global address. This is the reason why the Internet is not reachable, since link-local IPv6 address is only
valid on the link and cannot pass L3 device. You should troubleshoot why there are no global IPv6 address
on the interface. First check interface configuration on the PC1.
Check the interface configuration on the PC1. You can see that IPv6 address should be configured with
autoconfig.
To perform stateless address autoconfiguration, PCs must receive router advertisements from the first-hop
router. Router advertisements include /64 IPv6 prefix, which is used as network portion of the IPv6 address
on the host. The host portion is generated by the host itself. Apparently something is wrong with router
advertisements, therefore you should check IPv6 on the first-hop routers, which, according to the
documentation, are HQ1 and HQ2.
Check the interface Ethernet 0/0.10 on the routers HQ1 and HQ2. You can see from the output that subnet
mask /69 is configured on the interface. It is requirement for SLAAC to use a /64 masks. Therefore you
should correct the masks on the interface.
Correct the IPv6 addresses on the interface Ethernet 0/0.10 on both routers. IPv6 address should be removed
first and new IPv6 address should be added. The reason for removal is that there can be many IPv6
addresses on the interface. If you use command ipv6 address the old IPv6 address would stay on the
interface and new IPv6 address will be added.
To verify if your hypothesis was correct. Check the IPv6 address and Internet connectivity on both PCs.
First check if PC1 autoconfigures a global IPv6 address on the interface. You can see that the same /64
prefix is used as on the routers. To test if Internet connectivity was restored use ping command. You can see
that the ping was successful. You should repeat the same procedure on the PC2.
Note Since connectivity was restored, save the configurations, inform the customer engineer that
the problem is solved. You should also explain to the customer support engineer that the
HSRP for IPv6 is not necessary since both routers are configured to send Router
Advertisements offering all information required by the end stations. In case one router fails,
another will continue sending RA and acting as a default gateway. You can close the ticket.
Summary
Overview
Check
1. Refer to the exhibit. Presuming that policy based routing is correctly applied to the interface, what next-
hop IP address would be used when forwarding the traffic from the device with assigned IP address of
10.1.2.10? (Source: Debrief of the First Troubleshooting At RADULKO Transport Ltd.)
!
route-map CONTROL-POINT permit 10
match ip address PRB1
set ip next-hop 209.165.201.2
!
Extended IP access list PRB1
10 deny ip 10.1.2.0 0.0.0.255 any
!
A. The 209.165.201.2 IP address.
B. Next hop determined by the routing table, as the result of matching the route-map statement number
10.
C. The packet will be dropped.
D. Next hop determined in the routing table, as a result of matching the implicit deny route-map
statement at the end of the route map.
2. SW1 and SW2 switches have the portfast feature enabled globally. Their Ethernet 0/1 ports are
configured as access ports. At the future stage, they are going to be used to interconnect the switches
and both will be configured as trunk ports. What will be their portfast status then? (Source: Debrief of
the First Troubleshooting At RADULKO Transport Ltd.)
A. They will lose portfast status once they are connected and start sending BPDUs.
B. They will lose portfast status the moment they are converted to trunk.
C. They will keep their port status until BPDU guard or BPDU filter features are configured.
3. Which two features apply to MSTP? (Choose two.) (Source: Debrief of the First Troubleshooting At
RADULKO Transport Ltd.)
A. It groups a set of instances to a single VLAN.
B. It can group a set of VLANs to a single spanning-tree instance
C. A failure in one instance can cause a failure in another instance.
D. The total number of spanning-tree instances should match the number of redundant switch paths.
E. It is fully backward compatible with other versions of STP.
4. Which switchport mode must be used to propagate VTP information? (Source: Debrief of the Second
Troubleshooting At RADULKO Transport Ltd.)
A. access
B. trunk
C. etherchannel
D. none of above
5. How do you configure a specific interface for EIGRP in IPv6? (Source: Debrief of the Second
A. You configure it in global configuration mode.
B. It is configured under EIGRP configuration with the network command.
C. It is configured in interface configuration mode.
D. None of above.
6. Which command is used to display the IPv6 BGP table? (Source: Debrief of the Second
A. show ip bgp
B. show ipv6 bgp
C. show bgp ipv6 unicast
D. show bgp ipv6 summary
7. How can you include an interface into an OSPFv3 process for IPv4 address family? (Source: Debrief of
the Third Troubleshooting At RADULKO Transport Ltd.)
A. Using the global network command.
B. Using the ospfv3 processid ipv4 area areaid command on the interface, after first enabling IPv6 on
it.
C. Using the ospfv3 processid ipv4 area areaid command on the interface, after first disabling IPv6 on
it.
D. Using the global OSPFv3 configuration interface ipv4 area areaid command.
8. What can be concluded from the following output? (Source: Debrief of the Third Troubleshooting At
RADULKO Transport Ltd.)
HQ1# show ospfv3 interface brief
Interface PID Area AF Cost State Nbrs F/C
Et0/1 1 0 IPv4 10 BDR 1/1
Et0/2 1 0 IPv4 10 DR 1/1
Et0/2 1 0 IPv6 10 DR 1/1
Et0/3 1 0 IPv6 10 DR 1/1
A. Ethernet 0/2 interface has both IPv4 and IPv6 enabled.
B. Ethernet 0/1 interface has IPv6 disabled.
C. Router has only one neighbor adjacency for both address families on the Ethernet 0/2 interface.
D. Because all neighbors are in the same area – area 0, SPF recalculation in both address families will
occur for every adjacency state change.
E. Ethernet 0/3 interface must have IPv4 enabled because OSPFv3 runs over IPv4.
9. Which CLI commands will enable OSPFv3 authentication? (Select two.) (Source: Debrief of the Third
A. ospfv3 message-digest-key 1 md5 c1sc0
B. ospfv3 authentication ipsec spi 500 sha1 123456789A123456789B123456789C123456789D
C. area 0 authentication ipsec spi 1000 md5 1234567890ABCDEF1234567890ABCDEF
D. ospfv3 ipv4 authentication ipsec spi 500 md5 123456789A123456789B123456789C12
E. ospfv3 ipv6 authentication ipsec spi 501 md5 A123456789A123456789B123456789C1
10. Which LSA type is used to advertise external routes in OSPFv3? (Source: Debrief of the Fourth
A. Type 1
B. Type 2
C. Type 3
D. Type 4
E. Type 5
11. Which IP address is used as destination IP address for Hello messages when OSPFv3 is used to transfer
IPv4 routes? (Source: Debrief of the Fourth Troubleshooting At RADULKO Transport Ltd.)
A. 224.0.0.5
B. 224.0.0.6
C. FF02::5
D. FF02::56
12. Which command is used to display IPv6 – MAC mappings on the router? (Source: Debrief of the Fourth
A. show arp
B. show ip arp
C. show ipv6 neighbors
D. show ipv6 mac
Answer Key
1 D
2 B
3 B, D
4 B
5 C
6 C
7 B
8 A
9 B, C
10 E
11 C
12 C

Tshoot SG

Enviado por

Dados do documento

Título original

Direitos autorais

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Tshoot SG

Enviado por

Direitos autorais:

TSHOOT

Americas Headquarters Asia Pacific Headquarters Europe Headquarters

© 2014 Cisco Systems, Inc.

Upon completing this module, you will be able to:

Upon completing this lesson, you will be able to:

A troubleshooting method that is commonly deployed by both inexperienced and experienced

What are the drawbacks of this method?

What are the possible approaches for troubleshooting this issue?

Upon completing this lesson, you will be able to:

Upon completing this lesson, you will be able to:

R1(config)# username cisco privilege 15 password 0 cisco123

How many bits in the modulus [512]: 768

R1(config)# ip ssh time-out 120

R1# copy flash: scp:

Devices management IP addresses

DSW1 192.168.50.253 cisco/cisco

ASW1 192.168.50.100 cisco/cisco

ASW2 192.168.50.110 cisco/cisco

Maintaining a Documented Network

Step 1 From PC1, connect to R1 using telnet.

PC1# telnet 10.10.20.1

User Access Verification

ROM: Bootstrap program is Linux

Linux Unix (Intel-x86) processor with 265566K bytes of memory.

Configuration register is 0x0

Step 3 Investigate R1's CDP neighbors.

R1# show cdp neighbors

Device ID Local Intrfce Holdtme Capability Platform Port ID

Cisco Discovery Protocol

R1# show arp

Address Resolution Protocol

Step 5 On R1, list interfaces descriptions.

R1# show interfaces description

PC1# telnet 192.168.10.253

User Access Verification

DSW1# show cdp neighbors

Device ID Local Intrfce Holdtme Capability Platform Port ID

Directly connected neighbors of DSW1 are R1, ASW1, and ASW2.

DSW1# show mac address-table

Vlan Mac Address Type Ports

MAC Address Database

Step 8 Document the new identified connections.

Step 9 From PC1, connect to ASW1 and investigate its connection.

PC1# telnet 192.168.50.100

User Access Verification

ASW1# show cdp neighbors

Device ID Local Intrfce Holdtme Capability Platform Port ID

Vlan Mac Address Type Ports

Step 10 Investigate ASW1's connections.

ASW1# ping 192.168.50.255

Reply to request 0 from 192.168.50.254, 2 ms

Step 11 From PC1, connect to ASW2 and investigate its connections.

PC1# telnet 192.168.50.110

User Access Verification

ASW2# show cdp neighbors

Device ID Local Intrfce Holdtme Capability Platform Port ID

ASW2# show mac address-table

Vlan Mac Address Type Ports

ASW2# ping 192.168.50.255

Reply to request 0 from 192.168.50.10, 1 ms

ASW2# show arp

Step 13 What does your complete network topology look like?

ASW1(config)# interface range ethernet1/0-3, ethernet2/0-3, ethernet3/0-3,

Step 15 Investigate the configured interfaces descriptions on DSW1.

There are configured interface descriptions on DSW1.

DSW1# show interfaces description