CIS 188 CCNP TSHOOT (Troubleshooting)

Ch. 6 Troubleshooting Addressing

Services
Rick Graziani
Cabrillo College
graziani@cabrillo.edu
Fall 2014

Materials
Book:
Troubleshooting and Maintaining
Cisco IP Networks (TSHOOT)
Foundation Learning Guide:
Foundation learning for the CCNP
TSHOOT 642-832
By Amir Ranjbar
Book
ISBN-10: 1-58705-876-6
ISBN-13: 978-1-58705-876-9
eBook
ISBN-10: 1-58714-170-1
ISBN-13: 978-1-58714-170-6

NAT/PAT Operation

Network Address Translation (NAT) was designed for IP version 4 (IPv4)

Used for:
Address conservation
Address hiding
NAT usually operates at the border of a network
Translates the source address of the exiting IP packets that are private
addresses to public addresses before packets are forwarded.
3

NAT/PAT Operation Review Cont.

Order of operations with reference to NAT for incoming and outgoing
traffic:

NAT Example
1

DA
128.23.2.2

DA

SA
10.0.0.3
IP Header

....

128.23.2.2

Data

SA
179.9.8.80

....

Data

IP Header

The translation from Private source IP address to Public source IP address.

5

NAT Example
4

DA

SA

10.0.0.3

128.23.2.2

IP Header

DA
....

Data

179.9.8.80

SA
128.23.2.2

....

Data

IP Header

Translation back, from Public destination IP address to Private destination IP address.

PAT Example

Note: The router will try to maintain the original

TCP/UDP source port number if possible

NAT/PAT table maintains

translation of:
DA, SA, SP, Converted SP
DA
128.23.2.2

1
DA
128.23.2.2

SA
10.0.0.3
IP Header

SA
10.0.0.2
IP Header

DP
80

SP
1331

DA
Data

TCP/UDP
Header
DP
80

1555

TCP/UDP
Header

128.23.2.2 179.9.8.80

SP

DA
Data

SA

IP Header

SA

128.23.2.2 179.9.8.80
IP Header

DP
80

SP
3333

Data

TCP/UDP
Header
DP
80

SP
2222

TCP/UDP
Header

Data

PAT Example

NAT/PAT table maintains

translation of:
SA (DA), DA (SA), DP (SP)
DA
10.0.0.3

SA
128.23.2.2
IP Header

4
DA

SA

10.0.0.2

128.23.2.2
IP Header

DP

SP

1331

80

DA
Data

TCP/UDP
Header
DP
1555

80

TCP/UDP
Header

179.9.8.80 128.23.2.2

SP

DA
Data

SA

179.9.8.80

IP Header

SA
128.23.2.2
IP Header

DP
3333

SP
80

Data

TCP/UDP
Header
DP
2222

SP
80

TCP/UDP
Header

Data

 Static NAT:
Local and global addresses are mapped one to one.
Dynamic NAT:
Local addresses are translated to a group or pool of global addresses.
Still dealing with one-to-one translation once a global address has been
selected.
NAT overloading:
Special type of dynamic NAT in which addresses are translated in a
many-to-many fashion.
Also known as PAT, or Port Address Translation

Dynamic NAT
SA 172.16.10.1

Example: We want NAT to allow the first 31 hosts from each subnet on the
inside to communicate with devices on the outside using a public IP
address.
Note: We will use the 172.16.10.0/24 network to simulate a public
address
Range of public addresses: 172.16.10.1 through 172.16.10.63

10

Dynamic NAT (one-to-one)

interfaceethernet0
ipaddress10.10.10.1255.255.255.0
ipnatinside

172.16.10.1
172.16.10.2
172.16.10.3
172.16.10.4

interfaceethernet1
ipaddress10.10.20.1255.255.255.0
ipnatinside
interfaceserial0
ipaddress172.16.10.64255.255.255.0
ipnatoutside
ipnatinsidesourcelist7poolnooverload
!Indicatesthatanypacketsreceivedontheinsideinterfacethat
!arepermittedbyaccesslist7
!willhavethesourceaddresstranslatedtoanaddressoutofthe
!NATpool"nooverload".
accesslist7permit10.10.10.00.0.0.31
accesslist7permit10.10.20.00.0.0.31
!Accesslist7permitspacketswithsourceaddressesrangingfrom
!10.10.10.0through10.10.10.31and10.10.20.0through10.10.20.31.
ipnatpoolnooverload172.16.10.1172.16.10.63prefix24
!
!DefinesaNATpoolnamednooverloadwitharangeofaddresses
!172.16.10.1172.16.10.63

11

NAT Overloading (many-to-one)

interfaceethernet0
ipaddress10.10.10.1255.255.255.0
ipnatinside

172.16.10.1

interfaceethernet1
ipaddress10.10.20.1255.255.255.0
ipnatinside
interfaceserial0
ipaddress172.16.10.64255.255.255.0
ipnatoutside
ipnatinsidesourcelist7poolOVERLOADoverload
!
!Indicatesthatanypacketsreceivedontheinsideinterfacethat
!arepermittedbyaccesslist7willhavethesourceaddress
!translatedtoanaddressoutoftheNATpoolnamedovrld.
!Translationswillbeoverloadedwhichwillallowmultipleinside
!devicestobetranslatedtothesamevalidIPaddress.
accesslist7permit10.10.10.00.0.0.31
accesslist7permit10.10.20.00.0.0.31
!Accesslist7permitspacketswithsourceaddressesrangingfrom
!10.10.10.0through10.10.10.31and10.10.20.0through10.10.20.31
ipnatpoolOVERLOAD172.16.10.1172.16.10.1prefix24
!
!DefinesaNATpoolnamedovrldwitharangeofasingleIP
!address,172.16.10.1.

12

Static and NAT Overloading (many-to-one)

172.16.131.1

172.16.131.2

interfacee0
ipaddress10.10.10.254255.255.255.0
ipnatinside
interfaces0
ipaddress172.16.130.2255.255.255.0
ipnatoutside

ipnatinsidesourcelist7pooltestoverload
ipnatinsidesourcestatic10.10.10.1172.16.131.1
accesslist7permit10.10.10.00.0.0.255
ipnatpooltest172.16.131.2172.16.131.2netmask255.255.255.0

13

 Other examples
http://www.cisco.com/en/US/tech/tk648/tk361/technologies_tech_note09186
a0080094e77.shtml#topic1
14

Advantages and disadvantages of

implementing NAT

15

 Some applications or protocols have direct conflict with Network Address Translation
(NAT) or Port Address Translation (PAT).
Example: IPsec Virtual Private Networks (VPN)
IPsec protocols encapsulate the original IP packet
The protocol type on the IP header changes (to ESP or AH)
There is no TCP or UDP header next to the IP header.
Means that there is no port number for NAT/PAT to translate.
Some mechanisms have been invented to allow IPsec and NAT to coexist.
Those mechanisms include:
NAT Transparency or NAT traversal
IPsec over TCP
IPsec over UDP.
In certain cases you may still be required to disable NAT for VPN traffic, or create
exceptions for it.

16

Troubleshooting
Common NAT/PAT
Issues

Some of the important NAT issues and considerations to keep in mind are:
Having a diagram for the NAT configuration
ACLs are used to tell the NAT device:
"what source IP addresses are to be translated
IP NAT pools are used to specify:
"to what those addresses translate
Marking the IP NAT inside interfaces and the IP NAT outside interfaces
correctly is very important
NAT packets still have to obey routing protocols and reachability rules, so
make sure that every router knows how to reach the desired destinations.
17

Troubleshooting NAT problems:

clearipnattranslations
showipnattranslations
showipnatstatistics
debugipnat
debugippacket[accesslist]
debugconditioninterfaceinterface
showdebugcondition

18

Troubleshooting Example: NAT/PAT

Problems

19

Router R1 can ping R4, but router R1 cannot ping R3.

There are no routing protocols running in any of the routers.
R1 uses R2 as its gateway of last resort.
The objective is to restore end-to-end connectivity from R1 to all
destinations.

20

Looks good

pool

Looks good

This command also shows the full NAT configuration.

In this instance, we have both dynamic and static NAT configured.
The only indication that there are static translations configured is the reference to
1 static in the Total active translations line.
In order to see the static translation, the show ip nat translations
command is used.

21

10.10.10.1

172.16.6.1

The only entry in the NAT translation table is the static translation for R1's
Fa0/0 interface 10.10.10.1 into 172.16.6.1.
This translation may be causing the problem.
Typical issues with static translations occur when:
there is no route back to the statically translated address, or
when the statically selected global address overlaps with an available
address in the dynamic address pool.

22

?
R3# debug ip icmp
ICMP packet debugging is on

R1# ping 172.16.11.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.11.3, timeout is 2 seconds:
.....

Success rate is 0 percent (0/5)

R3#
*Aug
*Aug
*Aug
*Aug
*Aug

23
23
23
23
23

13:54:00.556:
13:54:02.552:
13:54:04.552:
13:54:06.552:
13:54:07.552:

ICMP:
ICMP:
ICMP:
ICMP:
ICMP:

echo
echo
echo
echo
echo

reply
reply
reply
reply
reply

sent,
sent,
sent,
sent,
sent,

src
src
src
src
src

172.16.11.3,
172.16.11.3,
172.16.11.3,
172.16.11.3,
172.16.11.3,

dst
dst
dst
dst
dst

172.16.6.1
172.16.6.1
172.16.6.1
172.16.6.1
172.16.6.1

The next step, verify whether packets leaving R1 actually reach R3 or not.
This can help us discover if the problem is with NAT a routing problem.
Use icmp debugging on R3 with the debug ip icmp command.
Next, we ping R3 from R1 and observe the out of the debug on R3.
Based on the output of the debug ip icmp command shown, the icmp echo
requests reach R3, but R3s icmp echo replies do not reach R1.
The NAT translation is working but there is a routing issue on R3 toward the
172.16.6.0 destination.

23

?
R3# show ip route 172.16.6.0 255.255.255.0
% Subnet not in table
R3# configure terminal
R3(config)# ip route 172.16.6.0 255.255.255.0 172.16.11.2
R3(config)# exit
R1# ping 172.16.11.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.11.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R1#

This is verified using the show ip route command on R3.

Because there are no routing protocols in use, we fix the routing problem by
entering a static route in R3s routing table.
Now we can check to see if the connectivity problem between R1 and R3
has been corrected.
The problem was not directly a NAT problem; it was a routing issue.
24

 When NAT is deployed, a set of inside local addresses are normally

translated to a set of inside global addresses.
The inside global addresses may not be assigned to any physical networks,
yet they need to be advertised so that outside devices know about those
addresses and know how to forward packets to those networks.
25

Troubleshooting Example: NAT Problem

Caused by an Inaccurate Access List

26

SSH
ICMP

ICMP
SSH

We are told that administrators are unable to use Secure Shell (SSH) from
the 10.10.10.0/24 network to routers R3 or R4
But they can accomplish connectivity (ping, etc) from the R1 loopbacks.
The routing protocol used is single area OSPF
Our mission is to restore end-to-end connectivity and make sure SSH is
operational to support management processes.

27

 Is SSH traffic is being filtered somewhere?

Or is this simply be a routing problem - traffic is either not reaching the
destination, or it is not coming back?
Not a routing problem
28

 Our follow-the-path strategy points to looking at the intermediate routers all

the way up to the destination (R3).
We will use a clever tool to discover the potential filter: ip tcp debug
transactions
Try the SSH connection again and observe the output of debug.
The results indicate that the attempt made by R1 to setup a TCP session
with R3 failed because the remote device (R3) responded with a TCP RST
(reset).
29

 Focus on R3: the output of show ip int serial 0/1/0 shows that an
access-list called FIREWALL-INBOUND is applied to serial 0/1/0
interface on the inbound direction.
show access-lists command and the access list looks correct:
statement number 30 permits TCP connection to 172.16.11.3 TCP port
number 22 (SSH).

30

 Cautiously make use of debug ip packet on R3.

Re-attempt the SSH session from R1 to R3 and observe the debug output on R3.
The SSH attempt from R1 fails again, but:
Security message (%SEC-6-IPACCESSLOGP) on R3 states that the denied
TCP with:
Source IP address 10.10.10.1 and source port number 29832
Destination IP address 172.16.11.3 and destination port number 2222
Now we know why the packet is denied, destination port is 2222 instead of 22.
31

 We have to find out which device has translated the port number from 22 to
2222.
Prime suspect is NAT on R2.
Cautiously, we use debug ip nat on R2
Re-attempt SSH from R1 to R3.
To confirm our findings, we also enter the show ip nat translations
command on R2.

32

R2# debug ip nat

IP NAT debugging is on
R2#
*Aug 23 16:28:31.731: NAT*: TCP s=555 55587, d=22->2222
R3# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R3(config)# ip access-list exten FIREWALL-INBOUND
R3(config-ext-nacl)# permit tcp any host 172.16.11.3 eq 22
R3(config-ext-nacl)# end
R3#

We are using TCP 2222 (destination port not the NAT source port); but the
access-list on R3 is permitting TCP 22.
The problem was not the NAT configuration, but a lack of synchronization
between the configuration teams:
The configuration on R2 is doing port mapping to a custom port (2222), but
the access-list configuration on R3 did not consider or account for the
custom port.
We need to correct the FIREWALL-INBOUND on R3

33

R3# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R3(config)# ip access-list exten FIREWALL-INBOUND
R3(config-ext-nacl)# permit tcp any host 172.16.11.3 eq 2222
R3(config-ext-nacl)# end
R3#

The next step is to correct the FIREWALL-INBOUND on R3 and re-attempt

SSH from R1 to R3.

34

R3# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R3(config)# ip access-list exten FIREWALL-INBOUND
R3(config-ext-nacl)# permit tcp any host 172.16.11.3 eq 2222
R3(config-ext-nacl)# end
R3#
R1# ssh -l user 172.16.11.3
Password:
*Aug 23 16:30:42.604: TCP: Random local port generated 43884, network 1
*Aug 23 16:30:26.604: TCB63BF854C created
*Aug 23 16:30:26.604: TCB63BF854C bound to UNKNOWN.43884
*Aug 23 16:30:26.604: TCB63BF854C setting property TCP_TOS (11) 62AF6D55
*Aug 23 16:30:26.604: Reserved port 43884 in Transport Port Agent for TCP IP type 1
*Aug 23 16:30:26.604: TCP: sending SYN, seq 1505095793, ack 0
*Aug 23 16:30:26.604: TCP0: Connection to 172.16.11.3:22, advertising MSS 536
*Aug 23 16:30:26.608: TCP0: state was CLOSED -> SYNSENT [43884 ->
172.16.11.3(22)]
*Aug 23 16:30:26.608: TCP0: state was SYNSENT -> ESTAB [43884 ->
172.16.11.3(22)]
*Aug 23 16:30:26.608: TCP: tcb 63BF854C connection to 172.16.11.3:22, peer MSS
536, MSS is 536
*Aug 23 16:30:26.608: TCB63BF854C connected to 172.16.11.3.22

SSH attempt is successful now. - The problem was not the NAT configuration.
The configuration on R2 was doing port mapping to a custom port (2222) but
the access-list configuration on R3 did not consider or account for the custom
port.

35

Reviewing DHCP
Operation

Reviewing DHCP Operation

These commands are most helpful to know during the troubleshooting process,
for example.
DHCPDECLINE client-to-server message - The client through some other
means has discovered that the IP address is already in use.
DHCPNAK - DHCP server refuses the request for a certain configuration
parameter.

37

Three roles router takes in DHCP

The most common reasons for problems are configuration issues.

This can result in a multitude of symptoms, such as:
Clients not obtaining IP information from the server
Client requests not reaching the server across a DHCP relay agent
Clients failing to obtain DHCP options and extensions.

38

Router(configif)#iphelperaddressaddress
Another issue related to DHCP Relay Agent is that enabling a router
interface with the ip helper-address interface command
Allows you to control which broadcast packets and which protocols are
forwarded.
Address: Destination broadcast or host address to be used when forwarding
UDP broadcasts.
Forwards six protocols (not just DHCP)
TFTP (port 69)
DNS (port 53)
Time Service (port 37)
NetBIOS Name Service and Datagram Service (ports 137 and 138)
TACACS (port 49)
DHCP/BOOTP Client and Server (ports 67 and 68)
If other protocols do not require this service, forwarding their requests
should be disabled using the no ip forward-protocol udp port-number
global configuration mode command.

39

 DHCP options deliver parameters in addition to the traditional IP address,

subnet mask, default-gateway, and DNS server address.
82 - Used for 911 purposes, in order to know the switch port that an IP
phone connects to the 911 callers exact location.

40

DHCP Troubleshooting Commands

show ip dhcp server statistics: Displays counts for server statistics and
messages sent and received for an IOS-based DHCP server.
show ip dhcp binding: Displays DHCP binding information for IP address
assignment and subnet allocation.
show ip dhcp conflict: Displays address conflicts found by a Cisco IOS
DHCP server when addresses are offered to the client.
show ip dhcp pool name: Displays the subnets allocated and the current
utilization level for the pool or all the pools if the name argument is not used.
show ip dhcp database: Displays server database agent information:
URL: Specifies the remote file used to store automatic DHCP bindings
Read/written: The last date and time bindings were read/written from the file
server
Status: Indication of whether the last read or write of host bindings was
successful
Delay: The amount of time (in seconds) to wait before updating the database
Timeout: The amount of time (in seconds) before the file transfer is aborted
Failures/Successes: The number of failed/successful file transfers
41

ip dhcp excluded-address 172.16.1.100 172.16.1.103

ip dhcp excluded-address 172.16.2.100 172.16.2.103
!
ip dhcp pool 0
network 172.16.0.0 /16
domain-name cisco.com
dns-server 172.16.1.102 172.16.2.102
netbios-name-server 172.16.1.103 172.16.2.103
netbios-node-type h-node
!
ip dhcp pool 1
network 172.16.1.0 /24
default-router 172.16.1.100 172.16.1.101
lease 30
!
ip dhcp pool 2
network 172.16.2.0 /24
default-router 172.16.2.100 172.16.2.101
lease 30

Three DHCP address pools are created:

network 172.16.0.0, subnet 172.16.1.0 and subnet 172.16.2.0
Attributes from network 172.16.0.0 - such as the domain name, DNS server, NetBIOS
name server, and NetBIOS node type are inherited in subnetworks 172.16.1.0 and
172.16.2.0.
In each pool, clients are granted 30-day leases and all addresses in each
subnetwork, except the excluded addresses, are available to the DHCP Server for
assigning to clients.
Excluded addresses include default-gateway, DNS and NETBIOS servers

42

DHCP Troubleshooting Issues

Troubleshooting can be related to DHCP security efforts.
Automatic addressing is accomplished through DHCP.
Security is accomplished through DHCP snooping.
Some specific issues related to DHCP snooping:
Improper configuration of the DHCP snooping trust boundaries
Failure to configure DHCP snooping on certain VLANs
Improper configuration of the DHCP snooping rate limits
Performance degradation
Poor planning of DHCP snooping can result in DHCP transactions
being blocked or affected.

DHCP Troubleshooting Issues Cont.

DHCP troubleshooting questions to ask:
Where are the DHCP servers and clients located?
Are DHCP relay agents configured?
What are the DHCP pool sizes? Are they sufficient?
Are there any DHCP option compatibility issues?
Are there any ACLs or firewalls filtering UDP port 67 or UDP port 68?
Are there any active DHCP DoS attacks?
Is forwarding disabled on the router acting as DHCP Relay Agent for
any UDP ports (using the Cisco IOS no ip forward-protocol
udp port command)?
Is the ip helper-address command applied to correct router
interfaces?
Is DHCP snooping configured?

Troubleshooting Example: DHCP Problems

45

 R1 provides DHCP services to the clients in the 10.1.1.0 subnet. (Not R4)
The DHCP clients in this example are routers R2 and R3.
It is reported that R1 is no longer providing reliable DHCP services:
The clients are unable to renew their IP addresses.

46

R2# show ip int brief

Interface
IP-Address
FastEthernet0/0 unassigned
FastEthernet0/1 unassigned
Serial0/0/0
unassigned
Serial0/0/1
unassigned

OK?
YES
YES
YES
YES

Method
DHCP
NVRAM
NVRAM
NVRAM

Status
Protocol
up
up
administratively down down
administratively down down
administratively down down

R3# show ip int brief

Interface
IP-Address
FastEthernet0/0 unassigned
FastEthernet0/1 unassigned
Serial0/0/0
unassigned
Serial0/0/1
unassigned

OK?
YES
YES
YES
YES

Method
DHCP
NVRAM
NVRAM
NVRAM

Status
Protocol
up
up
administratively down down
administratively down down
administratively down down

Check R2 and R3 to make sure that they are configured as a DHCP clients.
The output of the show ip interfaces brief command shows that interface
fa0/0 is configured as a DHCP client and it shows an unassigned IP
address.
Because multiple clients are having the same problem, it is reasonable to
suspect the problem originates elsewhere.

47

DHCP Troubleshooting Example

1 Cont.
R3# debug dhcp detail
DHCP client activity debugging is on (detailed)
R3#
*Aug 23 17:32:37.107: Retry count: 1 Client-ID: cisco-0019.5592.a442-Fa0/0
*Aug 23 17:32:37.107: Client-ID hex dump: 636973636F2D303031392E353539322E
*Aug 23 17:32:37.107: 613434322D4551302F30
*Aug 23 17:32:37.107: Hostname: R3
*Aug 23 17:32:37.107: DHCP: SDiscover: sending 291 byte length DHCP packet
*Aug 23 17:32:37.107: DHCP: SDiscover 291 bytes
*Aug 23 17:32:37.107: B cast on FastEthernet0/0 interface from 0.0.0.0
*Aug 23 17:32:40.395: DHCP: SDiscover attempt #2 for entry:
*Aug 23 17:32:40.395: Temp IP addr: 0.0.0.0 for peer on Interface: FastEthernet0/0
*Aug 23 17:32:40.395: Temp sub net mask: 0.0.0.0
*Aug 23 17:32:40.395: DHCP Lease server: 0.0.0.0, state: 1 Selecting
*Aug 23 17:32:40.395: DHCP transaction id: 13BA
*Aug 23 17:32:40.395: Lease: 0 secs, Renewal: 0 secs, Rebind: 0 secs
*Aug 23 17:32:40.395: Next timer fires after: 00:00:04
*Aug 23 17:32:40.395: Retry count: 2 Client-ID: cisco-0019.5592.a442-Fa0/0
*Aug 23 17:32:40.395: Client-ID hex dump: 636973636F2D303031392E353539322E
*Aug 23 17:32:40.395: 613434322D4551302F30
<output omitted>
*Aug 23 17:32:44.395: Hostname: R3
*Aug 23 17:32:44.395: DHCP: SDiscover: sending 291 byte length DHCP packet
*Aug 23 17:32:44.395: DHCP: SDiscover 291 bytes
*Aug 23 17:32:44.395: B cast on FastEthernet0/0 interface from 0.0.0.0
*Aug 23 17:32:48.395: DHCP: Qscan: Timed out Selecting state
%Unknown DHCP problem... No allocation possible
*Aug 23 17:32:57.587: DHCP: waiting for 60 seconds on interface FastEthernet0/0

debug dhcp detail command on R3:

Finds that the DHCP DISCOVER messages are generated out of interface
fa0/0, but no DHCP offers are received back from the DHCP server.
Timed out Selecting state message, followed by No allocation
possible.

DHCP Troubleshooting Example 1 Cont.

R1# show ip int brief

Interface
IP-Address
FastEthernet0/0 10.1.1.1
FastEthernet0/1 unassigned
Serial0/0/0
unassigned
Serial0/0/1
unassigned

OK?
YES
YES
YES
YES

Method
manual
NVRAM
NVRAM
NVRAM

Status
Protocol
up
up
administratively down down
administratively down down
administratively down down

Next, check router R1, the DHCP server.

Apply a bottom-up approach and use the show ip int brief command to verify
that Layers 1 and 2 are operational on the documented DHCP server.
R1's Fa0/0 interface is UP, with the IP address 10.1.1.1/24.

DHCP Troubleshooting
Example 1 Cont.
R1# show ip dhcp server statistics
Memory usage
9106
Address pools
1
Database agents
0
Automatic bindings 0
Manual bindings
0
Expired bindings
0
Malformed messages 0
shows
Secure arp entries 0
Message
BOOTREQUEST
DHCPDISCOVER
DHCPREQUEST
DHCPDECLINE
DHCPRELEASE
DHCPINFORM
Message
BOOTREPLY
DHCPOFFER
DHCPACK
DHCPNAK

that one address pool configured as part of the

service and very few DHCP messages sent and
Received received.
0
So it looks to be in good shape, just not very active.
1
1
0
0
0
Sent
0
1
1
0

DHCP Troubleshooting
Example 1 Cont.
R1# sh ip dhcp pool
Pool vlan10 :
Utilization mark (high/low) : 100/0
Subnet size (first/next)
: 0/0
Total addresses
: 254
Leased addresses
: 0
Pending event
: none
1 subnet is currently in the pool :
Current index
IP address range
10.1.1.12
10.1.1.1 -10.1.1.254

Leased addresses
0

Verify the address pool (scope) configuration to determine if it is either

misconfigured or exhausted, using the show ip dhcp pool command.
The results show one address pool, vlan10.
The number of leased addresses is 0, meaning that there are no leased
addresses.
There 254 addresses in this pool, but none are allocated at the moment.
Both the DHCP server and the DHCP clients have the correct configurations, and
are operationally UDP at physical and data-link layers.

DHCP Troubleshooting
Example 1 Cont.

R1# show ip sockets

Proto Remote
Port
Local
88
--listen-10.1.1.1
17
--listen-10.1.1.1
17
--listen-10.1.1.1
17
--listen-10.1.1.1
17
--listen---any-17
--listen---any-17
--listen---any--

Port
10
161
162
57767
161
162
60739

In Out
0
0
0
0
0
0
0
0
0
0
0
0
0
0

Stat TTY
0
0
1001
0
1011
0
1011
0
20001
0
20011
0
20011
0

OutputIF

We use the show ip socket command to see the active ports on R1, the
DHCP server.
The show ip socket command is not frequently used by network
administrators, but it is very handy in monitoring the open ports on a router
There is no entry for UDP port 67 (DHCP Server).
This is certainly a problem.

DHCP Troubleshooting
Example 1 Cont.
R1# conf t
R1(config)# service dhcp
R1# show ip sockets
Proto Remote
Port
88
--listen-17
--listen-17
--listen-17
--listen-17
--listen-17
--listen-17
--listen-17 0.0.0.0
0

Local
10.1.1.1
10.1.1.1
10.1.1.1
10.1.1.1
--any---any---any-10.1.1.1

Port
10
161
162
57767
161
162
60739
67

In Out
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

Stat TTY
0
0
1001
0
1011
0
1011
0
20001
0
20011
0
20011
0
2211
0

OutputIF

We enable the DHCP service using the service dhcp command.

After enabling this service you must wait a few seconds, because DHCP clients retry at
different intervals.
Use the show ip socket command again and now see port 67 as an active port on R1.
Finally, we check routers R2 and R3 (the DHCP clients) and see that they successfully
obtained IP addresses.

DHCP Troubleshooting Example: Duplicate

Client IP Addresses

54

 The IP address of router R1 on the fast Ethernet interface was changed

from 10.1.1.100 to 10.1.1.1 to comply with the new addressing scheme and
policies of the network.
This policy states that all branch routers will have the first IP address on any
subnet that is being assigned to a network segment.
After the change, some DHCP clients are reporting duplicated IP addresses.
Clients state that this happens sporadically, a few times a week.

55

DHCP Troubleshooting
Example 2 Cont.

R1# show running-config | beg ip dhcp pool

ip dhcp pool vlan10
network 10.1.1.0 255.255.255.0
default-router 10.1.1.1
lease 3

One piece of information we have is that the IP address duplication happens

sporadically, and one host at a time.
Knowing that, perhaps our first order of business is to look at the lease
times, and see if they match the frequency of the symptoms.
The Cisco IOS DHCP default values is 1 day.
The output of this command shows the vlan10 DHCP pool with a lease time
of 3 days.
The DHCP pool seems to be correct.

DHCP Troubleshooting
Example 2 Cont.

R1# show ip dhcp conflict

IP address
Detection method
10.1.1.1
Gratuitous ARP
10.1.1.3
Gratuitous ARP
10.1.1.3
Gratuitous ARP
10.1.1.4
Gratuitous ARP
10.1.1.5
Gratuitous ARP
10.1.1.6
Gratuitous ARP
10.1.1.7
Gratuitous ARP
10.1.1.8
Gratuitous ARP
10.1.1.9
Gratuitous ARP
10.1.1.10
Gratuitous ARP
10.1.1.11
Gratuitous ARP
--More--

Detection time
Aug 23 2009 06:28
Aug 23 2009 06:29
Aug 23 2009 06:29
Aug 23 2009 06:29
Aug 23 2009 06:29
Aug 23 2009 06:29
Aug 23 2009 06:29
Aug 23 2009 06:29
Aug 23 2009 06:29
Aug 23 2009 06:29
Aug 23 2009 06:29

PM
PM
PM
PM
PM
PM
PM
PM
PM
PM
PM

VRF

show ip dhcp conflict command will tell us whether the DHCP server has
found overlap or duplication in the IP addresses that it has assigned.
One of the many conflicting addresses is 10.1.1.1, which is the new ip
address of router R1 (the dhcp server itself) on interface fa0/0.
However, we know that the DHCP server should not provide its own IP
address to its clients!

R1# sh run | inc excluded

ip dhcp excluded-address 10.1.1.100
R1#

Many devices such as servers and printers are usually configured as DHCP
clients and have static IP addresses.
If their addresses are not excluded from the DHCP dynamic pool, there will
definitely be conflict problems.
We must check and verify which IP addresses are being excluded on R1,
the DHCP server.
We do that using the show running | include excluded command.
The only IP address excluded from the DHCP dynamic pool is 10.1.1.100,
which is R1s old address.

58

R1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)# no ip dhcp excluded-address 10.1.1.100
R1(config)# ip dhcp excluded-address 10.1.1.1 10.1.1.20
R1(config)# end
R1#

We need to exclude the range of addresses that are meant to be statically

assigned.
This range is 10.1.1.1 to 10.1.1.20
To ensure that the users will receive unique addresses from the DHCP
server and will not incur any more address conflicts, we must renew IP
address leases on all DHCP clients, especially those that have experienced
conflicts before. (Success!)
59

DHCP Troubleshooting Example 3: Relay Agent Issue

R4 is a centrally located DHCP server.

The DHCP clients in network segment 10.1.1.0 are unable to obtain IP
address and other parameters.
R2 is a DHCP client that is having trouble acquiring ip address.
R1 is supposed to act as a relay agent and forward DHCP messages
between local clients and the DHCP server (R4).

?
?

There are several possible causes:

The clients could be misconfigured or faulty
The relay agent could be not configured or misconfigured
The server could be misconfigured or exhausted (or faulty/disabled)
There could be network problems or filtering/security barriers.
Because multiple clients are having the same problem, it is possible that
they are all misconfigured, but that is not likely.
It is simpler to check the relay agent.
If we find no problem with the relay agent, we can then check the DHCP
server.
61

DHCP
Troubleshooting
Example 3 Cont.
R1# debug ip udp
UDP packet debugging is on
R1#
R1#
*Aug 23 19:01:05.303: UDP:
length=584
*Aug 23 19:01:05.303: UDP:
dst=192.168.1.255
*Aug 23 19:01:08.911: UDP:
length=584
*Aug 23 19:01:08.911: UDP:
dst=192.168.1.255
*Aug 23 19:01:12.911: UDP:
length=584
*Aug 23 19:01:12.911: UDP:
dst=192.168.1.255
<output omitted>

rcvd src-0.0.0.0(68), dst=255.255.255.255(67),

broadcast packet dropped, src=0.0.0.0,
rcvd src-0.0.0.0(68), dst=255.255.255.255(67),
broadcast packet dropped, src=0.0.0.0,
rcvd src-0.0.0.0(68), dst=255.255.255.255(67),
broadcast packet dropped, src=0.0.0.0,

One of the quickest ways to verify DHCP relay agent operations is using the debug
ip udp command.
R1 is certainly receiving DHCP requests but is dropping them.
The UDP/IP packets shown have a source address of 0.0.0.0, destination address of
255.255.255.255 with source UDP port of 68 (DHCP client) and destination UDP port
of 67 (DHCP server).
The problem could be that the fa 0/0 interface facing the DHCP client is missing the
ip helper-address command pointing to 192.168.1.4.

DHCP Troubleshooting Example 3 Cont.

R1(config)# int fa0/0

R1(config-if)# ip helper-address 192.168.1.4
R1(config-if)# end

Checking the configuration reveals that this command is indeed missing, so

adding this command is the first thing we have to do.

R4# debug ip udp

UDP packet debugging is on
R4#
*Aug 23 19:31:39.303: UDP:
*Aug 23 19:31:39.303: UDP:
*Aug 23 19:31:39.303: UDP:
*Aug 23 19:31:40.159: UDP:
*Aug 23 19:31:44.159: UDP:
*Aug 23 19:31:46.307: UDP:
*Aug 23 19:31:49.307: UDP:
<output omitted>
*Aug 23 19:32:28.439: UDP:
*Aug 23 19:32:31.439: UDP:
*Aug 23 19:32:35.439: UDP:
*Aug 23 19:32:37.011: UDP:

sent
rcvd
sent
rcvd
rcvd
rcvd
rcvd

src=0.0.0.0(67), dst=255.255.255.255(68),length=308
src=0.0.0.0(68), dst=255.255.255.255(67),length=584
src=0.0.0.0(67), dst=255.255.255.255(68),length=308
src=0.0.0.0(68), dst=192.168.1.4(67), length=584
src=0.0.0.0(68), dst=192.168.1.4(67), length=584
src=10.1.1.11(53470), dst=255.255.255.255(69),length=30
src=10.1.1.11(53470), dst=255.255.255.255(69),length=30

rcvd
rcvd
rcvd
rcvd

src=10.1.1.11(53470), dst=255.255.255.255(69),length=29
src=10.1.1.11(53470), dst=255.255.255.255(69),length=29
src=10.1.1.11(53470), dst=255.255.255.255(69),length=29
src=0.0.0.0(68), dst=192.168.1.4(67), length=584

debug ip udp command on R4, shows the DHCP Requests are being forwarded to
R1
DHCP messages fro the server are being received
Finally, we verify the status of the DHCP clients, such as R2, in the 10.1.1.0 subnet,
and see that they are acquiring IP address and other parameters from the DHCP
server.

IPv6

Next: Series of IPv6 videos from the Academy Conference I

presented at in 2012.
See my web page or summary email for links.
65

CIS 188 CCNP TSHOOT (Troubleshooting)

Ch. 6 Troubleshooting Addressing

Services
Rick Graziani
Cabrillo College
graziani@cabrillo.edu