Troubleshooting Cisco

Catalyst 4500 Series

Switches
Subhash Ramanathan – Escalation Engineer, Enterprise
Campus Switching Group
BRKCRS-3142
Session Goals

At the end of this session, you should be able to:

 Understand system resources and monitor their usage
 Identify all areas of packet loss
 Trace hardware packet path
 Make use of newer tools

This content is based on questions we see in the field. Feedback is welcome!

Agenda

 Products Overview
 Troubleshooting
– Method
– System Resources
– Packet path / loss
– Wired/Wireless Convergence
 Tools/Tips
 Products Overview

48 Gbps per slot

• +E Chassis support 12.2(53)SG4
onward
• Sup8E, Sup7E, Sup7L-E, 47xx line card
• 4507R+E, 4510R+E, 4503-E, 4506-E
4503-E 4507R+E 4510R+E 4506-E

See the appendix for supervisor, line card, and chassis product and compatibility details.
 Products Overview
 Intelligent Supervisors
Supervisor Engine 8-E, 7-E, 7L-E, 6-E, 6L-E Shared Packet CPU
Memory
 Transparent Line Cards TCAMs1 NFE2
Wire-rate, oversubscribed, PoE
10/100, 10/100/1000, GE, 10GE
Various physical media front panel ports Packet Forwarding
Processor Engine
Dedicated per-slot bandwidth to supervisor

 Switching ASICs Supervisor

Packet Processor
Line Card
Forwarding Engine
Stub ASICs
 Specialized Hardware
TCAM1s for ACLs, QoS, L3 forwarding Front Panel Ports
NetFlow2 (NFE) for statistics gathering
1. Ternary Content Addressable Memory
2. Integrated on Supervisor 7E, 7L-E, Sup8E
 Products Overview
 Intelligent Supervisors
Supervisor Engine 8-E, 7-E, 7L-E, 6-E, 6L-E Shared Packet CPU
Memory
 Transparent Line Cards Wireless
DC TCAMs1 NFE2
Wire-rate, oversubscribed, PoE
10/100, 10/100/1000, GE, 10GE
Various physical media front panel ports Packet Forwarding
Processor Engine
Dedicated per-slot bandwidth to supervisor

 Switching ASICs Supervisor

Packet Processor
Line Card
Forwarding Engine
Stub ASICs
 Specialized Hardware
TCAM1s for ACLs, QoS, L3 forwarding Front Panel Ports
NetFlow2 (NFE) for statistics gathering
1. Ternary Content Addressable Memory
2. Integrated on Supervisor 7E, 7L-E, Sup8E
Agenda

 Products Overview
 Troubleshooting
– Method
– System Resources
– Packet path / loss
– Wired/Wireless Convergence
 Tools/Tips
Troubleshooting Method
• General Recommendations

 Design with intent

– ideally, create a deterministic network
– engineers – not traffic – should control the network

 Baseline, monitor against baseline, alarm and/or adjust

– problems are solved faster when knowns can be eliminated

 Characterize issues quickly with a plan

Troubleshooting Method
Method

1. Define Problem Symptoms? System Messages? User

Input?
2. Gather Facts When? Frequency? Impact? Scope?
Documentation

3. Consider Possibilities • Need to have a good understanding about

how the system looks like when it is
4. Create Action Plan healthy

5. Execute Action Plan

6. Observe Results
Want to learn more? Check out CCNP Practical Studies: Troubleshooting by Donna Harrington.
CCNP TSHOOT 642-832 Official Certification Guide by Kevin Wallace.
Troubleshooting Method
Method

Category Possible Cause

1. Define Problem
Config/Design Mis-configuration
Reaching Capacity
2. Gather Facts
Traffic DOS Attack
Documentation

Traffic Pattern Change

3. Consider Possibilities
Bad peer/server
4. Create Action Plan Software Issue Software Limitation
Bug
5. Execute Action Plan Hardware Issue Hardware Limitation
Failed Hardware
6. Observe Results
Transient Hardware Issue
Troubleshooting Method
Method

1. Define Problem

2. Gather Facts
Documentation

3. Consider Possibilities

4. Create Action Plan What needs to be done to isolate each

potential root cause?
5. Execute Action Plan Make a change, measure results,
rollback change if problem persists
6. Observe Results Problem solved? If not, continue
action plan
Troubleshooting Method
Caution

 debug and show platform commands to follow

 Excessive debug output to console may disable switch

 show platform commands are intended for in-depth troubleshooting

 Use debug and show platform commands only when advised by TAC

 show platform CLIs are not officially supported IOS commands

Agenda

 Products Overview
 Troubleshooting
– Method
– System Resources
– Packet path / loss
– Wired/wireless Convergence
 Tools/Tips
 Shared Packet CPU
Memory
System Resources TCAMs1 NFE2

CPU
Packet Forwarding
• Linux based Operating System IOS-XE Processor Engine

Supervisor
• Runs IOS tasks
Line Card
• Runs 4500 platform-specific jobs Stub ASICs

• Sends/Receives control traffic

• Software-switches packets that can’t be hardware-switch IOS-XE

IOSd
• Elevated CPU == in-use CPU, does not impact data plane
Cat4500
jobs
 Troubleshooting CPU: show process cpu
Reference Document ID: 65591 on
CPU higher than IOS High CPU in IOS process or http://www.cisco.com for more
baseline Cat4k process? details

IOS-XE ios cat4k

sh proc cpu detail

show platform health show platform cpu packet stat
process iosd

Yes
High iosd use on IOS-XE?
No
Troubleshoot features related
to the process / open TAC SR
Yes
High CPU traffic driven?
(K5CpuMan Review)
No
Stop / alter traffic source,
open TAC SR if more detail
needed
Can the traffic be identified?
Yes No
monitor session 1 source cpu
OR
debug platform packet all buffer
show platform cpu packet buffer
Troubleshooting CPU: Narrowing Down Process
Quad Core
Switch#show proc cpu sort
Identify which process
Core 0: CPU utilization for five seconds: 9%; one minute: 8%; five minutes: 8% running high
Core 1: CPU utilization for five seconds: 5%; one minute: 8%; five minutes: 8%
Core 2: CPU utilization for five seconds: 8%; one minute: 5%; five minutes: 5%
Core 3: CPU utilization for five seconds: 5%; one minute: 5%; five minutes: 5%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

6989 3788661 13695505 309 7.21 6.73 6.73 0 iosd
6984 677640 11354599 5 0.06 0.06 0.06 0 wcm
6936 211124 554564 380 0.02 0.02 0.02 0 cpumemd
6936 211124 554564 380 0.02 0.02 0.02 0 cpumemd
11879 56609 555338 101 0.02 0.01 0.01 0 wnweb_paster.py

IOS-XE processes
Troubleshooting CPU: Narrowing Down Process
switch# show proc cpu detail process iosd sort
Switch#show process cpu detailed process iosd sorted
Core 0: CPU utilization for five seconds: 9%; one minute: 8%; five minutes: 8%
Core 1: CPU utilization for five seconds: 10%; one minute: 8%; five minutes: 8%
Core 2: CPU utilization for five seconds: 7%; one minute: 5%; five minutes: 5%
Core 3: CPU utilization for five seconds: 6%; one minute: 5%; five minutes: 6%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
6989 L 3884781 1372616 309 4.15 6.64 6.82 0 iosd
6989 L 1 6989 2797345 7018102 0 6.76 6.57 6.76 0 iosd
6989 L 0 10677 1065764 6669769 0 0.04 0.08 0.05 0 iosd.fastpath
6989 L 1 10678 19185 119427 0 0.00 0.01 0.01 0 CMI Thread
6989 L 0 10679 3288 261952 0 0.00 0.00 0.00 0 iosd.monitor
6989 L 3 10680 86 4203 0 0.00 0.00 0.00 34816 iosd.aux
123 I 3816054 2388033 0 7.66 7.99 8.99 0 Cat4k Mgmt LoPri
122 I 2256302 4346590 0 6.88 5.99 5.88 0 Cat4k Mgmt HiPri

Catalyst-4k Specific
Management Tasks
Troubleshooting CPU: Packet-Driven CPU
switch# show platform health K5CpuMan Over Target
…
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
K5CpuMan Review 30.00 70.81 30 17 100 500 91 66 9 19:17
…

Switch# show platform cpu packet statistics Recent flood of packets with IP Options
… (not HW routable)
Packets Dropped by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Ip Option 10715071 118803 71866 15919 0
…

(config)# monitor session 1 source cpu rx

(config)# monitor session 1 destination interface Gi1/48

If port is available, get a full capture from CPU

Troubleshooting CPU: SPAN not available?
• This debug does not require significant CPU overhead
switch# debug platform packet all buffer • Be sure to use “buffer” and not “log”
platform packet debugging is on

Switch# show platform cpu packet buffered

Total Received Packets Buffered: 1024
-------------------------------------
Index 0:
3 days 23:23:18:54927 - RxVlan: 1006, RxPort: Gi1/1
Priority: Normal, Tag: No Tag, Event: 11, Flags: 0x40, Size: 64
Eth: Src 00:00:0B:00:00:00 Dst 00:22:90:E0:D6:FF Type/Len 0x0800
Ip: ver:IpVersion4 len:24 tos:0 totLen:46 id:0 fragOffset:0 ttl:64 proto:tcp
src: 10.10.10.100 dst: 172.16.100.100 hasIpOptions firstFragment lastFragment
Remaining data:
0: 0x0 0x64 0x0 0x64 0x0 0x0 0x0 0x0 0x0 0x0
10: 0x0 0x0 0x50 0x0 0x0 0x0 0x8A 0x37Newer versions provide human-readable event
0x0 0x0
20: 0x0 0x1 0xB5 0x77 0x6A 0x7E Decode on older versions with:
switch# show platform software cpu events | i Code|11
CPU Event Code PE-Q
1 2 Ip Option 11 17
 Troubleshooting CPU: Common Punt Reasons
Common Cause Recommended Solution

Same interface forwarding no ip redirect, or alter topology

ACL logging disable ACL logging, use ACL matching stats or netflow

ACL deny causing switch to send no ip unreachables2

ICMP unreachable
Forwarding/Feature exception (out of
TCAM/adj space)
SW-supported feature (i.e.GRE)
IP packets with TTL<2, IP options
reduce TCAM usage
resize TCAM region (TCAM2/3)
disable the feature or reduce the amount of traffic
disable the offending traffic, regulate source with Control Plane Policing1

Unexpected control/data traffic Control Plane Policing1

1.CoPP supported on all legacy supervisors starting 12.2(31)SG, SUP6-E/6L-E /4900M/4948E on 12.2(50)SG , all Sup8E/7E/7L-E/4500X
2.Must be configured on all the L3 interfaces of the switch
System Resources
Memory

• Leak vs Large Usage

• Large usage goes away when condition is no longer present

• Leak never decreases

• Establish baseline

• Collect multiple iterations over recorded interval

• Correlate increase with any known activity

Troubleshooting Memory: Large Usage

switch# sh authentication session | count Runn

Number of lines which match regexp = 239 300Kb not leaked, simply used
switch# sh proc mem detail proc iosd sort | i Hold|Auth Manager
PID TTY Allocated Freed Holding Getbufs Retbufs Process
113 0 870624 125992 837216 0 0 Auth Manager

switch(config)# int ra gi 1/1 - 48 , gi 2/1 - 48 , gi 3/1 - 48 , gi 4/1 - 48

switch(config-if-range)# shut
switch(config-if-range)# int ra gi 7/1 - 48 , gi 8/1 - 48 , gi 9/1 - 48 , gi 10/1 - 48
switch(config-if-range)# shut
switch(config-if-range)# end

switch# sh authentication session | count Runn

Number of lines which match regexp = 0

switch# sh proc mem detail proc iosd sort | i Auth Manager

147 0 1434488 601760 514088 0 0 Auth Manager
 For Classic IOS, use:
• show process mem sort
Troubleshooting Memory • show process mem <pid>

switch# show proc mem sort

System memory : 3870600K total, 1250447K used, 2620153K free, 323704K kernel reserved
Lowest(b) : 2031687704
PID Text Data Stack Heap RSS Total Process
6989 152256 943268 100 680 1338232 1435556 iosd
6984 20464 580524 88 14140 179240 721848 wcm
6985 692744 22048 92 164 50028 98096 mgmte_tap
6956 112 93740 88 5200 48484 134924 cli_agent
switch# show proc mem detail proc iosd sort
Processor Pool Total: 805306368 Used: 645097888 Free: 160208480
I/O Pool Total: 20971520 Used: 361576 Free: 20609944 Auth Manager holding too much
Critical Pool Total: 4087852 Used: 40 Free: 4087812
Critical Pool Total: 106460 Used: 40 Free: 106420
PID TTY Allocated Freed Holding Getbufs Retbufs Process
153 0 1461539184 749742680 307884712 14266252 0 Auth Manager
0 0 304511544 14111208 272960272 0 0 *Init*
185 0 887586464 301222848 31368752 0 0 CDP Protocol

switch# show proc mem detail proc iosd task 153 Collect process memory
Process ID: 153 breakdown for TAC
Process Name: Auth Manager
Total Memory Held: 307882352 bytes
Processor memory Holding = 307882352 bytes
pc = 0x16FCD45C, size = 291258544, count = 4441
 Shared Packet CPU
Memory

System Resources TCAMs1 NFE2

TCAM
Packet Forwarding
%C4K_HWACLMAN-4-ACLHWPROGERR: Input VOIP_FROM_CE_IPv6 - Processor Engine
hardware TCAM limit, qos being disabled on relevant interface
Supervisor
%C4K_HWACLMAN-4-ACLHWPROGERR: Input Security: 101 - hardware
TCAM limit, some packet processing will be software switched Line Card
Stub ASICs
C4K_HWACLMAN-4-ACLHWPROGERRREASON: Input(75/Normal, 1/Normal)
Invalid Acl-based Feature - hardware TCAM policers exceeded

• Check TCAM usage for ACLs, security, L3 routes, PBR, DHCP Snoop, IPSG,
WCCPv2
Monitoring TCAM
switch# show platform hardware acl statistics utilization brief
Switch#show platform hardware acl statistics utilization brief
CAM Utilization Statistics
--------------------------
Used Free Total
--------------------------------
Input Security (160) 38 (1 %) 2010 (99 %) 2048
Input Security (320) 34 (1 %) 2014 (99 %) 2048
Input Qos (160) 15 (0 %) 2033 (100%) 2048
Input Qos (320) 8 (0 %) 2040 (100%) 2048
Input Forwarding (160) 7 (0 %) 2041 (100%) 2048
Input Forwarding (320) 24 (1 %) 2024 (99 %) 2048
Input Unallocated (160) 0 (0 %) 53248 (100%) 53248
Low utilization
switch# show platform hardware qos policer utilization
-------------------------------------------
Policer utilization summary:
Direction Assigned Used Free
-------------------------------------------
Input 2048 ( 12.5%) 4 ( 0.1%) 2044 ( 99.8%)
Output 2048 ( 12.5%) 1 ( 0.0%) 2047 ( 99.9%)
Free 12288( 75.0%) 0 ( 0.0%) 12288(100.0%)
System Resources
Transmit Queue Memory

%C4K_HWPORTMAN-3-TXQUEALLOCFAILED: Failed to allocate the needed queue entries for Gi6/13

• Reserved queue memory for each linecard, exceeding this eats into global pool

• When global pool exhausted, the above message appears

• Options:

• decrease queue depths on a per port basis

• combine classes under the same queue

Monitoring Queue Memory
Entry Sup6-E/6L-E/7L-E Sup8E
Total queue memory 512K 1M
Free Reserve: global pool 100K 50K
CPU, recirc, drop queues 20K 40K
Queue entries per slot1 x = 400K/ nSlots2 x = 910K/nSlots
Queue entries per port on a line card y = x / nPorts3 y = x/nPorts
Queue entries per class transmit queue z = y/nTxQs4 z = y/nTxQs
1. In a redundant chassis, two supervisor slots are treated as one
2. nSlots – number of Slots
3. nPorts – number of Ports in a line card
4. nTxQs – number of transmit queues in use
Monitoring Queue Memory
switch# show platform software qm
Drop port Tx Queue allocations (Size: 8184, Base: 0x019008)

CPU Subport Tx Queue allocations (TotalSize: 12304) Drop, Recirc, CPU reservations
Recirc Subport Tx Queue allocations(TotalSize: 12288)
…
Global TX Queue reservations
----------------------------

Slot Size Base Addr Current Unused

Addr Entries
---- ---- --------- ------- -------
0 101488 0x021010 0x021010 101488 • 101488 / 48 = 2114 entries/port
1 101488 0x039C80 0x039C80 101488
2 101488 0x0528F0 0x06B550 16 • >2114 entries will eat into global pool
3 101488 0x06B560 0x06B560 101488
4 101488 0x0841D0 0x096B00 25408
5 101488 0x09CE40 0x09CE40 101488
6 101488 0x0B5AB0 0x0B5AB0 101488
7 101488 0x0CE720 0x0CE720 101488
8 101488 0x0E7390 0x0E7390 101488
 Troubleshooting System Resources Commands
CLI Purpose

List IOS process CPU % on IOS-XE show proc cpu detail process iosd sort

Monitor Cat4k platform CPU statistics show platform health

show platform cpu packet statistics

SPAN packets to/from CPU monitor session 1 source cpu

monitor session 1 destination interface <int>

Enable/monitor Cat4k CPU buffer debug platform packet all buffer

show platform cpu packet buffered

Display process memory and buffer show proc mem sort

show process mem <pid>
holdings
show buffers

Display process memory and buffer show proc mem detail proc iosd sort
show proc mem detail proc iosd task <pid>
holdings on IOS-XE
show buffers detailed process iosd

Display Cat4k ACL and policer usage show platform hardware acl statistics utilization brief
show platform hardware qos policer utilization

Display Cat4k queue memory usage show platform software qm

Agenda

 Products Overview
 Troubleshooting
– Method
– System Resources
– Packet path / loss
– Wired/wireless Convergence
 Tools/Tips
Troubleshooting Packet Loss / Path
• Why is any packet sent to port(s), to CPU, or dropped?

Losing packets on the 4k without a clue why?

1. Collect “show tech” and iterations of the below • incrementing counters are most useful
• Some counters are normal
2. Step through the platform • Baseline data is useful

1. Identify counters outside of baseline, find an explanation based on counter meaning

2. Identify unexpected platform programming, work upwards

 Areas Of Investigation
HW-based PHY, stub, packet show interfaces <int> counters all
checks processor, forwarding show platform hardware interf <int> statis
engine show platform software interf <int> statis
Queue/buffer show platform software interf <int> stub statis
failure show platform software interf <int> stub cts statis all
show platform hardware ret rrq
show platform software drop-port
CPU queues CPU controller show platform cpu packet driver
show platform cpu packet statistics
STP L2 lookup show platform hardware stp vlan <vlan>
L3 entries forwarding lookup show platform hardware ip route [ipv4|ipv6] network <net> <mask>
show platform hardware ip route [ipv4|ipv6] host <ip or group>
ACL input classification, show access-list <*acl>
output classification show platform hardware acl input entries static
show platform hardware acl [input|output] entries interface <int> all
show platform hardware acl [input|output] entries vlan <vlan> all
show platform hardware acl [input|output] actions <action>
L2 entries, L2 lookup show plat hard mac add <mac>
floodsets show plat hard ret chain index <index>
show platform hardware floodset vlan <vlan>
 Troubleshooting Packet Loss / Path
PHY and Stub ASIC

 Layer 1 issues
 Malformed frames/packets
 Oversubscription
 Flow-control
 Storm-control Supervisor

Line Card
Stub ASICs

Front Panel Ports

 Troubleshooting Packet Loss / Path
Layer 1 Issues Switch#test cable-diagnostics tdr interface gigabitEthernet 3/1
Switch#show cable-diagnostics tdr interface gigabitEthernet 3/1
TDR
Interface Speed Local pair Cable length Remote channel Status
Gi3/1 1Gbps 1-2 0 m Unknown Terminated
• Match speed and duplex
• Isolate bad hardware using known good hardware
• Specific to end device? Patch/line cord? Front panel port? Linecard?
• Exclude patch panel if possible
• Peer misbehaving? Sniff wire for malformed frames

switch# show interfaces g5/5 count errors | exclude \ 0\ *0\ *0\ *0

Port CrcAlign-Err Dropped-Bad-Pkts Collisions Symbol-Err
Gi5/5 23736730 0 0 0
Port Undersize Oversize Fragments Jabbers
Port Single-Col Multi-Col Late-Col Excess-Col
Port Deferred-Col False-Car Carri-Sen Sequence-Err
See Appendix for Error descriptions
 Troubleshooting Packet Loss / Path
Layer 1 Issues

switch# show platform software interface gigabitEthernet 1/1 stub statistics

XgstubMan(0:N-0)Port( 1 ) Rx Stats:
…
OverrunPackets : 0
AlignmentErrorPackets : 0
Platform commands can narrow down stub
FcsErrorPackets : 0
SymbolErrorPackets : 0 ASIC vs packet processor
InvalidOversizePackets : 0
Ipv4HdrChecksumErrorPackets : 0
Ipv4HdrErrorPackets : 0
Ipv6HdrErrorPackets : 0
…

switch# show platform software interface gigabitEthernet 1/1 statistics

Superport8(Gi1/1-6) Non-Zero Software Statistics
…
RxSequenceErrors : 255 Note: counters may increment during plug / unplug
RxSymbolErrors : 255
 Troubleshooting Packet Loss / Path
Layer 1 Issues
Monitor for link flap via syslog
(config)# logging event link-status global
(config-if)# logging event link-status Configurable globally or per-interface
switch# show platform software interface all | inc downs:|PimPhyport
…
GalGlmPort(0:N/21), Active? : true, PimPhyport Name : Gi1/22, EpmPortMan Name : EpmPortMan(0:N/21)
Name( EpmPortMan(0:N/21) ), PimPhyport name( Gi1/22 )
#link downs: 41712
Get total number of flaps since switch boot
switch# show platform software interface gi1/1 mii Compare with switch uptime
…
0x00 ControlReg 0x1140
0x01 StatusReg 0x79C9
…
0x04 AutoNegAdvReg 0x01E1
0x05 AutoNegLinkPartnerAbilityReg 0x0000 This command should be run twice
0x06 AutoNegExpansionReg 0x0064
0x07 AutoNegNextPageTransmitReg 0x2001 Use the second results, decode standard
… 802.3 registers
0x09 1000BaseTControlReg 0x0F00
0x0A 1000BaseTStatusReg 0x0000
 Troubleshooting Packet Loss / Path
Oversubscription: stub/supervisor port buffers

completely even traffic flow does not occur in real-world

– 2:1 1Gbps != (real world) 500 Mbps x 2 ports
– 2:1 10bps != (real world) 5Gbps x 2 ports

max
 ingress traffic on oversubscribed ports avg
– control on the peer device min

 egress oversubscription
– consider multi-path
 Troubleshooting Packet Loss / Path
Flow control

 Switch may send pause toward end-device if rx buffer passes high watermark
 Stub will pause toward supervisor if end-device signals pause

3 Drops Packet
1. Device sends pause to stub
Processor
2. Stub sends pause to packet
processor 2
Pause
3. Packet processor pauses tx-
queue Stub ASICs

1 Pause
Front Panel Ports
 Troubleshooting Packet Loss / Path
Tx Oversubscription and Flow Control

 Tx oversubscription will result in tx-queue drops

 Pause frames from a peer will stop tx-queue processing

Queue 8 is the default queue with

no QoS Configured
switch# show interfaces g2/47 counters detail | begin Drops
Port Tx-Drops-Queue-5 Tx-Drops-Queue-6 Tx-Drops-Queue-7 Tx-Drops-Queue-8
Gi2/47 0 0 0 37748571

switch# show interfaces g2/47 counters detail | begin RxPause

Port Rx-No-Pkt-Buff RxPauseFrames TxPauseFrames PauseFramesDrop
Gi2/47 0 130 0 0
 Troubleshooting Packet Loss / Path
Rx Oversubscription

 RxFifo stub overrun will be seen during Rx oversubscription

 Packet buffer depletion can also cause Rx-No-Pkt-Buff

switch # show interface gi1/13 | include overrun

0 input errors, 0 CRC, 0 frame, 86432 overrun, 0 ignored

switch# show interface gi1/13 counter all | begin Rx-No

Port Rx-No-Pkt-Buff RxPauseFrames TxPauseFrames PauseFramesDrop
Gi1/13 206658 0 0 0

switch# show platform software interface g1/13 stub stat | in Overrun

OverrunPackets : 206658 (look for Rx Stats)
 Troubleshooting Packet Loss / Path
Packet Processor

%C4K_SWITCHINGENGINEMAN-4-IPPLLCINTERRUPTFREELISTBELOWHIPRIORITYTHRESHOLD: IPP
LLC freelistBelowHiPriorityThreshold interrupt FreeListCount: 2058,
lowestFreeCellCnt: 0
Shared Packet
Memory

 Central packet memory exhaustion Packet

Processor
 Deep transmit queues
 Egress oversubscription (example: SPAN) Supervisor
 Jumbo frames Line Card
 Troubleshooting Packet Loss / Path
Oversubscription: packet memory exhaustion

Shared Packet
Memory
Deep buffers and congestion Full

Limited gain (temporary buffering) 2

Switch-global expense (ingress and egress) Packet Processor

1 Deep Q
1. Deep egress queue fills
2. Packet memory consumed Drops
3. Packet memory unavailable for ingress 3
 Troubleshooting Packet Loss / Path
Oversubscription: packet memory exhaustion

Shared Packet
Memory
Reduced buffers during congestion Restricted

Limited expense (smaller threshold on given interface)

Large gain (no packet memory exhaustion) Packet Processor

Drops

Other solutions:
Even out packet port distribution
Egress policers
 Troubleshooting Packet Loss / Path
Packet memory: keeping the FreeList healthy

 128K 256 byte cells in Sup8E, Sup7E, Sup7L-E

switch# show platform hardware interface all | include FreeListCount

FreeListCount : 125062
switch# show platform hardware interface all | include FreeListCount
FreeListCount : 124904 Drop in FreeList will accompany
IPP log message
switch# show interfaces g2/47 counters detail | begin Drops
Port Tx-Drops-Queue-5 Tx-Drops-Queue-6 Tx-Drops-Queue-7 Tx-Drops-Queue-8
Gi2/47 0 0 0 37748571

(config)# policy-map egress_queue_limit

class class-default 1. Locate interfaces tail dropping
queue-limit 500
2. Reduce tx-queue size OR
3. Modify default queue size
(config)# hw-module system max-queue-limit <value>
 Troubleshooting Packet Loss / Path
Forwarding ASIC

 Stepping through forwarding ASIC stages CPU

 Identifying packet destiny TCAMs NFE

– Punt?
– Drop? Forwarding
– Forward to where? Engine

– Replicate to where?
Supervisor
 Working backwards from ASIC counters Line Card
 Packet Loss / Path: Forwarding ASIC
Location Purpose Most Common Platform Troubleshooting Need

IM Input mapping Vlan and port mapping

L2 L2 lookup Layer 2 destination

IC Input classification ACLs (especially static ACL, which evaluate *all* traffic)
For custom ACL, IOS-level CLI typically all that is needed
NF Netflow Platform troubleshooting not commonly required

IP Input policing IOS-level policer counters typically all that is needed

FL Forwarding lookup L3 Multicast replication

OC Output classification IOS-level CLI typically all that is needed

OP Output policing IOS-level policer counters typically all that is needed

OM Output mapping, Vlan re-mapping

replication Replication counters useful in very high density scenarios
QM Queueing Tx-queue programming
 Packet Loss / Path: Input Mapping
IM  Physical / aggregate port mapping
L2  Vlan mapping
Mapping information used in many platform CLI outputs
IC switch# show platform mapping ports
Interface Superport Subport CompactSubportId PortSet Phyport Aggport PimPhyport
NF Gi1/1 8 1 20 2 13 8 0
…
Gi7/48 35 4 210 8 402 Po1(417) 367
IP
switch# show platform hardware portvlan-map-table interface gigabitEthernet 1/1
FL Aggport( 8 ):
All ports on an Etherchannel share an Aggport
OC ----- PortVlanDirectTable -----
VlanId FwdVlanId SrcMissCtrl TxDropEn VlanTagStripEnOnTx
0 0 SrcMissCopyToCpu False False
OP … Vlan mapping in use
OM ----- PortVlanHashTable -----
Index PartialAggport VlanId FwdVlanId Dir SrcMissCtrl TxDropEn VlanTagStripEnOnTx
QM 1568 8 100 200 Rx SrcMissCopyToCpu - False
3188 8 100 200 Tx - False False
 Packet Loss / Path: Input Mapping / L2 Lookup
IM  Confirm if routing features are enabled on a vlan
L2 switch# show platform hardware rxvlan-map-table vlan 902
Vlan 902:
IC l2LookupId: 902 IPv4 unicast and multicast routing enabled
srcMissIgnored: 0
NF ipv4UnicastEn: 1
ipv4MulticastEn: 1
ipv6UnicastEn: 0
IP ipv6MulticastEn: 0
…
FL switch# show int vl 902 | i SVI
Hardware is Ethernet SVI, address is 001e.f73f.f5bf (bia 001e.f73f.f5bf)
OC switch# show mac address-table vlan 902 | i 001e.f73f.f5bf
902 001e.f73f.f5bf static ip,ipx,assigned,other Switch SVI MAC present in MAC
switch# show plat hard mac add 001e.f73f.f5bf vlan 902
OP … table
Index Mac Address Vlan Type SinglePort/RetIndex/AdjIndex
OM ----- -------------- ----- ---------- ----------------------------
63248 001E.F73F.F5BF 902 SinglePort Cpu aggport(4) ND RouterAddr
QM
 Packet Loss / Path: L2 Lookup
IM  STP state check
L2  SA Learning
IC
switch# show span int gi 7/48 state | i VLAN0002
VLAN0002 forwarding
NF
switch# show platform hardware stp vlan 2 | i Gi7/48
IP Gi7/48 (375) Forwarding

switch(config)# no mac address-table learning vlan 100

FL
switch# show platform hardware rxvlan-map-table vlan 100 | i srcMiss
srcMissIgnored: 1
OC no copies will be sent to CPU for MAC source address learning
switch# show mac add int gi 1/46 | i 902
OP 902 0000.0500.0000 dynamic ip,ipx,assigned,other GigabitEthernet1/46
902 ffff.ffff.ffff system Gi1/46,Gi7/48,Switch
OM HW matches SW
switch# show plat hard mac add 0000.0500.0000 | i 0500|Index
Index Mac Address Vlan Type SinglePort/RetIndex/AdjIndex
QM 27760 0000.0500.0000 902 SinglePort Gi1/46(53) ND SrcOrDst F
 Packet Loss / Path: L2 Lookup
IM • SA Lookup: port security
L2
switch# show run int gi 3/19
IC …
interface GigabitEthernet3/19
NF switchport access vlan 172
switchport mode access
switchport port-security
IP spanning-tree portfast

FL switch# show platform hardware mac vl 172

Flags are:
OC ---------- Traffic sourced from this MAC from any port
D - Drop
ND - Do not drop other than Gi3/19 will be dropped on vlan 172
OP
Index Mac Address Vlan Type SinglePort/RetIndex/AdjIndex
OM ----- -------------- ----- ---------- ----------------------------
2640 0017.9543.EA7F 172 SinglePort Gi3/19(74) ND SrcOrDst
QM 49300 0017.9543.EA7F 172 SinglePort WildcardAggport D SrcOrDst
 Packet Loss / Path: L2 Lookup
IM  DA Lookup: multicast, broadcast
L2 switch# show mac add multi vlan 902 | i 0100.5e01.0101
902 0100.5e01.0101 igmp Gi1/46,Switch
IC
switch# show plat hard mac add 0100.5e01.0101 | i 0100.5E01.0101|Index
NF Index Mac Address Vlan Type SinglePort/RetIndex/AdjIndex
20224 0100.5E01.0101 902 Ret 104444

IP switch# show plat hard ret chain index 104444

RetIndex 104444
FL RetWordIndex: 522220 Link: 1048575(0xFFFFF) FieldsCnt: 1
SuppressRxVlanBridging: true
Vlan: 902 BridgeOnly: N Gi1/46(53) Multicast traffic to 0100.5e01.0101 replicated
OC
here, unless overridden by L3/ACL
Switch# show platform hardware floodset vlan 902
OP Vlan 902:
Unicast Floodset:
OM FloodToCpu: - unknown unicasts will be flooded to these ports
RetIndex: 902
QM Gi1/46(53) Po1(417)
…
 Packet Loss / Path: L2 vs L3 vs ACL
IM  What HW programming will direct the packet?
L2

IC
L3 Entry
ACL Entry
> L2 entry > floodset
NF

IP
Fwdsel relevant to ACL (ic) only when there is a
redirect action
Depends on “fwdsel”
FL

OC switch# show platform hardware ip fwdsel summary Example:

L2Value == other (port/RET) (0):
OP IC L3 entry present, FwdSel=2
L3 0 1 2 3
0 l2 ic ic ic ACL redirect entry present, FwdSel=2
OM 1 l3 ic ic ic
2 l3 l3 ic ic
Winner = ACL (ic)
QM 3 l3 l3 l3 ic
 Packet Loss / Path: Input Classification
IM  SVI and ACL statistics require hardware resources
L2  Not enabled by default
IC switch# show run
…
NF interface Vlan902
ip address 92.92.92.1 255.255.255.0 Enable hardware counters
counter
IP …
ip access-list extended deny
FL deny ip any any Ensure resources are available
hardware statistics
OC …
switch# show platform hardware vlan statistic summary
Region Name First Last First LastUsed Entries Entries
OP Block Block Entry Entry Used Free
Size 2 Counters Region 0 510 0 0 1 2043
OM Size 4 Counters Region 511 1022 2044 - 0 2048

QM VlanStatsTable Programming Complete: Yes

 Packet Loss / Path: Input Classification
IM  ACL examples: static ACL, PBR, PACL
L2 • Watch for increment

IC switch# show platform hardware acl input entries static • Hit does not mean packet count
…
NF CamIndex Entry Type Active Apply QoS Hit Count
-------- ---------- ------ --------- ---------
2 IgmpToCpu Y N/A 14237 (estimate)
IP …
switch# show platform hardware acl input entries start 2 end 2 all
FL …
IP Src : 0.0.0.0 / 0.0.0.0
OC IP Dst : 224.0.0.0 / 240.0.0.0
IP Protocol : igmp / IpProtocolMask IGMP sent to 224/4
…
OP willPort:
ActIdx: 252 StatsIdx: 0 FwdIdx: (Cpu, Cpu: true, CpuEvent: 1, go to CPU
3)

OM switch# show platform hardware acl input actions 252 if FwdSel wins over L3
…
FwdSel: 3
L2Action: (0 = permit, 1 = drop, 2 = redirect)
QM
L2Action: 2
 Packet Loss / Path: Input Classification
IM  ACL examples: static ACL, PBR, PACL
L2
Note: PBR ACLs are removed if
IC switch# show platform hardware acl input entries vlan 901 all adjacency becomes unavailable
…
NF IP Src : 1.1.1.1 / 255.255.255.255
IP Dst : 0.0.0.0 / 0.0.0.0
…
IP ActIdx: 244 StatsIdx: 0 FwdIdx: (Adj, Adj: 8)
Packets sourced from 1.1.1.1/32
FL switch# show platform hardware acl input actions 244
…
will be redirected to adjacency 8 (Po1)
OC FwdSel: 2 If FwdSel wins over L3
…
L3Action: 2
OP
switch# show platform hardware ip adjacency entry 8
OM 000008: vlan: 192 port: Po1 (417) size: 1 ifaId: 20
fwdCtrl: 5 cpucode: 3 sifact4: FwdToCpu sifact6: FwdToCpu
QM sa: 00:1E:F7:3F:F5:BF da: 00:0C:29:6D:1A:ED rwFmt: Unicast
packets: 0 bytes: 0
 Packet Loss / Path: Input Classification
IM  ACL examples: static ACL, PBR, PACL
L2  Note: packets classified as non-IP, IPv4, IPv6 (cannot MAC ACL on an IP packet)
IC switch# show ip access deny
Extended IP access list deny
NF 10 deny ip any any (1056 matches)
switch# show ip int gi 1/2
Inbound access list is deny
IP switch# show plat hard acl inp entr int gi 1/2 all
…
FL IP Src : 0.0.0.0 / 0.0.0.0
IP Dst : 0.0.0.0 / 0.0.0.0
OC IP Protocol : IpProtocolNull / IpProtocolNull All IPv4 traffic will be dropped
…
ActIdx: 254 StatsIdx: 0 FwdIdx: (None, rep: 0) Fwdsel doesn’t matter
OP
L2Action: (0 = permit, 1 = drop, 2 = redirect)
switch# show plat hard acl inp act 254
OM …
FwdSel: 0
QM …
L2Action: 1
 Packet Loss / Path: Input Classification / Policing
IM  Order of operations Ingress Marking
Ingress Policing
Ingress Conditional
L2
Classification
IC flow record microflow Ingress Marking
match ipv4 source address Unconditional Forwarding
NF class-map match-all microflow
match flow record microflow
policy-map ingress
IP class voice-signalling
set dscp cs3 Microflow policing
FL police cir 32000 bc 8000
conform-action transmit Normal policer • Flexible Netflow
OC exceed-action set-dscp-transmit cs1 Conditional Marking • Class-map matching FNF
exceed-action set-cos-transmit 1
class microflow • Policer
OP police cir 100000
conform-action transmit
OM exceed-action drop
class class-default Classification
QM set dscp default
set cos 0 Unconditional Marking
 Packet Loss / Path: Input Classification / Policing
IM  Monitoring ingress Qos
L2
switch# show policy-map interface gigabitEthernet 1/46
GigabitEthernet1/46
IC Class-map stats are shared across interfaces with the
same policy map
Service-policy input: ingress
NF
Class-map: voice-signalling (match-all)
IP 28283457437 packets
Match: dscp ef (46)
QoS Set
FL
dscp cs3 • Ensure counters increment
police:
OC • Classification displays using the packet counts
cir 32000 bps, bc 8000 bytes
conformed 76128704 bytes; actions: • Policing displays using bytes
OP transmit
exceeded 1810581188160 bytes; actions:
set-dscp-transmit cs1
OM
set-cos-transmit 1
conformed 32000 bps, exceed 761238000 bps
QM
 Packet Loss / Path: Forwarding Lookup
IM  L3 unicast destination lookups
L2
Remember: unicast traffic won’t be destination-routed unless:
IC switch# show ip route 192.168.200.200 • routing is enabled on the vlan
Routing entry for 192.168.200.0/24
NF Known via "static", distance 1, metric 0 • traffic is sent to L3 MAC
Routing Descriptor Blocks:
* 192.168.100.100
• FwdSel of route wins over ACL
IP Route metric is 0, traffic share count is 1
switch# show ip arp | i 192.168.100.100
FL Internet 192.168.100.100 0 000c.296d.1aed ARPA Vlan192
switch# show mac address dynamic | i 000c.296d.1aed
OC 192 000c.296d.1aed dynamic ip,ipx,assigned,other Port-channel1
switch# show platform hardware ip route ipv4 network 192.168.200.0 255.255.255.0
Block: 0 En: true EntryMap: LSB Width: 80-Bit Type: Dst
OP …
000022: v4 192.168.200.0/24 --> vrf: Global Routing Table (0)
OM adjStats: true fwdSel: 2 mrpf: 0 (None) fwdIdx: 0 ts: 0
adjIndex: 8 vlan: 192 port: Po1 (417)
QM fwdCtrl: 5 cpucode: 3 sifact4: FwdToCpu sifact6: FwdToCpu
sa: 00:1E:F7:3F:F5:BF da: 00:0C:29:6D:1A:ED
 Packet Loss / Path: Output Classification / Policing
IM  Order of operations Output Marking
Output Policing
Output Conditional
L2
Classification
IC Output Marking
Unconditional Queuing
NF
policy-map egress
class voice Classification
IP set dscp ef Marking
set cos 5
FL priority
police cir percent 33 Policing
OC class voice-control
set dscp af31
set cos 3
OP bandwidth remaining percent 5
Queuing
class class-default
OM dbl

QM
 Packet Loss / Path: Output Classification / Policing
IM • Monitoring egress Qos
switch# show policy-map int g1/36 output
L2 GigabitEthernet1/36

IC Service-policy output: AutoQos-VoIP-Output-Policy Class-map stats are shared across interfaces with the
same policy map
NF Class-map: AutoQos-VoIP-Bearer-QosGroup (match-all)
625530530 packets
Match: qos-group 46
IP QoS Set
ip dscp ef
FL cos 5
priority queue:
OC Transmit: 32344068480 Bytes, Queue Full Drops: 0 Packets
police:
cir 33 %
OP cir 330000000 bps, bc 10312500 bytes • Ensure counters increment
conformed Packet count - n/a, 32335870400 bytes; actions:
• Classification display using the packet counts
OM transmit
exceeded Packet count - n/a, 7813435520 bytes; actions: • Policing display using bytes
QM drop
• Queue full drops are in packets
conformed 325185000 bps, exceed 97368000 bps
 Packet Loss / Path: Output Queuing
IM  DBL processing (if packet is not scheduled for drop)
L2  Descriptor enqueued in queue memory
IC

NF switch# show platform hardware interface gigabitEthernet 1/1 tx-queue

…
IP Phyport TxQ Head Tail Pre Empty Num BaseAddr Size Shape-Ok
Empty Packets TxQ Subport
-------------------------------------------------------------------------------
FL
Gi1/1 0 0x0000 0x0000 True 0 0x20D10 16 True True
Gi1/1 1 0x0000 0x0000 True 0 0x00000 0 True True
OC Gi1/1 2 0x0000 0x0000 True 0 0x00000 0 True True
Default queues configured
Gi1/1 3 0x0000 0x0000 True 0 0x00000 0 True True
OP Gi1/1 4 0x0000 0x0000 True 0 0x00000 0 True Currently empty
True
Gi1/1 5 0x0000 0x0000 True 0 0x00000 0 True True
Gi1/1 6 0x0000 0x0000 True 0 0x00000 0 True True
OM
Gi1/1 7 0x0000 0x0000 True 0 0x20D20 3152 True True

QM
 Packet Loss / Path: Output Queuing
policy-map egress_queueing
class dscp32-48 Tx Q Class
police cir 990000 Low priority queues can be
conform-action transmit starved, policer recommended 0 dscp32-48
exceed-action drop 5 dscp16-31
priority
class dscp0-15 6 dscp0-15
bandwidth 250000
7 dscp49-63, class-default
queue-limit 400
class dscp16-31
bandwidth 250000
queue-limit 512
class class-default First and last appear where expected, middle reversed

switch# show platform hardware interface g2/48 tx-queue

…
Phyport TxQ Head Tail Pre Empty Num BaseAddr Size Shape-Ok
Empty Packets TxQ Subport
-------------------------------------------------------------------------------
Gi2/48 0 0x0000 0x0000 True 0 0x5ECE8 352 True False
Gi2/48 1 0x0000 0x0000 True 0 0x00000 0 True False
Gi2/48 2 0x0000 0x0000 True 0 0x00000 0 True False Last queue is default queue
Gi2/48 3 0x0000 0x0000 True 0 0x00000 0 True False
Gi2/48 4 0x0000 0x0000 True 0 0x00000 0 True False In this example, it is non-empty
Gi2/48 5 0x0000 0x0000 True 0 0x5E958 512 True False
Gi2/48 6 0x0000 0x0000 True 0 0x5EB58 400 True False
Gi2/48 7 0x008A 0x0088 False 1421 0x5EE48 1520 True False
 Packet Loss / Path: ASIC Drop Categories
 show platform software drop-port shows global ASIC drop events (not per interface)
 these counters are frequently expected; baseline and/or high packet rate very useful

Common Drop Event Reason Typical Description

BridgeToRxPortDrop received in a vlan with no other ports, replicated to a floodset/entry where ingress port
was a member
DblDrop packets dropped by DBL (including DBL on CPU ports)
InpL2AclDrop, InpL3AclDrop, packets denied by ACL
OutL2AclDrop, OutL3AclDrop
rplErrDrop broadcast/multicast packets dropped while being replicated, many normal reasons to
increment, including: rpf failure, floodset containing drop port, packets replicated to the
CPU but also bridged to a floodset/entry containing the CPU
SptDrop spanning-tree drop; packets dropped because a port is not in a forwarding state
SrcHitDrop dropped at source learning stage; example: static MAC drop entry

TxQueFullDrop a tx port is oversubscribed

 Packet Loss / Path: CPU Queues
 Check for transient flooding / loss versus stuck queue
 Decode queue meaning with show platform software cpu events

• High “Kept” indicates high rate of traffic

switch# show plat cpu pack driv • Incrementing “Drop No Cell” indicates
Forerunner Packet Engine 1.83 (0)
queue oversubscription
Receive Queues: received packets summary
Qu Capac Guara CurPo Unpro Accum Kept BperP Packets
2 2512 112 610 0 2 2 73 610
58 512 256 37 12 5 511 216 591103

Receive Queues: dropped packets summary

Qu Total Packets Drop No Cell Drop Overrun
However, combine high “Kept” with:
Drop Underrun
58 591103 43623295103 0 • CurPo
0 does not increment
Transmit Queues • Drop No Cell does increment
Qu PosAdd Pendng Packets Bytes … queue 58 is stuck!
0 595 0 8633668179 663318795241
1 863 0 5315423 363150782
Agenda

 Products Overview
 Troubleshooting
– Method
– System Resources
– Packet path / loss
– Wired/wireless Convergence
 Tools/Tips
Wired/wireless Convergence
New capability on Sup8E with IOS-XE 3.7.0E

 Overview

 Converged Mode

 Troubleshooting

Quick Start
/Configuring
Wireless

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-4500-series-switches/guide-c07-733704.html
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/XE3-7-0E/wireless/configuration-guide/b_37e_4500sup8e_cg.pdf
Sup8E Wired/Wireless Convergence
Overview
 In-built daughter-card which enables wireless
capabilities

 Supported only with cat4500es8-universalk9*

(Crypto) images

 Not supported in VSS

 10th slot: No line card supported on 4510R-E

chassis, only 47xx on 4510R+E chassis

 ‘Install boot’ method is required

Sup8E Wired/Wireless Convergence
Converged Mode *.bin file needs to be copied into bootflash

rommon 1 >boot bootflash:cat4500es8-universalk9.SPA.03.07.00.E.152-3.E.bin

Loading image !!!!!!!!!!!!!!!!!!!!!!!
Checking digital signature....
[mem:/cat4500es8-firmware] Boot .bin file and run software
Digitally Signed Release Software with key version A
…
expand file command

Switch#software expand file bootflash:cat4500es8-universalk9.SPA.03.07.01.E.152-3.E1.bin

Preparing expand operation ...
[5]: Expanding bundle bootflash:cat4500es8-universalk9.SPA.03.07.01.E.152-3.E1.bin
[5]: Copying package files
[5]: Package files copied
[5]: Finished expanding bundle bootflash:cat4500es8-universalk9.SPA.03.07.01.E.152-3.E1.bin

Switch(config)#boot system bootflash:packages.conf

Switch#wr mem
Switch#reload
Set switch to boot packages.conf file
Sup8E Wired/Wireless Convergence
Converged Mode
Switch#show module
Chassis Type : WS-C4510R+E
…
Mod Ports Card Type Model Serial No.
…
5 12 Sup 8-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP8-E CAT1749L0M3

Mod Redundancy role Operating mode Redundancy status

Switch now in converged
----+-------------------+-------------------+---------------------------------- wireless
5 Active Supervisor SSO Active mode (install boot mode)

Mod Submodule Model Serial No. Hw Status

----+-----------------------+-----------------+------------+----+---------
5 Daughter Card WS-UA-SUP8E CAT1749L6FL 1.0 Ok

Switch#dir bootflash: | inc dc_console

32661 ---- 11678 May 21 2015 14:28:54 -07:00 dc_console_log-20150514-094417-UTC

DC console logs have clues if

wireless mode does not come up
Sup8E Wired/Wireless Convergence
Path of Packet Wired traffic
Shared Packet CPU
Memory
Wireless
DC TCAMs1 NFE2 Wireless Daughtercard with 2 10G links
to Packet Processor
20Gbps of throughput
Packet Forwarding
Processor Engine

Supervisor

Line Card
Stub ASICs

Front Panel Ports

Sup8E Wired/Wireless Convergence
Path of Packet Wireless traffic (1)
interface GigabitEthernet3/1
Shared Packet
Memory
CPU switchport access vlan 123
Wireless switchport mode access
DC TCAMs1 NFE2
interface Vlan123
ip address 192.168.21.1 255.255.255.0

wireless mobility controller

Packet Forwarding
wireless management interface Vlan123
Processor Engine

Supervisor

Line Card
Stub ASICs
Wireless packets are redirected to DC

Front Panel Ports

Sup8E Wired/Wireless Convergence
Path of Packet Wireless traffic (2)
Switch#show plat hardware acl input entries
Shared Packet CPU
Memory interface gigabitEthernet 3/1 all
Wireless
DC …
TCAMs1 NFE2
Idx: 7 Hit: false
IP Src : 0.0.0.0 / 0.0.0.0

Packet Forwarding IP Dst : 192.168.21.1 / 255.255.255.255

Processor Engine IP Protocol : udp / IpProtocolMask
UDP Src Port : 0 / 0
Supervisor UDP Dst Port : 5246 / 65535 …
ActIdx: 244 StatsIdx: 0 FwdIdx: (Adj, Adj: 10)
Line Card
Stub ASICs
Switch#show platform hardware adjacency entry 10
000010: vlan: 123 port: Po255 (671)
Front Panel Ports
Sup8E Wired/Wireless Convergence
Path of Packet Wireless traffic (2)
Shared Packet CPU
Memory
Wireless
DC TCAMs1 NFE2 Wireless operations complete packet is
returned again to Forwarding engine
pipeline

Packet Forwarding Packet is switched out front panel port

Processor Engine

Supervisor

Line Card
Stub ASICs

Front Panel Ports

 Sup8E Wired/Wireless Convergence
Wireless Daughtercard Etherchannel
Switch#show platform mapping ports | inc Te5/
Te5/1 66 0 16 17 204 392 384
Te5/2 67 0 18 17 206 393 385
Te5/3 68 0 20 18 208 394 386
Te5/4 69 0 22 18 210 395 387
Te5/9 64 0 24 16 212 Po255(671) 392
Te5/11 65 0 26 16 214 Po255(671) 394

Wireless mode only 4 uplinks available in

redundant/non-redundant configurations on 10-slot Te5/9 and 5/11 part of internal
chassis Portchannel configured between
Packet Processor and Wireless DC.
Sup8E Wired/Wireless Convergence
Packets Rx/Tx to Wireless DC
Switch#show platform hardware interface tenGigabitEthernet5/9 statistic
Switch Phyport Te5/9 Non-Zero Hardware Statistics
TxBytesTxQ0 : 192
TxBytesTxQ7 : 288

Superport64(Te5/9) Non-Zero Software Statistics

RxPackets64 : 7
TxPackets64 : 6
RxPackets65to127 : 1
TxPackets65to127 : 1
RxPackets128to255 : 1
RxMcastPackets : 5
TxMcastPackets : 3 Packet processor superport interface
RxUcastPackets : 4 to Wireless DC
TxUcastPackets : 4
RxGoodBytes : 672
TxBytes : 480
Sup8E Wired/Wireless Convergence
Wireless DC CPU, Memory usage

Switch#show processes cpu location active-dc

Core 0: CPU utilization for five seconds: 2%; one minute: 2%; five minutes: 2%
Core 1: CPU utilization for five seconds: 0%; one minute: 0%; five minutes: 0%
Core 2: CPU utilization for five seconds: 0%; one minute: 1%; five minutes: 1%
Core 3: CPU utilization for five seconds: 0%; one minute: 1%; five minutes: 1%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 1560 1093 1427 0.00 0.00 0.00 0 init
Location keyword introduced to
2 10 283 35 0.00 0.00 0.00 0 kthreadd
distinguish baseboard vs wireless
Switch#show processes memory location active-dc daughercard
System memory : 1934480K total, 847036K used, 1087444K free, 123824K kernel reserved
Lowest(b) : 640243296
PID Text Data Stack Heap RSS Total Process
1 324 384 88 276 1736 4604 init
2 0 0 0 0 0 0 kthreadd
Agenda

 Products Overview
 Troubleshooting
 Method
 System Resources
 Packet path / loss
 Wired/wireless Convergence
 Tools/Tips
 Tools: Wireshark

 Available on Sup8E, Sup7E, Sup7L-E, 4500X

 Onboard full packet capture, filter, decode / display
 Up to 8 instances supported

Wireshark Best Practices

 Do not display directly to console without a buffer, file or a duration limit

 Write to PCAP file on storage, display on switch or using laptop Wireshark GUI

 Only the core filter is implemented in hardware as ACLs. Use a restricted filter to avoid high CPU
Tools: Wireshark
Display
Filter Console

Forwarding IOS-XE Display

Engine
Ring Buffer Filter
Capture
Core Filter
Filter File

switch# monitor capture mycap int gi 1/46 in match ipv4 protocol tcp 10.1.1.1/32 any file location
bootflash:mycap.pcap limit duration 3

switch# monitor capture mycap start

*Apr 15 17:56:24.291: %BUFCAP-6-ENABLE: Capture Point mycap enabled.
*Apr 15 17:56:27.720: %BUFCAP-6-DISABLE_ASYNC: Capture Point mycap disabled. Reason : Wireshark session
ended

switch# show monitor capture file bootflash:mycap.pcap display-filter "ip.ttl == 100“

1 0.000000 10.1.1.1 -> 91.91.91.100 TCP [TCP ZeroWindow] 0 > 0 [<None>] Seq=1 Win=0 Len=2
 Tools: Wireshark

Troubleshooting Steps Commands

Create a monitor monitor capture mycap <interface | vlan | control-plane>

Add core filter monitor capture mycap [access-list <acl> | match <in-line match CLI>]
Display monitor details show monitor capture
Start/stop a monitor session monitor capture mycap start | stop

Display a pcap file show monitor capture file <filename>

Display a pcap file in detail show monitor capture file <filename> detailed
Display a pcap file with filter show monitor capture file <filename> display-filter “filter-detail”
Check if wireshark is running show proc cpu | inc dumpcap
 Tools: Embedded Event Manager
 Extremely versatile tool for monitoring, automating, working around issues
 (a) What do I want to detect? (b) What do I want to do after that?
event manager applet high-cpu
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.10.1 get-type exact entry-op ge entry-val “80" poll-interval 10
action 1.0 syslog msg "HIGH_CPU! CPU is at: $_snmp_oid_val“
action 2.0 cli command "enable"
action 2.1 cli command "show process cpu | redirect bootflash:cpu.txt"
Collect process CPU usage when CPU is high
action 2.2 cli command "configure terminal"
action 2.3 cli command "event manager scheduler suspend“

%HA_EM-6-LOG: TEST: HIGH_CPU! CPU is at: 99

event manager applet interface-flapping

event syslog pattern ".*UPDOWN.*GigabitEthernet1/1.*" occurs 4
action 1.0 syslog msg “GigabitEthernet Interface 1/1 changed state 4 times“
action 2.0 cli command "enable"
action 2.2 cli command "configure terminal"
action 2.3 cli command “interface GigabitEthernet1/1 “
Bring an interface down when it flaps too frequently
action 2.4 cli command “shutdown”
 Tools: EEM Netflow Integration
1. Packets with TTL=1 sent to the switch (TTL=1 streams can cause high CPU)
2. NetFlow Engine collects the flow capturing the TTL value:
switch# sh runn flow record ttl switch(config)# event manager applet ttl
match ipv4 ttl event nf monitor-name "ttl"
match ipv4 protocol event-type create event1 entry-value "2"
match ipv4 source address field ipv4 ttl entry-op lt
match ipv4 destination address
collect counter bytes action 1.0 syslog msg
collect counter packets "Flow Monitor $_nf_monitor_name reported Low TTL
collect timestamp sys-uptime first for $_nf_source_address $_nf_dest_address"
collect timestamp sys-uptime last
switch# sh runn flow monitor ttl
Current configuration: check – show flow monitor ttl cache format record for IP TTL: 1
flow monitor ttl
record ttl
cache timeout active 40
switch# sh runn int gi 6/1
3. EEM triggers a syslog
no switchport when flow is detected:
ip flow monitor ttl input %HA_EM-6-LOG: ttl: Flow Monitor ttl reported Low TTL for
ip address 10.10.10.2 255.255.255.254 10.10.10.3 10.10.10.4
Tips: Crashes Coredump not produced by default;
configure with exception coredump

Switch#show exception information

Exception configuration information
Coredump file - enabled,compressed
Maximum number of files

Switch#dir crashinfo:
Directory of crashinfo:/

24194-rw- 0 Dec 18 2013 04:13:06 -08:00 koops.dat

24198 -rwx 1679107 Oct 22 2014 14:38:41 -07:00 crashinfo_plogd_20141022-213819-UTC
24199-rwx 923370 Oct 22 2014 14:38:41 -07:00 fullcore_plogd_20141022-213819-UTC

Switch#dir kinfo:
Directory of kinfo:/
Gather latest files from both these
No files in directory directories

65624064 bytes total (65361920 bytes free)

koops.dat
Tips: Crashes (Wireless DC)
Switch#show exception files all
Exception crashinfo files all

NODE: LOCAL
============
Recent Crashinfo file:
crashinfo:crashinfo_plogd_20141022-213819-UTC
crashinfo:
crashinfo_iosd_20141022-213712-UTC Lists wireless DC crash files
crashinfo_plogd_20141022-213819-UTC
fullcore_plogd_20141022-213819-UTC
crashinfo-dc:

Switch#dir crashinfo-dc:
Directory of crashinfo-dc:/

12 -rw- 0 May 14 2015 14:52:48 -07:00 cilogs

13 -rw- 0 May 14 2015 14:52:49 -07:00 koops.dat
Tips: Miscellaneous
 Enable NTP to troubleshoot across switches
 Include date and time for debug and log messages
service timestamps [debug, log] msec localtime show-timezone

 Automatically output time and CPU utilization with each command (exec mode)
terminal exec prompt timestamp

 When logging the console, add comments and prefix with “!” to avoid error messages
switch#!!! show module after peer reload

switch# show module

Tips: Make Life Easier
 Search Bug Toolkit for known issues
 Output Interpreter to decode command output
 System Message Guide for mitigation recommendations
 Smart Call Home in 12.2(52)SG
 Catalyst 4000 Troubleshooting TechNotes
 Catalyst 4500 Configuration Guide and Release Notes
 NetPro discussion groups on http://www.cisco.com
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
• Complete your session surveys
though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online