BRKCRS 3147 Advanced Troubleshooting of The ASR1K and ASR4400 Made Easy 2014 Milan 90 Mins PDF

ASR1K and ISR445x
Troubleshooting Made Easy

BRKCRS-3147
Frederic Detienne
Agenda
 Platform and Hardware Architecture
 Software Architecture
 Day in the the Life of a Normal Packet
 Advanced Example: IPsec Control Plane Programming
 Debugging strategies
 Road to Simplification: Part I, Data Plane Debugging
 Understanding and Extracting ESP Logs
 Road to Simplification: Part II, Control Plane Unified Show Commands
 Road to Simplification: Part III, Deep Data Plane Debugging
 Future: Resource Consumption Monitoring
 Wrapping up...
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Objectives
 Understand the ASR 1K and ISR 445x architecture
– software
– hardware
– relationship between the two
 Understand how features process packets through IOS-XE
 Understand how to easily debug the platform
– long journey
– presentation of recent serviceability enhancements
Platforms and Hardware Architecture
Cisco ASR 1000 Series Routers: Overview
Compact, Powerful Router Business-Critical Resiliency Instant-on Services Delivery
Instant On
Service Delivery
 Line-rate performance 2.5G to  Fully separated control and  Integrated firewall, VPN,
200G+ with services enabled forwarding planes encryption, DPI, CUBE
 Investment protection with modular  Hardware and software  Scalable on-chip service
engines, IOS CLI and SPAs for I/O redundancy provisioning through software
licensing
 Hardware based QoS engine with  In-service software upgrades
up to 472K queues
One IOS-XE Feature Set

ASR 1013
ASR 1006
ASR 1004
ASR 1001 ASR 1002-X
2.5-5 5-36 10-40 10-100 40-200

Gbps Gbps Gbps Gbps Gbps
Chassis Options: ASR 1002-X
SPAs
4 x 1GE
ESP
2RU
RP/SI
P
Chassis Options: ASR 1004
SPAs
SIP
ESP 4RU
RP
Rack Mount &
Chassis Options: ASR 1006 Cable Management
SPAs
SIP
6RU
ESP
RP
ASR1K Building Blocks
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
QFP interconn. GE switch interconn. GE switch QFP
Crypto Crypto
QFP
Assist. PPE BQS Assist. PPE BQS
Route Processor
interconn. Handles control plane traffic interconn.
Manages system
Embedded Service Processor

Handles forwarding plane traffic Midplane
SIP interconn. SIP interconn. SIP interconn.
SPA SPA SPA

IOCP IOCP IOCP
Aggreg. Aggreg. Aggreg.
SPA SPA SPA SPA SPA SPA
SPA Interface Processor

Houses SPA’s
Queues packets in & out (FIFO)
System Architecture Control Plane
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
Crypto Crypto
EOBC switch in RP
interconn. interconn.
Midplane
Ethernet Out of Band Channel
(aka EOBC)
Inter Integrated Circuit (I2C) Bus 1Gbps Ethernet bus
SIP interconn. SIP interconn. SIP interconn. Used by RP to program system
Slow (few kbps)
Used for system monitoring Used by system to notify RP
(temp., OIR, fan speed,…)
SPA SPA SPA
IOCP IOCP IOCP
SPA Control Link

Works between the SPA’s and SIP
System Architecture Forwarding Plane
Hypertransport
10 Gbps Ethernet
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
Crypto Crypto
Embedded Service Interconnect
aka ESI Bus
interconn. 11.2 – 40 Gbps Forwarding Bus interconn.
Centralized Architecture Midplane

All traffic flows through ESP
SIP interconn. SIP interconn. SIP interconn.
SPA SPA SPA

IOCP IOCP IOCP
RP
CPU
Route Processor Architecture
interconn. GE switch
Highly Scalable Control Plane Processor
Route Processor
System Logging
Manages all chassis functions Not a traffic interface!
Core Dumps
Runs IOS Management only
Mgmt Console BITS

USB 2.5’’
Ethernet & Aux (input & output)
Hard disk
Card Infrastructure
Runs IOS, Linux OS
Manages boards and chassis
33MB
IOS Memory: RIB, FIB & NVRAM
other processes RP1: 1GB
Determines BGP routing CPU CPU Bootdisk RP2: 2GB
table size
RP1: 4GB Memory (1.5 – 2.66 GHz Dual-core)
RP2: 8&16GB Stratum-3 Network
clock circuit GE, 1Gbps
I2CChassis I 2C
Management Bus ESI EOBC SPA Control
Interconnect Gig Eth Switch Output Input SPA Bus
clocks clocks
ESI, 11.2-40 Gbps
SPA-SPI, 11.2Gbps
Hypertransport, 10Gbps
Other
SIPs ESPs RP Misc ESPs SIPs ESPs RP SIPs SIPs RP
Ctrl
ESP
FECP
ESP10 Block Diagram Crypto

Assist.
QFP
PP BQ
E S
intercon.
Reset / Pwr Ctrl Packet Buffer

TCAM Resource DRAM Part Len / BW
DRAM
(10Mbit) (512MB) SRAM
Temp Sensor (128MB)
EEPROM
QFP
DDRAM Packet Processor Engine BQS
Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5

(OBFL,…) FECP E-CSR
JTAG Ctrl PPE6 PPE7 PPE8 … PPE40
PCI* E-RP*
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430) SPI Mux
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Interconnect Hypertransport, 10Gbps
Other
RPs RPs ESP RPs SIPs
ESP
FECP
ESP10 Block Diagram (comments) Crypto

Assist.
QFP
PP BQ
E S
Quantum Flow Processor intercon.

Forwarding Engine Control Subsystem
ProcessorReset / Pwr Ctrl Responsible for forwarding
Packetpackets
Buffer
TCAM Resource DRAM Part Len / BW
Manages board (10Mbit) (512MB)
DRAM
SRAM
Programs QBS,
TempPPE,
SensorCrypto (128MB)
Linux Kernel
EEPROM
QFP
Buffering Queuing & Scheduling
Executes complex QoS scheduling
(shapers, LLQ’s,…)
Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5 Queues and schedules packets in
(OBFL,…) FECP E-CSR
due time
JTAG Ctrl PPE6 PPE7 PPE8 … PPE40
PCI* E-RP*
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430) SPI Mux
DRAM Interconnect Interconnect Hypertransport, 10Gbps
Other
ESP200 Block Diagram
Packet Buffer Packet Buffer
TCAM Resource DRAM Resource DRAM
DRAM DRAM
(80Mbit) (2GB) (2GB)
(512MB) (512MB)
Reset / Pwr Ctrl QFP QFP

Packet Processor Engine BQS Packet Processor Engine BQS
Temp Sensor PPE PPE PPE PPE PPE PPE PPE PPE PPE PPE
1 2 3 4 5 1 2 3 4 5
EEPROM PPE PPE PPE … PPE PPE PPE PPE … PPE

6 7 8 40 6 7 8 40
Dispatcher Dispatcher
DDRAM Packet Packet
Buffer Buffer
Boot Flash TCAM Resource DRAM

Packet Buffer
Resource DRAM
Packet Buffer GE, 1Gbps
(OBFL,…) FECP (80Mbit) (2GB)
DRAM
(512MB)
(2GB)
DRAM
(512MB)
I 2C
SPA Control
SPA Bus
JTAG Ctrl QFP QFP ESI, 11.5 or 23Gbps
Packet Processor Engine BQS Packet Processor Engine BQS SPA-SPI, 11.2Gbps
PPE PPE PPE PPE PPE PPE PPE PPE PPE PPE Hypertransport, 10Gbps
1 2 3 4 5 1 2 3 4 5
Other
PPE PPE PPE … PPE PPE PPE PPE … PPE
6 7 8 40 6 7 8 40
Dispatcher Dispatcher
Packet Packet
Buffer Buffer
Reset / Pwr Ctrl

Memory Crypto Pkt Re-
Dispatcher Interconnect order Logic
Memory Crypto
RPs RPs SIPs
BRKCRS-3147
ESP RPs
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
ESI Capacity by ESP-xxx and SIP-xxx
 Enhanced SerDes Interconnect (ESI) links over midplane
carry
– packets between ESP and other cards (SIPs, RP & other ESP)
QFP Complex – network traffic to/from SPA SIP’s
– punt/inject traffic to/from RP
11.2Gbps 11.2Gbps 25.6Gbps 40+G I/L – state synchronization to/from standby
SPI4.2 SPI4.2 eSPI
ESP-10G Interc.  Additional full set of ESI links to/from standby ESP (not
ESP-10G Interc. shown)
 CRC protection of packet contents

ESP-20G Interconnect
 ESP-10G: 1x11.5G ESI to each SIP slot
ESP-40 G Interconnect  ESP-20G: 2x11.5G ESI to two SIP slots; 1x11.5G to third
SIP slot
 ESP-40G:
– 2x23G ESI* to all three SIP slots
– could also support a 6-SIP chassis with 1 ESI to each (e.g. voice
application)
– also 23G between two ESP-40G’s
 SIP-10G: supports 1x11.5G mode only
 SIP-40G: supports 1x11.5G, 2x11.5G, 2x23G

Other RP1 RP0 SIP0 SIP1 SIP2
ESP
ASR1004 ASR1006
Embedded Services Processor – The Real Thing
Interconnect ASIC
SPI MUX
TCAM Crypto
Engine
FECP
QFP Subsystem CPU
PPE + BQS
FECP
DRAM
PPE BQS
DRAM Packet
DRAM
Cisco “Quantum Flow Processor”
Feature Summary
• Packet Processing Engine (QFP-PPE)

– 40 Packet Processors with 4 Contexts (threads) each; 160
simultaneous threads
– Up to 1.2GHz Tensilica ISA processors + DRAM packet
memory
– Single TCAM4 I/F; can cascade 1-4 devices
Multi-Core (40) Packet Processor
– C-language for feature development; extensive development
support tools
– HW assist for flow-locks, look-ups, stats, WRED, policers,
range lookup, crypto, CRC
• Buffer/queue subsystem (QFP-BQS)
– HW hierarchical 3-parameter (min, max & excess) scheduler
– Fully configurable # of layers based on HQF
– Priority propagation through the multiple layers
Traffic Manager (BQS)
ESP
FECP
Generic ESP Block Diagram Crypto

Assist.
QFP
PP BQ
E S
intercon.
Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex

(OBFL,…) FECP
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
SPI Mux
DRAM Interconnect Hypertransport, 10Gbps
Other
SIP intercon.
SPA
IOCP
Aggreg.
SIP10 Block Diagram SPA SPA
ESPs RPs RPs
Reset / Pwr Ctrl Interconnect

EV-RP
Temp Sensor EV-FC In ref
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler
Status
(OBFL,…) (SC854x SOC)
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) … distribution
…
Ingress buffers Egress buffers

(per port) (per port)
Network
clocks
Ingress Classifier
Reset / Pwr Ctrl SPA Agg. C2W
ESI, 11.2 Gbps GE, 1Gbps

RPs RPs SPA-SPI, 11.2Gbps 4 SPAs 4 SPAs I2C
4 SPAs 4 SPAs 4 SPAs
Hypertransport, 10Gbps SPA Control
Other SPA Bus
SIP intercon.
SPA
IOCP
Aggreg.
SIP10 Block Diagram (comments) SPA SPA
ESPs RPs RPs

EV-RP
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Status
(OBFL,…) SPA Aggregation
(SC854x SOC) Forwards and queues
packets (FIFO)
JTAG Ctrl
IO Control Processor clock
Manages SPA OIR & drivers …
Linux Kernel
Network
clocks
Ingress Classifier
ESI, 11.2 Gbps GE, 1Gbps

RPs RPs SPA-SPI, 11.2Gbps 4 SPAs 4 SPAs I2C
4 SPAs 4 SPAs 4 SPAs
Hypertransport, 10Gbps SPA Control
Other SPA Bus
SPA Interface Processor – SIP-10G
 Physical termination of SPA
 Supports up to 4 SPA's
– 4 half-height, 2 full-height, 2 HH+1FH
– full OIR support
 Does not participate in forwarding
 Limited QoS
– Ingress packet classification – high/low
– Ingress over-subscription buffering (low
priority) until ESP can service them. Up to
128MB of ingress oversubscription buffering
 Capture stats on dropped packets
 Network clock distribution to SPA's, reference
selection from SPA's
 IOCP manages Midplane links, SPA OIR, SPA
drivers
ISR 4451-X Hardware Diagram
DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

DRAM (4 cores) (10 core) FPGE
Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5
DDR3
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10
10 Gbps XAUI
System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash
NIM
NIM
NIM
ISR 4451-X Hardware Diagram (comments)
10 Cores, 1 thread / core Inline Cryptography
5 fwd cores by default No Crypto Assist chip
4 remaining cores license Crypto “locks” core
activated True run-to-completion
DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

DRAM (4 cores) (10 core) BQS onFPGE
a core
One Core dedicated to BQS
Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5
1 Control Plane Core Always active
RP and FECP-like roles DDR3(5+1 or 9+1 cores)
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10
3 Services Core
No hardware TCAM
10 Gbps XAUI
System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash
NIM
NIM
NIM
ISR 4451 System Layout (2RU Platform)
Dataplane
DIMM Control & Services Dual
DSP Slot
DIMM
External Dataplane Control & Services CPU
Serviceable CF CPU
1 SW-NIM or Dual HDD

Configurable Slot
(@ factory only)
30W PoE converter

for onboard GE’s
Airflow – Front to back
2RU, ~18” depth
4-GE (SFP)
AUX
MGMT
Service Modules and

2-GE (RJ-45)
Console: 2-GE (RJ-45) Network Interface Modules
Dual USB
Mini-USB /
Type-A
RJ45
Acronyms
 MCP – Midrange Converged Platform (codename for ASR1000 during development)

 RP – Route Processor
 FP – Forwarding Processor = ESP (Embedded Service Processor)
 CPP – Cisco Packet Processor Compex= QFP (Quantum Flow Processor)
 PPE – Packet Processing Engine
 IOCP – I/O Control Processor
 FECP – Forwarding Engine Control Processor
 SPA – Shared Port Adapter
 SIP – SPA Interface Processor
 IOSd – IOS image that runs as a process on the RP
 FMAN – Forwarding manager (FMAN-RP, FMAN-FP)
 Scbac – FW Session Control Block
 EOBC = Ethernet Out of Band Channels – Packet Interface for Card to Card Control Traffic
 IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP
Software Architecture
ASR1K Software Architecture
RP
CPU
RP
Chassis Manager
IOS CPU
ESI (10-40 Gbps)

Forwarding Manager
interconn.Linux GE switch
Kernel
ESP ESP
FECP
Chassis Manager
EOBC (1 Gbps) FECP
Drivers Forwarding Manager
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
Chassis Manager (CM)
RP  CM on RP communicates with CM processes
CPU on ESP and SIP
Chassis Manager – Distributed function
IOS
ESI (10-40 Gbps)
Forwarding Manager  Initializes hardware and boots other processes
Linux Kernel – CM on SIP queries SPA type and load SPA
drivers
 Manages hardware components
ESP FECP Chassis Manager – Manages EOBC on RP
– Manages ESI links on RP/ESP/SIP
EOBC (1 Gbps)

Drivers – Manages timing circuitry on RP
Drivers
I2C
Linux Kernel – Reset and power-down on RP/ESP/SIP
µ µµ
QFP  Communicates IOS hardware components
Crypto
µ BQS
Assist. – Static & OIR
µ µ
ESI (10-40 Gbps)
 Monitors environmental variables and alarms

SIP
IOCP
SPA Driver
SPA Driver
Chassis  Selects active/standby RP or ESP
Manager
SPA Driver – Coordinates switchover in case of failure or
operator command
Linux Kernel
SPA SPA SPA
Forwarding Manager (FMAN)
RP
CPU
 FMAN on RP communicates with
FMAN-RP
Chassis Manager FMAN process on ESP
IOS
– Distributed function
ESI (10-40 Gbps)
Forwarding Manager
Linux Kernel  Propagates control plane ops. to

ESP
FMAN-FP
ESP FECP ESP aka
Chassis Forwarding Plane
Manager – CEF tables, ACL’s, NAT, SA’s,…
EOBC (1 Gbps)
Drivers
Drivers
Drivers
Forwarding Manager
 FMAN-FP communicates
I2C
Linux Kernel information back to FMAN-RP

QFP – e.g. statistics
µ µµ Crypto
µ
µ µ
BQS
Assist. – FMAN-RP pushes info back to IOS
ESI (10-40 Gbps)
SIP
IOCP
 FMAN on active RP maintains
SPA Driver
SPA Driver
Chassis state for both active & standby
SPA Driver Manager
ESP’s
Linux Kernel
– Facilitates NSF after re-start with bulk
SPA SPA SPA
download of state information
PPE Microcode
RP
CPU  Written in C
ESI (10-40 Gbps)
IOS
Chassis Manager – proper features, no hack
Forwarding Manager
 Runs on each thread of the PPC
Linux Kernel
 Processes packets
ESP FECP Chassis Manager – run to completion
– assisted by various memories
EOBC (1 Gbps)

Drivers
Drivers – TCAM, DRAM,… various speeds
I2C
Linux Kernel
µ µµ
QFP
QFP
Packet Processor Engine
PPE PPE PPE PPE PPE
1 2 3 4 5
BQS
BQS
Crypto
 Features applied via FIA
µ …
– Feature Invocation Array
PPE PPE PPE PPE
6
µ µ
7 8 N
Assist.
Dispatcher
Packet Buffer
ESI (10-40 Gbps)
SIP
IOCP
SPA Driver Chassis
 FIA per interface
– input FIA, output FIA
SPA Driver
SPA Driver Manager
Linux Kernel – drop FIA (Null interface)

SPA SPA SPA
show platform hardware qfp active interface if-name GigabitEthernet 0/0/0
Feature Invocation Array (FIA)

 A per protocol array of functions/features to be executed in sequence
 The FIA is executed in PPE for every packet
 Input interface  Input FIA ; Output interface  Output FIA
Example Input FIA Example Output FIA Example Punt FIA
Dst Lookup Consume Output Inspect RP seens as an external device
Output Inspect from the ESP… connected to a
For Us Martian WCCP special interface. This interface
VFR Refrag has its own FIA.
RPF NAT
Security ACL (in) Drop Policy
Refragment
RPF Checks MQC Classify Internal Transmit Pkt
IPsec Classify (crypto map)
Lawful Intercept
NAT
Security ACL (out)
PBR
Tunnel Encapsulation
WCCP
Crypto (tunnel protection)
These are simple examples. Real
Input Lookup Process
FIA’s can be somewhat arcane...
IP Options Process
Day in Life of Normal Packet
SIP intercon.
SPA
IOCP
Aggreg.
SIP10 Block Diagram SPA SPA
ESPs

EV-RP
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Status
…
JTAG Ctrl
clock
…

Network
clocks
Ingress Classifier
SPA
ESP
FECP
Ingress packet through SIP Crypto

Assist. PP
QFP
BQ
E S
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex

(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
ESP
FECP
Packet dispatched to PPE core Crypto

Assist. PP
QFP
BQ
E S
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP Complex

(OBFL,…) FECP
Thread 1
Thread 2
Thread 3
Thread 4
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
ESP
FECP
Packet dispatched to PPE thread Crypto

Assist. PP
QFP
BQ
E S
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP Complex

(OBFL,…) FECP
Thread 1
Thread 2
Thread 3
Thread 4
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
ESP
FECP
Packet processed by PPE thread Crypto

Assist. PP
QFP
BQ
E S
intercon.
X-Connect Reset / Pwr Ctrl

L2 Switch IPv4 IPv6 MPLS
Packet Buffer Part Len / BW
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM Input FIA Output FIA

Netflow PPE2 QFP Complex
Netflow
BGP Accounting NAT
Boot Flash NBAR Classify PPE1 PPE2 PPE3 PPE4 NBAR
PPE5 Classify
(OBFL,…) FECP
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify IP Unicast …
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN
MQC Policing
NAT IP Multicast
MAC Accounting
PBR
Dispatcher WRED
Dialer IDLE Rst
Packet For
Packet Buffer
Us Queuing
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
ESP
FECP
Generic ESP Block Diagram Crypto

Assist. PP
QFP
BQ
E S
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM Input FIA Output FIA

Netflow PPE2 QFP Complex
Netflow
BGP Accounting NAT
Boot Flash NBAR Classify PPE1 PPE2 PPE3 PPE4 NBAR
PPE5 Classify
(OBFL,…) FECP
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify IP Unicast …
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN
MQC Policing
NAT IP Multicast
MAC Accounting
PBR
Dispatcher WRED
Dialer IDLE Rst
Packet For
Packet Buffer
Us Queuing
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
ESP
FECP
Packet proceeding to BQS then SIP Crypto

Assist. PP
QFP
BQ
E S
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex

(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
SIP intercon.
SPA
IOCP
Aggreg.
Egress packet through SIP SPA SPA
ESPs

EV-RP
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Status
JTAG Ctrl
clock
…

Network
clocks
Ingress Classifier
SPA
An Advanced Example:
IPsec control plane programming
IPsec SA – from IOS to FMAN-FP
show crypto ipsec sa interface virtual-access 1002
RP
CPU
interface: Virtual-Access1002
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1 Chassis Manager
IOS
protected vrf: (none)
local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0)
Forwarding Manager
remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0)
current_peer 17.0.0.26 port 500 Linux Kernel
… show platform software ipsec fp active flow identifier <flow_id>
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
conn id: 30214, flow_id: HW:28214, sibling_flags FFFFFFFF80000008, QFP SA handle: 1892
crypto map: Virtual-Access1002-head-0 …
Crypto SA ctx id: 0x000000002e02b9b6
outbound esp sas:
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
…
conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008,
crypto map: Virtual-Access1002-head-0
…
ESP FECP
Chassis Manager

Drivers
Drivers
Linux Kernel
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
TCAM DRAM DRAM

IPsec SA – from FMAN-FP to QFP TCAM
RP
CPU
IOS
Forwarding Manager
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
outbound esp sas:
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
…
…
show platform hardware qfp active classification feature-manager ESP FECP

Chassis Manager
class-group tcam ipsec <SPD-id> global detail
class-group [ipsec-cg:1008] (classes: 1, total number of vmrs: 4) Drivers
key name: 160_03 value size: 160 result size: 16 Drivers
region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0 Linux Kernel
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000
QFP
Result: : 40000000 905a0400 00000000 00000000 µ µµ Crypto
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000 µ µ Assist.
Result: : 20000000 8d458860 00000000 00000000
… TCAM DRAM DRAM
IPsec SA – from FMAN-FP to Crypto Engine
RP
CPU
IOS
Forwarding Manager
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
outbound esp sas:
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
… show platform software ipsec fp active
conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008, encryption-processor context 2e02b9b6
…
=======Context id: 0x02b249
…
SA word 0: 0x5ae0460fc201aa5
action bits: 0x001f84
show platform hardware qfp active classification feature-manager ESP FECP direction: outbound
Chassis Manager
class-group tcam ipsec <SPD-id> global detail mode: transport
protocol: esp
Drivers Forwarding Manager authentication: SHA-1
Drivers confidentiality: AES-128
key name: 160_03 value size: 160 result size: 16
Linux Kernel …
region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0
mfs: 1454
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 …
QFP sequence number: 306
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000 µ µ Assist. byte count: 25704
packet count: 306
Result: : 20000000 8d458860 00000000 00000000
… TCAM DRAM DRAM
IPsec SA – from FMAN-FP to QFP DRAM
RP
CPU
IOS
Forwarding Manager
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008 FMAN-FP knows
… … everything
crypto map: Virtual-Access1002-head-0 show platform hardware qfp active feature ipsec sa <qfp_sa_handle> …
outbound esp sas: QFP ipsec sa Information
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822) Also indexed by
… QFP sa id: 3623 class-group show platform software ipsec fp active
pal sa id: 32085 encryption-processor context 2e02b9b6
crypto map: Virtual-Access1002-head-0 QFP spd id: 3398
… QFP sp id: 1066 =======Context id: 0x02b249
QFP spi: 0x51E3FC8E(1373895822) …
crypto ctx: 0x000000002e02b9b6 SA word 0: 0x5ae0460fc201aa5
… action bits: 0x001f84
show platform hardware qfp active classification feature-manager ESP FECP direction: outbound
Chassis Manager
class-group tcam ipsec <SPD-id> global detail mode: transport
protocol: esp
Drivers Forwarding Manager authentication: SHA-1
Drivers confidentiality: AES-128
key name: 160_03 value size: 160 result size: 16
Linux Kernel …
region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0
mfs: 1454
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 …
QFP sequence number: 306
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000 µ µ Assist. byte count: 25704
packet count: 306
Result: : 20000000 8d458860 00000000 00000000
… TCAM DRAM DRAM
ESP
FECP
Egress IPsec Packet Flow (I) Crypto

Assist. PP
QFP
BQ
E S
Look up IPsec proxy-identities Lookup SA Handler by class-group ID
Obtain Crypto SA ctx ID intercon.
Obtain class-group ID

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex

(OBFL,…) FECP
Uses Crypto Context identified

by Context ID Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
ESP
FECP
Egress IPsec Packet Flow (II) Crypto

Assist. PP
QFP
BQ
E S
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
DDRAM PPE may be different butPacket
packetProcessor Engine BQS
processing continues where it
Boot Flash
stopped (right after crypto)
PPE1 PPE2 PPE3 PPE4 PPE5
(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
ESP
FECP
General Feature Dependencies QoS Mark/Police

NAT sessions
IPSec SA
Crypto
Assist.
QFP
PP BQ
E S
Netflow Cache
Per session data (FW, NAT, Netflow) QoS Queuing intercon.
Class/Policy Maps: QoS, DPI, FW NAT VFR re-assembly
ACL/ACE storage
Reset / Pwr Ctrl IPSec headers
IPSec Traffic Selectors, classes, rules TCAM Resource DRAM
DRAM SRAM
NAT Tables Temp Sensor
EEPROM Cores execute packet processing

QFP Complex
All features handled from here
CPU
Packet horsepower
Processor is here…
Engine BQS
DDRAM

(OBFL,…)
Memory for FECP FECP
QFP client / driver
OBFL
BQS offloads queuing and scheduling from
QoS Class maps
Crypto Assist chip offloads crypto cores.
FM FP
from the PPE cores 16000 Queues on ASR1001 & ESP 5
Statistics
Dispatcher 127000 Queues on ESP10+
ACL ACEs copy
Packet Buffer 470000 Queues on ESP 100+GE, 1Gbps
NAT config objects 2IC
IPSec/IKE SA SPA Control
NF config data Crypto System Bandwidth SPA Bus
ZB-FW config objects SPI Mux 5, 10, 20, 40, 100, 200 Gbps
DRAM Interconnect Hypertransport, 10Gbps
Other
Debugging strategies
Everyday situations
Traffic did not reach its target !

What happened to that packet ?
Why did that happen ?
Using statistics for troubleshooting packet drops
Not easy… not very practical either.
ESP Let’s dig deeper before making it simpler
SPA
 show platform hardware slot {f0|f1} serdes statistics
 show interfaces <interface-name>
 show platform hardware slot {f0|f1} serdes statistics internal
 show interfaces <interface-name> accounting
 show platform hardware qfp active bqs 0 ipm mapping
 show interfaces <interface-name> stats  show platform hardware qfp active bqs 0 ipm statistics channel all
SIP  show platform hardware qfp active bqs 0 opm mapping
 show platform hardware qfp active bqs 0 opm statistics channel all
 show platform hardware port <slot/card/port> plim statistics
 show platform hardware qfp active statistics drop [detail]
 show platform hardware subslot {slot/card} plim statistics
 show platform hardware qfp active interface if-name <Interface-name> statistics
 show platform hardware slot {slot} plim statistics  show platform hardware qfp active infrastructure punt statistics type per-cause |
exclude _0_
 show platform hardware slot {0|1|2} plim status internal
 show platform hardware qfp active infrastructure punt statistics type punt-drop |
exclude _0_
 show platform hardware slot {0|1|2} serdes statistics
 show platform hardware qfp active infrastructure punt statistics type inject-drop
| exclude _0_
RP
 show platform hardware qfp active infrastructure punt statistics type global-drop
| exclude _0_
 show platform hardware slot {r0|r1} serdes statistics
 show platform hardware qfp active infrastructure bqs queue output default all
 show platform software infrastructure lsmpi
 show platform hardware qfp active infrastructure bqs queue output recycle all
Debugging Strategies to Date
IOS Control Plane
Well Known
• show interface
• show ip route, show bgp …
Top Down
Platform Control Plane
Bottom Up
Very Difficult
• ESP “stuff”
• e.g. show platform … Let’s change
that!!
Data Plane
• ESP “stuff”
• e.g. show platform …
The Road to Simplification:
Part I, Data Plane Debugging
55
IOS 3.7
The Embedded Packet Capture
One way of capturing packets…
Device# monitor capture mycap start

Device# monitor capture mycap access-list v4acl
Device# monitor capture mycap limit duration 1000
Device# monitor capture mycap interface GigabitEthernet 0/0/1 both
Device# monitor capture mycap buffer circular size 10
Device# monitor capture mycap start Shows whether packets have been received or sent
Device# monitor capture mycap export tftp://10.1.88.9/mycap.pcap
Device# monitor capture mycap stop Shows what packets look like
Requires hex dump analysis or export to decoder (sniffer)
Does not tell us what happened to the packet
Device# show monitor capture mycap buffer dump
0
0000: 01005E00 00020000 0C07AC1D 080045C0 ..^...........E.
0010: 00300000 00000111 CFDC091D 0002E000 .0..............
0020: 000207C1 07C1001C 802A0000 10030AFA .........*......
0030: 1D006369 73636F00 0000091D 0001 ..example....... Excellent tool but insufficient in many cases
1
0000: 01005E00 0002001B 2BF69280 080046C0 ..^.....+.....F.
0010: 00200000 00000102 44170000 0000E000 . ......D.......
0020: 00019404 00001700 E8FF0000 0000 .............. http://www.cisco.com/en/US/docs/ios-
xml/ios/epc/configuration/xe-3s/asr1000/nm-packet-capture-
2 xe.html
0000: 01005E00 0002001B 2BF68680 080045C0 ..^.....+.....E.
0010: 00300000 00000111 CFDB091D 0003E000 .0..............
0020: 000207C1 07C1001C 88B50000 08030A6E ...............n
0030: 1D006369 73636F00 0000091D 0001 ..example.......
IOS 3.10
The Packet Tracer and FIA Debugger
TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
Condition determines packets DRAM SRAM
Temp Sensor
to be traced
Input FIA Output FIA Input ACL
EEPROM
Pak Trace ? PPE2 QFP Complex
MQC Classify
Output ACL NAT
Boot Flash Input ACL PPE1 PPE2 PPE3 PPE4 PPE5
(OBFL,…) FECP
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify NAT PBR
Output ACL
NAT Encaps
IP Unicast
Statistics and final action will be NAT
PBR Dispatcher Crypto
collected (matched packets dropped,
Packet Buffer punted to RP, forwarded to output
PPE2 Encaps
interface …)
Crypto Thread 3 Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
Optionally, FIA actions can logged per packet
System can capture several packets flows
RPs RPs ESP RPs SIPs Packet flows can be reviewed in show commands
Packet Tracer Demonstration
Packet-Trace: Configuration Commands
 The Pactrac (Packet Tracer) shows us what happens to a series of packets
– True inspection of IOS XE packet forwarding flow
 debug platform packet-trace enable
– Enables accounting
– Required for all levels of inspection
 debug platform packet-trace packet <pkt-num> \
[fia-trace | summary-only] [circular] [data-size <data-size>]
– Required for any per-packet data capture (e.g. necessary for packet copy to function)
– Specifies maximum number of packets maintained at one time (<pkt-num>)
– Always enables capture of summary data or only summary data (summary-only)
– Captures feature path data by default
– Optionally performs FIA trace (fia-trace) in addition to path data capture
– Allows specifying the size of the path data buffers (defaults to 2048)
 debug platform packet-trace copy packet {in | out | both} [L2 | L3 | L4] [size
<num-bytes>]
– Enables copy of the ingress and/or egress packets
– Optionally allows specifying where to start copy of the packet (L2 is default)
– Optionally allows specifying the maximum number of octets to copy (64 is default)
Available XE3.11 and forward
 debug platform packet-trace drop [code <code-num>]
– Enables retention only for dropped packets
– Optionally allows specifying retaining packets for a specific drop code
– Can be used without global/interface conditions to capture drop events*
*Drop event capture means the only the drop itself is traced not the life of the
packet, but, it still allows capture of summary data, tuple data and the packet to
help refine conditions or provide clues to the next debug step.
 clear platform packet-trace statistics
– Clears any collected statistics and data buffers
– Tracing must be stopped first (debug platform condition stop)
 clear platform packet-trace configuration
– Removes all debug platform packet-trace commands
 clear platform condition all
– Removes all debug platform condition and debug platform packet-trace commands
Packet-trace relies on the conditional infra to determine which packets are
interesting. The condition infra provides the ability to filter by protocol, IP address
and mask, ACL, interface and direction. A complete discussion of conditions is
not made here but some illustrative examples are:
 debug platform condition ingress
– Checks all incoming packets on all interfaces for all protocols
 debug platform condition interface g0/0/0 ipv4 ingress
– Checks all IPv4 packets arriving on interface g0/0/0
 debug platform condition interface g0/0/0 ipv4 access-list FOO ingress
– Checks incoming IPv4 packets on interface g0/0/0 that match access-list FOO
Conditions are activated or de-activated using debug platform condition start or
debug platform condition stop respectively.
NOTA BENE!!!!!
Conditions define what the filters are and when the filters are applied to a
packet. For example, debug platform condition interface g0/0/0 egress means
that a packet will be identified as a match when it reaches the output FIA on
interface g0/0/0 so any packet-processing that took place from ingress up to that
point is missed.
Best Practice
It is highly recommended to use ingress conditions for pactrac to get the most
complete and meaningful data. Egress conditions can be used but just be aware
of the limitation above.
Packet-Trace: Configuration Example
The following shows how one would trace the most recent 128 packets entering
GigabitEthernet0/0/0 including FIA trace and a copy of up to the first 2048 octets
of the input packet.
debug platform condition interface g0/0/0 ingress
debug platform packet-trace enable
debug platform packet-trace packet 128 fia-trace circular
debug platform packet-trace copy packet input size 2048
debug platform condition start
<…wait until you’ve captured the packets you think you want…>
debug platform condition stop
Packet-Trace: Configuration Highlights
 Pactrac buffers consume QFP DRAM
– Be mindful of how much memory a config needs and how much memory is available
 Configure as much detail as you want…more detail…more performance impact
for matched packets
 Each pactrac “config” change will temporarily disables pactrac and clears
counts/buffers
– “Cheap” way of ‘debug plat cond stop’, ‘clear plat pack stats’ and ‘debug plat cond start’
 Some configs require a ‘stop’ in order to display summary or per packet data
– Currently circular and drop tracing
 Conditions define where and when filters are applied to a packet
Packet-Trace: Show Commands
Show commands are used to display pactrac configuration and each level of data:
 show platform packet-trace configuration
– Displays packet-trace configuration including any defaults
 show platform packet-trace statistics
– Displays accounting data for all pactrac packets
 show platform packet-trace summary
– Displays summary data for the number of packets specified by debug platform packet-trace
packet
 show platform packet-trace packet { all | <pkt-num>} [decode]*
– Displays all path data for all packets or the packet specified
– Decode attempts to display packets captured by debug platform packet-trace copy in user
friendly way
– * decode was introduced in XE3.11
 NOTE: only a few protocol headers are supported initially (ARPA, IP, TCP, UDP,
ICMP)
Example of Packet-Trace Configuration
Example of Packet-Trace Accounting
Example of Packet-Trace Summary
in0/0/rp:0 is how the ESP sees the RP
Example of Packet-Trace Packet Details
Example of Clearing Packet-Trace Stats
Understanding and Extracting ESP Logs
72
ESP Tracing aka Logging
TEMP RAM FS
RP RP logs are first written
CPU
Chassis Manager here (efficiency)
IOS
NFS Shared Disk
ESI (10-40 Gbps)

Forwarding Manager
Hard disk is really here
Linux Kernel
ESP FECP TEMP RAM FS

Chassis Manager
ESP logs are first written
EOBC (1 Gbps)
Drivers Forwarding Manager here (efficiency)
Drivers
I2C Drivers
Linux Kernel
QFP Mounted NFS

µ µµ Crypto
µ BQS
Assist. ESP logs are committed
µ µ
here at regular intervals
ESI (10-40 Gbps)
SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
Linux Kernel
SPA SPA SPA
Important logs
RP
CPU
Chassis Manager
fman_rp_R[0|1]-0.log
IOS
ESI (10-40 Gbps)

Forwarding Manager Under /harddisk/tracelogs
Linux Kernel
fman_rp_R[0|1]-0.log.<timestamp>
fman-fp_R0.log.<timestamp>
cpp_cp_F[0|1]-0.log.<timestamp>
ESP FECP
Chassis Manager
fman_rp_R[0|1]-0.log
EOBC (1 Gbps)
Drivers Forwarding Manager fman_fp_F[0|1]-0.log
Drivers
I2C Drivers cpp_cp_F[0|1]-0.log
Linux Kernel
QFP Under /harddisk/tracelogs/

µ µµ Crypto fman-fp_R0.log.<timestamp>
µ BQS
µ µ Assist.
cpp_cp_F[0|1]-0.log.<timestamp>
ESI (10-40 Gbps)
SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
Linux Kernel
SPA SPA SPA
What log files are important?
 Important log files to get for security issues:

– fman_rp_R[0|1].log (under /tmp/rp/trace directory on RP)
– fman-fp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP
– cpp_cp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP)
 All these logs get rotated and are copied to /harddisk/tracelogs directory
on active RP.
 Look for the relevant log files depending on the time of the failure
 By default, all ERR messages are logged, these should be the first things
to look for
Example log files
The timestamp…
My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/
3768365 -rwx 1048934 Jan 6 2014 18:20:16 +00:00 cpp_cp_F0-0.log.7133.20140106182015
3768330 -rwx 551643 Jan 7 2014 09:27:51 +00:00 cpp_cp_F0-0.log.7133.20140107092751
3768335 -rwx 1048901 Jan 7 2014 08:56:44 +00:00 cpp_cp_F0-
0.log.7133.2014010708564339313059840 bytes total (30680653824 bytes free)
Rotating the log files
My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/traceMy-ASR1000-2#test platform software trace slot rp active forwarding-
manager rotate
Rotated file from: /tmp/rp/trace/stage/fman_rp_R0-0.log.13836.20140107094754, Bytes: 0, Messages:
6535
My-ASR1000-2#test platform software trace slot FP active cpp-control-process rotate
Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.7133.20140107093650, Bytes: 154027,
Messages: 786
My-ASR1000-2#test platform software trace slot FP active forwarding-manager rotate
Rotated file from: /tmp/fp/trace/stage/fman-fp_F0-0.log.8247.20140107093738, Bytes: 20170,
Messages: 210
OR use
My-ASR1000-2#request platform software trace rotate all Does not show the rotated file names w/
time stamp  have to hunt them down
The Road to Simplification – Part II
Control Plane Unified Show Commands
78
Simplifying the IPsec show commands
One show command to rule them all
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1
------------------ show platform software ipsec fp active flow identifier 34130 ------------------
protected vrf: (none) …
remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0) ------------------ show platform hardware qfp active feature ipsec sa 1427 ------------------
current_peer 17.0.0.26 port 500 …
PERMIT, flags={origin_is_acl,} ------------------ show platform software ipsec fp active encryption-processor context 6502aa4f ------------------
#pkts encaps: 25227, #pkts encrypt: 25227, #pkts digest: 25227
…
#pkts decaps: 25237, #pkts decrypt: 25237, #pkts verify: 25237
#pkts compressed: 0, #pkts decompressed: 0 ------------------ show platform software ipsec fp active flow identifier 34129 ------------------
#pkts not compressed: 0, #pkts compr. failed: 0 …
#pkts not decompressed: 0, #pkts decompress failed: 0 ------------------ show platform hardware qfp active feature ipsec sa 1867 ------------------
#send errors 0, #recv errors 0
…
local crypto endpt.: 172.18.0.1, remote crypto endpt.: 17.0.0.26 ------------------ show platform software ipsec fp active encryption-processor context 2e02aa4e -----------------
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0/2 …
current outbound spi: 0xA7B61FE5(2813730789)
PFS (Y/N): N, DH group: none
inbound esp sas:

spi: 0xA222F391(2720199569)
transform: esp-aes esp-sha-hmac ,
in use settings ={Transport, }
conn id: 36130, flow_id: HW:34130, sibling_flags FFFFFFFF80000008, crypto show crypto ipsec sa interface virtual-access 1002 platform
map: Virtual-Access1002-head-0
sa timing: remaining key lifetime (k/sec): (4607974/2137)
IV size: 16 bytes
or
replay detection support: Y replay window size: 512
Status: ACTIVE(ACTIVE)
show crypto ipsec sa peer 17.0.0.26 platform
…
outbound esp sas:

spi: 0xA7B61FE5(2813730789)
transform: esp-aes esp-sha-hmac ,
in use settings ={Transport, }
conn id: 36129, flow_id: HW:34129, sibling_flags FFFFFFFF80000008, crypto
map: Virtual-Access1002-head-0
sa timing: remaining key lifetime (k/sec): (4607974/2137)
IV size: 16 bytes
replay detection support: Y replay window size: 512
BRKCRS-3147
Status: ACTIVE(ACTIVE) © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Simplifying the ZBF show commands
Three Commands for ZBF under the sky
show policy-firewall config platform
--show platform software firewall FP active bindings--

--show platform software firewall RP active bindings--
--show platform software firewall FP active pairs--
--show platform software firewall RP active pairs--
--show platform software firewall FP active parameter-maps--
--show platform software firewall RP active parameter-maps--
--show platform software firewall FP active zones--
--show platform software firewall RP active zones--
show policy-firewall sessions platform | i show platform
--show platform hardware qfp active feature firewall datapath scb any any any any any all any --
show policy-firewall stats platform | i show platform
--show platform software firewall FP active statistics--

--show platform software firewall RP active statistics--
--show platform hardware qfp active feature firewall runtime--
--show platform hardware qfp active feature firewall memory--
--show platform hardware qfp active feature firewall drop--
--show platform hardware qfp active feature firewall client statistics--
The Road to Simplification
Part III, Deep Data Plane Debugging
81
IOS 3.11
The Packet Tracer and FIA Debugger
TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
DRAM SRAM
Temp Sensor
Input FIA Output FIA Input ACL

EEPROM
Pak Trace ? PPE2 QFP Complex
MQC Classify
Output ACL NAT
Boot Flash Input ACL PPE1 PPE2 PPE3 PPE4 PPE5
(OBFL,…) FECP
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify NAT PBR
Output ACL
NAT Encaps
IP Unicast
NAT
PBR Dispatcher Crypto
Packet Buffer Cond Dbg ?
PPE2 Encaps
Crypto If Conditional Debugging is on for Thread 3 Crypto
SPI Mux feature AND if packet needs to be
Reset / Pwr Ctrl
traced… feature will log its action step
SA table by step in cpp_cp_f0-0.log !!
DRAM Interconnect
Platform Conditional Debugging
BGL.D.16-ASR1000-1# debug platform condition feature ?
atm ATM feature
atom ATOM feature
bridge-domain Layer2 bridging feature
cft CFT feature
cxsc CXSC feature
evc EVC feature
fw FW feature Debugs get populated in cpp_cp_F0-0.log
ipsec IPSEC feature
nbar NBAR feature
otv OTV feature
subscriber Subscriber feature
vpls VPLS feature
Same match statement as
packet tracer…
BGL.D.16-ASR1000-1#debug platform condition ipv4 172.19.2.1/32 ingress
BGL.D.16-ASR1000-1#debug platform condition feature ipsec dataplane submode cce level info
BGL.D.16-ASR1000-1#debug platform condition start
Tells which feature to
debug
Start and stop debugging
Conditional Debugger Demonstration
Checking Resource Usage
Coming your way in an IOS-XE near you…
85
Unified show CPU platform summary
show processes cpu platform
Core 0: CPU utilization for five seconds: 1%; one minute: 1%; five minutes: 1%
Core 1: CPU utilization for five seconds: 1%; one minute: 1%; five minutes: 1%
PID Runtime(ms) uSecs 5Sec 1Min 5Min TTY Process
1 1102 1800 0.20% 0.50% 0.30% 0 init
3 100 1000 0.00% 0.00% 0.05% 0 events/0
4 100 200 0.00% 0.00% 0.00% 0 khelper
6 200 200 0.70% 0.10% 0.00% 0 kthread
…
 Simplified CPU usage visualization – system wide

 Will display all relevant CPU’s
Unified show memory platform summary
show memory platform summary
Total number of processes: 134

Virtual memory : 2822197248
Pages resident : 360197
Major page faults: 1921  Simplified memory consumption
Minor page faults: 1290831
 Will display all relevant memory
Memory (kB)
 Including TCAM consumption…
Physical : 4127744
Total : 3874992
Used : 2231964
Free : 1643028
Active : 1438412
Inactive : 694176
…
Wrapping up…
88
New Debugging Strategy
IOS Control Plane

Well Known
• show interface, show ip route, show bgp …
• Feature debugging
Platform Control Plane

• Unified show commands
Still Difficult • Platform show commands
(not overly) • Future: control plane conditional debugging
Data Plane
• Packet Tracer
Easy!! • Forwarding plane conditional debugging
• Embedded Packet Capture
Call to Action…
Visit the World of Solutions:-
 Cisco Campus
 Walk-in Labs
 Technical Solutions Clinics
 Meet the Engineer
 Lunch Time Table Topics, held in the main Catering Hall
 Recommended Reading: For reading material and further resources for this
session, please visit www.pearson-books.com/CLMilan2014
Complete Your Online Session Evaluation
 Complete your online session evaluation
 Complete four session evaluations
and the overall conference evaluation to receive your Cisco Live T-shirt

BRKCRS 3147 Advanced Troubleshooting of The ASR1K and ASR4400 Made Easy 2014 Milan 90 Mins PDF

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

BRKCRS 3147 Advanced Troubleshooting of The ASR1K and ASR4400 Made Easy 2014 Milan 90 Mins PDF

Enviado por

Direitos autorais:

Formatos disponíveis

ASR1K and ISR445x

Troubleshooting Made Easy

One IOS-XE Feature Set

2.5-5 5-36 10-40 10-100 40-200

Embedded Service Processor

SIP interconn. SIP interconn. SIP interconn.

SPA SPA SPA

SPA SPA SPA SPA SPA SPA

SPA Interface Processor

SPA SPA SPA SPA SPA SPA

SPA Control Link

Centralized Architecture Midplane

SIP interconn. SIP interconn. SIP interconn.

SPA SPA SPA

SPA SPA SPA SPA SPA SPA

Mgmt Console BITS

ESP10 Block Diagram Crypto

Reset / Pwr Ctrl Packet Buffer

Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5

JTAG Ctrl PPE6 PPE7 PPE8 … PPE40

RPs RPs ESP RPs SIPs

ESP10 Block Diagram (comments) Crypto

Quantum Flow Processor intercon.

JTAG Ctrl PPE6 PPE7 PPE8 … PPE40

RPs RPs ESP RPs SIPs

Reset / Pwr Ctrl QFP QFP

EEPROM PPE PPE PPE … PPE PPE PPE PPE … PPE

Boot Flash TCAM Resource DRAM

Reset / Pwr Ctrl

 CRC protection of packet contents

 SIP-10G: supports 1x11.5G mode only

 SIP-40G: supports 1x11.5G, 2x11.5G, 2x23G

• Packet Processing Engine (QFP-PPE)

Generic ESP Block Diagram Crypto

Reset / Pwr Ctrl Packet Buffer Part Len / BW

Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5

RPs RPs ESP RPs SIPs

SIP10 Block Diagram SPA SPA

ESPs RPs RPs

Reset / Pwr Ctrl Interconnect

Ingress buffers Egress buffers

ESI, 11.2 Gbps GE, 1Gbps

SIP10 Block Diagram (comments) SPA SPA

ESPs RPs RPs

Reset / Pwr Ctrl Interconnect

ESI, 11.2 Gbps GE, 1Gbps

DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5

DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

1 SW-NIM or Dual HDD

30W PoE converter

Service Modules and

 MCP – Midrange Converged Platform (codename for ASR1000 during development)

ESI (10-40 Gbps)

SPA SPA SPA SPA SPA

Drivers Forwarding Manager

Linux Kernel – Reset and power-down on RP/ESP/SIP

 Monitors environmental variables and alarms

SPA SPA SPA

Linux Kernel  Propagates control plane ops. to

Linux Kernel information back to FMAN-RP

Drivers Forwarding Manager

Linux Kernel – drop FIA (Null interface)