6500 Part 4

CATALYST 6500
BOOTCAMP
FORWARDING
FORWARDING
ARCHITECTURE
ARCHITECTURE
AGENDA
Introduction
Catalyst 6500 Architecture Review
Unicast Multi-Layer Switching
Multicast Multi-Layer Switching
Load-Balancing
Catalyst 6500 Internals

Introduction
What lies within the Catalyst

6500?
How does it perform its magic?
This section will look at some of the internals of the

Catalyst 6500
and then explore how it performs the task of forwarding

The Packet
Everything a Switch does is centered on the Data that it forwards

Ethernet Frame
Preamble
DEST MAC
A Layer 2 Switch
uses the Destination
MAC Address to
make forwarding
decisions
A Layer 3 Router
uses the Destination
IP Address to
make forwarding
decisions
SRC MAC
DATA
LEN/TYPE
FCS
IPv4 Header
Version
Version
IHL
Type
IHL
Type of
of Service
Service
Identification
Identification
Time
Protocol
Time to
to Live
Live (TTL)
(TTL)
Protocol
Flags
Flags
SOURCE
SOURCE IP
IP
DESTINATION
DESTINATION IP
IP
Options
Options
DATA
DATA (Payload)
(Payload)
Length
Length
Fragment
Fragment Offset
Offset
Header
Header Checksum
Checksum

Layer 2 vs. Layer 3 Forwarding
What is the difference between a Layer 2 & Layer 3 Forwarding operation?
Its all about the address which is used to forward the data...
Lets look at some Layer 2 & Layer 3 operations
C
SUBNET Y
D
A
B
VLAN 10
SUBNET X
SUBNET Z

The Layer 2 Boundary
A host can talk to any host at Layer 2 within the same VLAN (or Broadcast Domain)
without requiring a Layer 3 Switch (or router) to forward the data
A
Layer 2 operation
A <=> B
D <=> E
A <=> C
D <=> F
B <=> C
E <=> F
VLAN 10
Layer 3 operation
A/B/C <=> D D/E/F <=> A
A/B/C <=> E D/E/F <=> B
A/B/C <=> F D/E/F <=> C
VLAN 20

Basics of Layer 2 Forwarding
The MAC address-table is stored in a special (2-Value) Content Addressable Memory
The CAM is used to store MAC Addresses, and the Ports on which they were learnt
A
B
44
11
22
33
55
66
E
F
MAC Table
MAC Table also

stores the VLAN
within which MAC
was learnt
MAC
A
B
C
D
E
F
Port
1
2
3
4
5
6
MAC Tables range

in size between
different switch
platforms

The default CAM behavior for an existing (known) MAC entry is as follows
B
CAM
1
2
3
A
B
C
2
4
1
HIT!!!
DST
DST MAC
MAC == C
C
For MAC
addresses that
are FOUND in the
MAC Table, the
switch will
forward the
packet to ONLY
that host

The default CAM behavior for an unknown MAC address is as follows
B
CAM
1
2
3
A
B
C
2
4
1
MISS!!!
DST
DST MAC
MAC == D
D
For MAC
addresses that are
NOT found in the
MAC Table, the
switch will FLOOD
that frame to ALL
ports in the same
VLAN

We will explore a Layer 2 operation first Packets from host A to host B

IP: 10.1.1.1/24
MAC: C1
VLAN 10
11
A sends ARP - who is 10.1.1.3?

DMAC: All FFs / SMAC: A1
22
ARP is broadcast & DMAC: All FFs,

so switch forwards out all ports
B Replies to ARP
DMAC: A1 / SMAC:B1
A sends unicast to B
DMAC: B1 / SMAC:A1
DIP: 10.1.1.3 / SIP: 10.1.1.2
33
22
44
55
11
A
44
IP: 10.1.1.2/24
MAC: A1
VLAN 10
33
55
B
Switch does CAM lookup using

DMAC: B1 and forwards packet
directly to B1
IP: 10.1.1.3/24
MAC: B1
VLAN 10

Now lets look at a Layer 3 operation Packets from host A to host D
IP: 10.1.1.1/24
MAC: C1
VLAN 10
11
A sends ARP - who is 10.5.1.2?

DMAC: All FFs / SMAC: A1
22
Switch L3 interface replies to ARP saying send it to me
33
A sends to Switch
DMAC:C1 / SMAC:A1
DIP: 10.5.1.2 / SIP: 10.1.1.2
IP: 10.3.1.2/24
MAC: D1
33
22
11
IP: 10.3.1.1/24
MAC: C2
D
IP: 10.5.1.1/24
MAC: D2
IP: 10.1.1.2/24
MAC: A1
VLAN 10
IP: 10.5.1.2/24
MAC: F2

The operation proceeds as follows
IP: 10.1.1.1/24
MAC: C1
VLAN 10
44
55
IP: 10.3.1.1/24
MAC: C2
44
Switch1 does an L2 / L3 lookup
55
Packet forwarded...
DMAC:D1 / SMAC:C2
DIP: 10.5.1.2 / SIP: 10.1.1.2
66
77
Switch2 does an L2 / L3 lookup

Packet forwarded...
DMAC:F2 / SMAC:D2
DIP: 10.5.1.2 / SIP: 10.1.1.2
IP: 10.3.1.2/24
MAC: D1
66
77
IP: 10.5.1.1/24
MAC: D2
IP: 10.1.1.2/24
MAC: A1
VLAN 10
IP: 10.5.1.2/24
MAC: F2
ARCHITECTURE REVIEW
CATALYST 6500
ARCHITECTURE
REVIEW
Agenda Topics

32Gbps Bus
The 32Gbps (16Gbps Full-Duplex) Bus is a legacy backplane that originated with
the first release of the Catalyst 6500 back in 1999.
DBUS - Data Bus: Path

Path over
over which
which actual
actual
Supervisor
frame
frame Data
Data (and
(and // or
or Headers)
Headers) is
is transferred
transferred between
between
modules
modules and
and the
the Supervisor
Supervisor
RBUS - Results Bus: Path
Path over
over which
which
the
the Results
Results of
of forwarding
forwarding lookups
lookups by
by the
the Supervisor
Supervisor
are
are passed
passed back
back to
to modules
modules
DBUS
RBUS
Linecard
Linecard
Linecard
Catalyst 6500
The Supervisor 720 Supervisors
& Supervisor 2T support a Switch Fabric which offers each
module 1 or more dedicatedSwitch
communication
channels to every other module...
Fabric
Linecard
Slot #9
Linecard
Slot #1
Linecard
Slot #7
SWITCH
SWITCH
FABRIC
FABRIC
Linecard
Slot #2
Data
Flows
Linecard
Slot #8
Linecard
Slot #3
Linecard
Slot #4
Linecard
Slot #6
Supervisor
Slot #5

Ethernet Out-of-Band Channel (EOBC)
The Catalyst 6500 chassis provides an Out of Band connection for all internal
communication (e.g. programming & statistics) between the Supervisors & Modules
LCP
LCP
LCP
SLCP
SLCP
ACTIVE
SP/RP
Standby
SP/RP
The EOBC is NOT

used for Data Traffic...
LCP
LCP
LCP
LCP
EOBC
The Ethernet Out-of-Band Channel
is a 100MB (Half-Duplex) Bus
Each module has a unique EOBC (ethernet) Driver
(similar to a PC NIC), which transmits & receives
frames over the EOBC

Interaction between Supervisor & Modules
Two transport mechanisms exist:

SCP = Switch-module Configuration Protocol
IPC = Inter Process Communication
IOS uses SCP & IPC, and introduces:

ICC = Inter-Card Communication
The IPC (like SCP) communicates directly with the EOBC...
However, to offload work from IPC, the ICC handles inter-card
communications and then forwards this info to IPC
IPC/ICC & SCP are like TCP/UDP (L4 sessioncontrol) for various IOS applications
The EOBC uses an internal IP (L3) & MAC (L2)
address scheme, to forward between modules
Control Plane Protocol

Stack
How modules communicate
APPLICATION
(IOS Functions)
ICC SHIM
RPC
ICC
IPC
SCP
EOBC Driver
EOBC

Interface Management on PFC3
L2 Bridging based on
(mac,vlan) and L3 Routing
between VLANs
Need to logically separate all
traffic on a per - VLAN basis
The Total 4096 VLANs are
shared between L2 & L3 ports
VLANs are allocated for all L3
interfaces & other cases:
(Multicast, NDE, CoPP, etc.)
Use show vlan internal
usage

Interface Management on PFC4
In PFC4 (port_index, vlan) pair is
used by BD (bridge domain) for
Bridging, and LIF (logical interface)
for Routing
Total of 16K BDs and 128K LIFs
Decouples L2 & L3, and allows for
REUSE of VLANs & 128K interfaces
LIF & BDs are stored in an
external 512K * 72bit LIF
database or LDB
Each entry in LDB has LIF, BD, and
L2 lookup & result control bits
PFC or EARL?
What are these & what do they do?
The Enhanced Address Recognition Logic (EARL)

functions as a hardware-based forwarding engine for
the Catalyst 6500 switching platform
The reference to EARL is an Engineering term, but
from a Marketing or Customer standpoint, the EARL is
better known as the Policy Feature Card (PFC)
The primary function of the PFC (EARL) is to maximize the
forwarding of data (in hardware), by utilizing purpose-built
ASICs & special 3-value memory-tables called TCAMs
There have actually been many Generations of the EARL

concept...
A brief history of the PFC

How did we get here?
1. The EARL concept was first introduced on the Catalyst

5000 platform, to enable hardware Layer 2 switching
2.
The Route Switching Module (RSM) & later the Route Switch Feature Card (RSFC)
allowed the Catalyst to perform both L2 switching & L3 routing
3.
With the Multi-Layer Switching (MLS) & Cisco Express Forwarding (CEF) software
features, we were able to perform L2 switching & L3 routing in hardware tables
4.
The Catalyst 5500 RSFC along with the Netflow Feature Card (NFFC) later became
the Multi-layer Switch Feature Card (MSFC) & Policy-Feature Card (PFC)...
5. The PFC concept was first introduced on the Catalyst 6000

platform, to enable complete hardware (MLS / CEF-based)
Layer 2 & Layer 3 switching
PFC vs. DFC?

How are they different?
The PFC / EARL hardware (ASICs & Memory)

physically resides on the Supervisor module
Hence, all L2 / L3 forwarding decisions must be made ON
THE SUPERVISOR, before packets can be transmitted...
The Distributed Forwarding Card

(DFC) is an EARL daughter-board, for
individual modules
The DFC serves 3 basic roles:
Provides localized forwarding decisions for each module
Scales aggregate system performance to reach up to 400 mpps.

Supervisor720 / PFC3 Architecture
QoS TCAM
Fabric interface / Replication

engine to interface with fabric &
bus, and for multicast/SPAN
replication
Multicast
Expansion Table
1GE / 10GE
Uplinks
MSFC3 Complex
contains both the
SP & RP CPUs
contains QoS
ACL entries
NetFlow table
for stats &
features
MSFC 3
Flash
Flash
DRAM
DRAM
RP
CPU
1 Gbps
Flash
Flash
SP
CPU
1 Gbps
DRAM
DRAM
NetFlow
ADJ
contains
rewrite info
FIB contains
IPv4/IPv6 prefixes
and MPLS entries
Hardware for ACL
TCAM counters
QoS
TCAM
Adj
TCAM
FIB
TCAM
Port ASIC
ACL
TCAM
L3/4 Engine
ACE
ACE
Counte
rr
ACL TCAM
contains
security and
feature ACL
entries
MET
Switch
Switch Fabric
Fabric
18
x
20G
18 x 20G Traces
Traces
Traces # 1 to 16
Switch Fabric
20 Gbps
EOBC
Fabric
Fabric Interface
Interface
&&
Replication
Replication Engine
Engine
DBUS
RBUS
L2 Engine
PFC3
L2 CAM (64K)
16 Gbps
Bus
L2 CAM
contains MAC
entries

PFC3 (EARL7) Capabilities
Starting with PFC3 (EARL7), the EARL was designed

to support hardware forwarding of IPv6 & MPLS, in
addition to IPv4 & Layer 2
In addition to forwarding packets (@ 30/48Mpps), the
PFC also supports several hardware-accelerated
software features, to complement its capabilities
It has hardware support for features such as:
Access Control Lists (ACLs)
QoS Policies (Policing & Marking)
Netflow (Statistics & Forwarding)
Generic Route Encapsulation (GRE)

Path of a Packet in PFC/DFC3
L3 FE (TYCHO)
Netflow TCAM
FIB TCAM &

SSRAM
Netflow Table
Security ACL
TCAM
Netflow Statistics
QoS ACL TCAM
Adjacency Statistics
1. Packet
arrives at
the
L2 FE ASIC Superman
Adjacency Table
A MAC Address
lookup is
performed & this
determines if its an
L2 or an L3
packet
IP Packet Parse
L2 FE (SUPERMAN)
IP Packet Parse
L2 MAC Table

L3 FE (TYCHO)
Netflow TCAM
FIB TCAM &

SSRAM
Netflow Table
Security ACL
TCAM
Netflow Statistics
QoS ACL TCAM
Adjacency Table
2. Superman then
parses the IP
packet performing
CRC check
before passing the
packet over the bus
to the
L3 FE ASIC - Tycho
IP Packet Parse
2
L2 FE (SUPERMAN)
IP Packet Parse
L2 MAC Table

L3 FE (TYCHO)
Netflow TCAM
FIB TCAM &

SSRAM
Netflow Table
Security ACL
TCAM
Netflow Statistics
QoS ACL TCAM
Adjacency Table
3. Tycho parses
packet for
Destination
IP address /mask or
MPLS label...
Then it prepares to
perform FIB
lookups
IP Packet Parse
L2 FE (SUPERMAN)
IP Packet Parse
L2 MAC Table

Netflow TCAM
Netflow Table
Netflow Statistics
L3 FE (TYCHO)
FIB TCAM &

SSRAM
Security ACL
TCAM
Adjacency Table
4. Info parsed from

the packet is used
to do
Parallel lookups in
FIB, QoS & Security
ACL TCAMs
FIB result (SSRAM)
has a pointer to Adj
table
QoS ACL TCAM
IP Packet Parse
L2 FE (SUPERMAN)
IP Packet Parse
L2 MAC Table

Netflow TCAM
L3 FE (TYCHO)
FIB TCAM &

SSRAM
Netflow Table
Security ACL
TCAM
Netflow Statistics
QoS ACL TCAM
Adjacency Table
IP Packet Parse
L2 FE (SUPERMAN)
IP Packet Parse
L2 MAC Table
5. The Netflow Engine

performs a lookup in
Netflow TCAM
to retrieve an index
into the Netflow
Table
If the flow exists then
the results are sent
for Adjacency
processing...
If not, a flow is
created.

Netflow TCAM
L3 FE (TYCHO)
FIB TCAM &

SSRAM
Netflow Table
Security ACL
TCAM
Netflow Statistics
QoS ACL TCAM
6
IP Packet Parse
L2 FE (SUPERMAN)
IP Packet Parse
L2 MAC Table
Adjacency Table
6. The Adjacency
Engine performs a
lookup in
the Adjacency Table
to find the frame
Rewrite info
Lookup result is
passed to the
Rewrite process

Netflow TCAM
Netflow Table
Netflow Statistics
L3 FE (TYCHO)
7
FIB TCAM &

SSRAM
Security ACL
TCAM
4
6
IP Packet Parse
L2 FE (SUPERMAN)
IP Packet Parse
L2 MAC Table
QoS ACL TCAM

Adjacency Table
7. The Decision
Engine takes results
from FIB, Netflow,
QoS & Security
lookups and applies
Classification &
Policing to the
packet
Then adds rewrite
info

L3 FE (TYCHO)
Netflow TCAM
Netflow Table
FIB TCAM &

SSRAM
Security ACL
TCAM
QoS ACL TCAM
Netflow Statistics
8
Adjacency Table
IP Packet Parse
L2 FE (SUPERMAN)
IP Packet Parse
L2 MAC Table
8
8. The Decision
Engine updates the
Netflow & Adjacency
Statistics
Finally, the packet is
sent back to
Superman for a final
L2 lookup based on
new Destination MAC
& Outbound Port

CL1 TCAM
Supervisor2T / PFC4 Architecture
contains Security
Fabric interface / Replication

engine to interface with fabric &
bus, and for multicast/SPAN
replication
MSFC5 Complex
contains single
(combined SP/RP) CMP
CPU
Multicast
Expansion Table
1GE / 10GE
Uplinks
FIB contains IPv4 / IPv6

prefixes and MPLS
contains Security
entries
& Feature ACL
entries
NetFlow Hash
ADJ
CL2 TCAM
& Data tables
contains
contains
QoS
for stats &
Rewrite info
ACL
entries
features
MSFC 5
Central
Management
Processor
NetFlow
2 x 1 Gbps
ADJ
TCAM
FIB
TCAM
Port ASIC
CPU
DRAM
DRAM
CL1
TCAM
LIF
MAP
CL2
TCAM
RPF
Table
L3/4 Engine
Flash
Flash
Reverse Path
Forwarding
info
MET
Switch
Switch Fabric
Fabric
26
x
40G
26 x 40G Traces
Traces
Traces # 1 to 26
Switch Fabric
LIF Map contains

port mappings
40 Gbps
EOBC
Fabric
Fabric Interface
Interface
&&
Replication
Replication Engine
Engine
LIF Table
LIF Stats
PFC4
ACE
ACE
Counter
Counter
L2 Engine
L2 CAM (128K)
Hardware for
ACL TCAM
counters
L2 CAM
contains MAC
entries
DBUS
RBUS
Logical Interface
statistics
Logical Interface (LIF)

Table contains
entries

PFC4 (EARL8) Capabilities
The Next Gen forwarding-engine PFC4 (EARL8) will

offer a number of enhancements
Not only with increased performance (@ 60Mpps) and
extending the HW capabilities of existing features,
but also with brand NEW features
It has hardware support for new features like:
Virtual Private LAN Services (VPLS)
Cisco Trust Security (CTS)
Flexible Netflow (FnF)
Control-Plane Policing (CoPP)
Dual-RPF & Source-Register (Multicast)

Architecturally, the PFC/DFC4 is almost the same as the PFC/DFC3

What changes is the Single-Cycle Input (IFE) & Output (IFE) Processing
Here we perform the Input Forwarding Engine (IFE) pass...
IFE process:
Packet
Packet Header
Header
GV
GV
IF
IF
RP
RP
CL1
CL1
1. IF: Get Port & Ingress LIF QoS info

2. RP: Src FIB Lookup, Source QoS
PO
PO
CL2
CL2
3. CL1: Ingress ACL TCAM Lookup

4. CL2: Select Ingress Class & Policy
5. NF: Select Netflow Class & Policy
RI
RI
L2
L2 FE
FE
(EUREKA)
(EUREKA)
PL
PL
L3
L3
L3
L3 FE
FE
(LAMIRA)
(LAMIRA)
NF
NF
6. L3: Dst FIB Lookup, Dst QoS

7. PL: Apply Ingress Policing & Marking

Architecturally, the PFC/DFC4 is almost the same as the PFC/DFC3

What changes is the Single-Cycle Input (IFE) & Output (IFE) Processing
Here we perform the Output Forwarding Engine (OFE) pass...
OFE process:
RBUS
RBUS Result
Result
GV
GV
IF
IF
RP
RP
CL1
CL1
1. IF: Get Egress LIF QoS info

2. CL1: Egress ACL TCAM lookup
PO
PO
CL2
CL2
3. CL2: Select Egress Policy & Class

4. NF: Select NF Egress Policy & Class
5. PL: Apply Egress Policing & Marking
RI
RI
L2
L2 FE
FE
(EUREKA)
(EUREKA)
PL
PL
L3
L3
L3
L3 FE
FE
(LAMIRA)
(LAMIRA)
NF
NF
6. RI: Generate RBUS result
UNICAST
UNCAST MULTILAYER
FORWARDING
Agenda Topics

Layer 2 Forwarding on PFC
The PFC has an integrated
CAM Table that supports
4096 rows * X pages =
MAC address space
PFC
MAC Table
Table Port
MAC
A
B
C
D
E
F
16, 24, or
32 pages
1
2
3
4
5
6
4096
rows
MAC Table
PFC3A/B = 16 pages (64K entries)

PFC3C = 24 pages (96K entries)
PFC4 = 32 pages (128K entries)

Frame
VLAN
16, 24, or
32 Pages
MAC
A
B
C
D
E
F
0000.2222.7777 | 20
0000.1111.cccc | 10
0000.dddd.a112 | 30
MAC Table Row
Port
1
2
3
4
5
6
PFC
MAC
Hash
CAM Table
0000.bbbb.ac1c | 30
HIT!!!
MAC Table
1. Hash result identifies the starting Page and Row in MAC table
2. Lookup key (VLAN + MAC) compared to contents of indexed line on each page, sequentially
3. Destination lookup: Match returns destination interface(s), Miss results in Flood
4. Source lookup: Match updates age of matching entry, Miss installs new entry in table
4096
Rows

Layer 3 Forwarding is controlled by the CEF (Cisco Express

Forwarding) Software & Hardware interaction
Routing Protocols receive routing updates from the network...
Routing Protocols
OSPF, EIGRP, ISIS, BGP, etc
Control Plane (RP)
Holds routing tables in

Routing information Base (RIB)
from Static Routes
and all running
Routing Protocols
Software CEF
Takes RIB and builds a

Forwarding Information Base
(FIB) containing IP/mask prefixes
FIB (on PFC/DFC)
FIB & ADJ tables are used by EARL to

perform L3 lookups & forwarding
Hardware CEF
Loads FIB into PFC

& distributes to DFCs

Interacts with the external network where all the routing protocols operate
(e.g. OSPF, BGP, EIGRP, etc)
Control Plane (RP)
Data Plane (PFC/DFC)
Holds the FIB, Adjacency, ACLs and

Netflow Statistics - performs the
hardware based switching of packets
Hardware Based CEF Process

1. FIB lookup based on destination prefix (longest-match)
2. FIB Hit returns Adjacency pointer
3. Adjacency contains Rewrite (next-hop) information
4. ACL, QoS & NetFlow lookups occur in parallel, and effect final result

Located on the PFC are the FIB and Adjacency Table

The FIB contains:
L3 entries are arranged

logically from MOST to LEAST
specific (based on /mask)
Overall FIB hardware shared
by:
IPv4 Unicast
IPv4 Multicast
IPv6 Unicast
IPv6 Multicast
MPLS
The Adjacency Table:

L2 Re-Write information
and / or pointers for
replication
Hardware adjacency table
also shared among protocols
172.20.45.1
10.1.1.100
MASK (/32)
10.1.3.0
10.1.2.0
MASK (/24)
IF, MACs, MTU

IF, MACs, MTU
IF, MACs, MTU
IF, MACs, MTU
10.1.0.0
172.16.0.0
MASK (/16)
0.0.0.0
MASK (/0)
FIB TCAM
Adjacency Table

Assuming a lookup was performed for a packet with a destination of 10.1.5.2 /

24, then the following would occur
11
172.20.45.1
Packet
10.1.1.100
MASK (/32)
22
Key Gen
IF, MACs, MTU
IF, MACs, MTU
10.1.3.0
10.1.2.0
33
Lookup Key
Load-Sharing
Hash
MASK (/24)
HIT!
10.1.0.0
172.16.0.0
MASK (/16)
0.0.0.0
MASK (/0)
FIB TCAM
44
66
IF, MACs, MTU

IF, MACs, MTU
55
Adjacency Table
77
MULTICAST
MULTICAST MULTILAYER
FORWARDING
Agenda Topics

Multicast Forwarding on the PFC
Multicast is the act of forwarding a single packet that will reach multiple hosts
The Catalyst 6500 has a number of hardware resources which are used to
facilitate Multicast forwarding in hardware
Sender
Group A
Receivers Group B
Receivers Group A
Sender
Group B

The software & hardware elements used to facilitate Multicast forwarding are
RP/SP CPU
RP
RP builds
builds MROUTE
MROUTE Table
Table
SP
SP downloads
downloads into
into FIB
FIB (PFC)
(PFC)
FIB
Contains
Contains the
the Multicast
Multicast CEF
CEF info
info
Also
Also does
does ECMP,
ECMP, ifif available
available
ADJACENCY
TABLE
Contains
Contains the
the Rewrite
Rewrite Information
Information
and
and Index
Index into
into MET
MET table
table
MULTICAST
EXPANSION
TABLE (MET)
Contains
Contains Output
Output Interface
Interface Lists
Lists (OIL)
(OIL)
requiring
requiring replication
replication
Maps
Maps the
the OIL
OIL replication
replication list
list to
to the
the
physical
physical ports
ports

Multicast Hardware
CPU Daughter Card

RP DRAM
SP DRAM
RP Flash
SP Flash
RP CPU
maintains
state,
downloads
tables
RP CPU
(MSFC)
SP CPU
SP CPU programs
hardware, performs
IGMP snooping
PORT
ASIC
Counters
Layer 3 Engine
CAM
ACL TCAM
MET
Adjacency
Layer 2 Engine
FIB TCAM
Fabric Interface
and
Replication Engine
Netflow Table
Switch Fabric
L2 CAM contains
DBUS
Multicast MAC
RBUS
EOBC entries
Netflow TCAM
Fabric
Channels
MET
contains
OILs for
replication
QoS TCAM
Replication
Engine
replicates
packets to OILs
Policy Feature Card

FIB contains
(S,G) and (*,G)
entries, and
RPF VLAN
ADJ contains
rewrite info
and MET
index

IPv4 Multicast Lookup = 10.1.1.1, 239.1.1.1 /32
144 bits
36 bits
Forwarding Engine
Replication Engine
256 bits
172.21.4.19, 225.3.3.3
10.1.44.199, 240.9.8.1
10.1.1.1, 239.1.1.1
RW MACs, Encap,
MET Index
MASK (S,G) /32
RW MACs, Encap,
MET Index
*, 234.0.1.1
MASK BiDir Entries
RPF VLAN,
ADJ Pointer
10.1.1.0, 224.0.0.0
MASK IF 224/4 Entries
RW MACs, Encap,
MET Index
32 bits
RW MACs, Encap,
MET Index
OIL #1
OIL #2
OIL #3
OIL #4
*, 229.0.1.1
MASK PIM-SM (*,G) /32
Result Memory
FIB TCAM
Adjacency Table
MET
LOAD-BALANCING
LOAD-BALANCING
Agenda Topics

Load-Balancing Traffic
The Catalyst 6500 also supports traffic load-balancing
Two forms of load-balancing exist
Etherchannel
Up
Up to
to 88 Physical
Physical Links
Links
can
can be
be bundled
bundled together,
together,
to
to make
make them
them logically
logically
Look
Look && Work
Work like
like
11 single
single Larger
Larger link
link
Equal Cost Multi-path

(ECMP) Routing
Up
Up to
to 8*
8* Equal
Equal Cost
Cost Routes
Routes
(paths)
(paths) can
can be
be used
used to
to
forward
forward traffic
traffic to
to aa
remote
remote network
network
(e.g.
(e.g. from
from AA to
to B)
B)

Etherchannel - How it works?
Etherchannel uses a load-balancing algorithm to determine which link in the bundle to use
The inputs to the algorithm are a combination of L2, L3 addresses and / or L4 ports
6500(config)#
6500(config)# port-channel
port-channel load-balance
load-balance ??
dst-ip
Dst
dst-ip
Dst IP
IP Addr
Addr
dst-mac
Dst
dst-mac
Dst Mac
Mac Addr
Addr
dst-port
Dst
TCP/UDP
dst-port
Dst TCP/UDP Port
Port
mpls
Load
mpls
Load Balancing
Balancing for
for MPLS
MPLS packets
packets
src-dst-ip
Src
XOR
Dst
IP
Addr
src-dst-ip
Src XOR Dst IP Addr
src-dst-mac
Src
src-dst-mac
Src XOR
XOR Dst
Dst Mac
Mac Addr
Addr
src-dst-port
src-dst-port Src-Dst
Src-Dst TCP/UDP
TCP/UDP Port
Port
src-ip
Src
IP
Addr
src-ip
Src IP Addr
src-mac
Src
src-mac
Src Mac
Mac Addr
Addr
src-port
Src
TCP/UDP
src-port
Src TCP/UDP Port
Port
Why do we
support 8
Etherchannel
links?
We use a 3-bit (0-7) RBH (Results Bundle Hash) field in DBUS header, along with the
bndl_select register in the port ASIC, to decide which link should forward a given frame...

Etherchannel - Which Link is Used?
The output of the Hash algorithm is a HEX string which matches a bit out of an 8 bit string - this
in turn identifies the link in the bundle that will be used to transmit the data
Load Balance Option
Bit Result
HASH
Bit 7
Bit 6
Bit 5
Bit 4
Bt 3
Bit 2
Bit 1
Bit 0
2 Links
Link 1
Link 1
Link 1
Link 1
Link 2
Link 2
Link 2
Link 2
3 Links
Link 1
Link 1
Link 1
Link 2
Link 2
Link 2
Link 3
Link 3
4 Links
Link 1
Link 1
Link 2
Link 2
Link 3
Link 3
Link 4
Link 4
5 Links
Link 1
Link 1
Link 2
Link 2
Link 3
Link 3
Link 4
Link 5
6 Links
Link 1
Link 1
Link 2
Link 2
Link 3
Link 4
Link 5
Link 6
7 Links
Link 1
Link 1
Link 2
Link 3
Link 4
Link 5
Link 6
Link 7
8 Links
Link 1
Link 2
Link 3
Link 4
Link 5
Link 6
Link 7
Link 8
Example:
Example:
Hash result = 0x2 for 6 links
Hash result = 0x4 for 3 links
Result link is link 4
Result link is link 2

ECMP - How it works?
The Catalyst 6500 maintains an Adjacency entry for each link that can be used to
reach the destination - this can be viewed using the following command
G3/1
G3/2
G4/1
G4/2
6500#show
6500#show mls
mls cef
cef lookup
lookup 10.10.10.1
10.10.10.1
Codes:
Codes:
Index
Index
1874
1874
decap
decap -- Decapsulation,
Decapsulation, ++ -- Push
Push Label
Label
Prefix
Adjacency
Prefix
Adjacency
10.10.10.0/24
Gi3/1,
0000.0000.0013
10.10.10.0/24
Gi3/1,
0000.0000.0013
Gi3/2,
0000.0000.0014
Gi3/2,
0000.0000.0014
Gi4/1,
0000.0000.0015
Gi4/1,
0000.0000.0015
Gi4/2,
0000.0000.0016
Gi4/2,
0000.0000.0016
B
10.10.10.1

ECMP - Which Link is Used?
In the FIB, a matching prefix is found - this points to an Adjacency pointer that contains the
number of paths for that Adjacency Index
A hash is computed returning a result (an index offset) into the Adjacency table that points to the
rewrite info and the outbound interface
Prefix Entries
10.7.1.0
10.5.1.0
10.3.2.0
10.2.6.0
10.1.5.0
11.1.1.0
Mask /24
(TCAM)
Results Memory
Adjacency Entry #1
Adj Index: 15 / Number of paths: 3
Adjacency Entry #3
Adjacency Entry #4
Adjacency Entry #5
Adjacency Entry #6
Adjacency Entry #7
FIB
(SSRAM)
Source IP +
Destination IP +
L4 ports =
Unique ID
Load-Balancing
Hash
Result / Next Hop
Adj Index Offset: 0
Adj Index Offset: 1
Adj Index Offset: 2
New MAC and VLAN

New MAC and VLAN
New MAC and VLAN
Adj Idx 15: Rewrite info
Adj Idx 15+1: Rewrite info
Adj Idx 15+2: Rewrite info
New MAC and VLAN
New MAC and VLAN
New MAC and VLAN
New MAC and VLAN
New MAC and VLAN
New MAC and VLAN
Adjacency Table

ECMP - Some Caveats
IP CEF load-balancing is PerFlow
Per-Packet load-balancing is
NOT supported*
=/< 12.2SXF Up to 8 hardware
CEF load-sharing paths per
prefix
>/= 12.2SXH Up to 16
hardware CEF load-sharing
paths per prefix
=/< 12.2SXF default Loadsharing based on Source &
Destination IP addresses
>/= 12.2SXH default Loadsharing based on Src-Dst-IP &
L4 port (mixed)

ECMP - Polarization
Polarization is an issue that occurs in a multi-hop environment where a common

link is used continuously due to ECMP hashing
Switch X will ALWAYS chose link A
because the SRC & DST IP hash inputs
are the SAME for each packet between
these two hosts
IP: 10.5.3.2
AA
IP: 10.22.8.17
CC
BB
A Unique ID in Supervisor 720 prevents polarization

(can be changed with ip cef load-sharing algorithm universal command)
http://bock-bock.cisco.com/~tstevens/FAQs/pfc3-cef-polarization.ppt

6500 Part 4

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

6500 Part 4

Enviado por

Direitos autorais:

Formatos disponíveis

CATALYST 6500

Catalyst 6500 Internals

What lies within the Catalyst

This section will look at some of the internals of the

Catalyst 6500 Internals

Everything a Switch does is centered on the Data that it forwards

Catalyst 6500 Internals

Catalyst 6500 Internals

Catalyst 6500 Internals

MAC Table also

MAC Tables range

Catalyst 6500 Internals

Catalyst 6500 Internals

Catalyst 6500 Internals

We will explore a Layer 2 operation first Packets from host A to host B

A sends ARP - who is 10.1.1.3?

ARP is broadcast & DMAC: All FFs,

Switch does CAM lookup using

Catalyst 6500 Internals

A sends ARP - who is 10.5.1.2?

Switch L3 interface replies to ARP saying send it to me

Catalyst 6500 Internals

Switch1 does an L2 / L3 lookup

Switch2 does an L2 / L3 lookup

Catalyst 6500 Internals

DBUS - Data Bus: Path

Catalyst 6500 Internals

The EOBC is NOT

Catalyst 6500 Internals

Two transport mechanisms exist:

IOS uses SCP & IPC, and introduces:

Control Plane Protocol

Catalyst 6500 Internals

Catalyst 6500 Internals

The Enhanced Address Recognition Logic (EARL)

There have actually been many Generations of the EARL

A brief history of the PFC

1. The EARL concept was first introduced on the Catalyst

5. The PFC concept was first introduced on the Catalyst 6000

PFC vs. DFC?

The PFC / EARL hardware (ASICs & Memory)

The Distributed Forwarding Card

Catalyst 6500 Internals

Fabric interface / Replication

Catalyst 6500 Internals

Starting with PFC3 (EARL7), the EARL was designed

Catalyst 6500 Internals

FIB TCAM &

QoS ACL TCAM

Catalyst 6500 Internals

FIB TCAM &

QoS ACL TCAM

Catalyst 6500 Internals

FIB TCAM &

QoS ACL TCAM

Catalyst 6500 Internals

FIB TCAM &

4. Info parsed from

QoS ACL TCAM

Catalyst 6500 Internals

FIB TCAM &

QoS ACL TCAM

5. The Netflow Engine