Escolar Documentos
Profissional Documentos
Cultura Documentos
Cisco Public
1K
Cisco Nexus
x86
TECDCT-2001
Cisco Public
FCoE Storage
Ethernet
Fibre Channel
Dedicated FCoE Link
Converged Link
Nexus 7000
Nexus 7000
VNP
EvPC
MDS 9500
Nexus 5000
Nexus 7000
FCoE Storage
FCoE Storage
Nexus 2000
BRKSAN-2047 Melbourne Cisco LIve
Cisco Public
FCoE
Benefits
Encapsulation of FC Frames
over Ethernet
Fewer Cables
Both block I/O & Ethernet traffic
co-exist on same cable
Enables FC to Run
on a Lossless
Ethernet Network
Fibre
Channel
Traffic
TECDCT-2001
Cisco Public
No Gateway
Unified Fabric
Why?
TECDCT-2001
FC Economic Model
Embedded on Motherboard
Specialized Drivers
Many Suppliers
Few Suppliers
Mainstream Technology
Specialized Technology
Widely Understood
Special Expertise
Interoperability by Design
Interoperability by Test
Cisco Public
Unified Fabric
Why?
FCoE SAN
iSCSI
Gateway
NAS
Appliance
NAS
Gateway
Computer System
Application
Application
Application
Application
Application
File System
File System
File System
File System
File System
Volume
VolumeManager
Manager
Volume Manager
Volume Manager
I/O Redirector
I/O Redirector
FCoE Driver
NFS/CIFS
TCP/IP Stack
NFS/CIFS
TCP/IP Stack
NIC
NIC
NIC
NIC
NIC
BlockBlock
I/O
I/O
Ethernet
FCoE
TECDCT-2001
iSCSI
Appliance
Cisco Public
File I/O
Ethernet
Ethernet
Ethernet
Ethernet
NIC
NIC
NIC
NIC
TCP/IP Stack
TCP/IP Stack
TCP/IP Stack
TCP/IP Stack
iSCSI Layer
Bus Adapter
iSCSI Layer
FC HBA
File System
Device Driver
File System
FC HBA
FC
Block I/O
FC
Source: Infonetics
TECDCT-2001
Cisco Public
1K
Cisco Nexus
x86
TECDCT-2001
Cisco Public
FCoE
www.T11.org
IEEE 802.1
DCB
Fibre Channel
on network
media
PFC
ETS
IEEE 802.1Qaz
Priority Grouping
Enhanced Transmission Selection
FC-BB-5
IEEE 802.1Qbb
Priority-based Flow Control
TECDCT-2001
DCBx
Cisco Public
IEEE 802.1Qaz
Configuration Verification
ET = FCoE
Ver
Reserved
Reserved
SOF
Reserved
FCS
Byte 0
Cisco Public
FC
Header
Byte 2197
FCoE
Header
Ethernet
Header
2012 Cisco and/or its affiliates. All rights reserved.
Reserved
Reserved
FCoE is a standard - June 3rd 2009, the FCBB-5 working group of T11 completed its
work and unanimously approved a final
standard for FCoE
TECDCT-2001
Bit 31
FC Payload
11
CRC
EOF
FCS
Bit 0
Data Plane
http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html
TECDCT-2001
Cisco Public
12
FC-1 Encoding
TECDCT-2001
Cisco Public
13
FCoE Switch
FCF
Enode
Initiator
VLAN
Discover
y
VLAN
Discovery
FCF
Discovery
FCF
Discovery
FCoE
Initialization
Protocol
(FIP)
FLOGI/FDIS
C Accept
FLOGI/F
DISC
FC
Command
Responses
FC
Command
TECDCT-2001
Cisco Public
14
FCoE
Protocol
FCoE Switch
FC
port
FC Domain ID : 15
FC
port
FCF
FC
port
Ethernet Bridge
FC
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Cisco Public
Eth
port
Eth
port
15
FCF
Switch VE_Port
VE_Port
FCoE_
NPV
Switch
VF_Port
VNP_Port
VF_Port
VN_Port
VF_Port
VN_Port Node
End
Node
End
Cisco Public
16
FCF
FCF
Link
PCIe
Ethernet
Drivers
Fibre Channel
Ethernet
Ethernet Driver
bound to
Ethernet NIC PCI
address
FCoE VLAN
terminates on the
CNA
10GbE
10GbE
Data VLAN(s)
are passed to the
Ethernet driver
Fibre Channel
Drivers
Operating System
TECDCT-2001
Cisco Public
17
FC Driver
bound to FC
HBA PCI
address
Target
FC Fabric
capable switch
DCBX Negotiation
Access switch is a Fibre Channel Forwarder (FCF)
Dual fabrics are still deployed for redundancy
Unified Wire
CNA
ENode
TECDCT-2001
Cisco Public
18
Target
FC or FCoE Fabric
VF_Port
FC-MAP
(0E-FC-xx)
FC-MAC
Address
TECDCT-2001
FC-MAP
(0E-FC-xx)
VN_Port
CNA
FC-ID
10.00.01
FC-ID
7.8.9
Cisco Public
ENode
19
FIP Discovery
FIP snooping will use this as a mechanism in realizing the ACLs put in place to prevent
data corruption
FC-MAP
(0E-FC-xx)
FC-MAC
Address
BRKSAN-2047 Melbourne Cisco LIve
FC-MAP
(0E-FC-xx)
2012 Cisco and/or its affiliates. All rights reserved.
FC-ID
10.00.01
FC-ID
7.8.9
Cisco Public
20
The FCF-MAC address is configured on the Nexus 5K by default once feature fcoe has
been configured
This is the MAC address returned in step 2 of the FIP exchange
This MAC is used by the host to login to the FCoE fabric
Cisco Public
21
pwwn 50:06:01:61:3c:e0:1a:f6
Target
Fabric Zoning
Zoning is a feature of the fabric and is independent of
Ethernet transport
Zoning can be configured on the Nexus 5000/7000
using the CLI or Fabric Manager
FC/FCoE Fabric
pwwn 10:00:00:00:c9:76:fd:31
Initiator
TECDCT-2001
Cisco Public
22
Login complete
Flogi and FCoE Databases are populated
Login process: show flogi database and show fcoe database show the logins and associated
FCIDs, xWWNs and FCoE MAC addresses
Cisco Public
23
1K
Cisco Nexus
x86
Futures
TECDCT-2001
Cisco Public
24
Fibre
Channel
Traffic
Fibre Channel
Fibre
Channel
Traffic
or
1G Ethernet Copper/Fibre
10G Ethernet Copper/Fibre
10G DCB/FCoE Copper/Fibre
1/2/4/8G Fibre Channel
FC Attached
Storage
Servers
25
Eth Ports
Slot 1
Slot 3 GEM
Eth
Slot 4 GEM
Eth
FC
FC Ports
Eth Ports
BRKSAN-2047 Melbourne Cisco LIve
FC
Cisco Public
26
N2232PP
Cannot be cancelled
Re-Training is a major barrier to use of 10GBaseT for
block level storage (FCoE)
2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
27
I/O modules
TECDCT-2001
Cisco Public
28
Enables Cisco FabricPath for increased bisectional bandwidth for iSCSI and NAS traffic
FCoE License (N7K-FCOEF132XP)
$10,000 Cisco List
One license per F1/F2 module
Cisco Public
29
Ethernet
TECDCT-2001
Storage
VDC
FCoE &
FIP
Cisco Public
LAN
VDC
Storage
VDC
Ethernet
FCoE &
FIP
Converged I/O
30
Allocate FCoE vlan range from the Owner VDC to the Storage VDC. This is a necessary
step for sharing interfaces to avoid vlan overlap between the Owner VDC and the Storage
VDC
Cisco Public
31
Storage VDC
F2 line cards
Some restrictions when using mixed line cards (F1/F2/M1)
F2 ports need to be in a dedicated VDC if using shared ports
Shared Ports
Storage
VDC
Dedicated Ports
F1/M1
VDC
F2 VDC
F1
Storage
VDC
F1/M1
VDC
NX-OS 5.2
Storage
VDC
F2
F2
VDC
F2
F1
Any non-F2
Cisco Public
32
MDS 9500
T11 FCoE
IEEE DCBX, PFC, ETS
Connectivity FCoE Only, No LAN
VE to Nexus 5000, Nexus 7000, MDS 9500
VF to FCoE Targets
Optics Support
SFP+ SR/LR, SFP+ 1/3/5m Passive, 7/10m Active CX-1 (TwinAx)
Requirements
SUP2A
Fabric 2 modules for the backplane (applicable to 9513 only)
TECDCT-2001
Cisco Public
33
feature lldp
feature vlan-vsan-mapping
Cisco Public
34
1K
Cisco Nexus
x86
Futures
TECDCT-2001
Cisco Public
35
Networks
Connectionless
Logical circuits
Ethernet is non-deterministic.
Unreliable transfers
High connectivity
Higher latency
Longer distance
Software intense
Fibre-Channel is deterministic.
Flow control is source-based (B2B credits)
Channels
Connection service
Physical circuits
Reliable transfers
High speed
Low latency
Short distance
Hardware intense
TECDCT-2001
Cisco Public
36
Outside Data
Center cloud
Core
L3
L2
Virtual Port-Channel
(VPC)
STP
Access
Virtual Port-Channel
(VPC)
Cisco Public
37
Core
Core
FC
60 Servers with 4 Gb
HBAs
240 G
TECDCT-2001
Cisco Public
24 G
24 G
38
Converged FCoE
Link
FCoE
FC
CORE
L3
AGG
L2
MDS FC
SAN A
Dedicated FCoE
Links/Port-Channels
Access
Converged FCoE
Link
Nexus
Ethernet
Fibre Channel
Dedicated FCoE Link
Converged Link
Cisco Public
MDS FC
SAN B
39
Fabric A
vPC with Converged Links provides an ActiveActive connection for FCoE traffic
B
Fabric B
TECDCT-2001
Cisco Public
40
FC
Core
Core
FC
TECDCT-2001
Cisco Public
41
1K
Cisco Nexus
x86
Futures
TECDCT-2001
Cisco Public
42
Ethernet Enhancements
Can Ethernet Be Lossless?
Yes, with Ethernet PAUSE Frame
Ethernet Link
STOP
PAUSE
Queue Full
Switch A
Switch B
TECDCT-2001
Cisco Public
43
Ethernet Enhancements
IEEE DCB
Developed by IEEE 802.1 Data Center Bridging Task Group (DCB)
All Standards Complete
Standard / Feature
IEEE 802.1Qbb
Priority-based Flow Control (PFC)
Completed
IEEE 802.3bd
Frame Format for PFC
Completed
IEEE 802.1Qaz
Enhanced Transmission Selection (ETS) and
Data Center Bridging eXchange (DCBX)
Completed
CEE (Converged Enhanced Ethernet) is an informal group of companies that submitted initial
inputs to the DCB WGs.
TECDCT-2001
Cisco Public
44
Ethernet Enhancements
DCB Virtual Links
LAN/IP Gateway
VL1
VL2
VL3
Campus Core/
Internet
TECDCT-2001
Cisco Public
Storage Area
Network
45
DSCP is still marked but CoS will be required and used in Nexus Data Center designs
PCP/COS
Network
priority
Acronym
Traffic characteristics
0 (lowest)
BK
Background
BE
Best Effort
EE
Excellent Effort
CA
Critical Applications
VI
VO
IC
Internetwork Control
IEEE 802.1Q-2005
TECDCT-2001
Cisco Public
46
DCBX CNA
Adapter
https://www.cisco.com/en/US/netsol/ns783/index.
html
TECDCT-2001
Cisco Public
Ack no: 0
Other traffic assigned to other CoS values will continue to transmit and rely on upper layer
protocols for retransmission
Not only for FCoE traffic
Transmit Queues
Fibre Channel
Ethernet Link
Receive Buffers
One
One
Two
Three
R_RDY
Packet
B2B Credits
TECDCT-2001
Two
STOP
PAUSE
Three
Four
Four
Five
Five
Six
Six
Seven
Seven
Eight
Eight
Cisco Public
48
Eight
Virtual
Lanes
Prevents a single traffic class of hogging all the bandwidth and starving other
classes
When a given load doesnt fully utilize its allocated bandwidth, it is available to other
classes
Helps accommodate for classes of a bursty nature
Offered Traffic
3G/s
2G/s
3G/s
3G/s
3G/s
3G/s
4G/s
6G/s
t1
TECDCT-2001
3G/s
t2
t3
3G/s
HPC Traffic
3G/s
2G/s
3G/s
Storage Traffic
3G/s
3G/s
3G/s
LAN Traffic
4G/s
5G/s
t1
Cisco Public
t2
t3
49
Nexus QoS
QoS Policy Types
There are three QoS policy types used to define
system behavior (qos, queuing, network-qos)
There are three policy attachment points to apply
these policies to
Ingress interface
System as a whole (defines global behavior)
Egress interface
TECDCT-2001
Policy Type
Function
Attach Point
qos
system qos
ingress Interface
queuing
system qos
egress Interface
ingress Interface
network-qos
system qos
Cisco Public
50
N5k(config-cmap-qos)# match
Cisco Public
51
TECDCT-2001
Cisco Public
Network-QoS
Parameters
Class Type
MTU
Ingress Buffer Size
Marking
Default Value
Drop class
1538
20.4KB
No marking
52
TECDCT-2001
Cisco Public
53
On Nexus 5000 once feature fcoe is configured, 2 classes are made by default
policy-map type qos default-in-policy
class type qos class-fcoe
set qos-group 1
class type qos class-default
set qos-group 0
Enabling the FCoE feature on Nexus 5548/96 does not create no-drop policies automatically as
on Nexus 5010/20
Must add policies under system QOS:
system qos
service-policy type qos input fcoe-default-in-policy
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type network-qos fcoe-default-nq-policy
TECDCT-2001
Cisco Public
55
1Gig FC HBAs
Traditional Server
Best Practice: Tune FCoE queue to provide equivalent capacity to the HBA that
would have been used (1G, 2G, )
TECDCT-2001
Cisco Public
57
Template
TECDCT-2001
(Priority)
default-nq-8e-policy
0,1,2,3,4,5,6,7
5,6,7
default-nq-7e-policy
0,1,2,4,5,6,7
5,6,7
default-nq-6e-policy
0,1,2,5,6,7
5,6,7
3,4
default-nq-4e-policy
0,5,6,7
5,6,7
1,2,3,4
Cisco Public
NoDrop CoS
(Priority)
58
1K
Cisco Nexus
x86
TECDCT-2001
Cisco Public
63
FCoE Edge
N-Port Virtualizer (NPV)
N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a switch to act like a server performing
multiple logins through a single physical link
Physical servers connected to the NPV switch login to the upstream NPIV core switch
Physical uplink from NPV switch to FC NPIV core switch does actual FLOGI
Eth1/1
Server1
N_Port_ID 1
Eth1/2
Server2
N_Port_ID 2
Eth1/3
Server3
N_Port_ID 3
NP-Port
F-Port
F_Port
N-Port
Cisco Public
64
LAN Fabric
TECDCT-2001
Cisco Public
Fabric A
Fabric B
VSAN 3
VSAN 2
Nexus 5000
FCF
Nexus 5000
FCF
VLAN 10,20
VLAN 10,30
65
Fabric A Supporting
VSAN 20 & 40
Fabric B Supporting
VSAN 30 & 50
TF
VSAN 30,50
TNP
VF
VLAN 10,30
VN
VLAN 10,50
Server 1
VSAN 20 & 30
Server 2
VSAN 40 & 50
Cisco Public
66
FC
FCoE
Nexus 5000 as
FCF or as NPV
device
Cisco Public
FC
Nexus
5000/5500
Fabric B
67
LAN Fabric
Fabric A
Fabric B
Nexus 5000
FCF-A
Nexus 5000
FCF-B
VLAN 10,20
STP Edge Trunk
VLAN 10,30
TECDCT-2001
Cisco Public
68
FC
FCoE
N5K-A
N5K-B
FEX 101
FEX 100
TECDCT-2001
Cisco Public
69
SAN A
N5K(+N2K)
N5K(+N2K) as
as FCF
FCF
or
NPV
Device
or NPV Device
vfc 1
po 1
N5K(+N2K)
N5K(+N2K) as
as FCF
FCF
or
NPV
Device
or NPV Device
vfc 2
po 1
Eth1/1
Eth1/1
po-1 (vpc 1)
Configuration
SAN B
interface vfc 1
bind interface po-1
Configuration
interface vfc 2
bind interface po-1
1/2/4/8G FC
10G Ethernet
10G Unified I/O
10G FCoE
Cisco Public
70
As of NX-OS Release 5.1(3)N1(1) for N5K, new VFC binding models will be supported
In this case, we now support VF_Port binding to a member port of a given port-channel
Check the configuration guide and operations guide for additional VFC binding changes
TECDCT-2001
Cisco Public
71
Transparent Bridges?
FIP Snooping
What does a FIP Snooping device do?
FIP solicitations (VLAN Disc, FCF Disc and FLOGI) sent
out from the CNA and FIP responses from the FCF are
snooped
VF
FIP Snooping
ENode MAC
0E.FC.00.07.08.09
VN
ENode
Cisco Public
FCF MAC
0E.FC.00.DD.EE.FF
72
FCF
Fibre Channel Configuration and
Control Applied at the Edge Port
VNP
Proxy FCoE VLAN
Discovery
Proxy FCoE FCF Discovery
Cisco Public
VF
73
FCoE
NPV
MDS w/
release 5.2.x
TECDCT-2001
Cisco Public
74
FCoE NPV
For Your
Reference
Edge Capabilities
Benefits
DCB
FIP Snooping
FCoE NPV
FCoE
Switch
Scalability
(Server connectivity)
FC to FCoE Migration
(Ease of FCoE device
migration from FC fabric to
FCoE network)
SAN Administration
(VSAN, VFC visibility for SAN
Administration)
TECDCT-2001
Cisco Public
75
1K
Cisco Nexus
x86
TECDCT-2001
Cisco Public
76
FCoE
FC
VN
VF
VE
VE
VE
VE
VF
VN
Cisco Public
77
Cisco Public
78
In FCoE there cannot be more than one VSAN mapped over a VLAN.
The net result is that trunking is done at the Ethernet level, not at the FC level.
FC trunking is not needed and the Fibre Channel Exchange Switch Capabilities(ESC)
& Exchange Port Parameters (EPP) processing is not required to be performed as on
the MDS
Cisco Public
79
FCIP
IP
FCoE
FCoE
3 km
Point-to-point FC
3 km
Speed
(Gbps)
Max Distance
(KM)
8000
4000
FCoE Convereged
2000
FCoE Dedicated
1000
10
680
20 km
10
km1
FC
Cisco Public
80
FC
CORE
L3
AGG
L2
N7K
N7K
MDS FC SAN
A
MDS FC SAN
B
Fibre Channel
Access
N7K
N7K
Cisco Public
81
Storage on MDS
Extending FCoE to MDS SAN from Access
FC
FCoE
CORE
AGG
L3
L2
MDS FC
SAN A
MDS FC
SAN B
N5K
Ethernet
Fibre Channel
Dedicated FCoE Link
Converged Link
Cisco Public
82
CORE
Multiple VDCs
FCoE SAN
LAN Agg
LAN Core
L3
AGG
L2
SAN A
SAN B
N5K
Access
N5K
Ethernet
Fibre Channel
Dedicated FCoE Link
Converged Link
BRKSAN-2047 Melbourne Cisco LIve
Cisco Public
83
Fabric Manager
FMS
DCNM
(Converged)
DCNM
DCNM-LAN
DESKTOP
CLIENT
Cisco Public
84
LAN/SAN Roles
Data Center Network Manager
Collaborative management
FCoE Wizards
Nexus
7000
Tasks
Tools
LAN
Admin
DCNM-LAN
SAN
Admin
DCNM-SAN
Cisco Public
85
Recommended Reading
Cisco Public
86
Q&A
#CiscoPlusCA
BRKSAN-2047 Melbourne Cisco LIve
Cisco Public
87
Cisco Public
88