Designing Data Centers With The Nexus 7000: Ben Basler Technical Marketing Engineer January 2009, v2.1

Designing Data Centers with the Nexus 7000
Ben Basler Technical Marketing Engineer January 2009, v2.1

2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Disclaimer
Network design is not black and white but, some design approaches have proven to work better than others
Design options in this presentation follow existing ESE design recommendations with a focus on Nexus 7000 Designs presented here are based on NX-OS 4.0 and 4.1 releases only
Does not address SAN designs
Does not address Unified I/O, FCoE
Recommendations are based on what customers have deployed since FCS of the platform
2009 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
Agenda Data Center Design

Design Aspects and Physics
NX-OS Best Practices Oversubscription
Advanced Design Options

Conclusion
Cisco Confidential
3-Tier Data Center Design

Core
Aggregation
Access
Layer Core Aggregation (Distribution) Access
Switch Type Modular Modular Fixed or Modular
Port Speed 10GE 10GE GE/10GE
Configuration Layer 3 only L2/L3 boundary Layer 2 only
Oversubscription Low to medium Medium to high Medium to high
Other Campus hand off Services (optional) ToR, MoR, Blade Switch
4
Table values are considered typical for a green field deployment

Layer 2 Domain
Use Rapid-PVST+ or MST Typically doesnt span beyond aggregation layer Pay attention to logical port count (MST scales better)
Cisco Confidential
Aggregation Layer (no Core Layer):

Modular Chassis
High 10GE Density Layer 2 Trunks to Access Default Gateway for Servers Typically Attaches Services
How far does it scale? To be discussed later

Access Layer Options

Top of Rack (ToR)
Typically 1-RU servers 1-2 GE LOMs Mostly 1 sometimes 2 ToR switches Copper cabling stays within rack Low copper density in ToR Higher chance of East-West traffic hitting aggregation layer Drives higher STP logical port count for aggregation layer Denser server count
Middle of Row (MoR aka EoR)

1-RU or multi-RU servers Multiple GE or 10GE NICs Horizontal copper cabling for servers High copper cable density in MoR Larger portion of East-West traffic (Server-to-Server) stays in access Larger subnets less address waste Keeps agg. STP logical port count low (more EtherChannels, less trunk ports) Lower # of network devices to manage
Mapping to Data Center Floor Plan
SAN
Pod section using MoR
Pod section using ToR
Cisco Confidential
Pod 1
Pod 2
Pod: Repeatable Network Environment with Predictable Scalability
Pod 1
Pod 2
Cisco Confidential
Zones
Cisco Confidential
10
Cabling Options
Typically <100m
In-Rack Cabling
10GBase-CX1 Up to 7m
In-Row and X-Row Cabling

10GBase-USR* (100m using OM3 fiber)
10GBase-SR (300m using OM3 fiber)

10GBase-LR (10km using SMF)
*Available Q2CY09 11

NX-OS Best Practices
Layer 2 Layer 3 Traffic flow optimizations Security High Availability System Management
Oversubscription Advanced Design Options Conclusion

12
A Closer Look at Layer 2

Haves:
Hardware based MAC learning with software synchronization MST Rapid-PVST+ Backwards compatibility to PVST+ and 802.1d Stateful HA (process restart and switchover) for STP, LACP, CDP, IGMP, etc. VTP Transparent Mode NX-OS 4.1(2)
Have-nots:
VTP Client/Server/v3 support LLDP-MED FlexLink
Cisco Confidential
13

Dos:
Use MST for large scale L2 environments CPU load is lower even with large number of trunks/VLANs Up to 75,000 logical ports (RPVST+ up to 16,000) Use newest STP enhancements (e.g. Bridge Assurance and STP Guards) Avoid STP by using vPC NX-OS 4.1(3)
Donts:
Tune timers in dual Sup environments w/o testing switchover
Cisco Confidential
14

Haves:
IPv4, IPv6 and GRE tunnel hardware forwarding Full unicast and multicast routing protocol support Graceful restart (GR) for routing protocols, SSO for OSPF (aka NSR) Full VRF-lite support (255 VRFs per VDC) all features are VRF-aware! Policy Based Routing Full and Sampled Netflow
Have-nots:
MPLS Import/export capabilities for VRFs/BGP BFD Sub-second timers for all routing protocol NAT, any feature that includes software forwarding
Cisco Confidential
15

Dos:
Use VLANs, VRFs and VDCs for virtualization needs Use PBR set VRF for shared service environments
Donts:
Peer with an ISP and get full BGP feed (current TCAM cant hold 272k prefixes)
Cisco Confidential
16
Traffic Flow Optimizations

Haves:
Scalable 802.3ad/LACP implementation (up to 768 port channels in 4.1(2)) Improved EtherChannel hashing (adding VLAN to key) Cross-module EtherChannel (including consistency checkers) 16-way ECMP (per-flow or per-packet)
Have-nots:
PAgP/ePAgP support
Dos:
Use cross-module EtherChannels and vPC Modify EtherChannel hash input values to your needs Use port channels with 2, 4 or 8 members for optimal traffic distribution
Donts:
Use per-packet ECMP without considering out-of-order possibility
17
Cisco Confidential
Security Aspects
Haves:
Control Plane Policing (enabled by default) Atomic ACL commits Cisco TrustSec (IEEE 802.1AE MAC Sec)
Have-nots:
Reflexive ACL support Firewall feature set ACS server managing Security Group ACLs
Dos:
Tune CoPP according to specific requirement of environment Use configuration sessions for large ACL configurations
Cisco Confidential
18
High Availability Aspects

Haves:
Stateful process restarts, SSO, ISSU, fabric failover with no packet loss True modularized OS Graceful restart (GR) for EIGRP, OSPF, ISIS, BGP
NSR for OSPF

Dedicated management Ethernet port plus CMP Fully redundant hardware architecture
Have-nots:
Cisco-proprietary version of graceful restart (aka NSF) only standards based graceful restart is available/supported. Exception is built by EIGRP
Cisco Confidential
19
High Availability Aspects

Dos:
Use install all to upgrade software using ISSU Tune protocol timers to lower values for single Sup systems
Donts:
Perform ISSU upgrade @ the busiest hour (we are confident in the solution but it is just not a best practice)
Modify default timers for dual sup environment (subject to testing)
Cisco Confidential
20
System Management Aspects

Haves:
Configuration sessions plus rollback Roles-based access control for management Connectivity Management Processor (CMP)
Relevant MIB support Have-nots:

Centralized AAA role definition and per-command authorization Access classes for VTY (ability to limit VTY access via ACL) Workaround using CoPP ERSPAN
Dos:
Use CMP as backup mechanism to access the system Use mgmt0 as primary interface for SSH/telnet and SNMP, XML, NTP, NDE, etc.
Cisco Confidential
21

Where does it happen? Scaling examples
Conclusion
Cisco Confidential
22
Where are Oversubscription Points?

Network
Uplink to downlink ratio If services are deployed total throughput?
Client-to-Server (North-South) Server-to-Server (East-West)
23
Line card & system architecture
Servers
Internal bus architecture # of CPUs/cores and network interfaces
Active vs. standby interfaces

Disk I/O and applications
Requirements might be different for North-South vs. East-West traffic

Why does a blocking switch architecture make sense? check PPT by Carl Solder here
Network Oversubscription
Nexus 7000 offers
32 port 10GE card with variable oversubscription Use flexible oversubscription to your advantage!
8 Ports 16 Ports 32 Ports 1:1 2:1 4:1
High performance 48 port GE I/O card MoR designs keep large amounts of East-West traffic away from Aggregation layer
Cisco Confidential
24
Oversubscription on Servers
Server Type Ratio GE 10GE
Single attached server

Mostly 1 RU server For GE mostly LOM
10GBaseT LOM not before 2H CY10
1:10
1:5 1:2
100 Mbps
200 Mbps 500 Mbps
1 Gbps
2 Gbps 5 Gbps
Dual attached servers

1 or multi-RU server 2 NICs (active-standby) 4 NICs (active-standby)
1:10
1:5 1:2
100 Mbps / 200 Mbps

200 Mbps / 400 Mbps 500 Mbps / 1 Gbps
VM
1 Gbps / 2 Gbps
2 Gbps / 4 Gbps 5 Gbps / 10 Gbps
VM
Servers with Virtual Machines

1 or multi-RU server 4 NICs total 2x NIC user traffic (active-standby) 2x NIC VMWare (Vmotion & mgmt) Assumption: 10 VMs Total Bandwidth
VM
VM
VM
VM
1:10
1:5 1:2
10 Mbps / 100 Mbps

20 Mbps / 200 Mbps 50 Mbps / 500 Mbps
100 Mbps / 1 Gbps

200 Mbps / 2 Gbps 500 Mbps / 5 Gbps
25
Oversubscription What Is Acceptable?

Questions to ask:
Server-to-Server or Client-to-Server traffic? Understand ratio of East-West vs. North-South Transaction based or bulk file transfer? Web based traffic or backup jobs Are there peak times and if so what causes those? Backup jobs, usage patterns, etc. What generation of Servers are used (bus/CPU technology)?
How does traffic behavior change in failover case? What compromises are you willing to make in event of failure?
Cisco Confidential
26
Recap of Port Densities

Nexus 7010
Supervisor Configuration GE Copper GE Fiber 10GE Ports (4:1 oversub) Single 384 384 256 Dual 384 384 256
Nexus 7018
Single 768 768 512 Dual 768 768 512
Catalyst 6509
Single 384 384 130 Dual 336 336 116
10GE Ports (linerate)
64
64
128
128
34
32
So how far do we scale with that port density?
Cisco Confidential
27
Scalability of 2-Tier Design

Assumptions:
Servers use 2x GE LOM (active/standby) 24 1-RU Servers/Rack @ 300W/server 7.2kW/rack (!) Each server can drive up to 500 Mbps total 1x Catalyst 4948-10G in each rack Each Catalyst 4948-10G has 24 GE links (local servers active) and 24 GE links (servers from adjacent rack backup) 4 port 10GE port channel (dedicated mode) between aggregation switches Each Aggregation switch has 2x 10 Gbps to Campus (dedicated mode) /24 subnets for servers @ 50% utilization = ~ 128 servers/VLAN
Oversubscription Calculation
Server: Access Layer: 500 Mbps/server over two GE NICs (active/standby) 1Gbps/500Mbps = 1:2 24 active server connections into 1x 10G uplink = 2.4:1 1:2 * 2.4:1 * 4:1 = 4.8:1 this is a low oversubscription typically 4:1 up to 12:1
Aggregation Layer: I/O module oversubscription in aggregation = 4:1 Overall:
Cisco Confidential
28
Scaling with Catalyst 6500

Using a Catalyst 6509 the previous example would look like this:
Calculation:
112 ports (4:1) and 4 sup ports (1:1) - 2x 1:1 ports for uplink to campus core
- 4x 1:1 ports to aggregation switch

= 104x 10GE ports (4:1) left 104 racks = 104 racks * 24 servers/rack = 2,496 servers = ~20 VLANs Rack with 24x Servers and 1x Catalyst 4948-10G
Cisco Confidential
Scalability based on 12.2(33)SXH 29
Scaling with Catalyst 6500

Using a Catalyst 6509 the previous example would look like this:
Calculation:
112 ports (4:1) and 4 sup ports (1:1) - 2x 1:1 ports for uplink to campus core

= 104x 10GE ports (4:1) left 104 racks = 104 racks * 24 servers/rack = 2,496 servers = ~20 VLANs
STP logical ports Rapid-PVST+ limit: 10,000 MST limit: This design: 50,000 2,100
Cisco Confidential
Scalability based on 12.2(33)SXH 30
Scaling with Nexus 7010

Using a Nexus 7010 the previous example would look like this:
Calculation:
256 ports (4:1) - 4x 1:1 ports for uplink to campus core

= 208x 10GE ports (4:1) left 208 racks = 208 racks * 24 servers/rack = 4992 servers = 39 VLANs
Cisco Confidential
31

Calculation:

= 208x 10GE ports (4:1) left 208 racks = 208 racks * 24 servers/rack = 4992 servers = 39 VLANs
Cisco Confidential
32

Calculation:

Cisco Confidential
33

Calculation:

Cisco Confidential
34


Services Integration vPC Segmentation/Virtualization
Conclusion
Cisco Confidential
35

The following slides assume an already existing DC and highlight how the Nexus 7000 can be inserted
Customer might expand their DC or deploy next generation servers. Both can change:
Level of oversubscription
Port speed requirements Port density requirements
The above represent opportunities to position Nexus 7000 Remember: the competition will take this opportunity too!
Cisco Confidential
36
Migration from Inline Services

The Need: Higher performance/scalability required in aggregation and/or core
The Solution: Use the Nexus 7000 to satisfy higher port densities The Migration: Move Catalyst 6500 chassis with service modules to an on-thestick configuration and re-use high speed links to connect to the Nexus 7000
Cisco Confidential
37
Service Chassis on-the-stick Successfully Validated by ESE !!!

ESE tested and documented Implementing Nexus 7000 in the Data Center Aggregation Layer with Services
This Cisco Validated Design Guide (CVD) is published on Cisco.com
http://www.cisco.com/en/US/docs/solutions/E nterprise/Data_Center/nx_7000_dc.html
Document contains integration of Catalyst 6500 service chassis as well as ASA 5580. ESE currently verifies advanced design options using vPC as well as VDCs. Available late Q1CY09.
Cisco Confidential
38


Services Integration vPC Virtual Port Channel Segmentation/Virtualization
Conclusion
Cisco Confidential
39
Spanning Tree in the Data Center

Mission of Spanning Tree
Prevent layer 2 loops by blocking redundant links
Problem with Spanning Tree

If BPDU exchange/processing fails, blocked links may go forwarding BROADCAST STORM
The vPC Approach

Recommended vPC topology doesnt have links actively blocked by STP Single link-failure doesnt trigger a STP convergence stability is improved Configuration error or wiring mistake doesnt cause layer 2 loop
Cisco Confidential
40
How does vPC help with STP?

Before vPC
STP blocks redundant uplinks VLAN based load balancing Re-convergence relies on STP Protocol Failure
Primary Root Secondary Root
With vPC
No blocked uplinks Lower oversubscription EtherChannel load balancing (hash) Convergence sub-second

Cisco Confidential
41
How does vPC help with STP?

Before vPC
STP blocks redundant uplinks VLAN based load balancing Re-convergence relies on STP
Protocol Failure
With vPC
No blocked uplinks Lower oversubscription EtherChannel load balancing (hash) Convergence sub-second
Replay of vPC DCVT Update

42
Combining vPC with Services and VSS

Services can be
attached using EtherChannel Appliance based Services-chassis based (standalone or VSS)
vPC
VSS
ACE Appliance
ASA
NAM Appliance
Services Chassis
Cisco Confidential
43


Services Integration vPC Segmentation/Virtualization
Conclusion
Cisco Confidential
44
Segmentation in the Data Center

Drivers
Consolidate data centers from multiple organizational units (OUs) Higher utilization of dedicated data center spaces
Needs
Partition routing so groups cant see/access each other (compliance/organizational boundaries)
Separate management for subset of services
Options
VLANs/VRFs VDCs Service contexts
Cisco Confidential
45
Segmentation Within the Data Center

Allows partitioning of data center services such as SLB, firewalling, etc.
Security policy management and deployment by user group/VRF
Campus/WAN
Global Table (Default VRF) Campus/WAN Nexus 7000
Back-to-back VRFs
802.1Q trunk
Cisco Confidential
46
Shared Services VRFs

Import/Export using PBR-set-VRF
Shared services typically implemented for Backup, Email, DNS, etc. Must provide access to various VRFs Must NOT interconnect VRFs PBR-set-VRF helps with shared services VRF scenario Use ACLs and route-maps to define allowed paths
Logical Physical Prod-App
Backup Prod-DB
Prod-App Backup Prod-DB

47
Segmentation Across the Whole Network

Allows partitioning of data center services such as SLB, firewalling, etc.
Security policy management and deployment by user group/VRF Extends user groups from campus/WAN all the way to the DC
802.1Q trunk Back-to-back VRFs MPLS Core MPLS Core Catalyst 6500
Cisco Confidential
48
Segmentation using VDCs

Using VDCs, management of both services and switching infrastructure kept completely independent
Security-policy management and deployment by user group VRFs/VLANs can be deployed within context of each VDC
Campus/WAN
Campus/WAN Nexus 7000
VDCs and Contexts
Cisco Confidential
49


Conclusion
Cisco Confidential
50
Conclusion
Position Nexus 7000 in Data Center designs TODAY!
Nexus 7000 offers superior 10GE density & performance Nexus 7000 supports DC relevant features today
ISSU (zero packet loss) vPC VDCs Etc.
ESE has verified Nexus 7000 in the data center design
Properly position customer for Unified Fabric in future

51
Core Layer:
Modular Chassis High 10GE Density Low Oversubscription Layer 3 Links to Aggregation Hand-off to Campus
53
Aggregation Layer:
Modular Chassis High 10GE Density Layer 2 Trunks to Access Default Gateway for Servers Typically Attaches Services
54
Access Layer:
Top of Rack (ToR) Blade Switches as Access Layer End/Middle of Row (EoR/MoR) GE or 10GE to Servers Layer 2 Switching
55

Designing Data Centers With The Nexus 7000: Ben Basler Technical Marketing Engineer January 2009, v2.1

Enviado por

Dados do documento

Descrição original:

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Designing Data Centers With The Nexus 7000: Ben Basler Technical Marketing Engineer January 2009, v2.1

Enviado por

Direitos autorais:

Formatos disponíveis

Designing Data Centers with the Nexus 7000

Ben Basler Technical Marketing Engineer January 2009, v2.1

Does not address Unified I/O, FCoE

2009 Cisco Systems, Inc. All rights reserved.

Agenda Data Center Design

Advanced Design Options

2009 Cisco Systems, Inc. All rights reserved.

3-Tier Data Center Design

Layer Core Aggregation (Distribution) Access

Switch Type Modular Modular Fixed or Modular

Port Speed 10GE 10GE GE/10GE

Configuration Layer 3 only L2/L3 boundary Layer 2 only

Oversubscription Low to medium Medium to high Medium to high

Table values are considered typical for a green field deployment

3-Tier Data Center Design

2009 Cisco Systems, Inc. All rights reserved.

2-Tier Data Center Design

Aggregation Layer (no Core Layer):

How far does it scale? To be discussed later

Access Layer Options

Middle of Row (MoR aka EoR)

Mapping to Data Center Floor Plan

Pod section using MoR

Pod section using ToR

2009 Cisco Systems, Inc. All rights reserved.

Mapping to Data Center Floor Plan

Pod: Repeatable Network Environment with Predictable Scalability

2009 Cisco Systems, Inc. All rights reserved.

Mapping to Data Center Floor Plan

2009 Cisco Systems, Inc. All rights reserved.

In-Row and X-Row Cabling

10GBase-SR (300m using OM3 fiber)

Agenda Data Center Design

Oversubscription Advanced Design Options Conclusion

A Closer Look at Layer 2

2009 Cisco Systems, Inc. All rights reserved.

A Closer Look at Layer 2

2009 Cisco Systems, Inc. All rights reserved.

A Closer Look at Layer 3

2009 Cisco Systems, Inc. All rights reserved.

A Closer Look at Layer 3

2009 Cisco Systems, Inc. All rights reserved.

Traffic Flow Optimizations

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

High Availability Aspects

NSR for OSPF

2009 Cisco Systems, Inc. All rights reserved.

High Availability Aspects

2009 Cisco Systems, Inc. All rights reserved.

System Management Aspects

Relevant MIB support Have-nots:

2009 Cisco Systems, Inc. All rights reserved.

Agenda Data Center Design

Advanced Design Options

2009 Cisco Systems, Inc. All rights reserved.

Where are Oversubscription Points?

Line card & system architecture

Active vs. standby interfaces

Requirements might be different for North-South vs. East-West traffic

2009 Cisco Systems, Inc. All rights reserved.

Single attached server