Trans Oceanic 6up

Trans-Oceanic
Trans-Oceanic Internet
Backbones Basic
Backbones
Techniques
Scaling to Tomorrow
version 2.0
Presentation_ID
1378_07F8_c1 © 1999, Cisco Systems, Inc. 1 Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 2
Trans-Oceanic Backbones Trans-Oceanic Backbones

Basic Techniques Basic Techniques
• Six Techniques used by the Internet
Easiest
Easiestway
wayto toscale
scalebandwidth
bandwidthisisto
toupgrade
Community to scale bandwidth: the
thecircuit.
circuit.There
Thereare
arenot
notmany
manyOC-12s
OC-12s
upgrade
(633Mbps)
(633Mbps)circuits
circuitsacross
acrossthe
theAtlantic.
Atlantic.
3 Bigger Circuits
3 Inverse Multiplexing
3 Clear Channel E3 or DS3
Problem:
Problem:Many
Many Telcos
Telcosfail
failin
intheir
their
3 PPP over SDH engineering
engineeringand
transmission
andcapacity
capacityplanning
planningfor for
transmissioncapacity
capacity--leading
leadingtotolimits
limitson
on
the
the upgrade capability (i.e. lockedto
upgrade capability (i.e. locked tonxE1
nxE1
3 Asymmetrical Satellite Systems upgrades).
upgrades).
3 Hybrid Systems
Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 3 Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 4

“Pearl: It helps to work directly with • Inverse Multiplexing (iMux) takes several
circuits and bundles them into one or
more logical circuits.
the people who do the international
transmission capacity planning and • Two major techniques:
purchasing. They get to see your 3 Use protocol/forwarding features in the router
projections, you get on time 3 Use an external inverse multiplexer
upgrades.
” Router Multiple Circuits Router
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr 1
• Several Techniques:
• Router protocol/forwarding features as iMux 3 Static Route - Per Packet
3 Parallel Links Across the Ocean. n x E1 circuits 3 OSPF
between the routers using the routing protocols to
3 eBGP Multihop - Per Flow (w/ Netflow & CEF) or Per Packet
perform the load balancing and bundling of the (w/ CEF)
parallel circuits - works up to 4 to 8 E1s.
3 Multi-Link PPP (MLPPP) - Tighter Bundling Options (up to
Data In Data Out 8)
3 CEF Load Balancing - Per packet or per flow
B B2 A2 Telco A2 B
B2 3 BGP Maximum Paths (up to 6 - different routers)
Lease Line
A A1 B1 Network B1 A
A1
Sequencing and Sequencing and

Fragmentation Re-Assembly

3 Several iMUX bundles can be grouped together on the
same router to build 34M and 45M equivalent circuits
3 Many Telcos have not provisioned facilities to cater to
oceanic circuits above E1. 3 eBGP Multihop is the preferred load balancing technique.
3 Many E1s grouped together into a larger pipe via iMUX 3 Telstra Internet is now over 100M of iMUX backbone
technology. bandwidth!
3 Defacto Industry practice is to use Cisco Routers with HSSI
ports connected to Larscom’s Orion 4000 iMUXes .
8 E1 Circuits n x E1 Circuits
HSSI to Router HSSI to Router

n x E1 Circuits
Trans-Oceanic Backbones Trans-Oceanic Internet

Basic Techniques Backbones
• What do you do after DS3 (45M)?
• Clear Channel E3 (34M) or DS3 (45M)
3 Multiple DS3?
3 Preferred method for high speed backbone links is a clear
channel circuit. 3 ATM at OC-3 (155M) or OC-12 (622M)?
3 Configuration is simple - connect the CSU/DSU to a 3 PPP over SDH at OC-3 (155M) or OC-12 (622M)?
HSSI/T!/E1 port on a Cisco router.
• ATM vs PPP over SDH (POSIP)

E3 or DS3 Circuit
3 ATM is not the best choice when all you
are doing is Internet traffic across the
ocean.
Trans-Oceanic Internet Trans-Oceanic Backbones
Backbones Basic Techniques
• Line utilization Mbit/s
160 OC-3 Payload POSIP Overview
3Packet over SONET
provides 98% utilization 140 3 Packet-oriented serial interface, OC3/STM-1, OC-12, OC-48
120
320% - 30% overhead over PPP over 3 Supports either SONET or SDH interface
100 ATM “Cell Tax” SONET
an ocean is a lot of money! 3 PPP packets are encapsulated in SONET STS-3c
80
3 Provides superior line utilization and data efficiency
• Goodput verses 60
é POS available information bandwidth: 149.76 Mbps
40
throughput 20 é ATM available information bandwidth: 128.36 Mbps
é Cell tax ranges from 14-50% depends on the packet distribution
3Sufficient buffering
0
for large TCP flows 50 100 200 700 1000 3 Encapsulations
3Congestion avoidance Bytes per Packet é RFC 1619 point-to-point protocol over SONET/SDH
é RFC 1662 point-to-point protocol in HDLC-like framing
3ATM and TCP/IP Headers
are part of “throughput,” 3 IP protocol only
not “goodput”
Trans-Oceanic Internet Trans-Oceanic Backbones

Backbones Basic Techniques
• If an ISP’s trans-oceanic traffic pattern is always
• Pensacken NY (Sprint asymmetrical, then why pay for that idle bandwidth
NAP) to Stockholm going out?
Sweden (D-GIX).
3 Asymmetrical traffic pattern is when the ISPs is pulling
• First 155 M Internet down more information than sending out. Today a typical
link across an ocean! ISP is somewhere between a 80:20 - 60:40 traffic ratio to
the US.
• First operational 155M
POSIP connection! • Satellite Services allow ISPs to buy a circuit with
different speeds in the two different directions.
• Second is from Japan
3 For example - 2 Mbps in / 256 Kbps out
to US
• Hence the ISPs only pays for what they need - no
idle bandwidth giving the other side a free ride.

• Minimizing Latency on Asymmetrical
• Trans-Oceanic ISP links are a reality. They Satellite Links:
work and are used to backup terrestrial
trans-oceanic links. 3 Good Traffic Engineering (i.e. avoid
congestion)
• Latency issues (around 700 ms RTT) can
be minimized through creative 3 Hybrid Asymmetrical Links
engineering. 3 Hybrid Simplex Links
45Mbps 3 WWW Caching
2 Mbps
Router Router 3 Content Routing

Asymmetrical Circuit
• Hybrid Asymmetrical Satellite links • Telstra and Teleglobe were the first two ISPs
combine terrestrial and satellite together. who pioneered this technique.
3 Reduces the latency by 1/3 to 1/2. 3 Several other ISPs in Asia and Europe are using
this technique.
3 Static, BGP filtering, or eBGP multihop is used
to manage the links 3 Very few Tier 1 & 2 NSPs in the US will terminate
these trans-oceanic systems - hence the growth
of co- lo business (I.e. AboveNet)
45Mbps
2 Mbps 45 Mbps
Router Router Simplex
ILC ILC Link
4 Mbps
16 Mbps - Terrestrial Link
Telstra Teleglobe
Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 19 Presentation_ID Receiver
© 1999, Cisco Systems, Inc. www.cisco .com Sender 20

• Hybrid Simplex Satellite links combine terrestrial • WWW Caching adds an additional “buffer” to the higher
and satellite’s capability to for a circuit with only latency.
direction.. 3 Caches content as it comes over the link.
3 Takes advantage of uni-directional nature of satellite circuits 3 Minimizes the download of the same content over and over again.
3 Reduces the latency by 1/3 to 1/2. 3 WWW Caching is a main stream Internet technology - all issues
with it’s use have been resolved via technical means via IETF and
3 Static, BGP filtering, or eBGP multihop is used to manage the W3O.
links with new protocols coming.
45Mbps
45Mbps
Router
ILC ILC
4 Mbps
Router Router
ILC ILC
4 Mbps
Trans-Oceanic Backbones Backbones
Basic Techniques
• Since asymmetrical systems will limit any “free Asymmetric
AsymmetricSatellite
SatelliteService
Service--Scenario
Scenario1A
1A
rides” from people who should be paying for the ISP
ISP Router
Routerin in Earth
EarthStation
Station
Receive
Receiveand
andTransmit
TransmitLinks
Links(less
(lessthan
thanor
orequal
equalto
to88Mbps)
Mbps)
traffic from the ISP to the world.
3 The system is designed around the ISP’s traffic profile.
Single Single Up Converter, HPA Down Converter, HPA Single Single
Serial Port Lower Speed Serial Port Lower Speed Serial Port Lower Speed Serial Port
3 Choke point in the ISP’s space limits excessive pull from Transmit
Custom
IF
RF
Link
RF
IF Receive
Custom
the other side. Cable

Assembly
IF IF
Cable
Assembly
ISP Router Low Speed Low Speed Teleglobe Router

RF Higher Speed RF
Higher Speed Satellite Satellite Higher Speed
Receive Modem Link Modem Transmit
Down Converter, HPA (< 8 Mbps) Up Converter, HPA
Satellite Earth Station Operator Teleglobe Earth Station
45Mbps
Single low-speed satellite modem supports
up-link and down-link speeds up to 8 Mbps
Router
ILC ILC
4 Mbps
Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 23 Slides courtesy

Presentation_ID © 1999,of Teleglobe
Cisco Systems, Inc. www.cisco .com 24
Trans-Oceanic Internet Trans-Oceanic Internet
Backbones Backbones
Asymmetric
AsymmetricSatellite
SatelliteService
Service--Scenario
Scenario1B
1B Teleglobe
Teleglobe&&Intelsat
IntelsatAsymmetric
AsymmetricE1/512
E1/512Kbps
KbpsSatellite
Satellite
ISP
ISPRouter
RouterininEarth
EarthStation
Station Link
Link
Receive Link to ISP (greater than 8 Mbps)
Receive Link to ISP (greater than 8 Mbps) for Africa Telecom `98 Internet Cafe
for Africa Telecom `98 Internet Cafe
Transmit
TransmitLink
Linkfrom
fromISP
ISP(greater
(greaterthan
than88Mbps)
Mbps)
Lower Speed
Up Converter, HPA Down Converter, HPA

512 Kbps 512 Kbps
(Channelized (Channelized
Lower Speed IF IF
Up Converter, HPA Link Down Converter, HPA E1 Circuit) E1 Circuit)
Lower Speed Lower Speed G703 G703 RF RF G703 G703
IF (> 8 Mbps) IF Teleglobe
Transmit Receive ISP Router
Router G703 G703
G703 G703
RF RF IF IF
HSSI A d a p t e r ECL E C L Adapter HSSI Satellite Modem RF RF Satellite Modem E1
E1 EF Data SDM-8000
EF Data SDM-8000
Down Converter, HPA Up Converter, HPA Drop & Insert
Drop & Insert
IF IF G703Interface
G703Interface
ISP Router High Speed Higher Speed High Speed Teleglobe Router
RF RF
Higher Speed Satellite Satellite Higher Speed INTERNETCAFE TELKOM S.A. EARTH STATION T E L E G L O B E EARTH STATION T E L E G L O B E GATEWAY
Receive Modem
Link
Modem Transmit Higher Speed
Down Converter, HPA (> 8 Mbps) Up Converter, HPA
Satellite Earth Station Operator Teleglobe Earth Station
Single high-speed satellite modem supports

up-link and down-link speeds up to 45 Mbps
Slides courtesy
Presentation_ID © 1999,of Teleglobe
Cisco Systems, Inc. www.cisco .com 25 Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 26
Backbones
• Equipment Issues
3 Need a interface card in a router with
lots of buffering. VIP2-50 with max
New Trends
memory connected to a HSSI PA or POS
PA (DS-3) are known to work. What others are doing with their
3 Need Random Early Detection (RED). Trans-Oceanic Links
Needed to insure effective utilization of
the link.
New Trends New Trends
• ISP’s Trans-Oceanic Backbones are

migrating into systems designed to get
maximized efficiency from the link.
• Co-Location or Lease of Router in the
US. • High Cost of trans-oceanic bandwidth,
Exponential Growth, and new demand for
• Dual Sided Content Routing. Value Added Services (QoS, Content
Routing, and VoIP) are all driving factors.
• Heavy Localization of traffic (IXPs)
iBGP RR Client iBGP RR Client iBGP RR Client
and RR Server
• These Trans-Oceanic Systems will Satellite Satellite GW Peering GW1

consist of:
ISP's
3 Mix of Satellite and Terrestrial Circuits Satellite
GW Internet
3 WWW Caching and Performance

Customers
Enchanting Proxies ISP's
Terrestial GW Peering GW2
Terrestial
GW
3 QoS Services (Premium and Basic)
iBGP Route iBGP RR Client
Reflector
3 Application Redirection (Voice and and RR Server iBGP RR Client
Multicast)

Cache Fram WCCP Intercepts
Points Default Traffic
over Satellite
Cache Cache
Farm Satellite Satellite GW Peering GW1 Farm Satellite Satellite GW Peering GW1
ISP's ISP's
Satellite Satellite
GW Internet GW Internet
Customers Customers
ISP's ISP's
Terrestial Terrestial GW Peering GW2 Terrestial GW Peering GW2
Terrestial
GW GW
Traffic Between the Caches

WCCP Intercepts WCCP Intercepts
Traffic
are locked up with HTTP
Traffic
1.1 Persistent Connections

• Any QoS, CoS, or DiffServ tools need to be applied on the
upstream router’s interface.
• HTTP 1.1 Persistent connections between
3 Applying the tools on the downstream side would force the ISP to pay
the two caches move the average packet for the packets before they are dropped.
size from ~40 bytes to 512 - 1500 bytes. 3 Most US NSPs are reluctant to apply any special configurations on
the US side.
3 More Goodput vs Overhead 3 That means the ISP outside of the US needs to co-locate their own
router in the US (or lease one from a NSP in their facilities).
3 Satellite Modifications to TCP increase
ISP’s The Rest
efficiencies (RFC 2488 or equivalent) Customers of the
Internet
• Technique is also called Performance

Enhancing Proxies (PEP)
Tools Tools
3 draft-ietf-pilc-pep-00.txt
WFQ, dTS, or some other QBPR and CAR
technique to soft limit to Mark and Hard
Limit Flows
• Use BGP to propagate

Service Cache
precedence-setting policy Farm Satellite
Provider Satellite GW Peering GW1
as needs change Content
AS Source
• BGP community ISP's
label used to bind Satellite
precedence Premium
GW Internet
polices to Subscribers iBGP Peers

destination prefixes IP Precedence
for this Customers
• Return traffic classified Routing Entity ISP's

Pkt. Flow Terrestial Terrestial GW Peering GW2
with appropriate QoS GW
Data IP Header
QBPR and CAR to Mark WFQ, dTS, or some other QBPR and CAR
and Communicate QOS ID technique to soft limit to Mark and Hard
Precedence ToS to Other Routers Limit Flows
Type of
Service Field
New Trends
• What’s Next? Collecting and

3 VoIP and IP Telephony. Using routing
tricks or NPR (Netflow Policy Based Reporting Capacity
Routing) to keep voice traffic on the
terrestrial circuits.
Information
3 Content Replication. Pushing content
from the international side to the US
side of the link.
609
Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 39 Presentation_ID
1042_05F9_c2 © 1999, Cisco Systems, Inc. 40
Performance Management and

Internet Traffic Measurement Capacity Planning Definitions
• Aggressive collections and analysis
of network data is critical to any ISPs • Capacity planning
who wishes to tackle the problems of
CoS, QoS, and I3F 3 The process of determining the likely
• Ironically, most ISPs do not collect future network resource requirements
this information, even when most of to prevent a performance impact on
the tools are public domain on the business critical applications
Internet.
• Performance management
• The concern is that so many people
are talking about buzzword and not 3 The practice of managing network service
enough about the fundamentals of response time, consistency
what is actually happening on the and quality for individual services
Internet. and services overall
Increasing Importance of
Capacity Related Risks
Capacity Planning
• Frequent application deployment failure

• Increased reliance on network services • Network degradation and failure
for business applications
• Application timeouts and failure
• Exponential growth in business and
• Application performance degradation
nonbusiness related traffic
• Network Failure is typically capacity
related
Effective Capacity Resource Constraints

Management or Bottlenecks
Performance
Gather
Baselining • CPU
Configuration
and Traffic Observe Statistics • Memory
Information Collect Capacity Data
Analyze Traffic • Buffering, queuing and latency
Solve • Interface and pipe sizes
Problems
Plan Changes
Implement Evaluate
• Speed and distance
Changes
What-if • Application characteristics
Analysis
Collecting and Reporting

Information to Collect
Capacity Information
Gather • Link utilization • Frame Relay DE,
• Development of Configuration Performance
Baselining FECN, BECN,
information collection plan and Traffic
Information
Observe • CPU traffic-shaping
Statistics
• Tools for collecting Collect Capacity
Data
• Memory parameters
capacity information Analyze Traffic
Solve
• Performance (ping • NetFlow statistics
• Defining capacity areas Problems
Plan Changes
• Reporting and Implement Evaluate response time) • RMON

Changes What-if
interpreting results Analysis
• Queue/buffer drops
• Broadcast volume
Link Utilization CPU Utilization
Avg. Util Peak Util Polling Avg. Util Peak Util

Resource Address Segment (%) (%) Resource Address (%) (%)
JTKR01S2 10.2.6.1 128 Kbps 66.3 97.6 FSTR01 10.28.142.1 60.4 80
JYKR01S0 10.2.6.2 128 Kbps 66.3 97.8 NERT06 10.170.2.1 47 86
FMCR18S4/4 10.2.5.1 384 Kbps 51.3 109.7 NORR01 10.73.200.1 47 99
PACR01S3/1 10.2.5.2 384 Kbps 51.1 98.4 RTCR01 10.49.136.1 42 98
Performance
(Ping Response Time)
AvRes T (mS) AvRes T (mS) AvRes T (mS) AvRes T (mS)

Resource
AADR01
Address
10.190.56.1
09-09-98
469.1
09-09-98
852.4
09-24-98 10-01-98
461.1 873.2
Capacity and
ABNR01
APRR01
10.190.52.1
10.190.54.1
486.1
490.7
869.2
883.4
489.5
485.2
880.2
892.5
Performance
ASAR01
ASRR01
10.196.170.1
10.196.178.1
619.6
667.7
912.3
976.4
613.5
655.5
902.2
948.6
Tools
ASYR01S 503.4
AZWRT01 10.177.32.1 460.1 444.7
BEJR01 10.195.18.1 1023.7 1064.6 1184 1021.9
609
Three Essential Tools Traffic Management Elements
• Data collection
Data Collection mechanisms on
• Simple Network Management network equipment
Protocol (SNMP) • Data export
Data Export
mechanisms to
• R emote MON
MONitoring Protocol (RMON) applications
Data Analysis
and
• NetFlow - Flow Based TCP/IP Visualization
• Data analysis
Analysis and visualization
SNMP is everywhere in the Internet Network Monitoring with RMON
Service Provider Service Provider
SwitchProbe
• Host and matrix for link,
network and application
layers
Firewall • Address translation
Tap
• ISL VLAN monitoring
Box
Trunk SwitchProbe
Backbone
WAN and/or
Router/Switch LAN
Switch
Server
SNMP Work Station SPAN

Building Per Port RMON: Port
Router Statistics History
Alarms Events • Filter/capture
• Protocol
distribution
Wiring Closet • User history
Traffic Analysis on: Link, Network

Top Hosts
and Application Layers
• Aggregate and • Top hosts by any of

historical statistical the following metrics:
analysis for 3 Packets sent or
switched segments received
3 Bandwidth utilization 3 Octets sent or received
3 Erroranalysis
3 Broadcasts sent
3 Broadcast levels
3 Multicastssent
3 Baseline analysis
3 Errors generated
Measuring International Links

Measuring International Links
Most
MostISPs
ISPsoutside
outsidethe
theUS
USonly
onlymeasure
measureone
side
one • ISPs Outside of the US
sideof
ofthe
thelink.
link.To
Toget
getaaaccurate
accuratepicture
pictureof
of
what
whatisishappening
happeningon onthe
thelink
linkthat
thatthey
theyare
are should require the
paying
paying$$$$
$$$$for,
for,SNMP
SNMPneeds
needsto tobe
beopen
openonon
the
theupstream
upstreamsite
siteof
ofthe
thelink.
link.
upstream provider to:
3 Create a special SNMP
Community with ACL to
poll the router on the
US side.
• OR
SNMP
SNMPon onthe
theupstream
upstreamside
sideof
ofthe
thelink
linkwill
will
tell
tellthe
theISP
ISPif:
if: 3 Create a special Web
•• Packets
Packetsare aredropped
droppedononthe
theUSUSside
side
•• In/out bytes per second
In/out bytes per second
page that displays the
•• Other
Otherinformation
information SNMP data. MRTG or
Looking Glass with
access security would
work. www.cisco .com
© 1999, Cisco Systems, Inc. 60
Measuring International Links Measuring International Links
Measuring International Links Measuring International Links
Fundamental Tools are Cheap! Fundamental Tools are Cheap!
• The tools to create a simple network

manage system that will give an ISP • Example of what can be done with
the basics comprise of the following: another Shareware tool - MRTG ….
3 PC with LINUX (free UNIX)
3 CMU SNMP (free SNMP)
3 PERL5 (free UNIX script language)
3 GNU Plot (free graphic plot tool)
3 Printer
Fundamental Tools allow for the
Limitations of SNMP and
Baseline!
RMON
• Baseline Quality Levels are critical for any ISP Server.
Average Utilization and Packet Loss need to be monitored on the
3
entire network.
• SNMP and RMON will tell you what is
3 QoS Threshold need to be set and acted on to maintain any sort happening on the network (I.e. load,
of foundation to build advanced IP services. This is ISP 101 which
most new ISPs forget!
PPS, packet drops).
All you need is SNMP! It’s not rocket science.
3
• SNMP and RMON will not tell you
who is doing what to where and
Threshold
when.
• For that sort of details, TCP/IP Flow
Upgrade! Based Analysis is needed.
Flow Based Analysis Flow Based Analysis

Connectionless IP But…
The Network Is Full of Flows (Conversations) • Key IETF work:
Conventional Packet Flow Based Analysis
POV
3 Real Time Traffic Flow (RTFM) working
POV
group
é http://www.auckland.ac.nz/net/Internet
/rtfm/
• Flows are Unidirectional 3 IP Provider Metrics (IPPM)
• Flows are Granular
- IP address and app. port# pairs • Public Domain and Commercial Tools
- (TOS/Protocol/Interface) now available.
Flow Based Analysis Flow Activation Locations

• Key Tools Used Today
é NetScarf - Global and Regional Internet Analysis
project (http://www.merit.edu/~netscarf) Ingress Egress
Aggregation Aggregation
é Traffic Flow Analysis - NLANR (http://www.nlanr.net)
é NetFlow - Analysis and IP switching technology build
into Cisco’s IOS.
é NeTraMet - Free Flow analysis software.
(http://www.auckland.ac.nz/net/Accounting/ntm.Release WAN Access Router
.note.html)
é Many new Netflow based commercial tools ….
Transit
NetFlow Data Record (V5) Netflow Empowerment
Usage • Packet Count • Source IP Address • Before • With Netflow

• Byte Count • Destination IP Address 3 Flat Rate Charging 3 Detailed Volume
Charging
Time • Start Timestamp • Source TCP/UDP Port 3 SNMP Volume Charging
of Day 3 QoS Charging
• End Timestamp • Destination TCP/UDP Port 3 Time Based Dial-up
Charging 3 Application Based
Port • Input Interface Port • Next Hop Address Application Charging
Utilization • Output Interface Port • Source AS Number 3 No data on where you
Routing
customer go on the Net 3 Distance Based
• Dest
Dest.. AS Number and
Peering Charging
QoS • Type of Service
• TCP Flags 3 Time of Day Charging
• Protocol 3 Details on where and
what you customers are
doing on the Net
Cache Management and Data

NetFlow Switching Statistics
NetFlow Statistics
IP NetFlow Switching Cache, 29999 active, 2769 inactive, 58411388 added
Export
statistics cleared 141949 seconds ago
Protocol Total Flows Packets Bytes Packets Active Sec Idle Sec
Flows / Sec. / Flow / Pkt / Sec. / Flow / Flow
TCP - Telnet 267,034 1.8 233 75 439.3 182.6 36.5

FTP 1,030,837 7.2 10 78 76.6 22.6 43.7
FTPD 554,967 3.9 164 345 641.3 52.7 15.7
WWW 32,107,858 226.2 15 247 3610.6 13.5 28.1
SMTP
X
BGP
3,526,231
9,600
111,096
24.8
0.0
0.7
13
121
14
159
129
77
323.1
8.2
11.5
10.2
148.2
229.2
23.6
55.1
61.1
Header
other 5,729,172 40.3 70 220 2858.1 71.0 41.3 • Sequence number Flow Flow
UDP - TFTP
DNS
other
2,398
12,875,077
1,489,072
0.0
90.7
10.4
3
2
30
62
110
293
0.0
195.4
321.8
13.4
5.4
28.5
69.5
43.6
68.7
• Record count Record … Record
ICMP
IGMP
665,771
5,144
4.6
0.0
13
18
259
278
62.8
0.6
75.7
82.4
66.8
64.3
• Version number
IPINIP 4,450 0.0 933 377 29.2 166.7 61.0
IP - other 2,693 0.0 11 136 0.2 80.8 65.7
TOTAL 58,381,400 411.3 20 227 8579.4 0.0 0.0

NetFlow Cache
Src
Intf
Source
IP Address
Dest
Intf
Dest
IP Address
Prt Src
Port
Dest
Port
Pkts Bytes/
/ Pkt
Act Sec
/ Flow
Idle Sec
/ Flow • Cache manager expires flows
Hs3/0 204.119.134.49 Fd0/0 142.35.4.36 6 0050 0610 1 44 0.0 0.6
Fd0/0
Hs3/0
206.42.156.2
125.160.1.24
Hs3/0
Fd0/0
206.52.126.29
200.246.225.8
6
6
0439
BB81
0050
0DB7
12
745
105
542
9.1
323.0
1.0
0.0 No traffic/long life/TCP flags/cache full/etc.
... ... ... ... ... ... ... ... ... ... ...
• Intelligent cache aging

• Router exports groups of expired flows
• Extensive statistics maintained on L3 • Export uses UDP datagrams with sequence numbers
device
Presentation_ID • CLI summary traffic
© 1999, Cisco Systems, Inc. characterization
www.cisco .com 75 Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 76
NetFlow Metering
Netflow Statistics and RMON
Infrastructure
RMON/RMON2 Netflow Statistics
Network Planning
TopN Users
Flow Start Time
TopN Conversations Packets
Flow End Time
Packet Capture Bytes
Accounting/Billing Protocol Distribution Input/Output Interfaces
S/D IP
Packet Size Distribution Next Router Hop
S/D Port
Alarm Thresholds and Events AS #
Port/Segment Stats
Flow Profiling
History
Network Monitoring
Flow Switching Flow
and Data Export Flow Collection Consolidation Flow Consumers
Presentation_ID © 1999, Cisco Systems, Inc. www.cisco
Cisco .com
Systems Confidential 77 Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 78
NetFlow Provides Open NetFlow Provides Open
Interfaces Interfaces
Cflowd NetFlowMet
by ANS & BBN by Nevil Brownlee
and maintained by
CAIDA
éUses the work from
the IETF’s Realtime
Traffic Flow
Measurement
(RTFM) WG
Http://www.caida.org
ftp://ftp-eng.cisco.com/ftp/NetFlow/fde/README http://www.
http://www.auckland
auckland.ac.
.ac.nz
nz/net/Accounting/
/net/Accounting/
auckland.ac.nz/net/Accounting/
ftp://ftp-eng.cisco.com:/ftp/NetFlow/fde/netflowv5.tools.tar.Z http://www.
http://www.auckland
auckland.ac.
.ac.nz
nz/net/Internet/
/net/Internet/rtfm
rtfm//
auckland.ac.nz/net/Internet/rtfm/
NetFlow Distance-Based NetFlow Distance-Based

Accounting Accounting
• General Information page for Cisco Netflow
Global services:
AS 3 http://www.cisco.com/warp/public/732/netflow/
Customers • Cisco's NetFlow FlowCollector v2.0 and NetFlow
Regional FlowAnalyzer v2.0:
AS
3 http://www.cisco.com/warp/public/732/netflow/netan_o
v.htm
Domestic • Case Study - How to implement Netflow in a

AS network. Traffic Accounting Using Netflow and
Internal Cflowd by Roberto Sabatino (DANTE/TEN-34)
3 http://www.dante.net/pubs/dip/32/32.html
NetFlow Distance-Based
Netflow as a tool
Accounting
Netflow statistics empowers all
• 3rd Party Solutions: ISPs with the ability to know the
3 Belle Systems http://www.belle.dk
who, what, where, and how
3 Solect http://www.solect.com
3 XACCT Technologies http://www.xacct.com
much.
3 Apogee Networks, Inc. http://www.Apogeenet.com $ $
$ $
3 RODOPI http://www.rodopi.com 1/2 circuit 1/2 circuit
ISP NSP
Asia Pacific Netflow Netflow US
Conclusions
• Aggressive measurement and

analysis is critical to an ISPs and the Capacity and
Internet’s survival. Performance
• Not enough measurement and actual Best Practices
data analysis is taking place on the
Internet. Too many people are
speculating with weak data to back
up their claims.
609
What-If Analysis Service Level Management
• Simulation applications
Gather
• Define performance requirements
Performance
• Lab application modeling Configuration Baselining
3Protocol analyzer,
and Traffic
Information Observe • Define Upgrade criteria by
Statistics
WAN emulator, packet Collect Capacity
Data
capacity area
generator, NETSYS Analyze Traffic
performance analyzer
Solve • Measure capacity and performance
• Lab network modeling Problems
Plan Changes
3NVS/NVT, lab network Implement
Changes
Evaluate • Review thresholds and baseline
modeling
What-if
Analysis • Take action!
Service Level Management Peak and Average Utilization

• Solution to narrow
Threshold WAN LAN collection interval
CPU 75-90% 75-90% • Low collection interval
Link = high overhead (Threshold)
80-90% 40-90%
Memory 50% 50% • Recommend >=5 Capacity
minutes Parameter
Output Queue 200 25 Green Bar Represents
• Peak values not quite SNMP Value Returned
Buffer Misses Any Any
what they seem
Broadcast Vol 10/Sec 300/Sec
Time in
• Close to threshold Collection Intervals
FECN/BECN 10/Sec N/A
indicates likely
exceed condition
Capacity Exception Performance and
Management Capacity Baselining
• Interface utilization Gather
Performance
Configuration
• Device CPU, Baselining
• Alarm critical capacity thresholds memory, buffer,

and Traffic
Information Observe
Statistics
(CPU, critical link) I/O utilization Collect Capacity
Data
Analyze Traffic
• Develop notification, escalation and • Network overhead Solve

Problems
action plan for threshold violations • Raw performance Plan Changes
Evaluate
Implement
characteristics Changes
• Take action! What-if
• Monthly or quarterly Analysis
baseline report
Upgrade Planning QoS Management
• Prioritize applications by
business impact
• Understand lead times for circuits,
equipment, planning and design • Understand networked application
behavior (packet size, timeouts,
• upgrade criteria based on service
flows, bandwidth requirements)
level management
• Develop QoS management plan
Problems with Congestion
Controlled Congestion
Throughput
Managing TCP Traffic Uncontrolled Congestion
Congestion
Moving Mountains of Data • Uncontrolled congestion will seriously degrade system

performance
Without Incurring the World 3 The system buffers fill up
Wide Wait 3
3
Packets are dropped, resulting in retransmissions
This causes more packet loss and increased latency
319
3 The problem builds on itself until the system collapses
Presentation_ID
Backbone Traffic Mix TCP Technology Issues
Transport Breakout TCP Applications
• Single drops communicate from

network to sending host
3 “You need to slow down”
• Multiple drops in round trip trigger
time-outs
3 “Something bad happened out here”
Source: MCI/NSF OC-3MON via http://www.nlanr.net, 1998
Behavior of a TCP Sender Behavior of a TCP Receiver

• Sends as much as N N
credit allows N+
N+1
2
• When in receipt of “next N+
N+1
2
N+
3 message,” schedules N+
3
• Starts credit small kN
+1
an ACK Ac
N+
1
3 Avoid overloading Ac
k
+1
network queues • When in receipt of Ac
kN
something else,
• Increases credit acknowledges all it
exponentially can immediately
3 To gauge network capability
Sender Response to ACK Multiple Drops in TCP

N
N+
1
N+
• If ACK acknowledges N
• In the event of multiple drops N+
2
N+ 3
something N+1
2 within the same session: N+
4 ck N
A
+1
N+
+1
3 Update credit and send 3
+1 Ac
k N
Ac
kN 3 Current TCPs wait for time-out +1
kN
• If not, presume it indicates k N+1 N+
Ac
Ac 1
a lost packet +1 3 Selective acknowledge may
kN
Ac
N+ work around (but see
3 Send first unacknowledged 1
+4
INFOCOM ’98) kN
Ac
message right away World
N+
+4 4
3 Halve current credit Ac
kN 3 New Reno “fast retransmit Wide
phase” takes several RTTs Wait!
3 Increase linearly to gauge
+5
to recover kN
network throughput Ac
How Can We Make TCP in a Fundamental FIFO Queue
Network Act Predictably? Management Technologies
• Predictable amount of traffic • Tail drop
in the network:
3 Network standard behavior
3 Well-written TCP implementations 3 Causes session synchronization when
manage their rates to the available waves of traffic experience
bandwidth correlated drops
• Router needs to • Random Early Detection (RED)
3 Provide predictable treatment of packets 3 Random drops used to desynchronize
3 Queue delay and drop characteristics TCP sessions and control rates
Tail Drop Large Packets “Freeze Out” Voice
Voice Packet Voice Packet Voice Packet

Queue 60 bytes 60 bytes 60 bytes
Packets Every 20ms Every >214ms Every >214ms
Arriving
~214ms Serialization Delay
Queue Voice 1500 bytes of Data Voice Voice 1500 bytes of Data Voice Voice 1500 bytes of Data Voice
Pointer at the tail
10mbps Ethernet 10mbps Ethernet
56kb WAN
• Without RED, when the queue

fills up all packets that arrive are • Large packets can cause playback buffer
underrun, tail drops
dropped—Tail Tail drop
• Jitter or playback buffer can accommodate
some delay/delay variation
Session Synchronization Session Synchronization
• Session 100%
Queue
Utilization
synchronization
results from
synchronized
losses Tail Drop
Time
3 Traffic Flows Start Another Traffic Flow

• Tail drop from at Different Times Starts at This Point
• TCP is a sliding window protocol that uses self-clocking to adjust its

waves of traffic use of the network to match available bandwidth using slow-start and
synchronizes congestion avoidance algorithm
• Session synchronisation is when many TCP connections go through
losses TCP Slow-Start mode at the same time
Random Early Detection (RED) Effect of
Packets
Queue
Random Early Detection
Arriving
Queue
Pointer
Courtesy of Sean Doran, Ebone RED Enabled

• With RED, as oppose to doing a tail
drop, • One day, below 100% throughput
the router monitors the average queue 3 Simple FIFO with tail drop
size and using randomization it
chooses connections to notify that a • Starting 10:00 second day, 100%
congestion throughput
is impending 3 Random Early Detection enabled
Presentation_ID 3 Note: Avg. queue size
© 1999, Cisco Systems, Inc.
is.com
www.cisco not an 109 Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 110
Was that a Fluke? FIFO Traffic Timings

400
350
300
Courtesy of Sean Doran, Ebone RED Enabled Mean

250
Latency
Correlates
Ns RTT
200
with
150
Maximum
• No, here’s what happened that week… Queue
100 Depth
• Session synchronization reduced 50
throughput until RED enabled 0

Elapsed Time
Mean RTT Min RTT Max RTT STD DEV
Where to apply RED?

RED Traffic Timings
400
RED Turned
ON!
350
Additional
300 Capacity
to Absorb
250
Bursts
Upstream
Ms RTT
200
Mean Downstream
150
Latency
Correlates
100 with RED
REDneeds
needsto tobebeconfigured
configuredononthe
the
Minimum upstream
upstreamrouter’s
router’sinterface.
interface.This
Thisisis
50 Drop router that will drop packets when
router that will drop packets when
Threshold congestion
congestion
0 is
isreached.
reached.
Elapsed Time
Mean RTT Min RTT Max RTT STD DEV
Therefore—TCP QoS
Applying RED/WRED Definition:
• Enabling WRED
3 [no] random-detect <weight-constant>
3 weight-constant = <1-16> is an integer used in weighted
average to mean 2^weigh-constant. 10 is the default. • Normally at most one drop per
round trip
• Tuning weight constant affects loss rate
3 rule-of-thumb: • Mean variation in latency bounded
3 DS-3/OC-3 Links: Value of 10 might achieve ~10^-4 drop rate,
recommended for DS-3/OC-3 link. by predictable network
3 T1/E1 Links: Value of 7 might achieve a loss rate around 10^-
3.
3 Actual recommended value should be determined in real
operational network.
An Interesting Common
TCP Flow Statistics Fallacy about RED:
• >90% of sessions have ten packets • “RED means you will have more drops”
each way or less
3 Statement derives from observed statistics
3 Transaction mode (mail, small web
• RED means that you will have
page)
3 Closer to 100% utilization of your line
• >80% of all TCP traffic results from
3 Less average delay per packet
<10% of the sessions, in high
rate bursts • But queuing theory?
3 As a line approaches 100% utilization, drops will
3 It is these that we worry about increase, even though served load increases
managing
TCP Traffic Management TCP Bandwidth Policy

Issues Questions to Answer
• Applications
3 Often have site-specific policy • Particular site or application wants at
associated with them least a certain bandwidth
3 Traffic
often identifiable by port • Particular site or application wants at
numbers most a certain bandwidth
• Sites • Particular site or application wants to
3 Generally identifiable by address prefix average about a certain bandwidth
or interface traffic is received on
Preparation
What to look for in an

• List all potential providers
Upstream Provider
• Get Maps of oceanic cable systems
• Get Maps of satellite foot prints.
609
Presentation_ID
What to ask from the

Example - Network Map
prospective providers?
• Network Maps with
landing/termination points of your
links.
• List of IXPs and Private Peers
• URLs of NOC Pages
• Do they lease routers and/or co-
locations space?
• Do they have upstream caches?
What to require from the

Upstream Provider
• Statistics Page and Weekly Reports
• 24x7 NOC Contacts
• RED or WRED on their router’s
interface
• CAR ICMP Rates Limits for DoS
Protection
• Back-up contingencies in writing.

Trans Oceanic 6up

Enviado por

Dados do documento

Descrição original:

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Trans Oceanic 6up

Enviado por

Direitos autorais:

Formatos disponíveis

Trans-Oceanic

Trans-Oceanic Backbones Trans-Oceanic Backbones

Trans-Oceanic Backbones Trans-Oceanic Backbones

Sequencing and Sequencing and

Trans-Oceanic Backbones Trans-Oceanic Backbones

HSSI to Router HSSI to Router

HSSI to Router HSSI to Router

Trans-Oceanic Backbones Trans-Oceanic Internet

• ATM vs PPP over SDH (POSIP)

Trans-Oceanic Internet Trans-Oceanic Backbones

Trans-Oceanic Backbones Trans-Oceanic Backbones

Router Router 3 Content Routing

Trans-Oceanic Backbones Trans-Oceanic Backbones

the other side. Cable

ISP Router Low Speed Low Speed Teleglobe Router

Satellite Earth Station Operator Teleglobe Earth Station

Presentation_ID © 1999, Cisco Systems, Inc. www.cisco .com 23 Slides courtesy

Up Converter, HPA Down Converter, HPA

Satellite Earth Station Operator Teleglobe Earth Station

Single high-speed satellite modem supports

New Trends New Trends

• ISP’s Trans-Oceanic Backbones are

• These Trans-Oceanic Systems will Satellite Satellite GW Peering GW1

3 WWW Caching and Performance

New Trends New Trends

Traffic Between the Caches

New Trends New Trends

• Technique is also called Performance

• Use BGP to propagate

polices to Subscribers iBGP Peers

• Return traffic classified Routing Entity ISP's

• What’s Next? Collecting and

Performance Management and

• Frequent application deployment failure

Effective Capacity Resource Constraints

Collecting and Reporting

• Reporting and Implement Evaluate response time) • RMON

Avg. Util Peak Util Polling Avg. Util Peak Util

JYKR01S0 10.2.6.2 128 Kbps 66.3 97.8 NERT06 10.170.2.1 47 86

FMCR18S4/4 10.2.5.1 384 Kbps 51.3 109.7 NORR01 10.73.200.1 47 99

PACR01S3/1 10.2.5.2 384 Kbps 51.1 98.4 RTCR01 10.49.136.1 42 98

AvRes T (mS) AvRes T (mS) AvRes T (mS) AvRes T (mS)

Three Essential Tools Traffic Management Elements

SNMP Work Station SPAN

Traffic Analysis on: Link, Network

• Aggregate and • Top hosts by any of

Measuring International Links

Measuring International Links Measuring International Links

Fundamental Tools are Cheap! Fundamental Tools are Cheap!

• The tools to create a simple network

Flow Based Analysis Flow Based Analysis

Flow Based Analysis Flow Activation Locations

Usage • Packet Count • Source IP Address • Before • With Netflow

Cache Management and Data

TCP - Telnet 267,034 1.8 233 75 439.3 182.6 36.5

TOTAL 58,381,400 411.3 20 227 8579.4 0.0 0.0

• Intelligent cache aging

NetFlow Distance-Based NetFlow Distance-Based

Domestic • Case Study - How to implement Netflow in a

• Aggressive measurement and

What-If Analysis Service Level Management

Service Level Management Peak and Average Utilization

• Alarm critical capacity thresholds memory, buffer,