International Conference on Computing, Networking and Communications

(ICNC 2017)

Technology and Architectural Approaches

to Address Continued Explosive Growth
in Network Traffic
Jane M. Simmons
Monarch Network Architects
www.monarchna.com
January 28, 2017
Santa Clara, CA

MNA Copyright 2017 Monarch Network Architects LLC

Overview
Explosive Growth in Network Traffic
Huge Growth in System Capacity
Impending Fiber Capacity Limits
Technology Approaches to Address Fiber Capacity Limits
Architectural Approaches to Address Fiber Capacity Limits

Focus is on Transmission in the Long-Haul Network

Expanded and updated version of: Saleh and Simmons, "Technology and Architecture to Enable the
Explosive Growth of the Internet," Communications Magazine, Vol. 49, No. 1, Jan. 2011, pp. 126-132
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 2
MNA
Explosive Growth in Network Demand
If traffic growth is exponential
Traffic Volume (PB/mo) If traffic growth rate is hyperbolic

Traffic Volume (PB/mo)

G(y) = 10a (y yo)b

a = 3, b = -1.14, yo = 1994
Actual Global Internet
Traffic Growth

Traffic Growth is Best Modeled by a

Hyperbolic Compound Annual Growth Rate (CAGR)
Explosive Growth, not Exponential Growth S. Korotky, Bell Labs Technical
Journal, Vol. 18, No. 3, Dec. 2013
Current CAGR between 20 and 25% 2013 Lucent Technologies Inc.

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 3

MNA
 Accompanied by Huge Growth in System Capacity
1995: Systems from this timeframe supported 8 wavelengths of 2.5 Gb/s on one fiber
20 Gb/s Total Bandwidth
Today: 80 wavelengths of 100 Gb/s on one fiber
8 Tb/s Total Bandwidth
Factor of 400 increase in system capacity in ~20 years
Enabled by advances in:
Modulation formats
Transmitter / Receiver technology (advanced signal processing & high-speed electronics)
Filtering technology
Forward Error Correction (FEC)
Accompanied by more than 3 orders of magnitude decrease in cost per Gb/seckm
~$1,000/(Gb/seckm) less than $0.50/(Gb/seckm) (Backbone Network)

Can these trends regarding system capacity and cost continue?

The capacity of a fiber has always been viewed as being almost infinite
But

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 4

MNA
Fiber Capacity Isnt Infinite After All !
System Spectral Efficiency metric
The ratio of the information bit-rate to the total bandwidth consumed
Current state-of-the-art systems: 100 Gb/s with 50 GHz wavelength spacing
and dual-polarization
Overall spectral efficiency is 2 bits/s/Hz

Theoretical analysis of the maximum

supportable capacity on conventional fiber
Assumptions:
Raman amplification
Single polarization
Optical reach of 2,000 km
(system capacity increases as the optical reach decreases)
Optical reach is the distance an optical signal can travel before it
Essiambre et al., Capacity limits of optical fiber networks, J. of
degrades to a level that necessitates it be regenerated Light. Tech., February 15, 2010, pp. 662-701. 2010 IEEE

Conclusion: Conventional-fiber Spectral Efficiency limit is 4 to 6 bits/s/Hz per polarization

Systems likely to be dual polarization Max Spectral Efficiency of 8 to 12 bits/s/Hz
Todays state-of-the-art systems are within a factor of ~5 of the limit !
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 5
MNA
How Should We Address This Impending Limit ?
Assuming a 20% compound annual growth rate,
a 5 growth in traffic will occur in about 9 years
Something new is needed !
The end-game is not simply providing more capacity
The solution(s) must be efficient with respect to cost, power consumption & space
Technology approaches
Expand into other regions of the spectrum
Space Division Multiplexing (SDM)
Multiple fiber-pairs per link
Multiple cores per fiber For each approach, we discuss
Multiple modes per fiber the possible benefits and the
Architectural approaches implementation challenges
Improved packing of IP traffic
Asymmetric traffic
Caching
Multicasting
Some of the architectural
Dynamic networking approaches are dependent upon
new technology
Gridless/Elastic optical networks
Optical reach vs. capacity
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 6
MNA
 Technology Approaches for Addressing
Fiber Capacity Limits

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 7

MNA
Expand into Other Regions of the Spectrum
0.30

Fiber Loss (dB/km)

0.25

0.20
S-Band C-Band L-Band C-band = central or conventional band
S-band = shorter-wavelength band

L-band = longer-wavelength band
1460 1530 1565 1625

Wavelength (nm)

Currently, most systems use the C-band (and perhaps part of the L-band or S-band)
Expand further into S and/or L bands to increase the system capacity
Perhaps will increase system capacity by a factor of 2 to 3
Desired components:
An optical amplifier that can operate over the whole expanded system
Requiring multiple amplifiers will not improve on the system economics
Transponders that are tunable over the whole spectral region
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 8
MNA
Space Division Multiplexing (SDM) Option 1: Multiple Fiber Pairs Per Link

Light up N fiber-pairs per link instead of one fiber-pair

Increases the capacity of a link by a factor of N
Does not require new fiber types (in contrast to the SDM options that are discussed next)
Two architectural options:
Deploy parallel systems, where N ROADMs are deployed at a node
Maintain a single system, where all fiber pairs feed into one ROADM at a node
This may lessen wavelength contention
Not all links have to be upgraded to multi-fiber at once
But the required ROADM degree may become very large
Disadvantages
Does not improve on the system economics (i.e., cost per bit/sec) or the system power
consumption (i.e., energy per bit)
E.g., The number of optical amplifiers increases by N
(However, there is ongoing research into developing amplifiers that can amplify signals in more than one fiber)

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 9

MNA
Space Division Multiplexing (SDM) Option 2: Multiple Cores Per Fiber
Multicore Fiber (MCF)
Current conventional fibers have a single core (SCF)
MCF: Increase the number of cores to N
Benefits
The system capacity theoretically increases by N
Significant improvement in system economics and power consumption
Amplification of all cores with a single amplifier has been demonstrated
The cores are tightly packed and can be processed together
A single connector can interconnect multicore fibers rather than requiring one connector per core
Concerns and Challenges
Need to install new fiber
Cross-talk between the cores; Tradeoff between low cross-talk and high core density
Various techniques to reduce cross-talk (see next slide)
Non-linear impairments worsen with more cores (reduces the optical reach)
Due to the accompanying decrease in fiber effective area
Two types of MCF systems
Weakly-coupled: Signals in the cores remain uncoupled can route the signals in the
cores independently. Perhaps 12 cores (or more ?) in a long-haul network.
Strongly-coupled: Signals in the cores become coupled as the number of cores
increases e.g., the lis in each one of the cores must be routed together
Pay attention to the achievable distance Some experimental systems with many cores
have an optical reach of 10s of kms not suitable for long-haul transmission
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 10
MNA
Multi-Core-Fiber Cross-Section Examples
2009: K. Imamura et al., Multi-core holey fibers for
Cross-section of a the long-distance (>100 km) ultra large
fiber with 7 cores capacity transmission, OFC/NFOEC 2009.
2009 OSA
~100 km

2013:
Cross-section of a fiber with Saitoh et al., JLT, Jan. 1, 2016. 2016 IEEE
12 cores with Trenches

2013:
Cross-section of a fiber with
12 cores, T. Kobayashi et al., ECOC 2013. 2013 ECOC
Bidirectional
~1500 km

2015:
Cross-section of a fiber with
Heterogeneous cores; 30 cores Saitoh et al., JLT, Jan . 1, 2016. 2016 IEEE
Remains Weakly Coupled
at 100 km

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 11

MNA
Space Division Multiplexing (SDM) Option 3: Multiple Modes Per Fiber

Mode Division Multiplexing

Current conventional long-distance fibers have a single mode (SMF)
Consider using fiber with multiple modes: Few-Mode Fiber (FMF)
Small number of modes (e.g., 3 or 6), not hundreds of modes
Benefits
The system capacity theoretically increases in proportion to the number of modes
Improvement in system economics
Amplification of all modes with a single amplifier has been demonstrated
Amplified 5 modes with one EDFA (Genevaux et al., JLT, Jan. 2016)

Concerns and Challenges

Need to install new fiber
The modes become coupled together
Requires electronic multiple-input multiple-output (MIMO) processing at the receiver
May consume a significant amount of power
Minimizing inter-mode cross-talk allows less complex MIMO DSP to be used
All modes of a given wavelength need to be routed together; a ROADM cannot drop a
subset of the modes

As with MCF, need to pay attention to the achievable optical reach

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 12

MNA
 Are Coupled Spatial Carriers Good or Bad ?
The modes (and possibly the cores) of a fiber will be coupled together
Example: if there are 6 modes, then all 6 modes of l1 will be coupled together
A ROADM will not be able to pick out just one of the modes
All modes will likely be routed as an inseparable unit, whether desired or not

Glass Half Full View: Coupling results in Spatial Super-Channels

Allows for efficient transmission of very high bandwidth connections
Example: A 600 Gb/s connection can be carried as one 6100 Gb/s super-channel
More cost-effective ROADMs and transponders
One MEMS mirror can steer all of the wavelengths in a superchannel
Component sharing across a superchannel in the transponders

Glass Half Empty View: Coupling is similar to Wavebands in standard SMF

Waveband is a group of wavelengths routed as a bundle
Wavebands were first proposed in the late 1990s to reduce switching costs
Wavebands require more complex algorithms to use the bandwidth efficiently
Service providers did not like wavebands 20 years ago (due to loss of flexibility)
Will they like coupled modes / cores any better? May not have a choice !
With wavebands, traffic analyses were performed to determine optimal band size
With coupled modes, the size is determined by the limits of the technology

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 13

MNA
Space Division Multiplexing (SDM) Option 4:
Multiple Cores and Multiple Modes Per Fiber

Multiple Modes can be Combined with Multiple Cores (FM-MCF)

Multiplicative capacity increase
Inter-core cross-talk increases with higher-order modes
To maintain weak coupling among cores, larger inter-core distance is required
Fewer cores

Example: 12 cores, each with 3 modes (527-km reach) (Shibahara et al., OFC 2015)

The potential capacity benefits of multi-core and/or few-mode fiber are very large

But a lot of implementation challenges still need to be addressed

Unlikely to be deployed in the near-term

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 14

MNA
 Architectural Approaches for Addressing
Fiber Capacity Limit

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 15

MNA
Architectural Approaches for Addressing Capacity Limits
Architectural strategies do not increase the system capacity to handle more traffic
In contrast to the technology options
Rather, they reduce the effective traffic load so that more traffic can be carried

A variety of architectural approaches can be utilized to reduce capacity requirements

Improved Packing of IP Traffic
Asymmetric Traffic
Caching
Multicasting
Dynamic Networking
Gridless/Elastic Optical Networks
Trade off optical reach vs. capacity

Most of these approaches are likely more feasible in the near-term than the technology
options that were presented

For each approach, we estimate the capacity benefits and the amount of traffic in the
network that can take advantage of the approach

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 16

MNA
Architectural Approaches for Addressing Capacity Limits:
Improved Flow Management in IP Networks

Reduce the need for excessive headroom when packing IP traffic onto wavelengths
Some carriers run IP wavelengths at ~35% fill, to allow for burstiness of IP traffic and
rerouting during failures
Large pipes (e.g., 40 Gb/s,100 Gb/s) carry a large number of flows, which allows for
smoother statistical multiplexing of the traffic
Increase fill of IP wavelengths to more than 65%
Still need some headroom for restoration from failures

Estimate a 2 Benefit in Capacity

Estimate 80% of the traffic is carried in IP Packets
Remainder may be, for example, wavelength services carried directly in the optical layer

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 17

MNA
Architectural Approaches for Addressing Capacity Limits:
Asymmetric Connections
Typically, symmetric connections are established in a network
X Gb/s from Node A to Node Z, and X Gb/s from Node Z to Node A
However, many applications are asymmetric
X Gb/s from Node A to Node Z, and Y Gb/s from Node Z to Node A, where Y X
Video distribution, data backup, distributed data processing, etc.

IP Link Direction with Higher Peak Traffic

AT&T Study
Peak Traffic
8 Weeks of IP Data IP Link Direction with Lower Peak Traffic

Approximately a 2:1
asymmetry ratio S. L. Woodward et al., Asymmetric optical connections for
network-wide improved network efficiency, JOCN, vol. 5. no. 11, Nov.
2013. 2013 Optical Society of America

IP Link
Establish asymmetric connections accordingly (e.g., 10 Gb/s A to Z, 5 Gb/s Z to A)
Will require changes to provisioning systems, element management systems, etc.
Equipment savings are possible as well, in addition to capacity savings
Transponders typically include both a transmitter (Tx) and a receiver (Rx) (or N of each)
If Txs and Rxs are deployed on separate cards, then some number of Txs and Rxs can be removed
Separating the Tx and Rx is likely less efficient in terms of cost, power, space (on a normalized basis)
However, if enough Tx and Rx can be removed, the overall benefit may be positive

Estimate a 1.3x Benefit in Capacity

Estimate this applies to 80% of the traffic
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 18
MNA
Architectural Approaches for Addressing Capacity Limits:
Multicasting
Typically, unicast connections are established in a network
One source, one destination
Some applications are inherently multicast in nature; e.g., video distribution
More efficient to route the multicast traffic over one tree rather than over multiple unicast connections

Assume that Node A is transmitting the same data to Nodes W, X, Y, and Z

A A
E E

B B

F F
C D C D

W X Y W X Y

G H Z G H Z

4 Separate Unicast Connections One Multicast Tree

Backbone network study comparing capacity requirements with a multicast tree vs. multiple
unicast connections:
If number of destinations is uniformly distributed between 5 and 15: factor of ~3 savings
If number of destinations is uniformly distributed between 2 and 6: factor of ~1.5 savings
Estimate a 2x Benefit in Capacity
Estimate this applies to 20% of the traffic
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 19
MNA
Architectural Approaches for Addressing Capacity Limits:
Caching
Distributed caching via Content Distribution Networks (CDN)
Content is stored on multiple servers to be closer to the consumers of the content
Fewer connections are needed; connections are routed over shorter distances
Lower latency is an additional benefit
Percentage of traffic that is distributed via CDNs is increasing
Cisco Visual Networking Index 2013: 34% percent of global Internet traffic crossed CDNs in 2012
45% percent of global Internet traffic crossed CDNs in 2015
Cisco Visual Networking Index 2016: 64% percent of global Internet traffic will cross CDNs in 2020
Caching algorithm improvements increase the probability that the desired data are stored on a
nearby server
A study by AT&T estimates CDNs reduce capacity requirements by a factor of 3 as compared
with content distribution via a centralized server
A. Gerber and R. Doverspike, Traffic types and growth in backbone networks, OFC/NFOEC11, Paper OTuR1

Estimate a 3x Benefit in Capacity

Estimate this applies to 50% of the traffic

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 20

MNA
Architectural Approaches for Addressing Capacity Limits:
Dynamic Optical Networks
Networks have historically been quasi-static; connections remain in place for months / years
Networks have gradually become more configurable
Operations personnel initiate the provisioning process
Connections are established remotely through software control
Takes advantage of flexible technology such as ROADMs and tunable transponders
The next step in this evolution is Dynamic Networking
Deliver bandwidth when & where needed, instead of having a fixed pipe
Connections are rapidly established and torn down without the involvement of operations personnel
The higher layers of the network automatically request bandwidth from the optical layer, which is then
reconfigured accordingly; completely under software control
Connections are provisioned and torn down in seconds, or possibly sub-seconds
Software Defined Networking (SDN) may enable more dynamic networking
There are still challenges
Finding the correct balance between centralized and distributed operation
Dealing with resource contention and latency. Tradeoff with optimization.
Strategies for pre-deployment of equipment: Cost vs. blocking

Backbone network study compared capacity requirements using a dynamic vs. a static optical layer
A subset of the traffic was assumed to have ON/OFF cycles, where it was ON 10% of the time
If this traffic subset is 25% of the total traffic: factor of ~5 savings (for this traffic)
If this traffic subset is 10% of the total traffic: factor of ~4 savings (for this traffic)

Estimate a 4x Benefit in Capacity

Estimate this applies to 20% of the traffic
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 21
MNA
Architectural Approaches for Addressing Capacity Limits:
Gridless/Elastic Optical Networks
Assign spectral resources to meet the required rate of the carried services
The spectrum on each fiber is sliced up as needed: Optical-layer Grooming
Example: a 15 Gb/s connection is assigned 15 Gb/s worth of spectrum as opposed to grooming
the connection with other connections to fill up a 100 Gb/s wavelength
Allow the assigned spectral resources to grow or shrink: Elastic Network
If the connection now requires 20 Gb/s as opposed to 15 Gb/s, the spectrum allotted to that
connection expands (assuming the spectrum is available)
Do not adhere to the 50-GHz ITU-T grid: Gridless Architecture

Jinno et al., Spectrum-Efficient and Scalable Elastic Optical Path Network: Architecture, Benefits, and Enabling
Technologies, IEEE Communications Magazine, November 2009. 2009 IEEE

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 22

MNA
Gridless/Elastic Optical Networks Pose
Numerous Management and Practical Challenges
Adds network management complexity
Need to track how the spectrum is sliced up on each fiber
Routing and Spectrum Assignment (RSA) vs. Routing and Wavelength Assignment (RWA)
May lead to stranded spectral resources (especially in a dynamic network)
The available spectrum may consist of narrow, non-contiguous spectral regions
May limit opportunities for optical bypass (especially in a dynamic network)
The spectrum may be sliced up differently on each fiber entering a ROADM
Need to Defragment the network to create larger blocks of free spectrum
Defragmentation involves modifying live connections; needs to be done carefully
Need guardbands between each optical slice Wasted bandwidth
Need bandwidth-variable transponders (BVTs) and bandwidth-variable ROADMs
Likely not problematic: e.g., transponders based on OFDM subcarriers, ROADMs using LCoS
Many bandwidth-variable transponders are required
If transponders transmit/receive just one connection, then many more transponders are required as
compared to a standard grid-based system
Alternative: Multi-Flow Transponders one transponder can support multiple independent connections
The granularity of the optical slices cannot be arbitrarily fine
E.g., Filtering limitations in the ROADM
Electronic grooming is still needed for the relatively low-rate connections (e.g., 10 Gb/s)

Estimate a 1.5x Benefit in Capacity

Estimate this applies to ~100% of the traffic
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 23
MNA
Architectural Approaches for Addressing Capacity Limits:
Trade Off Optical Reach vs. Capacity
State-of-the-art transponders typically include powerful Digital Signal Processing (DSP)
Example: DSP required for coherent detection, a key enabler of 100 Gb/s transmission
Take advantage of the DSP to enable software-adaptable transponders
Adaptable transponders can trade off bandwidth versus optical reach
Short connection Requires reduced optical reach Utilize less bandwidth to carry it
Use a modulation format with higher spectral efficiency so that less bandwidth is required
Modulation
Line Rate Bandwidth Optical Reach
Format
100 Gb/s DP-BPSK 75 GHz 3,800 km DP-BPSK Dual Polarization
Binary Phase Shift Keying
100 Gb/s DP-QPSK 50 GHz 2,500 km DP-QPSK Dual Polarization
Quadrature Phase Shift Keying

Use the same modulation format but with reduced bandwidth

Modulation
Line Rate Bandwidth Optical Reach
Format
QAM Quadrature Amplitue
400 Gb/s 16 QAM 100 GHz 400 km Modulation

400 Gb/s 16 QAM 75 GHz 300 km

Teipen, Griesser, and Eiselt,
ICTON 2012
Use in conjunction with the ITU-T Flex-Grid option
Channel spacing is a multiple of 12.5 GHz instead of the current 50-GHz-spacing

Estimate a 1.3x Benefit in Capacity

Estimate this applies to ~25% of the traffic
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 24
MNA
Summary of Architectural Approaches
Approximate
Approximate Percentage of Effective Capacity
Benefit Factor Traffic Subject to Multiplier
Benefit
More Efficient IP Packing 2 80% 1.7
Asymmetric Connections 1.3 80% 1.2
Multicast 2 20% 1.1
Caching 3 50% 1.5
Dynamic Optical Networking 4 20% 1.2
Gridless/Elastic Optical Networks 1.5 100% 1.5
Optical Reach vs. Capacity 1.3 25% 1.1
Total Effective Capacity Multiplier ~6

~6 more traffic can be carried as compared to today,

using the same amount of bandwidth

Benefits are not as significant as those provided by technology advancements

But likely more feasible in the near-term (using existing fiber infrastructure)

These are Rough Estimates !!

(some of the benefits may overlap, so full multiplicative effect may not be achieved)
Copyright 2017 Monarch Network Architects LLC J.M. Simmons 25
MNA
Conclusions
Fiber Capacity Limits Need to be Addressed
Technology Advancements:
Numerous implementation challenges
Most of the advancements presented are not near-term solutions
Architectural Options
Most of the advancements presented are near-term solutions
Require changes in network management
Gridless / elastic networks more challenging to implement
Only Transmission and Fiber Capacity Limits were Discussed
Scalable switching is needed as well

Need to attack the problem from both

technology and architectural perspectives

Combined, may provide an overall 250 effective capacity increase

(Well be fine for 30 years !)

Copyright 2017 Monarch Network Architects LLC J.M. Simmons 26

MNA