Multipath TCP

Jennifer Rexford
Fall 2014 (TTh 3:00-4:20 in CS 105)
COS 561: Advanced Computer Networks
http://www.cs.princeton.edu/courses/archive/fall14/cos561/

Multipath
Mobile user
WiFi and cellular at the same time

High-end servers
Multiple Ethernet cards

Data centers
Rich topologies with many paths

Benefits of multipath
Higher throughput
Failover from one path to another
Seamless mobility

Bringing Multipath to the End Host

Multiple addresses
One or more addresses at an end host
E.g., one per interface card

Multiple paths
Sequence of links between sender and receiver
E.g., four-tuple of source and dest address and port

Multiple subflows
Flow of TCP segments over an individual path
All associated with a single TCP connection

Review of TCP Protocol

Establishing a TCP Connection

A

B
Each host tells its Initial
Sequence Number (ISN)
to the other host.

Three-way handshake to establish connection

Host A sends a SYN (open) to the host B
Host B returns a SYN acknowledgment (SYN ACK)
Host A sends an ACK to acknowledge the SYN ACK
5

Initial Sequence Number (ISN)

Sequence number for the very first byte
E.g., Why not a de facto ISN of 0?

Practical issue: reuse of port numbers

Port numbers must (eventually) get used again
and an old packet may still be in flight
and associated with the new connection

Security issue: adversary injecting packets

Adversary may try to inject packets in a connection
by guessing the Initial Sequence Number
to send counterfeit packets to the receiving host
e.g., counterfeit packets that reset the connection
Some firewalls change the ISN to further randomize

Step 1: As Initial SYN Packet

As port

Bs port

As Initial Sequence Number

Flags: SYN
FIN
RST
PSH
URG
ACK

Acknowledgment
20

Flags

Checksum

Advertised window
Urgent pointer

Options (variable)

A tells B it wants to open a connection

7

Step 2: Bs SYN-ACK Packet

Bs port

As port

Bs Initial Sequence Number

Flags: SYN
FIN
RST
PSH
URG
ACK

As ISN plus 1
20

Flags

Checksum

Advertised window
Urgent pointer

Options (variable)

B tells A it accepts, and is ready to hear the next byte

upon receiving this packet, A can start sending data
8

Step 3: As ACK of the SYN-ACK

As port

Bs port

Sequence number
Flags: SYN
FIN
RST
PSH
URG
ACK

Bs ISN plus 1
20

Flags

Checksum

Advertised window
Urgent pointer

Options (variable)

A tells B it is okay to start sending

upon receiving this packet, B can start sending data
9

Sequence Number
Host A
ISN (initial sequence number)

Sequence
number = 1st
byte

Host B

TCP Data

TCP Data

10

TCP Header
Source port

Destination port

Sequence number
Flags: SYN
FIN
RST
PSH
URG
ACK

Acknowledgment
HdrLen 0

Flags

Advertised window

Checksum

Urgent pointer

Options (variable)

Data

11

Receive Buffering: Flow Control

Receive window size
Amount that can be sent without acknowledgment
Receiver must be able to store this amount of data

Receiver tells the sender the window

Tells the sender the amount of free space left
Window Size

Data ACKd

Outstanding
Un-ackd data

Data OK
to send

Data not OK
to send yet
12

TCP Header: Receive Window

Source port

Destination port

Sequence number
Flags: SYN
FIN
RST
PSH
URG
ACK

Acknowledgment
HdrLen 0

Flags

Advertised window

Checksum

Urgent pointer

Options (variable)

Data

13

Tearing Down the Connection

B

time

Closing (each end of) the connection

Finish (FIN) to close and receive remaining bytes
And other host sends a FIN ACK to acknowledge
Reset (RST) to close and not receive remaining bytes
14

Extending TCP: TCP Options

TCP header
Ten mandatory fields
Optional extension field (usually during handshake)

Examples
Maximum segment size (MSS)
Window scaling
Support for Selected ACKs
But, some middleboxes:

Unknown options
Ignored by receiving host

(i) strip TCP options from

some packets or (ii) drop
packets with TCP options

Routers and TCP options

Should ignore them, passing them through unchanged

15

Incremental Deployment
Challenges

16

Keeping the Same Socket API

Backwards compatibility with existing apps
Present the same socket API and expectations

Establish the TCP connection in the same way

Create a socket to a single remote IP address/port
and then add more subflows to the connection

Work in all scenarios where regular TCP works

If a subflow fails, the connection should continue
as long as some other subflow has connectivity

17

MPTCP in the Network Stack

From http://queue.acm.org/detail.cfm?id=2591369

18

Negotiating MTTCP Capability

How do end-points know they both speak
MPTCP?
During the 3-way SYN/SYN-ACK/ACK handshake

What if middleboxes strip the TCP option?

On the SYN? On the SYN-ACK?
19

Negotiating MTTCP Capability

Include capability on the ACK of the SYN-ACK?
What if the ACK is lost?
Carry on all subsequent packets

What if the middlebox drops SYN packets with

unfamiliar options?
Sender can retransmit lost SYN without the option
and fall back to regular TCP behavior

20

Adding Subflows, Idealized

How to associate a new subflow with the
connection?
Use the source/destination IPs and ports

How to start using the new subflow?

Simply start sending packets with new IP/port pairs
and associate them with the existing connection

How could two end-points learn about extra IP

addresses for establishing new subflows?
Implicitly: one end-point establishes a new subflow, to
already-known address(es) at the other end-point
21

Challenges: NAT
Network Address Translators (NAT)
Problem: NAT changes the IP address and port number
WiFi

LTE

NAT1
NAT2

How to identify a connection?

Using a token established during connection set-up

How to establish new subflows?

Allow one end-point to tell another about its addresses

22

Challenges: Security
Security
Malicious parties creating subflows
To highjack (part of) the connection

How to bootstrap security?

Include a random key during connection set-up
and use it to verify authenticity of new subflows

How to identify the connection on new subflows?

A token generated from the key

How to authenticate the addition of subflows?

Exchanging nonces and computing message
authentication codes using the keys
23

Establishing New Flows, Reality

24

Sequence Numbers
Challenges across subflows
Out-of-order packets due to RTT differences
Access networks that rewrite sequence numbers
Middleboxes upset by discontinuous TCP byte stream
Need to retransmit lost packets on a different subflow

Two levels of sequence numbers

Sequence numbers per subflow
Sequence numbers for the entire connection

Enables
Efficient detection of loss on each subflow
Retransmission of lost packet on a different subflow

25

Sending the Second Sequence #

Mapping of subflow bytes to data sequence space
To associate segments in a subflow
with their position in the connection

Encoding the information

Send the additional data in TCP options
Carry the additional data in the TCP payload

Problems with using TCP payload

Flow control may keep receiver from sending a data
ACK
Simply exclude the data-ACKs from flow control?
but middleboxes wont know to do this
26

Sending the Second Sequence #

So, encode in TCP options
Encode as a mapping
Subflow sequence # to connection sequence #?

But, middleboxes may rewrite subflow sequence #

Instead, map as an offset
Offset from the subflows initial sequence number
robust to middleboxes that shift the sequence
numbers

Middleboxes that modify contents (and length)?

Include checksum in the mapping
and terminate the subflow if modifications occur

27

Receive Buffer Space

Each TCP connection has a receive buffer
Buffer space to store incoming data
until it is read by the application

TCP flow control

Receiver advertises the available buffer space
using the receive window

Should each subflow have its own receive window?

Starvation of some subflows in a connection?
Fairness relative to other TCP connections?
Fragmentation of the available buffer space?

Instead, use a common receive window

28

Use of Multipath TCP in iOS 7

Multipath TCP in iOS 7
Primary TCP connection over WiFi
Backup TCP connection over cellular data

Failover
If WiFi becomes unavailable
iOS 7 will use the cellular data connection

For destinations controlled by Apple

E.g., Siri

29

Multipath Congestion Control

Slides from Damon Wischik

30

Goal #1: Fair at Shared Bottlenecks

A multipath
TCP flow with
two subflows
Regular TCP

To be fair, Multipath TCP should take as much capacity

as TCP at a bottleneck link, no matter how many paths
it is using.
31

Use Efficient Paths

12Mb/s

12Mb/s
12Mb/s

Each flow has a choice of a 1-hop and a 2-hop path.

How should split its traffic?

32

Use Efficient Paths

12Mb/s

8Mb/s
12Mb/
s

8Mb/s
12Mb/
s

8Mb/s

If each flow split its traffic 1:1 ...

33

Use Efficient Paths

12Mb/s

9Mb/s
9Mb/s

12Mb/s

If each flow split its traffic 2:1 ...

12Mb/s

9Mb/s

34

Use Efficient Paths

12Mb/s

12Mb/s
12Mb/s

12Mb/s
12Mb/s

12Mb/s

Better: Each connection on a one-hop path

Each connection should send all traffic on the leastcongested paths

35

Use Efficient Paths

12Mb/s

12Mb/s
12Mb/s

12Mb/s
12Mb/s

12Mb/s

Better: Each connection on a one-hop path

Each connection should send all traffic on the leastcongested paths
But keep some traffic on the alternate paths as a probe
36

Discussion

37