Introduction to

Single Root I/O Virtualization (SR-IOV)

Brian Johnson Networking Division, Intel Corporation
April 2016
Legal Notices and Disclaimers
Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service
activation. Learn more at intel.com, or from the OEM or retailer.

No computer system can be absolutely secure.

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration
will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more
complete information about performance and benchmark results, visit http://www.intel.com/performance.

Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and
configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost
reduction.

This document contains information on products, services and/or processes in development. All information provided here is subject to
change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced
web site and confirm whether referenced data are accurate.

Intel, the Intel logo, Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be
claimed as the property of others.

2016 Intel Corporation.

Network Platforms Group 2

Agenda

Chapter 1 Networking in a Virtual Server Overview

Chapter 2 Introduction to Single Root I/O Virtualization (SR-IOV)
Chapter 3 Under the Hood of SR-IOV on the Intel Ethernet Controller XL710
Chapter 4 Use Cases Examples
Chapter 5 Summary

Network Platforms Group 3

Agenda

Chapter 1 Networking in a Virtual Server Overview

Chapter 2 Introduction to Single Root I/O Virtualization (SR-IOV)
Chapter 3 Under the Hood of SR-IOV on the Intel Ethernet Controller XL710
Chapter 4 Use Cases Examples
Chapter 5 Summary

Network Platforms Group 4

Server Virtualization Overview
Physical Servers Virtualized Servers

OS OS
VM VM VM VM

Hypervisor
Virtual Switch
OS OS
Physical
Hardware

Network Switch Network Switch

1:1 relationship between MANY:1 relationship between

applications and hardware applications and hardware

Network Platforms Group 5

 Networking Virtual Machines
Connecting VMs to a network
VM VM VM VM Directly assign to the NIC hardware
Assigned devices are physical devices that are exposed to the virtual
machine. This method is also known as 'passthrough'. Device
assignment allows virtual machines exclusive access to PCI devices for
a range of tasks, and allows PCI devices to appear and behave as if
Operating
Virtual Switch they were physically attached to the guest operating system.
System NIC Driver

Intel VT-d Use software devices and a software virtual switch

PCI Express* Emulated devices are purely virtual devices that mimic real hardware,
allowing unmodified guest operating systems to work with them
Physical
Function
Physical
Function
Physical
Function
Physical
Function
using their standard in-box drivers.

Virtio devices are purely virtual devices designed to work optimally in

a virtual machine.

Network Platforms Group 6

 Virtual Machine (VM) Direct Assignment

VM VM VM VM The Virtual Machine owns the network device

Data is sent directly to/from the VM via Direct
Memory Access (DMA) VMM does not touch it.
Operating
System Near-native performance due to direct connectivity
Intel VT-d to each VM
PCI Express*

The Device can no longer be shared

Physical Physical Physical Physical
Function Function Function Function

Does not Scale - Must have a dedicated network

device for each VM using direct assignment

Network Platforms Group 7

 Direct Assignment Using PCI-SIG* Standard
Single Root I/O Virtualization and Sharing (SR-IOV)
Objective of the Specification
VM VM VM

Virtual NIC Virtual NIC

VF Driver
Single Root I/O Virtualization (SR-IOV) and Sharing is a PCI
Name Name
Space Space feature which allows virtual functions (VF) to be created that
share the resources of a physical function (PF).
VF Driver

Operating
Virtual Switch
System PF Driver
Function Definitions
Intel VT-d
PCI Express* Physical Functions (PFs): These are full PCIe functions that
include the SR-IOV Extended Capability. The capability is
Virtual
Function 1
Virtual
Function 2
Virtual
Function n
Physical
Function used to configure and manage the SR-IOV functionality.
Virtual Functions (VFs): These are lightweight PCIe*
Hardware Virtual Ethernet Bridge (VEB) functions that contain the resources necessary for data
Intel Ethernet with SR-IOV movement but have a carefully minimized set of
configuration resources.
LAN
Traffic Intel Virtualization Technology for Directed I/O (Intel VT-d)

Network Platforms Group 8

Agenda

Chapter 1 Networking in a Virtual Server Overview

Chapter 2 Introduction to Single Root I/O Virtualization (SR-IOV)
Chapter 3 Under the Hood of SR-IOV on the Intel Ethernet Controller XL710
Chapter 4 Use Cases Examples
Chapter 5 Summary

Network Platforms Group 9

 Single Root I/O Virtualization and Sharing (SR-IOV)
The Foundation for I/O Virtualization and Flexible Port Partitioning
VM VM VM

Virtual NIC Virtual NIC

VF Driver

Name Name
Space Space
VF Driver

Virtual Switch
Operating
System PF Driver
Seen as PCI Express* Device Ethernet Controller in OS
Intel VT-d
Virtual Function Driver
PCI Express*

Virtual Virtual
Virtual Physical Seen as PCI Express* Device Ethernet Controller in OS
Physical Function Driver
Function 1 Function 2 Function n Function

Hardware Virtual Ethernet Bridge (VEB)

Intel Ethernet with SR-IOV Hardware resources that are available for SR-IOV
defined Virtual Functions utilizing a Hardware VEB
LAN
Traffic

Network Platforms Group 10

 Single Root I/O Virtualization and Sharing (SR-IOV)
The Foundation for I/O Virtualization and Flexible Port Partitioning
VM VM VM

Virtual NIC Virtual NIC

VF Driver

Name Name
Intel Virtualization Technology for Directed I/O
Space
VF Driver
Space
(Intel VT-d) provides the following capabilities:
Operating
Virtual Switch Improved reliability and security through device
System
isolation using hardware assisted remapping
PF Driver

Intel VT-d
PCI Express*
Improved I/O performance and availability by direct
assignment of devices
Virtual
Function 1
Virtual
Function 2
Virtual
Function n
Physical
Function

Hardware Virtual Ethernet Bridge (VEB)

Intel Ethernet with SR-IOV

LAN
Traffic

Network Platforms Group 11

 Single Root I/O Virtualization (SR-IOV) Benefits

Single Root I/O Virtualization and Sharing (SR-IOV)

VM VM VM

VF Driver
Virtual NIC Virtual NIC
Is a PCI feature which allows virtual functions (VF) to be
Name
Space
Name
Space
created that share the resources of a physical function (PF)
VF Driver

Virtual Switch
Benefits of Virtual Functions
Operating
System PF Driver Supports PCI-SIG* SR-IOV standards
Intel VT-d Near-native performance using direct connectivity to each VM by-
PCI Express* passing software virtual switch
Provides data protection via Intel VT for Directed I/O (Intel VT-d)
Virtual
Function 1
Virtual
Function 2
Virtual
Function n
Physical
Function Automatic Receive and Transmit load-balancing with Round-Robin
Rx/Tx scheduling with bandwidth Tx rate-limiting per VF
Hardware Virtual Ethernet Bridge (VEB) Each VF has dedicated resources, Tx/Tx buffers/queues, Tx/Rx
Intel Ethernet with SR-IOV Descriptors, etc.

LAN Flexible Port Partitioning with Bandwidth Rate-Limiting and QoS

Traffic

Network Platforms Group 12

Network Traffic Between VMs

Connectivity Scenarios
1. Same Host (East-West) Open vSwitch (br-tun) Open vSwitch (br-tun)

Virtual Switch NVE NVE

Virtual Functions
VM1 VM2 VM3 VM4 VM5 VM6 VM1 VM2 VM3 VM4 VM5 VM6

2. Host-to-Host (North-South)
Virtual Switch
Virtual Functions
VF VF VF VF VF VF PF VF VF VF VF VF VF PF

Hardware Virtual Ethernet Bridge (VEB)

PCI Express v3.0 x8
Hardware Virtual Ethernet Bridge (VEB)
PCI Express v3.0 x8

3. Combined (N-S + E-W)

Virtual Switch
Virtual Functions

Network Platforms Group 13

Agenda

Chapter 1 Networking in a Virtual Server Overview

Chapter 2 Introduction to Single Root I/O Virtualization (SR-IOV)
Chapter 3 Under the Hood of SR-IOV on the Intel Ethernet Controller XL710
Chapter 4 Use Cases Examples
Chapter 5 Summary

Network Platforms Group 14

Intel Ethernet Controller XL710 Family
I/O Virtualization Diagram
VM VM VM VM VM VM VM VM
VSI VSI VSI VSI VSI VSI VSI VSI 384 VSIs
for the device

SW VEB
Hypervisor VSI

128 VFs
PF VF VF VF VF VF VF for the device

16 VEB/VEPA
NIC HW VEB for the device
802.1 VEPA Floating VEB

4 S-Components for the device (1 per LAN port)

S-Component 384 S-Channels for the device
512 S-tags for the device

MAC
Intel Ethernet Controller XL710

Definitions:
Virtual Ethernet Bridge (VEB)
Virtual Station Interfaces (VSIs)
Virtual Ethernet Port Aggregator (VEPA)

Network Platforms Group 15

 Network Traffic using VF Direct Assignment VMs

Same Host (East-West)

2 Virtual Functions (VFs)
Tx Only netperf default traffic
VM1-VM2 3 instances 44
Gbps

VM1 VM2 VM3 VM4 VM5 VM6 VM1 VM2 VM3 VM4 VM5 VM6

VF VF VF VF VF VF PF VF VF VF VF VF VF PF
Hardware Virtual Ethernet Bridge (VEB) Hardware Virtual Ethernet Bridge (VEB)
PCI Express v3.0 x8 PCI Express v3.0 x8

1. Netperf numbers are representations of max bandwidth of the PCI Express* v3.0 with
8-lanes (x8) and network throughput. Actually throughput may be lower depending on 3 netperf sessions transmitting from on VF to another VF1
packet size, system configuration and capabilities.

Network Platforms Group 16

 Network Traffic using VF Direct Assignment VMs

Same Host (East-West)

3 Virtual Functions (VFs)
Tx Only netperf default traffic
VM1-VM2 3 instances 22 22
Gbps Gbps

VM2-VM3 3 instances
VM1 VM2 VM3 VM4 VM5 VM6 VM1 VM2 VM3 VM4 VM5 VM6

VF VF VF VF VF VF PF VF VF VF VF VF VF PF
Hardware Virtual Ethernet Bridge (VEB) Hardware Virtual Ethernet Bridge (VEB)
PCI Express v3.0 x8 PCI Express v3.0 x8

1. Netperf numbers are representations of max bandwidth of the PCI Express* v3.0 with
8-lanes (x8) and network throughput. Actually throughput may be lower depending on 3 netperf sessions transmitting from on VF to another VF1
packet size, system configuration and capabilities.

Network Platforms Group 17

 Network Traffic using VF Direct Assignment VMs

Same Host (East-West)

4 Virtual Functions (VFs)
Tx Only netperf default traffic
VM1-VM2 3 instances 14 14 14
Gbps Gbps Gbps

VM2-VM3 3 instances
VM1 VM2 VM3 VM4 VM5 VM6 VM1 VM2 VM3 VM4 VM5 VM6
VM3-VM4 3 instances

VF VF VF VF VF VF PF VF VF VF VF VF VF PF
Hardware Virtual Ethernet Bridge (VEB) Hardware Virtual Ethernet Bridge (VEB)
PCI Express v3.0 x8 PCI Express v3.0 x8

1. Netperf numbers are representations of max bandwidth of the PCI Express* v3.0 with
8-lanes (x8) and network throughput. Actually throughput may be lower depending on 3 netperf sessions transmitting from on VF to another VF1
packet size, system configuration and capabilities.

Network Platforms Group 18

 Network Traffic using VF Direct Assignment VMs

Same Host (East-West)

5 Virtual Functions (VFs)
Tx Only netperf default traffic
VM1-VM2 3 instances 10 10 10 10
Gbps Gbps Gbps Gbps

VM2-VM3 3 instances
VM1 VM2 VM3 VM4 VM5 VM6 VM1 VM2 VM3 VM4 VM5 VM6
VM3-VM4 3 instances
VM4-VM5 3 instances
VF VF VF VF VF VF PF VF VF VF VF VF VF PF
Hardware Virtual Ethernet Bridge (VEB) Hardware Virtual Ethernet Bridge (VEB)
PCI Express v3.0 x8 PCI Express v3.0 x8

1. Netperf numbers are representations of max bandwidth of the PCI Express* v3.0 with
8-lanes (x8) and network throughput. Actually throughput may be lower depending on 3 netperf sessions transmitting from on VF to another VF1
packet size, system configuration and capabilities.

Network Platforms Group 19

 Network Traffic using VF Direct Assignment VMs

Same Host (East-West)

6 Virtual Functions (VFs)
Tx Only netperf default traffic
VM1-VM2 3 instances 6.5 6.5 6.5 6.5 6.5
Gbps Gbps Gbps Gbps Gbps

VM2-VM3 3 instances
VM1 VM2 VM3 VM4 VM5 VM6 VM1 VM2 VM3 VM4 VM5 VM6
VM3-VM4 3 instances
VM4-VM5 3 instances
VM5-VM6 3 instances VF VF VF VF VF VF
Hardware Virtual Ethernet Bridge (VEB)
PF VF VF VF VF VF VF
Hardware Virtual Ethernet Bridge (VEB)
PF

PCI Express v3.0 x8 PCI Express v3.0 x8

1. Netperf numbers are representations of max bandwidth of the PCI Express* v3.0 with
8-lanes (x8) and network throughput. Actually throughput may be lower depending on 3 netperf sessions transmitting from on VF to another VF1
packet size, system configuration and capabilities.

Network Platforms Group 20

 Network Traffic using VF Direct Assignment VMs

Combined (N-S + E-W)

1 Physical + 4 Virtual Functions
Tx Only netperf default traffic
Host 1 to Host 2 3 instances 11 11 11
Gbps Gbps Gbps

VM1-VM2 3 instances
VM1 VM2 VM3 VM4 VM5 VM6 VM1 VM2 VM3 VM4 VM5 VM6
VM2-VM3 3 instances
VM3-VM4 3 instances
VF VF VF VF VF VF PF VF VF VF VF VF VF PF
Hardware Virtual Ethernet Bridge (VEB) Hardware Virtual Ethernet Bridge (VEB)
PCI Express v3.0 x8 PCI Express v3.0 x8

37
11
Gbps

1. Netperf numbers are representations of max bandwidth of the PCI Express* v3.0 with
8-lanes (x8) and network throughput. Actually throughput may be lower depending on 3 netperf sessions transmitting from on VF to another VF
packet size, system configuration and capabilities.
Plus Host 1 to Host 2 transmit traffic1

Network Platforms Group 21

Tx Rate Limiting on Virtual Functions
VF attributes are set using ip-route2 commands

rate TXRATE - change the allowed transmit bandwidth, in Mbps, for the specified VF. Setting this parameter to 0 disables rate
limiting. The vf parameter must be specified.
Example of VF Rate Limiting

# ip -d link show
6: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc mq portid 6805ca30dfd8 state UP mode DEFAULT
group default qlen 1000
link/ether 68:05:ca:30:df:d8 brd ff:ff:ff:ff:ff:ff promiscuity 0
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

# ip link set enp3s0f0 vf 0 rate 250

# ip -d link show
6: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc mq portid 6805ca30dfd8 state UP mode DEFAULT
group default qlen 1000
link/ether 68:05:ca:30:df:d8 brd ff:ff:ff:ff:ff:ff promiscuity 0
vf 0 MAC 00:00:00:00:00:00, tx rate 250 (Mbps), max_tx_rate 250Mbps, spoof checking on, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

Network Platforms Group 22

Agenda

Chapter 1 Networking in a Virtual Server Overview

Chapter 2 Introduction to Single Root I/O Virtualization (SR-IOV)
Chapter 3 Under the Hood of SR-IOV on the Intel Ethernet Controller XL710
Chapter 4 Use Cases Examples
Chapter 5 Summary

Network Platforms Group 23

Intel Ethernet Controller XL710/XXV710 NVO/NFV Models
Control Path Network
Virtualization NVE
Data Path
Edge (NVE) OpenStack OpenStack OpenStack
Control OpenDaylight OpenDaylight OpenDaylight
NVE NVE NVE

VM VM VM VNF VNF VNF VNF VNF VNF VNF VNF VNF

Virtio I40evf I40evf
PMD PDM PDM
virtio virtio virtio virtio virtio i40evf i40evf i40evf i40evf

DPDK

OvS NVE
OvS NVE OvS OvS
CP DP CP DP CP DP CP DP

DPDK-netdev
Kernel Kernel Kernel VF Kernel VF

Stack Stack Stack VF Stack VF

VF VF

i40e Driver i40e PMD i40e Driver i40e Driver

SR-IOV SR-IOV

Intel Ethernet XL710 / XXV710 CP NVE

Switch CP NVE
Switch

Model 1 Model 2 Model 3 Model 4

Kernel Data Path User Mode Data Path Bypass + Trusted VNFs Bypass + Switch NVE
Network Platforms Group 24
Agenda

Chapter 1 Networking in a Virtual Server Overview

Chapter 2 Introduction to Single Root I/O Virtualization (SR-IOV)
Chapter 3 Under the Hood of SR-IOV on the Intel Ethernet Controller XL710
Chapter 4 Use Cases Examples
Chapter 5 Summary

Network Platforms Group 25

Summary

Primary use of SR-IOV

Bypass the Hypervisor and vSwitch by directly assigning a Virtual Function to a
Virtual Machine or Container
Benefits
Near-native performance using direct assignment to each VM bypassing
software virtual switch
Provides data protection via Intel VT for Directed I/O (Intel VT-d)
Transmit load-balancing with Round-Robin Rx/Tx scheduling with bandwidth
Tx rate-limiting per Virtual Function
Each VF has dedicated resources, Tx/Tx buffers/queues, Tx/Rx Descriptors, etc.

Network Platforms Group 26

Additional Resources
Intel Ethernet Controller Datasheets
http://www.intel.com/ethernet
FAQ for Intel Ethernet Server Adapters with SR-IOV
https://www-ssl.intel.com/content/www/us/en/support/network-and-i-
o/ethernet-products/000005722.html?wapkw=82599+sr-iov
Intel SR-IOV Explanation
http://www.intel.com/content/dam/www/program/support/us/en/videos/Int
el-SR-IOV-Explanation.mp4
PCI-SIG* SR-IOV Primer: An Intro to SR-IOV Technology
http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-
sr-iov-technology-paper.html

Network Platforms Group 27