Quality of Service in a Software-Defined Network

Dennis Chan
University of California, Los Angeles

Abstract

that of changing traffic patterns, in which

modern applications often access information
from an array of geographically distributed
databases or servers rather than a single entity.
Second, the consumerization of IT requires
additional flexibility and security to handle
mobile devices in the corporate environment.
Cloud computing and cloud services have
also enjoyed wide acceptance, which has lead
to the need for tools that can provide secure
on-demand access to elastic resources. Finally,
the emergence of big data science in the fields
of artifical intelligence and machine vision
necessitates massive, scaling infrastructure
that can handle large amounts of parallel
processing, infrastructure that can be achieved
through SDN architecture.

In this paper, we construct a software-defined

network (SDN) of a single controller node and
several client nodes using Python scripts. We
begin by outlining the network topology for this
experiment and our messaging scheme for internodal communication. We then proceed to
describe the functions of our scripts and the various worker threads they rely on, as well as design decisions that were made. Using this custom SDN architecture, we apply flow optimization techniques to attempt to provide quality of
service (QoS) by redirecting flows from prioritized paths1 .

Introduction

SDN architecture involves three separate

planes: the application plane, the control plane,
and the data plane [2]. The data plane consists
of the logical network devices that cede control
of their data processing and forwarding to the
SDN controller, which resides in the control
plane. The application plane consists of the
SDN application which interacts with the
SDN controller to request specific network
requirements.

Software-defined networking has become a hot

topic in the field of computer networking in
recent years due to the increasing prevalence
of high bandwidth dynamic networks. In
response, major companies such as Cisco and
IBM now offer SDN solutions to fulfill the
rising demand.
The demand for this type of networking
stems from several trends in computing and
storage that break away from the conventional networks of the past [3]. The first is
1 Source

code:

SDN implementations can vary by vendor

and are not restricted to the same general
archetype [1]. The common solution is to use

https://github.com/dchandc/mptcp-

sdn.git

the OpenFlow protocol, which is maintained

by the Open Networking Foundation and
allows for remote communication with the
forwarding plane of routers and switches in
order to observe and modify packet flows.
Another type of SDN implementation relies on
communicating via device APIs, and a third
type makes use of an overlay network with
virtual switches and hypervisors to achieve
centralized control.

address that will be used in the Python scripts

to establish communication for reports and
commands. Thus, all the client nodes have
an interface configured to some 10.10.6.2XX
address.
The first client node, arbitrarily designated
server, is connected only to leon on the eth0 interface with the address 10.0.3.1. On the other
end, leon has eth0 configured with 10.0.3.3.
leon is also set up to connect with the other
two intermediate nodes, celaya and irapuato,
with eth2 configured to 10.0.23.3 and eth1 configured to 10.0.13.3 to enable communication
with the respective nodes. Similarly, celaya
has eth2 with 10.0.23.2 for leon and eth1 with
10.0.12.2 for irapuato, while irapuato has
eth0 with 10.0.13.1 for leon and eth2 with
10.0.12.1 for celaya. Lastly, the fifth client
node is labeled client. To communicate with
this node, celaya has a wired connection on its
eth0 interface with the address 10.0.2.2, while
irapuato has a wireless connection on its wlan1
interface with the address 10.0.1.1. In addition
to having an interface facing the controller,
client has eth0 configured with 10.0.2.100 for
celaya and wlan1 configured with 10.0.1.100
for irapuato. Figure 1 shows how the client
nodes are connected.

Our SDN implementation with Python

scripts is similar to the overlay network approach. On each node in the network, we
run a single Python script which handles all
incoming and outgoing messages for the node.
For the SDN clients, we collect flow data and
query neighbors to capture a local snapshot
of the network. For the SDN controller,
we listen for reports from SDN clients and
assemble a merged view of the entire network.
The controller then sends commands to the
clients to modify the network as appropriate to
ensure quality of service and flow optimization.
Section 2 of this paper details the network topology that we set up for testing.
Section 3 goes over our choice of protocol
buffers as a messaging scheme. Sections 4
and 5 cover the Python scripts that define
our SDN architecture. We present our flow
distribution algorithm in Section 6, followed
by our thoughts on future work in Section 7.

Protocol Buffers

In order to have the SDN clients communicate

reliably with each other and with the SDN
controller, we need to use a structured and
consistent message format that can be easily
serialized for sending via sockets. Rather
than go through the process of designing our
own serialization process for Python, we take
advantage of Googles protocol buffers, which
was designed to allow for flexible and efficient
serialization of structured data for Python,
Java, and other languages.

Network Topology

The network testbed is comprised of five SDN

client nodes and one SDN controller node. All
client-client communication is done on the
10.0.0.0/16 subnet, while all client-controller
communication is done on the 10.10.6.0/24
subnet. The controller node, rio, connects to
all the other nodes on its eth0 interface, which
has the IP address 10.10.6.210. It is this IP
2

Figure 1: Network configuration of client nodes and neighbor connections.

To use protocol buffers, we maintain a
.proto file that contains all the messages and
fields that we need. We call the protocol
buffer compiler, protoc, on the file to produce
a Python module which we can include in
our client and controller scripts. Messages
are easily constructed as any other object
and subsequently serialized with the SerializeToString() method for sending. The data
fields are then retrieved from received socket
messages with the ParseFromString() method.
Figure 2 shows the types of messages that are
encoded in protocol buffers.

responsible for broadcasting query messages

to determine the nodes neighbors, listening for
query messages from other nodes and replying
to those queries, analyzing the packet flows
at the node itself, and finally reporting all the
necessary info to the SDN controller.
The script contains options for specifying
the controllers IP address and port as well as
options needed for capturing and parsing flows.
For debugging purposes, it also allows the user
to specify the debug output location, either to
the console or to a file. The main function of
the script consists of creating a worker thread
for each of the responsibilities, passing in a
shared thread event, and waiting for the threads
to complete upon keyboard interrupt.

SDN Client

The SDN client script, sdnclient.py, is run

on all of the nodes in the network with the
exception of the controller node. The script is
3

Figure 2: Messages exchanged between nodes.

4.1

Query worker thread

specifically, we want to avoid broadcast needlessly to the overlying wireless communication

subnet. It also makes little sense to broadcast
query messages to the loopback interface.
Thus, we filter out broadcast addresses that do
not fall within the desired SDN subnet. The
netifaces module allows us to find the IP addresses of all the interfaces of the node. From
there, we convert the addresses to integers,
apply the subnet mask and keep only the appropriate interfaces. Since both the query and
listen worker threads need this information, we
perform this filtering in the main thread and
pass in the broadcast interfaces as arguments
to the worker threads.

The query worker thread periodically broadcasts query messages to the client nodes
neighbors in order to determine who those
neighbors are and to measure the RTT to
those neighbors. To achieve this, a single
non-blocking UDP socket is first created.
The query message is then constructed from
the update message of our Protocol Buffers
specification. It is assigned the query type,
then given a timestamp corresponding to the
current system time, as well as the alias of the
node.
It would be simple to broadcast the query
message out every interface, but this is
problematic given our network setup. More

Once the query message is built and the

broadcast addresses determined, we loop
4

through the addresses. The query message is

updated with the address and then serialized
and sent to the neighbors. Finally, we have the
thread sleep for a pre-defined interval before
repeating the process.

4.2

neighbor.
The listen worker thread also constantly
checks its list of neighbors to determine which
neighbors, if any, have not recently sent a
query message. If the amount of time elapsed
since the last refresh time exceeds some
QUERY TIMEOUT value, then the neighbor
is considered dead and consequently it is
removed.

Listen worker thread

The listen worker thread listens for messages

from other client nodes and responds if necessary. It creates a non-blocking UDP socket
for each broadcast interface and proceeds to
poll each socket until it receives some data to
be read. From there, it reads the data from the
socket, then checks the sender of the message
to make sure the message did not originate
from the node itself. Messages from new
neighbors prompt a new neighbor entry to be
created. The entry contains the alias of the
neighbor if found in the message, as well as the
interface the neighbors message arrived on.
The last refresh time, which is used to ascertain
which neighbors are still alive, is updated for
new and old neighbors alike.

4.3

Flow worker thread

The flow worker thread measures and analyzes

the flow traffic on the node by using the
tcpdump and captcp commands. Tcpdump is
a command-line packet sniffer based on the
libpcap library, while Captcp is an open-source
TCP packet analyzer that uses the output from
tcpdump to produce statistics on the flows and
their associated throughput.
The main loop of the flow worker thread
begins by calling tcpdump and having it sniff
for TCP packets on all interfaces. We make the
assumption here that any other packet traffic is
negligible and not necessary to observe for our
experiments. In order to ensure that packets
are not dropped, we restrict the snapshot length
to 96 bytes so that the header information is
captured but not the payload. We also set a
packet count limit of 1000 packets in light of
the observation that a large packet count limit
can delay the flow analysis. tcpdump writes its
output to a .pcap file that is then used by captcp.

Query messages indicate that a neighbor

is trying to determine its own neighbors and
the RTT to those neighbors. In reply, the listen
worker thread constructs a response message
with the same timestamp and IP address. This
allows the querying neighbor to verify that the
response is to its own query and to compare the
timestamp against the time of the receipt of the
response. The response message is then sent
back on the broadcast interface it was received
on.

Captcp contains several modules for analyzing packets, but we use only the statistic
module to obtain the throughput and flow
information from tcpdump. A peculiar aspect
of this module is that it runs into an error
if its input contains zero packets. Thus, we
examine the summary output from tcpdump to
ensure that there is at least one packet that was
captured. After running the captcp command,

Response messages, on the other hand,

require no reply. Instead, if the IP address in
the response message corresponds to the listening node, then the RTT value for the neighbor
that sent the response message is updated
according to a running mean calculation. For
the calculation, it is also necessary to maintain
how many such responses were heard from that
5

we find the flow information presented in a

human-readable format along with highlighting and indentation. In order to have this
information accessible to the controller, we
must first parse it line by line into a JSON
object, so that the different fields and values
can be accessed as properties of that object.
Lastly, the object is dumped as a string into a
file to be read by the update worker thread.

4.4

attempt to receive the route change command

message itself, but with the timeout reduced
according to how much time has elapsed. After
the command message is received, the timeout
is reduced again for the next socket receive
call. It is possible that the timeout occurs
between the receipt of the length message and
the command message, so the thread maintains
the message buffer at each step. Once the
complete command message is received, the
thread can process the command.

Update worker thread

The update worker thread is responsible for

communicating with the SDN controller. It
sends reports of information the node has collected and receives commands from the SDN
controller to change its routes. It maintains
communication with the SDN controller via a
TCP socket to avoid the unreliability of UDP.

SDN Controller

The SDN controller script, sdncontroller.py, is

run solely on the controller node. The script
is responsible for listening for new clients,
receiving reports from clients, and maintaining
a consistent view of the network. It also needs
to determine what routing actions need to be
taken by client nodes in order to optimize the
flows or assure quality of service.

The update worker thread begins constructing

the report message by adding the current
system time and the client nodes alias. It
appends all its known neighbor information
to the message, including the aliases of those
neighbors, the interfaces on which those neighbors are reached and the RTT values to those
neighbors. The client nodes flow information
is then read from the file created by the flow
worker thread. The report message further
contains parsed output from the IP routing
table and network interfaces, both of which
are necessary for the SDN controller to make
routing decisions. Finally, the update worker
thread sends the completed report message.

The script contains options for specifying

the IP address and port for clients to connect
to. As in the SDN client script, there are also
debugging options. Unlike the SDN client
script, however, this script does not require
root permissions, as all the route changes are
done remotely. The main function of the script
consists of creating a shared thread event and
a shared Network object, starting up worker
threads with these shared entities, and waiting
for the threads to complete upon keyboard
interrupt.

The report message is sent to the controller at regular intervals. In between these
sends, however, the update worker thread
needs to listen for any route change commands
that could come from the SDN controller.
To achieve this, we attempt to receive data
from the controller with a timeout set for the
length of the update interval. If the initial
length message is received, then the thread will

5.1

Network class

The Network class contains the complete view

of the network that the controller has collected
from the client nodes. It holds a dictionary for
mapping node aliases to the Node objects, and
a thread lock object for updating the network
safely. The Network class also contains several
6

5.2

important methods for acting on the nodes in

the network.

Server worker thread

The server worker thread is responsible for

listening for incoming connections from SDN
client nodes. For each new client, it starts
a receive worker thread with the address
and socket of the client. It also appends
a tuple consisting of the socket and the
thread to a list of clients. The server worker
thread constantly checks this list to determine
which threads have died, signalling clients that
have been lost, and updates the list accordingly.

The update node() method does as its name

suggests, which is to update the status of a
particular node within the network. It takes as
arguments the alias, the incoming IP address,
the received report message, and the connecting socket for the node. Before applying any
updates, it must acquire the class thread lock
object to avoid creating inconsistencies. The
same goes for every method in the Network
class. If the node already exists in the network,
then its last update time is modified and any
routing or interface info in the report message
is added. Otherwise, a new node is created with
all the given info. In addition, any neighbors
that are contained within the report message
are updated for the node. Finally, the lock is
released.

Each client is bound to a receive worker

thread that does a blocking receive for new
report messages from the client. When a report
message is received, it is parsed into an object
and passed to the Network object to update
the appropriate node. The thread ends when
the socket receive encounters an error or the
clients socket has closed.

The refresh node list() method is the garbage

collector for the Network class. It iterates
through all the client nodes in the network
and checks their last update time. If enough
time has elapsed since the last update without
hearing further updates from the node, and the
node is not considered a neighbor of any of its
neighbors, then the node is removed from the
network.

5.3

Input worker thread

The input worker thread serves as the SDN application in that it allows the user to have the
SDN controller send route change command
messages to the SDN clients or run the flow
distribution algorithm on the current state of
the network. It uses the argparse module to
continuously read and parse commands. The
key commands that can be input from the user
are state, route, and run. The state command simply prints out the current network
state. From there, the user can use the route
command to manually add or delete routes, or
the run command to run the algorithm.

Finally, there are the small helper methods. The set node lost() method removes the
reference to the socket of a client node, which
will eventually signal its removal upon the next
refresh. send from node() is a helper method
for sending a route change command message
to a particular client node, if the node exists in
network. And the debug print() method helps
show the states of all of the client nodes and
their neighbors to aid in debugging.

Algorithm

With our software-defined network of Python

scripts set up, we seek to develop an algorithm
that would provide quality of service for a set
of user-defined TCP flows while balancing the
overall network utilization, thereby minimizing
7

C(E) is the capacity of the edge, Nflows (E) is

the number of flows on the edge, and Cmin is
the capacity of the lowest capacity edge in the
network. Cmin is included only to normalize the
qualities of all the edges in the network. For
example, the qualities of three edges having
capacities of 1 Gbps, 100 Mbps, and 50 Mbps
in a network with one flow per edge would be
20, 2, and 1, respectively.

the throughput degradation that results from

any necessary route changes.
Maintaining quality of service for a flow
requires ensuring that a minimum level of
throughput can be achieved for the flow. The
level of throughput that can be obtained for
an individual flow is determined by the route
that it takes in the network and the presence of
other flows. The flow traverses edges of fixed
capacity along its route. Since TCP seeks to
maximize throughput, the throughput for the
flow increases until it reaches the capacity of
the lowest capacity edge along its route. This
assumes, however, that the flow has sole access
to the network edges. In reality, a network
edge can be utilized by several flows at the
same time, each vying for the limited available
capacity. Given that TCP attempts to achieve
fairness, and if we consider the particular case
where the network is only populated by TCP
traffic, then we see that the throughput allowed
for any flow on an edge can be considered
proportional to the number of flows passing
through the edge.

Instead of trying to determine the shortest

path from the source to the destination, our
algorithm needs to find the path which guarantees the highest gain, or highest quality
of throughput. For any path in the network,
this quality value is determined by the edge
with the lowest quality, as that edge becomes
the bottleneck. Thus, rather than sum up the
weights of the edges to derive a distance value
for a node, we maintain only the minimum
quality value found on the path to each node
for the relaxation step of the algorithm. When
the destination node is finally visited, it should
be updated with the highest quality value
possible.
Theorem 1. The modified Dijkstras algorithm
is correct.

In terms of balancing the overall network

utilization, we essentially have a maximum
flow problem with multiple flow sources and
destinations in the network (see Figure 3).
Normally, such a problem might solved via the
Ford-Fulkerson method, but in our case, the
TCP flows cannot have a split path. Thus, we
choose to instead modify Dijkstras algorithm
for finding shortest paths into a version which
allows us to find the path which can provide
the highest throughput for a particular flow.

6.1

Proof. We prove the correctness of our modified Dijkstras algorithm by induction on

the number of visited nodes. Our invariant
hypothesis is as follows: for each visited node
u, q(u) is the highest quality value from source
to u; and for each unvisited node v, q(v) is the
highest quality value via visited nodes only
from source to v.
The base case, in which the only node is
the visited source node, is trivial. For N - 1
visited nodes, we choose an edge (u, v) where
v has the highest q(v) of any unvisited node
and the edge (u, v) is such that q(v) = min(q(u),
Q((u, v))). q(v) must be the highest quality
value from source to v because if there were
a path from source to v that provides a higher

Modified Dijkstras Algorithm

For our modified algorithm, we assign to each

edge a quality variable Q(E), where
(
C(E)
Cmin (N f lows (E)) N f lows (E) > 0
.
Q(E) = C(E)
otherwise
Cmin
8

Figure 3: Maximum flow problem.

6.2

quality value, then the last unvisited node w on

that path must have q(w) > q(v), with Q((w,
v)) >= q(w). Because every node on the path
going back to the source from v must have a
quality value at least as high as the previous
one, we have the first unvisited node y on the
path with q(y) >= ... >= q(w) > q(v), which is
a contradiction of the hypothesis. If there were
a path without unvisited nodes that yielded a
higher quality value, then we would have q(v)
> min(q(u), Q((u, v)).

Selection Order

We also need to consider the order in which we

select the flows for finding the optimal routes.
The ordering matters because each flow finds
the network in a different state based on the
calculations of the routes of the previous flows.
While the ordering strategy applies to both
QoS flows and regular network flows (in the
absence of absolute priorities), we prioritize all
of the QoS flows first before moving on to the
other flows.

Once the node v has been processed, for

each remaining unvisited node w, q(w) is the
highest quality value from source to w using
visited nodes only, since a higher value of q(w)
would have been found already if the path
does not contain q(v). If the path does contain
q(v), then q(w) would have been updated in the
process.

One solution might be to assign routes

based on the hop distance between the source
and destination for each flow. Thus, flows
between neighbors would be assigned the
direct route first, and flows that cross over
more of the network would wait to get routes
last. Since later routes may encounter lower
quality edges and require more circuitous
routes, assigning these routes to flows with
high hop distances has the benefit of keeping
the hop count proportional.
9

quality threshold, S(E), that must be met for

each edge in the path. While S(E) is normally
set to 0 for regular network flows, it is updated
whenever a QoS flow is added if the new flow
has a higher quality requirement. If adding
the flow to the network using the calculated
path would cause Q(E) to fall below S(E) for
any edge on the path, then the path must be
rejected for the next best alternative. Thus,
there is higher priority for QoS flows that
are selected first, but as we discussed in the
Selection Order section, we can only attempt
to achieve fairness. If there are no alternatives
to the rejected path, then we cannot guarantee
quality of service for that particular flow.

Alternatively, we could define clusters of

flows based on location within the network.
We would then randomly select flows with
probabilities proportional to their hop distances
to their destinations. This gives flows with high
hop distances a better chance to be selected
versus the other scheme.
Both solutions unfortunately run into the
problem of fairness, since flows with higher
hop distances never have priority in the first
solution, and flows outside the clusters have
a lower chance of being selected than those
within the clusters. Our approach is to instead
group the flows by destination. As flows are
bidirectional, there is not a true destination for
each flow, but since we have implemented a
way to observe the throughput of both directions of each flow, we choose the direction with
heavier traffic as that of source to destination.
Then for each destination, we randomly choose
flows for the algorithm.

Once the routes of all the QoS flows have

been set, we can then move on to handling the
remaining flows in the same manner. Although
these flows do not impose quality requirements
on the network, they nonetheless must still
adhere to the minimum quality thresholds of
the QoS flows.

This choice of selection order provides a

mechanism for flow balance because all the
flows for a particular destination will be
allocated first. This creates a bottleneck near
the destination, but allows other parts of the
network to be relieved of traffic, thus new flows
will be able to circumvent the bottlenecks.

6.3

As flows are added to the network, we

need to adjust the quality variable for each
edge that is affected. When we update Q(E)
on a particular edge for the route of a flow,
that quality change can affect the overall route
qualities of other flows. Thus, whenever we
determine a route for a flow, we must also
check any and all flows that share edges with
that route and update them accordingly.

Distributing Flows

For example, in Figure 4, suppose we

randomly select the flow C-A-E. Because
Q(C-A) is 1/3, it is the bottleneck for the route.
While Q(A-E) remains at 1 after the flow is
added, we keep track of the remaining 2/3
quality value that is available to other flows.
When we next consider the flow B-A-E, it
can achieve an overall 2/3 quality value for
its route, despite the fact that it would have
otherwise only been able to get 1/2 due to the
presence of the first flow. The observed Q(E)

With our modified Dijkstras algorithm in

hand and our selection order determined, we
begin distributing flows on an initially empty
network. We want to maintain the priority
of QoS flows, so we group all the QoS flows
and process them first before dealing with any
of the other regular network flows. For each
QoS flow, we apply our modified Dijkstras
algorithm to find the highest quality path from
source to destination. In finding this path,
we must take into consideration the minimum
10

for a flow can thus be described as max(Q(E),

Qrem (E)). Finally, we consider what happens
when flow D-A-E is added to the mix. Qrem (E)
no longer applies, so we are left with 1/3
for Q(E). In order to provide this quality, the
quality for f low B-A-E must be reduced from
2/3 to 1/3.

6.4

We do not believe our algorithm provides an

optimal solution to the flow distribution problem. While we have attempted to achieve fairness between the flows by appealing to randomness, there are still problems with having flows
take circuitous routes due to their selection order. In addition, we believe that the order in
which we update affected routes may have an
effect on how the qualities of the routes are determined. The problem may even ultimately
be an NP-complete problem. Regardless, these
questions warrant further investigation.

Implementation

Our implementation of the flow distribution algorithm is defined as a function in our SDN
controller script which is called with the run
command. We maintain several dictionaries to
keep track of the matrices of Q(E), S(E), and
flow routes, among other statistics. The function also calls a separate function which contains the modified Dijkstras algorithm.

6.5

We have presented in this paper our work in

building a custom software-defined network using Python scripts and developing a flow distribution algorithm for providing quality of service and balancing network utilization. The
majority of our efforts were directed towards
the construction of the SDN architecture - collecting the necessary node data and coordinating the messaging between all the nodes proved
to be a more difficult task than we had originally thought. The flow distribution algorithm
for quality of service also took a considerable
amount of time to understand and implement.
Despite the challenges and imperfections of our
network, we nonetheless feel like we have accomplished a substantial amount of work over
the course of this project.

Experiment

To test our SDN architecture, we devise a simple experiment with leon, celaya and irapuato.
We create a large 10 gigabyte file on leon and
using three IP aliases on leon and on celaya,
we initiate three scp commands from celaya
to retrieve that file. The routing is initially set
up to have all three flows on the direct route
between leon and celaya. We collect the view
of the network at rio, as show in Figure 5.
We then assume that one of the flows is
requesting a quality of service equal to half of
the throughput of the link. We walk through
our algorithm and see that a route change must
be made to shift one of the other flows to go
through irapuato. As we can see in Figure 6,
irapuato now is aware of the new flow.

Conclusion

Acknowledgements

We would like to thank our mentor Jorge Mena

for his guidance and support in devising this
project for the comprehensive exam. In addition to formalizing the underlying flow distribution algorithm, Jorge was extremely helpful
in setting up the testbed network and maintaining clear goals for the project. We would also
like to thank Justin Morgan for his collaborative efforts on this project.

Future Work

There is a good deal more that can be done to

improve upon the work presented in this paper.
11

Figure 4: Example diagram for edge quality changes.

References

org / sdn - resources / sdn definition.

[1] Susan
Fogarty.
http://www.networkcomputing.com/cloudinfrastructure/7-essentials-softwaredefined-networking/1672824201. URL:
http : / / www . networkcomputing .
com / cloud - infrastructure / 7 essentials - software - defined networking/1672824201.
[2] Open Networking Foundation. SDN
Architecture Overview. URL: https :
/ / www . opennetworking . org /
images / stories / downloads / sdn resources/technical-reports/SDNarchitecture-overview-1.0.pdf.
[3] Open Networking Foundation. SoftwareDefined Networking (SDN) Definition.
URL: https://www.opennetworking.

12

Figure 5: SDN controller output with three scp flows.

13

Figure 6: SDN controller output with three scp flows after route change.

14