Live Migration of Virtual

Machines
A Seminar Report
done in partial fulfilment of
the requirements for the award of the degree of

Bachelor of Technology
in

Computer Science & Engineering

by
Bobba Venkata Praveen
B120678CS

Department of Computer Science & Engineering

National Institute of Technology Calicut

Kerala 673601
Monsoon 2015

National Institute of Technology Calicut

Department of Computer Science & Engineering
Certified that this Seminar Report entitled

Live Migration of Virtual

Machines
is a bonafide record of the Seminar presented by
Bobba Venkata Praveen
B120678CS

in partial fulfilment of
the requirements for the award of the degree of

Bachelor of Technology
in

Computer Science & Engineering

Ibrahim A
(Seminar Co-ordinator)
Department of Computer Science & Engineering

Acknowledgement
This work was done as part of the seminar at National Institute of Technology
Calicut under the guidance of Ibrahim A. The major part of the report has been
finished using the reference of three papers by Clark[1], Michael[2] and Medina[3].

ii

Contents
Acknowledgement

ii

1 Introduction

2 Literature Survey

3 Design
3.1 Memory Migration . . . . . . . . . . . . .
3.1.1 Pre-Copy approach . . . . . . . . .
3.1.2 Post-Copy approach . . . . . . . .
3.2 Local Resources Migration . . . . . . . . .
3.2.1 Network Migration . . . . . . . . .
3.2.2 Block Device Migration . . . . . . .
3.3 Division of Responsibilities . . . . . . . . .
3.3.1 Managed Migration (VMM-based)
3.3.2 Self Migration (OS-Based) . . . . .
3.4 Security Problems . . . . . . . . . . . . . .

3
3
4
5
6
6
7
7
7
8
8

4 Conclusion

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

References

10

iii

List of Figures
1
2

5-step view of the pre-copy technique . . . . . . . . . . . . . . . . .

Timeline of (a)pre-copy vs (b)post-copy migration . . . . . . . . . .

iv

4
5

Abstract
Migrating operating system instances across distinct physical hosts is
a useful tool for administrators of data centers and clusters. It allows a
clean separation between hardware and software, and facilitates fault management, load balancing, and low level system maintenance.
This report addresses the following questions: Why live os migration but
not process migration ? How to migrate memory ? Various mechanisms in
memory migration. How to manage or migrate local resources like Network
Resources & Local Storage (for example block devices) ? Finally, security
problems that arise in VM migration.

Introduction

Cloud computing has currently attracted considerable attention from both the industrial community and academic community. In this new computing paradigm,
all the resources are delivered as the services (Infrastructure Service, Platform Service, and Software Service) to the end users via the Internet. Virtualization is a
core technique to implement the cloud computing paradigm.
Virtualization is a technology that started in 1960s. Virtualization introduces
a software abstraction layer between the hardware and the operating system and
applications running on top of it. This abstraction layer is called a Virtual Machine Monitor (VMM) or hypervisor. We can say virtualization as a technique
of partitioning or dividing the resources of a single server into multiple separated
execution environments. These environments are called virtual machines.
A key benefit of virtualization is the possibility of applying some sort of consolidation strategy to allocate several virtual machines (VMs) on the same host.
Consolidation can be statically or dynamically managed. In the former one, VMs
run on the same host. But, in the latter one a VM can be migrated from one
physical host to the other.This migration happens in such a way that the client
has no idea of this.
Live migration concerns with the migration of VM and all applications running on it from one physical host to the other. But, this involves the problem of
maintaining of memory state, network state and also maintaining the connection
with the client. This also takes care of downtime and migration time.
Here comes the haunting question why os migration? But not process migration. The problem of process migration was thoroughly studied during the
90s. Unfortunately, the fact that applications are strongly connected with the OS
by way of open sockets,file descriptors and other resource pointers makes process
migration very difficult. In some cases (e.g. shared mem- ory between processes),
such migration is not even possible. Migrating an entire OS with its applications
is a much more manageable procedure, especially in the presence of a Virtual Machine Monitor (VMM).
In the following sections, various designs for memory migration, network state
maintenance and migration of block devices, security issues will be discussed.

Literature Survey

The literature survey for this report is as follows:

Various designs for live migration presented by Clark [1].
Post-copy live migration and different ways of handling page fetching in
post-copy approach by Michael [2].
Local resources migration in WAN by Medina [3].

Design

VM can be described as an encapsulation to physical resources. Migration of VMs

without any noticeable downtime is known as Live Migration of Virtual Machines.
Migrating VMs can help data center administrators for system maintenance. This
section presents various designs of live VM migration.

3.1

Memory Migration

Memory migration is classified into three phases:

Push phase: The source VM continues running while certain pages are
pushed across the network to destination VM. The pages that modified during this process must be resent.
Stop-and-copy phase: As the name indicates, the source VM is stopped and
all the pages are copied across the network to the destination VM, then the
destination VM is started.
Pull phase: In this phase, the destination VM starts its execution and accesses a page which is a page fault. Then, it copies the page from source.
The practical solutions selects only one or two phases. For example, pure
stop-and-copy involves halting the original VM, copying all pages to the destination, and then starting the new VM. This has advantages in terms of simplicity
but means that both downtime and total migration time are proportional to the
amount of physical memory allocated to the VM.
One more example is pure demand-migration. In this, the source VM is stopped
for very small duration which is stop-and-copy phase and transfers essential kernel
data structures to the destination. The destination VM is then started, and the
accesses the pages first time which causes page faults. These pages are copied
from source to destination VM. This process considerably reduces downtime and
increases total migration time unacceptably.
Above all live migration follows two types of approaches:
pre-copy approach: It combines push with stop-and-copy.
post-copy approach: It combines pull with stop-and-copy.

3.1.1

Pre-Copy approach

The pre-copy algorithm uses an iterative push phase, followed by a minimal stopand-copy. The iterative approach is because of the dirty pages. These dirty pages
are because of the modification of memory pages since the last transfer. These
modified pages must be resent to the destination VM. It also maintains iteration
i with less number of dirty pages than that of i-1. But in some of the instances,
some pages get updated at a faster rate than the rate they can be transferred. In
such cases, stop-and-copy approach must be executed.

Figure 1: 5-step view of the pre-copy technique

The stop-and-copy phase is when the CPU state and any remaining inconsistent
pages are sent to the new host, leading to a fully consistent state. But determining
when to stop a VM to copy pages is a bit difficult. Because if it is stopped to early,
a lot pages have to be copied which results in high downtime. If it is stopped too
late, migration takes too long because of rapid page dirtying which results in
4

high migration time. The frequently updating pages called writable working set
(WWS) or hot pages and CPU state can only be transferred during the stop-andcopy phase. To maintain high availability migration must start with the static
pages that consume less bandwidth and CPU.
3.1.2

Post-Copy approach

The migration of pages after complete transfer of CPU state to the destination
VM and starting the VM. During migration, in pre-copy source VM handles client
requests. But in post-copy technique, destination host takes care of that. In
general, post-copy first suspends the source VM and copies minimal processor
state(PCB information etc.,) to the target VM and resumes it and then starts
copying memory pages from the source. The advantage of post-copy migration is
that each memory page is transferred not more than once.

Figure 2: Timeline of (a)pre-copy vs (b)post-copy migration

There are three ways of handling page fetching in post-copy approach:
Post-Copy via Demand Paging: after the VM resumes at the target, page
faults are serviced by requesting the referenced page over the network from
5

the source node. Usually, this results in unacceptable total migration times
and application degradation.
Post-Copy via Active Pushing: proactively push the pages from the source
to the target across the network, even as the VM continues executing at the
target machine. But if a page fault occurs, it has to be served first.
Post-Copy via Pre-Paging: this approach is an extension of active pushing
technique. In this approach, VM memory access pattern is observed to expect
the page faults so that the rate of pages being transferred can be controlled.

3.2

Local Resources Migration

The three main physical resources in cluster environment are memory, network
and disk. Memory as discussed above can be copied from one host to another but
local disk and network interface migration is not trivial.
3.2.1

Network Migration

To preserve open network connections and to avoid network redirection mechanisms, a VM should retain its protocol state and original IP address after migration. If the migration is within the same LAN, which is the norm in a clustered
server environment, this can be done by generating an unsolicited ARP reply advertising the new location for the migrated VMs IP. This will reconfigure all users
or peers to the new location. This involves very few packet losses.
But to prevent IP spoofing, some of the routers are configured not to accept
broadcast ARP. To deal with this migrated OS should be aware of the migration
so as to inform the interfaces that it is migrated and not to block broadcast ARP.
VM migration in local area networks concerns with efficient methods of memory migration only. Because, in LAN VM uses network-attached storage (NAS)
which is discussed in the following sections. But in case of wide area networks
(WAN), migration should take care of VMs file system and its open network connections.
In WAN migration, network technologies such as Virtual Private Networks,
tunneling and DNS servers has to be used. One of the solutions in WAM migration
is using Mobile IP, which is being used for laptops, mobiles and all moving devices
from one network to the another. This involves a node on the home network

(home agent) forwards packets destined for the client (mobile node) to a careof address on the foreign network. Among all, WAN migration is facing many
practical problems than LAN like bandwidth, firewall, performance issues etc.,
3.2.2

Block Device Migration

Local disk migration should not be needed inside a server farm. Data centers use
network-attached storage (NAS) devices, which can be accessed from anywhere inside the cluster. Thus, secondary storage doesnt have to be migrated with the VM.
Consequently, in a clustered server environment, the Live VM Migration problem
is reduced to finding a way of consistently transferring VM memory state from one
host to another.
In WAN, migration of block devices i.e, copying of data over network increases
the migration time unacceptably. To solve this issue many mechanisms has been
proposed. One of them involves a target server and a proxy server of a block-level
I/O protocol which uses Network Block Device (NBD) technology.
NBD connects the source and destination host nodes to the target and proxy
server, respectively, using TCP/IP. In this connection, disk blocks are copied using
proxy server to the destination while memory migration is taking place. The proxy
server copies remote disk blocks using two methods, working in parallel on-demand
fetching and background copying. In on-demand fetching, if the VM tries to access
a disk block which is not available in the destination, proxy server retrieves it from
the source. Whereas the background copying mechanism copies the blocks that
still remain at the source site.

3.3

Division of Responsibilities

When performing Live VM Migration, the operation can be either VMM-intensive

or OS-intensive. Depending on the decision made, either the OS kernel or the
VMM will need to be modified. Under some circumstances, the developer is forced
to go for one option or the other. For example, OS-intensive procedures might not
be possible if a closed-source OS is being used, such as Windows 7.
3.3.1

Managed Migration (VMM-based)

In case of managed migration, the migration is performed by the daemons running

in the management VMs of the source and the destination. These daemons are
responsible for creating a new VM on the destination machine, and coordinating

transfer of live system state over the network.For example, dom0 in xen hypervisor
logs dirty pages in every iteration to inform the destination VM.
3.3.2

Self Migration (OS-Based)

In this design, no modifications are required to the VMM. The implementation

aspects are present within the OS itself. The destination machine must run a
migration stub to listen for incoming migration requests, create an appropriate
empty VM, and receive migrated system state.

3.4

Security Problems

Live VM migration includes a lot of state transfer through the network. During the procedure, protecting the contents of the VM state files is an important
consideration as the volatile state being transferred may contain highly sensitive
information like passwords and encryption keys. A secure channel is at times not
enough for protection. Mutual validation among the hosts involved in the migration might even be a more important issue to be considered.
Live VM Migration, like any other network-bound process, is susceptible to
network attacks such as ARP spoofing, DNS poisoning, and route hijacking. If an
attacker somehow manages to place himself between the source and the destination
host, he can then conduct passive (sniffing) or active (man-in-the-middle) attacks.
The fact that the live migration procedure is usually carried out inside a LAN
makes it even more likely for a network attack to be successful, especially in
situations where different third-parties run their VMs inside the same network
subnet, which is the case in cloud computing.

Conclusion

In this paper, we have discussed various schemes in memory migration. But implementation of these schemes is completely hypervisor dependent. Live migration
is playing major role in cloud environment especially in case of load balancing.
Local resource migration plays role as important as memory in case of wide area
networks. Experiments are going on to reduce the performance overhead.
Above all, the major factor to be concerned is security. Secure Live VM Migration has not received the amount of attention that it deserves. There is a clear
trend towards delegating computation to the Cloud and, at the same time, Live
VM Migration is becoming an everyday operation inside clustered server environments. Considering that security is defined by the lowest common denominator,
the existence of vulnerabilities in current migration mechanisms nullifies strong
security guarantees provided by secure hardware and hypervisors.

References
[1] Christopher Clark et al. Live Migration of Virtual Machines. In: Proceedings
of the 2nd Conference on Networked Systems Design & Implementation Volume 2. NSDI05. Berkeley, CA, USA: USENIX Association, 2005, pp. 273
286. url: http://dl.acm.org/citation.cfm?id=1251203.1251223.
[2] Michael R. Hines, Umesh Deshpande, and Kartik Gopalan. Post-copy Live
Migration of Virtual Machines. In: SIGOPS Oper. Syst. Rev. 43.3 (July
2009), pp. 1426. issn: 0163-5980. doi: 10 . 1145 / 1618525 . 1618528. url:
http://doi.acm.org/10.1145/1618525.1618528.
[3] Violeta Medina and Juan Manuel Garca. A Survey of Migration Mechanisms
of Virtual Machines. In: ACM Comput. Surv. 46.3 (Jan. 2014), 30:130:33.
issn: 0360-0300. doi: 10.1145/2492705. url: http://doi.acm.org/10.
1145/2492705.

10