TAKE STOCK OF YOUR RAM

I N S I D E

EVALUATE CPU NEEDS

Hypervisor management and optimization

All hypervisors essentially do the same job, but not in the same way. The three top hypervisors all have differences that require specific management tactics.

SIZE UP YOUR NETWORK

HYPERVISOR TEARDOWN: VENDOR BY VENDOR

ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

management
TAKE STOCK OF YOUR RAM

HYPERVISOR

and optimization
BY MIKE LAVERICK

EVALUATE CPU NEEDS

SIZE UP YOUR NETWORK

VMWARE ESX

MICROSOFT HYPER-V

CITRIX XENSERVER

few best practices and recommendations you can follow to optimize and maintain the three core data center hypervisors VMware ESX, Microsoft Hyper-V and Citrix XenServer. On the surface, the discussion of hypervisor optimization might seem trivial since hypervisors are fairly basic and dont come with the usual baggage that a full-blown operating system has. But it is possible to modify key default settings and tweak various vendors hypervisors. A little research reveals that there is a marked difference in recommendations and best practices from one hypervisor to another. No matter which hypervisor you select to manage your virtual machines (VMs), all three platforms have similar resource demands as well as challenges to solve. The easiest way to control these demands is to systematically exTHERE ARE A

plore the four-core resources of any system: memory, CPU, disk utilization and network resources.

TAKE STOCK OF YOUR RAM

RAM is the key resource to consider before purchasing a hypervisor, mainly because its what most people run out of first. Its very common for virtualized environments to run low on memory long before they run out of CPU cycles. Failure to purchase the correct amount of memory for current and future demands can lead to unwanted and expensive consequenceshaving to upgrade your memory and having to purchase additional hosts. In general, you can say that the right amount of memory to purchase is as much as you can afford. But thats a rather simplistic view considering that a server has a finite number of DIMM slots, and the

ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

TAKE STOCK OF YOUR RAM

EVALUATE CPU NEEDS

jump from 2 GB, 4 GB, 8 GB and 16 GB DIMMs is not a small one from a cost perspective. In the long run, having fewer servers with a large amount of memory will be more cost efficient than having more smaller servers, especially when you consider the total cost of power, cooling and licensing. In the case of VMware, for example, this total cost is bound to the number of completed sockets in a rackmounted server or chassis of blades.

machines and the software applications they contain and ask if virtual symmetric multiprocessing, or SMP, will really pay any dividends. A legacy service that isnt multiprocessor aware will simply not use all of the CPU efficiently.

SIZE UP YOUR NETWORK

EVALUATE CPU NEEDS

VMWARE ESX

MICROSOFT HYPER-V

CITRIX XENSERVER

Despite the rise of four-core and sixcore CPUs, the more VMs you create with more than one vCPU, the more work the underlying hypervisor has to do to schedule each CPU within the guest vCPU. Therefore, on a VM level, be careful about the excessive use of multiple vCPUs within a VM because it may not produce the performance dividend youre looking for. Additionally, the excessive use of vCPU can increase scheduling collisionsmore than one vCPU executes on the same core at the same time. In general, this isnt a bad thing in its own right; it forms the bedrock of server consolidation where two, three and four VMs share the same underlying physical CPU. But resources do not scale to infinity and beyond, so its important not to overuse the physical CPUs. Closely examine your virtual

Pound for pound, x64 OSes come with greater virtualization overhead, requiring more memory just to power on the VM.
Additionally, you may be attracted to the performance gains that x64 OSes can deliver from both a memory and CPU perspective. But you may discover that your application doesnt notice all of the benefits of the new architecture because it is 32-bit infrastructure. Pound for pound, x64 OSes come with greater virtualization overhead, requiring more memory just to power on the virtual machine. The question to ask is, Will my application run on 32bit architecture in a way that the vendor will support? Finally, if youre working with a hypervisor that doesnt allow for intelligent placement of the virtual machine on the correct host, then
ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

its up to you to blend the VMs on an array of hosts. It makes no sense to put two or more resource-intensive VMs on the same physical host. You want to avoid excessive over-

TAKE STOCK OF YOUR RAM

EVALUATE CPU NEEDS

It makes no sense to put two or more resourceintensive VMs on the same physical host.
subscription or contention, which degrades performance when too many virtual machines try to access the same physical CPU. In short, if you think your workload is going to be CPU bound, then the amount of onboard memory cache on the CPU will make more of a difference than the overall amount of CPU cycles. Additionally, the new Intel Nehalem and AMD Shanghai processors offer significant CPU gains over

SIZE UP YOUR NETWORK

VMWARE ESX

older Xeon and Opteron chipsets. In the past, most virtualization specialists would have recommended disabling Hyper-Threading on physical CPUs, but recent enhancements on the Intel Nehalem CPUs indicate that Hyper-Threading is back and worth enabling for most workloads. Most vendors often have a confusing array of different CPU specifications for the same rack-mounted server or blade. Often, there can be a huge price difference between one CPU type and anotherwith an almost negotiable gain in performance. You can find some helpful information online regarding which processor works well in virtualized environments.

MICROSOFT HYPER-V

SIZE UP YOUR NETWORK

CITRIX XENSERVER

From a network perspective, I recommend using discrete and different network interfaces for each type of traffic that you intend to push in and out of the physical box. Each type of traffic should reside in its own broadcast domain, ideally using

VLAN TAGGING is a process in which the hypervisor adds an additional identifier to Ethernet frames. These packets are tagged in such a way that when the packets arrive at the physical network switch, it can read the data and send the frame to the appropriate network. The process of VLAN tagging significantly reduces the number of physical network interface cards (NICs) that a hypervisor needs.

ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

TAKE STOCK OF YOUR RAM

VLANs and VLAN tagging to maintain network boundaries. The features you select will ultimately dictate the number of physical network interface cards (NICs) you need and whether the traffic they carry requires redundancy and fault-tolerance. Generally, the following numbers are recommended:

EVALUATE CPU NEEDS

SIZE UP YOUR NETWORK

VMWARE ESX

1 At least one NIC for management traffic 1 At least two NICs connected for VM traffic 1 At least two NICs connected for any IP-based storage system used (network-attached storage or iSCSI) 1 At least one NIC for any live migration or VMotion feature
Hypervisors network demands alone can be daunting. Its possible for network traffic to be blended together, so its not uncommon for organizations to also use the management network to carry out live migration tasks. As a network gateway, the management network isnt heavily used, which is a shame because it would be suitable as spare bandwidth for something else the network needs. However, most hypervisor vendors caution that this is against best practices. So companies will inevitably have to make compromises based on cost, performance and security. For example, its less expensive to drive live migration traffic across a man-

agement network, but that has its security consequences. Most live migration traffic is unencrypted and contains sensitive VM memory data. With that said, most management networks are secured and private. Few customers have experienced poor network performance in the virtual data center. In these few instances, however, the root cause was attributed to external configuration issues such as poor DNS queries and structures or out-ofdate routing table settings that needed immediate attention.

HYPERVISOR TEARDOWN: VENDOR BY VENDOR

MICROSOFT HYPER-V

CITRIX XENSERVER

All hypervisors essentially have the same jobspawn virtual machines and allocate a slice of memory, CPU, disk space and network usage to the workload. And while there are some fundamental aspects of hypervisors where performance optimization will be very similar, not all hypervisors are the same architecturally. In some cases, vendor-specific tweaking is required to gain the most from the virtualization layer you have selected. Heres a look at the three biggest hypervisors by vendor and considerations when implementing each.
1 VMware ESX

Generally recognized to have one of the smallest hypervisor footprints on the market, the actual amount of
ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

TAKE STOCK OF YOUR RAM

EVALUATE CPU NEEDS

SIZE UP YOUR NETWORK

space VMware ESX consumes depends on which flavor of ESX you purchase. ESX Classic, which has a fully interactive service console, is commonly recognized as the more mature cousin to ESXi. Despite VMware Inc.s publically stated position that ESXi is the future of its hypervisor, youre more likely to see ESX Classic in most data centers. The Classic edition allocates a block of memory to the ancillary management front end called the service console. With the amount of memory in most hosts moving from 32 GB to the 64 GB or 72 GB range,

its good practice to max out the amount of memory assigned to it. Currently, VMware assigns a block of memory to the service console on a sliding scale with 300 MB allocated on an 8 GB server and up to 703 MB allocated to a 128 GB server. I recommend allocating the maximum of 800 MB to the service console when possible to reduce the impact of management agents that are installed on it. The value can be adjusted by modifying the settings in the Configuration tab as well as the Memory option in the Hardware pane (Figure 1). Adopting the ESXi version of the

VMWARE ESX

FIGURE 1:
MICROSOFT HYPER-V

Use VMware ESX's Configuration tab to adjust the value of memory allotted to the host.

CITRIX XENSERVER

ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

TAKE STOCK OF YOUR RAM

EVALUATE CPU NEEDS

SIZE UP YOUR NETWORK

platform will avoid this issue altogether. Since it doesnt have a service console, you dont need to worry about configuration issues. ESXi can be run completely from either USB or SD memory cards, which complete the boot process. Once loaded in memory, ESXi presents a smaller attack service area that should leave it less vulnerable to attacks and easier to patch, and it makes VMware ESXi an attractive option for many data centers. However, VMware claims that it will not produce the service console edition forever.

Additionally, if you want to take advantage of VMwares fault tolerance feature, you should look for a CPU that supports the vLockstep attribute, which allows for the successful mirroring of the primary and mirrored secondary virtual machine. A common problem with enabling VMware fault tolerance is that using power-saving features like Intel SpeedStep and AMD PowerNow can undermine the features reliability. As a general rule, review the BIOS settings that control these features sets.

VMWARE ESX

MICROSOFT HYPER-V

CITRIX XENSERVER

3 Buying the right amount of CPUs: From a CPU perspective, you must select an x64 process because VMware no longer produces a 32-bit version of its platform. I recommend buying the CPU that has the largest onboard memory cache you can reasonably afford. With that said, available memory probably matters more than overall speed. Although ESX will manage the mapping of VM memory to physical memory using its internal memory management unit (MMU), its better to have a CPU that offers this MMU functionality as a hardwareassist feature. Intel VT-X CPU, which possesses Extended Page Tables (EPTs) and the AMD CPU, which possesses AMD-V and Rapid Virtualization Indexing (RVI) support, both have this capability.

3 Storage recommendations for ESX: I also recommend that businesses using ESX Classic edition use a boot-from-SAN configuration as well as USB or SD cards to boot ESXi systems. The goal should be to use a stateless ESX host in which there is no configuration or enduser data held on the host itself. When comparing SAN boot or USB boot, SAN booting is more complicated and troublesome from a configuration perspective. It involves a complex interlocking series of settings from the server BIOS, HBA and LUN as well as LUN masking. In the future, VMware will likely support the Preboot Execution Environment (PXE) booting for ESX.

2 Microsoft Hyper-V

Microsoft Hyper-V is available in several formats: as a standalone

ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

TAKE STOCK OF YOUR RAM

EVALUATE CPU NEEDS

platform, in graphical user interface (GUI) mode Standard edition, Enterprise edition and Datacenter edition. You have the option to run them all in a core mode where the graphical shell is suppressed in favor of a command-line-only distribution. For corporate customers, the standalone model will probably be insufficient given that it is limited to addressing 32 GB of RAM. A more appealing version would be to run the server core versions of either the

Enterprise or Datacenter edition, which come with generous licensing conditions that allow unlimited copies of the Datacenter edition. That way, companies can create VMs without worrying about the licensing implication of VM sprawl. As is the case with VMware ESXi, running a stripped down version of Hyper-V also reduces the attack surface from a security perspective and frees up valuable resources. Currently, Microsoft Hyper-V

SIZE UP YOUR NETWORK

VMWARE ESX

The history of storage protocol wars

perspective, although storage vendors battle over which is better Fibre Channel, iSCSI or NFSeach method offers about the same performance characteristics. The correct argument is more about usability, manageability, features and quality of support than raw throughput. As is the case with CPU resources, you want to avoid oversubscribing the same LUN/volume while storing too many VMs on the same underlying spindles. In the case of excessive IOPS, its possible to dedicate an entire LUN/volume and the bandwidth of your Fibre Channel HBAs or network interface cards to service read/ writes. Disk I/O rarely constitutes a bottleneck unless excessive queue depths are being created from storing too many virtual machines on too few LUNs/volumes without sufficient spindles. Organizations often ask what the optimum LUN/volume size is. I dont recommend any hard and fast numbers; however, between 500 GB and 1,500 GB is a safe bet. Using less than 500 GB creates a situation where you need to manage too many volumes that will fill up rapidly. Additionally, creating LUNS/volumes in excess of 1.5 TB usually wastes space, which causes you to run out of IOPS before reaching capacity. I
FROM A DISK

MICROSOFT HYPER-V

CITRIX XENSERVER

ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

does not support memory overcommitment, so optimizing your memory efficiency is critical. Microsoft has indicated that Service Pack 1 will offer a feature for handling memory over-commitment called Dynamic Memory.
TAKE STOCK OF YOUR RAM

servers. There also may be extreme cases where youll want to adjust the guest OSs ReceiveBufferSize and SendBufferSize in the Microsoft registry. If you have a large network load, you might experience heavy traffic loss combined with subsequent retransmits. Microsoft recom-

EVALUATE CPU NEEDS

SIZE UP YOUR NETWORK

VMWARE ESX

MICROSOFT HYPER-V

CITRIX XENSERVER

3 Network selection and performance: From a network perspective, its important to choose synthetic devices over emulated devices when possible. Microsoft doesnt use this specific terminology in the Hyper-V suite, but in parts of the user interface that concern networking, these appear as Network Adapter and Legacy Network Adapter. You can switch from one to the other after you have installed the Integration Components to the VM. Use both synthetic devices with integration tools to get the best performance from a CPU perspective. Microsoft recommends using versions of Windows that have been enlightened for use on Hyper-V such as Windows Vista, Windows 7 and Windows Server 2008. Older Windows OSes such as Windows 2000 and NT4 lack these enhancements. At the physical level, Hyper-V will need a significant number of physical NICsespecially if youre using IP-based technologies such as iSCSI. As a rule of thumb, use up to two quad-port NICs to complement the two onboard network interfaces common on most rack-mounted

Although dynamic disks offer better disk space utilization, theyre not without a performance hit. Use fixed disks for production VMs and dynamic virtual disks for test and dev.
mends increasing the buffer size on a per-virtual-machine basis from 1 MB to 2 MB to alleviate this. Although dynamic disks offer better disk space utilization, theyre not without a performance hit. The best bet is to use fixed disks for production VMs and use dynamic virtual disks for desktops, test and development environments where raw I/O may be less of a pain point. At the upper end, many people
ADVANCED VIRTUALIZATION E-BOOK

HYPERVISOR MANAGEMENT AND OPTIMIZATION

TAKE STOCK OF YOUR RAM

EVALUATE CPU NEEDS

SIZE UP YOUR NETWORK

VMWARE ESX

MICROSOFT HYPER-V

recommend using the Pass-Thru Disk format (PTD), which gives a Hyper-V host the capacity to present LUNs/volumes directly to the virtual machine. At the disk-level, Microsoft recommends using fixed virtual disks over the newer Dynamic VHD format. With fixed disks, a virtual disk is created upfront as a flat file whenever a VM is created. This means that a 30 GB virtual disk would consume 30 GB of physical storage. With a dynamic disk, a 30 GB virtual disk would use just a couple of megabytes of physical storage and would grow dynamically on demand as data is written inside it. This represents a better use of physical disk space because you dont waste space storing empty blocks of data that are waiting for the VM to write to them.

1 Default and custom VM configuration directory, such as C:\ProgramData\ Microsoft\Windows\Hyper-V 1 Custom VM configuration directories, such as C:\Users\ Public\Documents\Hyper-V\ Virtual Hard Disks 1 Snapshot directories 1 Vmms.exe and Vmwp.exe

3 Citrix XenServer

CITRIX XENSERVER

3 Hyper-V trouble spots: Antivirus software configuration can be an area of contention in Microsoft Hyper-V because it can degrade performance. When correctly configured, Hyper-V should not be used as a workstation nor should it access the Internet from its management network. Of course, some data center policies may be non-negotiable, so if youre forced to install an AV product, the general guidelines should not include these locations or these executables until a systemwide scan has been completed:

There are several tricks for tuning XenServer specifically for virtual desktop loads, and many of Citrix Systems Inc.s performance optimization methods center on the companys long history of delivering a client experience to end users. One of the downsides of virtual desktops is that they consume resources even when end users arent connected to them. In XenServer 5.6, Citrix introduced Dynamic Memory Control (DMC), which frees an administrator from allocating a fixed amount of memory to the virtual desktop before powering it on. This could possibly solve the virtual machine memory management paradox allocating a large amount of memory to VMs will improve their performance, but reduce the number of VMs per host. Allocating a small amount of memory increases the consolidation ratio but reduces performance. The problem is that you never quite
ADVANCED VIRTUALIZATION E-BOOK

10

HYPERVISOR MANAGEMENT AND OPTIMIZATION

know how much memory a VM needs, when it needs it or if it will only need it for a certain period of time, such as when the user is connected to his virtual desktop.

comes from the Dom0 partition can add up. At boot up, Dom0 receives an allocation of 752 MB; Dom0

TAKE STOCK OF YOUR RAM

EVALUATE CPU NEEDS

SIZE UP YOUR NETWORK

VMWARE ESX

3 Configuring XenServers Dom0: XenServer has a privilege VM called Dom0. It acts like a broker for all network communication from VMs to the physical world. Each VM lives inside a DomU, and network communication occurs from the netfront component in DomU to its netback component. So, a bottleneck could occur if the Dom0 isnt configured correctly. Each VM thats created (DomU) will use 6 MB of RAM at power on, and although this isnt a huge amount of memory, the RAM allocation that

Y never quite ou know how much memory a VM needs, when it needs it or if it will only need it for a certain period of time.
takes 400 MB of this for itself, leaving approximately 352 MB from which each VM (DomU) can take its 6 MB. Once you reach about 60

MICROSOFT HYPER-V

CITRIX XENSERVER

Storage and capacity planning library

researching storage optimization and planning your virtual environment capacity for Citrix XenServer, look to these resources for help.
WHEN YOU START

q XenServer Performance Monitoring for Scalability Testing q Evaluate Performance of Block Type Storage Repositories for XenServer q Proper Storage Allocation for Hosted Virtual Desktops q Software iSCSI Storage Repository Detachment

ADVANCED VIRTUALIZATION E-BOOK

11

HYPERVISOR MANAGEMENT AND OPTIMIZATION

TAKE STOCK OF YOUR RAM

EVALUATE CPU NEEDS

SIZE UP YOUR NETWORK

VMWARE ESX

MICROSOFT HYPER-V

CITRIX XENSERVER

VMs, the memory pool is full. This situation is common in a hypervisor; organizations that previously used different versions of VMware ESX will know it has a memory reservation for its management layer. Of course, Microsoft Hyper-V needs a chunk of memory just to be viable. The memory allocations in Citrix XenServer are much more specific. However, incorrect planning can reduce VM performance and degrade the manageability of the host through the XenAPI. And, in a worst-case scenario, it can crash XenServer. If your performance woes are not RAM related but IOPS bound, I recommend that you avoid individual commands that let you monitor network and desk activity such as iostat, hdparm, dd, tcpdump, netstat and ifconfig. Focus instead on Citrixs various shell scripts, which pull all of these tools together into handy scripts that send out all of the data you actually need. I
Mike Laverick is a professional instructor with 15 years experience in technologies such as Novell, Windows and Citrix, and he has been involved with the VMware community since 2003. Laverick is a VMware forum moderator and member of the London VMware User Group Steering Committee. He is also the owner and author of the virtualization website and blog RTFM Education, where he publishes free guides and utilities aimed at VMware ESX/VirtualCenter users. In 2009, Laverick received the VMware vExpert award and helped found the Irish and Scottish user groups. He has had books published on VMware Virtual Infrastructure 3, VMware vSphere4 and VMware Site Recovery Manager.

Cathy Gagne Editorial Director cgagne@techtarget.com

Jo Maitland Executive Editor jmaitland@techtarget.com

Colin Steele Site Editor csteele@techtarget.com

Michelle Boisvert Managing Editor mboisvert@techtarget.com

Martha Moore Copy Editor mmoore@techtarget.com

Linda Koury Art Director of Digital Content lkoury@techtarget.com

Marc Laplante Publisher mlaplante@techtarget.com

2010 TECHTARGET. ALL RIGHTS RESERVED.

ADVANCED VIRTUALIZATION E-BOOK

12

FROM OUR SPONSOR

q Virtualization: Consolidation to Cloud Foundation q Forecast: Cloudy with a Chance of Virtualization q Implementing and Expanding a Virtualized Environment

About Dell, Inc.: Dell Inc. (NASDAQ: DELL) listens to customers and delivers innovative technology and services they trust and value. Uniquely enabled by its direct business model, Dell is a leading global systems and services company and No. 34 on the Fortune 500. For more information, visit www.dell.com, or to communicate directly with Dell via a variety of online channels, go to www.dell.com/dellshares. To get Dell news direct, visit www.dell.com/RSS.

ADVANCED VIRTUALIZATION E-BOOK

13