Server Consolidation

with Solaris Containers and ZFS

Davide Restivo

Creative Commons Attribution ShareAlike

Agenda
● Server consolidation overview
● Traditional filesystem and RAID vs. ZFS
● ZFS overview
● ZFS usage examples
● Containers overview
● A simple zone configuration
● An advance example of zone configuration
● How to add a dataset to a zone
● Migrate a zone from one server to another
● Q&A
2
Server Consolidation (SC) Overview
● Reduce hardware cost by running multiple
services on the same system
– Better hardware utilization
– Reduced infrastructure cost
– Low administration cost
● Security requirement
– Resources control
– Security isolation
– Failures containment
3
Possible approach to SC

● Technologies enabling consolidation:

– Hard Partitions
– Virtual Machines
– OS virtualization
– Resources Management

4
 Hard Partitions

For examples:

● SUN Dynamic System APP 0 APP 1

Domains Applicaton
● IBM Scalable

Systems Manager ENV 0 ENV 1

Environment

OS 0 OS 1

Operating System

HW 0 HW 1
Hardware

5
Very expensive!!!
 Virtual Machines

For examples:

● Xen APP 0 APP 1

Applicaton
● VMware

ENV 0 ENV 1
Environment

OS 0 OS 1

Operating System

Hardware

6
 OS Virtualization
For examples:

● Solaris Zones APP 0 APP 1

● Unix Chroots Applicaton

ENV 0 ENV 1

Environment

Operating System

Hardware

7
 Resource Management
For examples:

● SRM (Projects) APP 0 APP 1

Applicaton

Environment

Operating System

Hardware

8
ZFS Overview
● ZFS is a new fs from SUN available from Solaris 10
(06/06)
● ZFS was designed by Jeff Bonwick
● ZFS is an open source project (CDDL)
● ZFS eliminates the need of a volume manager
● ZFS is a 128-bit fs (unlimited capacity)
● 2^48 — Number of snapshots
● 2^48 — Number of files in any individual filesystem
● Automatically repairs corrupted data (in redundant
configuration only)
9
ZFS Overview
● 16 exabytes (2^64 byte) — Maximum size of a
filesystem
● 16 exabytes (2^64 byte) — Maximum size of a single
file
● Copy-on-write model
● Transactional object model
● Portable between paltforms (x86 from/to SPARC)
● Simple administration
● All data and metadata are checksummed
P.S.: The name originally stood for "Zettabyte File System"
but is now an pseudo-acronym. 10
ZFS vs. Traditional filesystems
● Traditional filesystem
– resides on a single device
– requires a “Volume Manager” or “Raid Controller” to use more than one device
– Not portable between platforms
– Offline fsck utility
– Difficult to grow/shrink

● ZFS
– no partitions to manage
– is built on top of virtual storage pools called zpools.
– grow/shrink automatically
– No need to edit /etc/vfstab (automatically mounted)
– Online scrubbing (data available to users)
– Portable between paltform (e.g. X86 to/from SPARC) 11
ZFS vs. Raid
● RAID 5
– Suffers “Write Hole” (loss of power between data and parity)
– Workaround to “Write Hole”: NVRAM (..expensive!!!)
– Expensive controllers

● ZFS
– Everything is copy-on-write (no live data modified)
– Everything is transactional (related changes succeed or fail
as a whole, in other words Atomic)
– No extra hardware!!!

12
ZFS Pools
● Eliminates the notion of volume
● A pool is constructed from virtual devices (vdevs)
vdevs
● A vdev describes a single device or a collection of devices
organized according to certain performance and fault
characteristics
● It is possible to use whole disk or slice as vedv in a storage pool
(slices are not portable)
● The storage capacity of all vdevs are then available to all of the
file systems in the zpool
● The zpool command configures ZFS storage pools
● All datasets in a storage pool share the same space
13
Vdevs (Virtual DEVices)
● Vdevs can not be nested (no mirror or mirror, raidz of mirror,
ecc... )
● The following vdevs are supported:
– disk (/dev/dsk/c0t0d0, /dev/dsk/c0t0d0s2, c0t0d0)
– slices
– file (only for testing purpouse)
– mirror (minimum 2 disks)
– raidz ( raidz1), raidz2
– spare

14
Manage ZFS pool
Create a single root vdev (mirror)
# zpool create pool1 mirror c0t0d0 c1t0d0

Create two root vdevs (each a mirror)

# zpool create pool1 mirror c0t0d0 c1t0d0 mirror c0t0d1 c1t0d1

Create a single root vdev (raidz)

# zpool create pool1 raidz c0t0d0 c1t0d0 c2t0d0

Add more space to an existing pool

# zpool add pool1 mirror c0t0d1 c1t0d2

15
 Manage ZFS pool
Create a single root vdev (a mirror) with one hot spare disk
# zpool create pool1 mirror c0t0d0 c1t0d0 spare c2t0d0

Create a single root vdev (a mirror) and then add a hot spare disk
# zpool create pool1 mirror c0t0d0 c1t0d0
# zpool add pool1 spare c2t0d0
( # zpool remove pool1 c2t0d0 )

Multiple pools can share hot spare devices

# zpool create pool1 mirror c1t1d0 c2t1d0 spare c1t2d0 c2t2d0
# zpool create pool2 mirror c3t1d0 c4t1d0 spare c1t2d0 c2t2d0

Destroy a pool
# zpool destroy pool1
16
List all available zpools and show info of a particular zpool
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
zones 68G 665M 67.4G 0% ONLINE -
# zpool status zones
pool: zones
state: ONLINE
scrub: scrub completed with 0 errors on Wed Dec 27 15:13:36 2006
config:
NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
17
errors: No known data errors
 IOStat

# zpool iostat zones 1 10

capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
zones 665M 67.4G 0 0 84 3.56K
zones 665M 67.4G 0 0 0 0
zones 665M 67.4G 0 0 0 0
zones 665M 67.4G 0 0 0 0
zones 665M 67.4G 0 0 0 0
zones 665M 67.4G 0 0 0 0
^C

Option -v gives information on each root vdev

18
Migrating ZFS Storage pool
In certain situations you might need to move a storage pool between
two machines.
ZFS enables you to export the pool from one machine and import it
on the destination machine. Migration consists of two steps:

1 - Export a ZFS Storage Pool

● Attempt to unmount any mounted file systems

● Flush any unwritten data to the disk
● Write metadata indicating exporting is done succefully
● Remove knowledge of the pool from the system
● Note: Place attention on volumes
# zpool export -f pool1

19
2 - Import a ZFS Storage Pool

# zpool import

pool: pool1
id: 3778921145927357706
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
Config:
pool1 ONLINE
mirror ONLINE
c1t0d0 ONLINE
c1t1d0 ONLINE

# zpool import 3778921145927357706 new_pool

20
.. if something
.

goes wrong :( ...

21
1 - Recovering Destroyed ZFS Storage pools

# zpool create pool1 mirror c0t0d0 c1t0d0

# zpool destroy pool1

# zpool import -D

pool: pool1
id: 3778921145927357706
state: ONLINE (DESTROYED)
action: The pool can be imported using its name or numeric identifier. The pool was
destroyed, but can be imported using the ’-Df’ flags.

config:
pool1 ONLINE
mirror ONLINE
c1t0d0 ONLINE
c1t1d0 ONLINE

# zpool import -Df pool1

< output omitted >
22
2 - Repairing a Missing Device
If a device cannot be opened, it displays as UNAVAILABLE in zpool
status output.

Let's find pool with problems:

# zpool status -x
pool: pool1
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ’zpool online’.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Thu Aug 31 11:45:59 MDT 2006

config:
NAME STATE READ WRITE CKSUM
pool1 DEGRADED 0 0 0
mirror DEGRADED 0 0 0
c0t1d0 UNAVAIL 0 0 0 cannot open
c1t1d0 ONLINE 0 0 0

23
Let's change broken disk an bring the new disk online
# zpool online pool1 c0t1d0

Confirm that the pool with the replaced device is healthy.

# zpool status -x pool1
pool ’pool1’ is healthy

24
3 - Replacing a broken device with another (on different channel)
# zpool replace pool1 c1t0d0 c2t0d0

# zpool status pool1

pool: pool1
state: DEGRADED
reason: One or more devices is being resilvered.
action: Wait for the resilvering process to complete.
see: http://www.sun.com/msg/ZFS-XXXX-08
scrub: none requested
config:

NAME STATE READ WRITE CKSUM

pool1 DEGRADED 0 0 0
mirror DEGRADED 0 0 0
replacing DEGRADED 0 0 0 52% resilvered
c1t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0

Replaces old_device with new_device. This is equivalent to attaching new_device,

25
waiting for it to resilver, and then detaching old_device.
4 - Activating and Deactivating Hot Spares
Hot spare can be actived in the following way:

Manually:
Manually using zpool replace command
Automatically:
Automatically When a fault is received, an FMAagent examines the
pool to see if it has any available hot spares. If so, it
replaces the faulted device with an available spare.

26
Manually spare activation
# zpool create pool1 mirror c1t2d0 c2t1d0 spare c1t3d0 c2t3d0

# zpool status pool1

pool: pool1
state: ONLINE
scrub:

config:
NAME STATE READ WRITE CKSUM
pool1 ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
spares
c1t3d0 AVAIL
c2t3d0 AVAIL
errors: No known data errors

27
# zpool replace pool1 c2t1d0 c2t3d0

# zpool status pool1

pool: pool1
state: ONLINE
scrub: resilver completed with 0 errors on Fri Jun 2 13:44:40 2006

config:
NAME STATE READ WRITE CKSUM
pool1 ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
spare ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
spares
c1t3d0 AVAIL
c2t3d0 INUSE currently in use
errors: No known data errors

28
After the faulted device is replaced and the resilver is completed, use
the zpool detach command to return the hot spare back to the spare
set.
# zpool detach pool1 c2t3d0
# zpool status pool1
pool: pool1
state: ONLINE
scrub: resilver completed with 0 errors on Fri Jun 2 13:44:40 2006

config:
NAME STATE READ WRITE CKSUM
pool1 ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
spares
c1t3d0 AVAIL
c2t3d0 AVAIL
errors: No known data errors
An in-progress spare replacement can be cancelled by detaching the hot spare. If
the original faulted device is detached, then the hot spare assumes its place in the
29
configuration, and is removed from the spare list of all active pools.
...for all the other problems
start to pray
an search for a
valid backup!!!
30
Managing ZFS datasets
The zfs command configures ZFS datasets.
A storage pool is also the root of the ZFS file system hierarchy, it
can be accessed as a file system, inluding mounting, unmounting,
taking snapshots, ecc...
A dataset can be one of the following:
– file system
– volume (simulates a block device)
– snapshot (read-only)
– clone

31
Managing ZFS datasets
Create a dataset
# zfs create pool1/data1
● Directories are created and destroyed as needed.
● The default mountpoint for pool1/data1 is /pool1/data1.
● The mountpoint properties is inherited
● A file system mountpoint property of “none” prevents the file system from being
mounted.
● If a file system's mount point is set to "legacy", ZFS file systems can be managed
with traditional tools (mount, umount, /etc/vfstab).

Change default mount point

# zfs set mountpoint=/export/data1 pool1/data1
32
Get the list of ZFS datasets
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool1 77K 354M 24.5K /pool1
zones 24.0G 42.9G 28.5K /zones
zones/iacc1prd 8.00G 42.9G 24.5K /zones/iacc1prd
zones/iacc1prd/opt_iona 614M 4.40G 614M /opt/iona

Automatically NFS-export
# zfs set sharenfs=rw pool1/data1

Set compression
# zfs set compression=on pool1/data1

Set quota
# zfs set quota=10g pool1/data1
33
 Managing ZFS datasets
Set reservation
# zfs set reservation=2g pool1/data1

Get zfs proprerties

# zfs get all pool1
NAME PROPERTY VALUE SOURCE

pool1 type filesystem -

pool1 creation Wed Jan 10 11:21 2007 -

pool1 used 77K -

pool1 available 354M -

pool1 referenced 24.5K -

pool1 compressratio 1.00x -

pool1 mounted yes -

pool1 quota none default

<continue> 34
<continue>

pool1 reservation none default

pool1 recordsize 128K default

pool1 mountpoint /pool1 default

pool1 sharenfs off default

pool1 checksum on default

pool1 compression off default

pool1 atime on default

pool1 devices on default

pool1 exec on default

pool1 setuid on default

pool1 readonly off default

pool1 zoned off default

pool1 snapdir hidden default

pool1 aclmode groupmask default

pool1 aclinherit secure default

35
ZFS Snapshots
Note: Snapshots are read-only!
Take a snapshot of pool1/data1
# zfs snapshot pool1/data1@first

Roll back a snapshot

# zfs rollback pool1/data1@first
Note: this command delete all intermediate snapshots

Generate full backup of a snapshot

# zfs backup pool1/data1@first > /backup/data1_first

Generate incremental backup of a snapshot

# zfs backup -i pool1/data1@first pool1/data1@second > /backup/second-first

36
ZFS Snapshots
Take a look at file foo.txt in a snapshot
# cat /export/data1/.zfs/snapshot/first/foo.txt
Note: The visibility of the ".zfs" directory can be controlled by the "snapdir" property.

Destory a zfs dataset

# zfs destroy pool1/data1

Displays all ZFS file systems currently mounted.

# zfs mount

Mounts/Unmounts a specific ZFS file system

# zfs mount pool1/data1
# zfs unmount pool1/data1

Mounts all available ZFS file systems.

zfs mount -a
37
To unmount all currently muounted fils systems use: zfs unmount -a
ZFS clones
Note: Clones are read-write!
Take a clone of a snapshot
# zfs clone pool1/data1@first pool1/data1first

38
Container Overview
Solaris Container consists of several technologies
that work together. In particular:

– ZFS
– Dynamic Resource Pool
– FSS (Fair Share Scheduler)
– Solaris Zones

39
Dynamic Resource Pools
Dynamic Resource Pools enables CPU resources
to be allocated to specific applications.

One Application Server Two Web Server

Resource Pool 1 Resource Pool 2

(1 CPU) (3 CPU)

The two web server will share the 3 CPU

40
Fair Share Scheduler (FSS)
Share Allocated to Zones
1

2 4
WebServer
Mail Server
Database
AS

4 / (1 + 2 + 3 + 4 ) = 0.4 = 40 %

This software enables CPU resources to be allocated

proportionally to applications. Each application get its own
share. 41
Configure Resource Pools
1 - Enable the resource pools feature

globalzone# pooladm -e

2 - Save the current configuration and activate it

globalzone# pooladm -c

3 - See default pool config

globalzone# pooladm

system my_system
string system.comment
int system.version 1
boolean system.bind-default true
int system.poold.pid 638

<continue> 42
pool pool_default
int pool.sys_id 0
boolean pool.active true
boolean pool.default true
int pool.importance 1
string pool.comment
pset pset_default

pset pset_default
int pset.sys_id -1
boolean pset.default true
uint pset.min 1
uint pset.max 65536
string pset.units population
uint pset.load 7
uint pset.size 8
string pset.comment

cpu
int cpu.sys_id 1
string cpu.comment
string cpu.status on-line
<omitted> 43
4 - Create a processor set containing one CPU
globalzone# poolcfg -c 'create pset as-pset (uint pset.min=1; uint pset.max=1)'

5 - Create a resource pool for the processor set

globalzone# poolcfg -c 'create pool as-pool'

6 - Link the pool to the processor set

globalzone# poolcfg -c 'associate pool as-pool (pset as-pset)'

7 - Activate the configuration.

globalzone# pooladm -c

8 - Enabling the Fair Share Scheduling

globalzone# poolcfg -c 'modify pool pool_default (string pool.scheduler="FSS")'

globalzone# pooladm -c
44
globalzone# reboot
Solaris Zones
● Solaris Zones provide separate environments on a
machine and logically isolate applications.
● Two type of zones:
– Global zone
– Non-global zone
● A standard zone automatically shares the
– /usr (ro)
– /lib (ro)
– /platform (ro)
45
– /sbin (ro)
Create a simple zone
1 - Zone definition
globalzone# zonecfg -z zone1
zone1: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:zone1> create
zonecfg:zone1> set zonepath=/export/home/zones/zone1
zonecfg:zone1> add net
zonecfg:zone1:net> set address=10.0.0.2
zonecfg:zone1:net> set physical=eri0
zonecfg:zone1:net> end
zonecfg:zone1> set pool=pool_default
zonecfg:zone1:net> verify
zonecfg:zone1:net> commit
zonecfg:zone1:net> exit

46
2 - Install the zone
globalzonel# zoneadm -z zone1 install

3 - Boot the zone

globalzonel# zoneadm -z zone1 boot

4 - Login in the new zone and configure it

globalzone # zlogin -C zone1

Configure it as a fresh Solaris installation

47
An advanced example
1 – Zone definition

globalzonel# zonecfg -z zone2

zone2: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:zone2> create
zonecfg:zone2> set zonepath=/export/home/zones/zone2
zonecfg:zone2> add net
zonecfg:zone2:net> set address=10.0.0.3
zonecfg:zone2:net> set physical=eri0
zonecfg:zone2:net> end
zonecfg:zone2> set pool=pool_default
zonecfg:my-zone> set autoboot=true
zonecfg:zone2> add rctl
zonecfg:zone2:rctl> set name=zone.cpu-shares
zonecfg:zone2:rctl> add value (priv=privileged,limit=2,action=none)
zonecfg:zone2:rtcl> end
zonecfg:zone2> add fs
zonecfg:zone2:fs> set dir=/cdrom
zonecfg:zone2:fs> set special=/cdrom
zonecfg:zone2:fs> set type=lofs
48
zonecfg:zone2:fs> set options=[nodevices]
<continue>
<continue>

zonecfg:zone2:fs> end
zonecfg:zone2> add device
zonecfg:zone2:device> set match=/dev/dsk/c0t0d0s6
zonecfg:zone2:device> end
zonecfg:zone2> add device
zonecfg:zone2:device> set match=/dev/rdsk/c0t0d0s6
zonecfg:zone2:device> end
zonecfg:zone2> verify
zonecfg:zone2> commit Block device
zonecfg:zone2> exit

Character device

49
2 - Install the zone
globalzonel# zoneadm -z zone2 install

3 - Boot the zone

globalzonel# zoneadm -z zone2 boot

4 - Login in the new zone and configure it

globalzone # zlogin -C zone2

Configure as a fresh Solaris installation

50
How to add a dataset to a zone
1 - Create a dataset

globalzone# zfs create mypool/myzonefs

globalzone# zfs umuont mypool/myzonefs

2 - Change default mount point

# zfs set mountpoint=/export/myzonefs mypool/myzonefs

3 - Add the dataset to the zone

globalzone# zonecfg -z myzone
zonecfg:myzone> add dataset
zonecfg:myzone:dataset> set name=mypool/myzonefs
zonecfg:myzone:dataset> end
zonecfg:myzone> commit
zonecfg:myzone> exit
Reboot the zone
globalzonel# zoneadm -z myzone reboot :( 51
Migrate a zone from one server to
another

zone1 zone2 zoneA ZoneB

zone3 zone4

52
Server 1 Server 2
1 - Halt the zone

globalzones1# zoneadm -z zone4 shutdown -y -i 0

2 - Detach the zone

globalzones1# zoneadm -z zone4 detach

Use -p
3 - Create the archive
globalzones1# cd /export/zones/zone4
globalzones1# tar cvpf /tmp/zone4.tar ./zone4

4 - Move the archive to the other server

globalzones1# scp /tmp/zone4.tar server2: /tmp/zone4.tar
globalzones2# cd /export/home/zone ; tar xvpf /tmp/zone4.tar
globalzones2# zonecfg -z zone4
zonecfg:zone4>create -a /export/home/zone/zone4
zonecfg:zone4>exit
53
Means: attach the zone
5 - attach the zone

globalzones2# zoneadm -z zone4 attach

6 – Boot the zone

globalzones2# zoneadm -z zone4 boot

54
Reference Documentation
● “Solaris ZFS Administration Guide”
● The Sun BluePrints - “Guide to Solaris Containers”
● Solaris 10 How to Guides - “How to move a Solaris Container”
● Solaris 10 How to Guides - “Consolidating servers and
applications”
● Solaris 10 How to Guides - “ZFS in Solaris Containers”
● man pages

55
”Time for questions”

56