Solaris Disk Mirroring

Contents 1 Introduction................................................................................................3 2 Initial Requests...........................................................................................3 3 Tape Backup Procedure............................................................................3 4 Disk Mirroring.............................................................................................5 5 References................................................................................................12

1
1.1

Introduction
This document describes the procedure we used for recreating HDD mirroring on relay server nodes that are part of Ericsson EMO system. Procedure shows how to perform a tape backup of root partition, and how to set up a two disk mirror of root and swap slices.

Initial Requests
Two out of three EMO relay server nodes needed to have disk mirroring restored. Relay servers are set up using Sun Fire V210 servers, with Solaris 9 being used as an OS. Each of the servers has two internal 72GB SCSI HDDs. Original set up was performed as to include mirroring of both root and swap partitions. First disk data was in order on all three relay servers, and the nodes were operational all the time. Initial requests were:

1.2 1.3 1.4

Back up root partition data on all three relay servers, before performing any disk mirroring operation. Restore disk mirroring on two relay servers, as to reflect original mirroring set up. No service downtime was a prerequisite, because the operations were performed out of Maintenance window. Since at least two relay nodes were operational at each moment, it was not difficult to meet this request.

Tape Backup Procedure

Since no tape backup device was included in EMO system, we provided an additional tape device, SUN StorEdge SDLT320 tape drive. The following steps show the general procedure we used for backing up root file system using the ufsdump command. 01. Login to relay server using system console. Once you attach SUN StorEdge SDLT320 to external SCSI port of Sun Fire V210 server, devfsadm command needs to be issued, for Solaris to detect newly attached tape drive:
/usr/sbin/devfsadm -c tape

After performing devfsadm, the content of /dev/rmt directory should look something like this:
root@rel3 # ls -l /dev/rmt/

total 48 lrwxrwxrwx 1 root other 44 Jan 15 11:13 0 -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0: lrwxrwxrwx 1 root other 45 Jan 15 11:13 0b -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:b lrwxrwxrwx 1 root other 46 Jan 15 11:13 0bn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:bn lrwxrwxrwx 1 root other 45 Jan 15 11:13 0c -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:c lrwxrwxrwx 1 root other 46 Jan 15 11:13 0cb -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:cb lrwxrwxrwx 1 root other 47 Jan 15 11:13 0cbn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:cbn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0cn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:cn lrwxrwxrwx 1 root other 45 Jan 15 11:13 0h -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:h lrwxrwxrwx 1 root other 46 Jan 15 11:13 0hb -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:hb lrwxrwxrwx 1 root other 47 Jan 15 11:13 0hbn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:hbn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0hn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:hn lrwxrwxrwx 1 root other 45 Jan 15 11:13 0l -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:l lrwxrwxrwx 1 root other 46 Jan 15 11:13 0lb -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:lb lrwxrwxrwx 1 root other 47 Jan 15 11:13 0lbn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:lbn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0ln -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:ln lrwxrwxrwx 1 root other 45 Jan 15 11:13 0m -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:m lrwxrwxrwx 1 root other 46 Jan 15 11:13 0mb -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:mb lrwxrwxrwx 1 root other 47 Jan 15 11:13 0mbn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:mbn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0mn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:mn lrwxrwxrwx 1 root other 45 Jan 15 11:13 0n -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:n lrwxrwxrwx 1 root other 45 Jan 15 11:13 0u -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:u lrwxrwxrwx 1 root other 46 Jan 15 11:13 0ub -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:ub lrwxrwxrwx 1 root other 47 Jan 15 11:13 0ubn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:ubn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0un -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:un

/dev/rmt ls output can be used to identify the device name of the tape drive. The default tape drive is the /dev/rmt/0 device. 02. Feed a tape to the tape drive, and check the tape drive status.
root@rel3 # mt status Quantum Super DLT 320 tape drive: sense key(0x0)= No Additional Sense residual= 0 retries= 0 file no= 0 block no= 0

03. Bring the system to run level S (single-user mode).

root@rel3 # shutdown -g30 -y Shutdown started. Fri Jan 15 11:59:44 CET 2010 Broadcast Message from root (pts/2) on rel3 Fri Jan 15 11:59:44... The system rel3 will be shut down in 30 seconds showmount: rel3: RPC: Program not registered Broadcast Message from root (pts/2) on rel3 Fri Jan 15 12:00:04... THE SYSTEM rel3 IS BEING SHUT DOWN NOW ! ! ! Log off now or risk your files being damaged showmount: rel3: RPC: Program not registered Changing to init state s - please wait root@rel3 # INIT: New run level: S

04. Check for available hard disks.

root@rel3 # format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> /pci@1c,600000/scsi@2/sd@0,0 1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> /pci@1c,600000/scsi@2/sd@1,0

In our case root and swap file systems were mounted on c0t0d0 disk. This can be checked by using /etc/vfstab file.
root@rel3 # cat /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/dsk/c0t0d0s1 swap no -

/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 swap /tmp tmpfs yes -

ufs

no

05. Optionally, check the root file system for consistency.

root@rel3 # fsck /dev/rdsk/c0t0d0s0

06. To backup file system, ufsdump command is used:

#

ufsdump options arguments filenames

When performing a full root file system backup, issue the following command:
root@rel3 # ufsdump 0ucf /dev/rmt/0 / DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: Date of this level 0 dump: Fri 15 Jan 2010 12:35:27 PM CET Date of last level 0 dump: the epoch Dumping /dev/rdsk/c0t0d0s0 (rel3:/) to /dev/rmt/0. Mapping (Pass I) [regular files] Mapping (Pass II) [directories] Writing 63 Kilobyte records Estimated 19736342 blocks (9636.89MB). Dumping (Pass III) [directories] Dumping (Pass IV) [regular files] Tape rewinding

DUMP: 19736260 blocks (9636.85MB) on 1 volume at 17889 KB/sec DUMP: DUMP IS DONE DUMP: Level 0 dump on Fri 15 Jan 2010 12:35:27 PM CET

- 0 specifies that this is a 0 level dump (or a full backup) - u specifies that the /etc/dumpdates file is updated with the date of this backup - c identifies a cartridge tape device - f /dev/rmt/0 identifies the tape device - / is the file system being backed up 07. To verify that the backup was successful, use ufsrestore.
root@rel3 # ufsrestore tf /dev/rmt/0 2 . 3 ./lost+found 6016 ./var 6017 ./var/sadm 6018 ./var/sadm/install 6019 ./var/sadm/install/admin

08. To bring the server back to run level 3, use ctrl-d.

4
1.5

Disk Mirroring
Since one of initial requests was to perform disk mirroring as to reflect original mirroring set up, here are outputs of metadb i and metastat commands, and of /etc/vfstab file that show the original setup.
root@rel2 # metadb -i flags first blk block count

a m p luo 16 8192 /dev/dsk/c0t0d0s7 a p luo 8208 8192 /dev/dsk/c0t0d0s7 a p luo 16400 8192 /dev/dsk/c0t0d0s7 a p luo 16 8192 /dev/dsk/c0t1d0s7 a p luo 8208 8192 /dev/dsk/c0t1d0s7 a p luo 16400 8192 /dev/dsk/c0t1d0s7 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors

metadb output shows that slices 7 (/dev/dsk/cxt0d0s7) are used for storing metastat data bases, and that each slice contains three copies of meta DB.
root@rel2 # metastat d10: Mirror Submirror 0: d11 State: Okay Submirror 1: d12 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 16780224 blocks (8.0 GB) d11: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s1 0 No Okay Yes d12: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s1 0 No Okay Yes d0: Mirror Submirror 0: d1 State: Okay Submirror 1: d2 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 125510784 blocks (59 GB) d1: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s0 0 No Okay Yes d2: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s0 0 No Okay Yes Device Relocation Information: Device Reloc Device ID c0t1d0 Yes id1,sd@w4849544143484920485553313033303733464c3338303020563357595a394841 c0t0d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575956485641

root@rel2 # cat /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no #/dev/dsk/c0t0d0s1 swap no /dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no /dev/md/dsk/d10 swap no swap /tmp tmpfs yes -

d0 mirror is used for / (root) file system, with d1 being c0t0d0 based submirror and d2 being c0t1d0 based submirror of d0. d10 is used for swap file system, with d11 being c0t0d0 based submirror and d12 being c0t1d0 based submirror of d10. Though our servers run Solaris 9 operating system, one should be able to use the following procedure on servers running Solaris 8 and Solaris 10 as well. 1.6 Root partition mirroring 01. Before you actually start performing hard disk mirroring, create backup copies of /etc/vfstab and /etc/system files:
cp -p /etc/system /etc/system.orig cp -p /etc/vfstab /etc/vfstab.orig

This can show as a useful thing if something goes wrong with /etc/system file during the process of mirroring. By issuing boot -a from the OK prompt and specifying /etc/system.orig as a system file that should be used, you should be able to boot Solaris at any moment. 02. Check slices structure (partition table) on c0t0d0 (boot) disk.
root@rel3 # format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> /pci@1c,600000/scsi@2/sd@0,0 1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> /pci@1c,600000/scsi@2/sd@1,0 Specify disk (enter its number): 0 selecting c0t0d0 [disk formatted] Warning: Current Disk has mounted partitions. FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels save - save new disk/partition definitions inquiry - show vendor, product and revision volname - set 8-character volume name !<cmd> - execute <cmd>, then return quit format> ver Primary label contents: Volume name = < >

ascii name = <SUN72G cyl 14087 alt 2 hd 24 sec 424> pcyl = 14089 ncyl = 14087 acyl = 2 nhead = 24 nsect = 424 Part Tag Flag Cylinders Size Blocks 0 root wm 1649 - 13982 59.85GB (12334/0/0) 125510784 1 swap wu 0 - 1648 8.00GB (1649/0/0) 16780224 2 backup wm 0 - 14086 68.35GB (14087/0/0) 143349312 3 unassigned wm 13983 - 14034 258.38MB (52/0/0) 529152 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 unassigned wm 14035 - 14086 258.38MB (52/0/0) 529152

03. Transfer the partition table from drive c0t0d0 to drive c0t1d0.
root@rel3 # prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2 fmthard: New volume table of contents now in place.

VTOC is volume table of contents. 04. Next, metastat data bases should be created.
root@rel3 # metadb -a -c 3 -f c0t0d0s7 c0t1d0s7 mdmonitord daemon pid 20005 already running

The -c 3 creates three copies of the metastat data base in space specified after f i.e. in slices c0t0d0s7 and c0t1d0s7. 05. Check metastat data bases status.
root@rel3 # metadb -i flags first blk block count a u 16 8192 /dev/dsk/c0t0d0s7 a u 8208 8192 /dev/dsk/c0t0d0s7 a u 16400 8192 /dev/dsk/c0t0d0s7 a u 16 8192 /dev/dsk/c0t1d0s7 a u 8208 8192 /dev/dsk/c0t1d0s7 a u 16400 8192 /dev/dsk/c0t1d0s7 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors

06. Initialize submirrors of root partition mirror.

root@rel3 # metainit -f d1 1 1 c0t0d0s0 d1: Concat/Stripe is setup root@rel3 # metainit -f d2 1 1 c0t1d0s0 d2: Concat/Stripe is setup

07. Create root mirror.

root@rel3 # metainit d0 -m d1 d0: Mirror is setup

08. Update /etc/system and /etc/vfstab.

metaroot d0

This command is used to let your OS know that from now on you are using a meta device as your root partition (slice). 09. Check updated /etc/vfstab.
cat /etc/vfstab #device device mount #to mount to fsck point # fd /dev/fd fd no /proc /proc proc no /dev/dsk/c0t0d0s1 swap /dev/md/dsk/d0 /dev/md/rdsk/d0 / swap /tmp tmpfs yes root@rel3 # lockfs fa FS type fsck mount mount pass at boot options

ufs -

no 1

no

10. Reboot the server.

lockfs -fa reboot

While rebooting, the server will be down, but the system would not face any downtime, since two other relay servers are fully operational. 11. After the server reboots, check the status of the root mirror.
root@rel3 # metastat d0: Mirror Submirror 0: d1 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 125510784 blocks (59 GB) d1: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s0 0 No Okay Yes d2: Concat/Stripe Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase Reloc c0t1d0s0 0 No Yes Device Relocation Information: Device Reloc Device ID c0t1d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575959413941 c0t0d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575958385941

12. Attach the other root submirror.

root@rel3 # metattach d0 d2 d0: submirror d2 is attached

13. Check the status of submirrors.

root@rel3 # metastat d0: Mirror Submirror 0: d1

State: Okay Submirror 1: d2 State: Resyncing Resync in progress: 1 % done Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 125510784 blocks (59 GB) d1: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s0 0 No Okay Yes d2: Submirror of d0 State: Resyncing Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s0 0 No Okay Yes Device Relocation Information: Device Reloc Device ID c0t1d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575959413941 c0t0d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575958385941

Once the syncing is done, d2 submirrors state changes from Resyncing to Okay.
root@rel3 # metastat d0: Mirror Submirror 0: d1 State: Okay Submirror 1: d2 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 125510784 blocks (59 GB) d1: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s0 0 No Okay Yes d2: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s0 0 No Okay Yes Device Relocation Information: Device Reloc Device ID c0t1d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575959413941 c0t0d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575958385941

1.7

Swap partition mirroring 01. Initialize submirrors of swap partition mirror.

root@rel3 # metainit -f d11 1 1 c0t0d0s1 d11: Concat/Stripe is setup root@rel3 # metainit -f d12 1 1 c0t1d0s1 d12: Concat/Stripe is setup

02. Create swap mirror.

root@rel3 # metainit d10 -m d11 d10: Mirror is setup

03. Attach the other swap submirror.

metattach d10 d12 d10: submirror d12 is attached

04. Check the status of submirrors.

root@rel3 # metastat d10: Mirror Submirror 0: d11 State: Okay Submirror 1: d12 State: Resyncing Resync in progress: 1 % done Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 16780224 blocks (8.0 GB) d11: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s1 0 No Okay Yes d12: Submirror of d10 State: Resyncing Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s1 0 No Okay Yes

Once the syncing is done, d12 submirrors state changes from Resyncing to Okay.
root@rel3 # metastat d10: Mirror Submirror 0: d11 State: Okay Submirror 1: d12 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 16780224 blocks (8.0 GB) d11: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s1 0 No Okay Yes d12: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s1 0 No Okay Yes

05. Edit /etc/vfstab and update swap related line as to include path to d10 meta device, instead of previously used c0t0d0s1 swap slice.
root@rel3 # cat /etc/vfstab #device device mount FS fsck mount mount

#to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/dsk/c0t0d0s1 swap no /dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no swap /tmp tmpfs yes root@rel3 # vi /etc/vfstab (vi is used for file content editing) root@rel3 # cat /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/md/dsk/d10 swap no /dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no swap /tmp tmpfs yes -

06. Reboot the server.

root@rel3 # reboot Jan 15 20:04:50 rel3 reboot: rebooted by root Jan 15 20:04:50 rel3 syslogd: going down on signal 15 syncing file systems... done rebooting...

While rebooting, the server will be down, but the system would not face any downtime, since two other relay servers are fully operational. Once the server reboots, the mirroring is done. Procedure used for swap partition mirroring can be used to perform mirroring of other partitions that might be in use. This excludes root partition (procedure previously described) and partitions used for metastat data base. In addition, appropriate eeprom parameter values can be changed (updated), as to enable Solaris to boot even if one of mirrored disks is not available. For more information on this subject, please consult mirroring references, listed in chapter 5.2.

5
5.1

References
Tape Backup
http://docs.sun.com/app/docs/doc/817-1658/6mhcgsubh?a=view#bkuptasks2-26447

5.2

Mirroring
http://www.sun.com/bigadmin/content/submitted/svm_mirroring.jsp http://wikis.sun.com/display/BigAdmin/Mirroring+root+With+Solaris+Volume+Manager +in+the+Solaris+9+and+10+OS http://www.math.uwaterloo.ca/mfcf/internal/procedures/OS/Solaris/mirror.shtml