Veritas Support Guide

Entities
Cluster Highest level entity which contains all service groups supported across all related systems
that have membership within a specific cluster implementation. The database cluster contains several
systems(hosts) used to support the database server in the event of failure of the system running the
database. There is a database cluster and for some implementations there may be an application server
cluster.
Service Group A service group is a logical grouping of resources. There is a service group called
oracle-ha-grp within the database cluster. There is a service group called APP-Serv within the
application server cluster.
System a system is one node within a service group. A machine is usually a system. In a two node
system there are two systems. Ideally one should be in an ONLINE state, the other in an OFFLINE
state. Failovers are from one system to another. Veritas will detect the failure of one system and bring
the other online.
Resource Any service, program, hardware, volume manager object or other entity within the
computing environment placed under the control of the cluster within a Service Group.

Veritas Cluster Server Commands (VCS)

Run these as root/root
To See Status
hastatus sum
The state will show FAULTED if the service group did not start properly

To start Cluster
hastart -force
This will try to start the vcs cluster on a particular node/system
To Clear Fault
hagrp clear <service group name> -sys <system name>
for example for the database systems:
hagrp clear oracle-ha-grp -sys pcsvcs-db1
for the app server:
hagrp clear APP-Serv sys pcsvcs-as1
To Bring a Service Group Offline and Online
hagrp online <service group name> -sys <system name>
for example for the database systems:
hagrp offline oracle-ha-grp -sys pcsvcs-db1
hagrp online oracle-ha-grp -sys pcsvcs-db1
for the app server:
hagrp offline APP-Serv sys pcsvcs-as1
hagrp online APP-Serv sys pcsvcs-as1

To Bring a Resource Offline and Online

hares offlline <resource name> -sys <system name>
for example for the database systems:
hares offline listener -sys pcsvcs-db1 (Perform these two steps in this order!!)
hares offline oracle -sys pcsvcs-db1
hares online oracle -sys pcsvcs-db1 (Perform these two steps in this order!!)
hares online listener -sys pcsvcs-db1
for the app server:
hares offline ps_init sys pcsvcs-as1

To Switch a System
hagrp switch <service group name> -to <system name>
for example, assume the database has failed over and you want to bring the processing database back to the
primary machine:
hagrp switch oracle-ha-grp -to pcsvcs-db1
To Stop a System
hastop [-all]
hastop will shut down a system
hastop all will shut down all systems within a cluster

VCS Directories
Installation:
Logs:
/var/VRTSvcs/log

(engine_0.log oracle_0.log)

Configuration files:
/etc/VRTSvcs/conf/config (main.cf)

Veritas Volume Manager Operations

Check the status of volume manager objects (volumes, plex, subdisk, disk group)
vxprint ht | more (status should be ENABLED for all)
To start Volume Manager GUI
/opt/VRTSvmsa/bin/vmsa
Example.

set DISPLAY to your IP

export DISPLAY=10.10.10.10:0.0 (bash or korn shell)

vmsa &

To check disk status

vxprint list
Sample output:
# vxdisk list
DEVICE
TYPE
c1t0d0s2
sliced
c1t1d0s2
sliced
c3t0d0s2
sliced
c3t1d0s2
sliced
c3t8d0s2
sliced
c3t9d0s2
sliced
c4t0d0s2
sliced
c4t1d0s2
sliced
c4t2d0s2
sliced
c4t8d0s2
sliced
c4t9d0s2
sliced
c4t10d0s2
sliced

DISK
rootdisk
rootmirror
c3t0
c3t1
c3t8
c3t9
c4t0
c4t1
c4t8
c4t9
-

GROUP
rootdg
rootdg
oradg
oradg
oradg
oradg
oradg
oradg
oradg
oradg
-

STATUS
online
online
online failing
online failing
online
online
online
online
online
online
online
online

NOTE: The above output shows two disk flagged as failing. This status indicates that communication was lost
at one point with the disk but has been restored. This status can be cleared and the disks monitored. If it recurrs
there is an underlying problem with the disk and it most likely needs to be replaced.
To clear the failed status for a disk
vxedit -g diskgroupname set failing=off diskname
or in this case
vxedit -g oradg set failing=off c3t0
vxedit -g oradg set failing=off c3t1

To check the status of mirrors

vxprint lp
Each mirrored volume contains at least two plexes and each should have a state of ACTIVE as in the following
output:
Plex:
info:
type:
state:
assoc:
flags:

voldata-01
len=10498504 contiglen=10498120
layout=STRIPE columns=4 width=128
state=ACTIVE kernel=ENABLED io=read-write
vol=voldata sd=c3t0-15,c3t1-15,c3t8-15,c3t9-15
busy complete

Plex:
info:
type:
state:
assoc:
flags:

voldata-02
len=10498504 contiglen=10498120
layout=STRIPE columns=4 width=128
state=ACTIVE kernel=ENABLED io=read-write
vol=voldata sd=c4t0-15,c4t1-15,c4t8-15,c4t9-15
busy complete

To check the overall status

vxprint ht | more
To resize a volume containing a file system
/usr/lib/vxvm/bin/vxresize -F vxfs -g oradg voldata2 8g
To administer disks under volume manager (add disk , replace disk, etc)
# vxdiskadm
Volume Manager Support Operations
Menu: VolumeManager/Disk
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
list

Add or initialize one or more disks

Encapsulate one or more disks
Remove a disk
Remove a disk for replacement
Replace a failed or removed disk
Mirror volumes on a disk
Move volumes from a disk
Enable access to (import) a disk group
Remove access to (deport) a disk group
Enable (online) a disk device
Disable (offline) a disk device
Mark a disk as a spare for a disk group
Turn off the spare flag on a disk
Unrelocate subdisks back to a disk
Exclude a disk from hot-relocation use
Make a disk available for hot-relocation use
List disk information

?
??
q

Display help about menu

Display help about the menuing system
Exit from menus

Select an operation to perform:

Change the log size

haclus
haconf
haclus
haconf
haclus

-display (to display the current log size)

-makerw (to make the configuration writable)
-modify LogSize 10240000 (~10MB)
-dump - makero
-display

Change a resource attribute example (NetworkHosts)

haconf -makerw
hares -modify Multi-eri0-qfe0 NetworkHosts "addressone" "addresstwo"
"addressthree"
haconf -dump -makero
hares -display Multi-eri0-qfe0
Change a type attribute
You can view/verify your changes with:
hatype -display MultiNICA
Modify with:
haconf
hatype
hatype
haconf

-makerw
-modify MultiNICA MonitorInterval 60
-modify MultiNICA MonitorTimeout 60
-dump -makero

Create a new Volume Manager volume and file system:

- You must first create a volume (concatenated or striped) then add a file system to it. large file support should
be enabled in the file system as in the examples.
remove old
first remove volume from cluster See Remove volume resource from cluster below.
umount /psprd1-rdo
[umount file system]
vxassist remove volume MNT-psprd1-rdo
[remove old volume]
create new
vxassist -g oradg make MNT-psprd1-rdo1 200m oradg09

[create volume on a specific disk]

mkfs -F vxfs -o largefiles /dev/vx/rdsk/oradg/MNT-psprd1-rdo1 [add a file system to the volume]

mount -F vxfs -o largefiles /dev/vx/dsk/oradg/MNT-psprd1-rdo1 /MNT-psprd1-rdo1
[mount file system enable large files]
add mirror
vxassist g oradg mirror psprd1-rdo1 oradg01
Remove volume resource from cluster:
hares makerw
hares delete VOL-psprd-rdo
hares delete MNT-psprd-rdo
hares dump makero

[add mirror on specific disk]

Add new volume and file system to cluster:

haconf makerw
hares add MNT-psprd1-rdo1 Mount oracle-ha-grp
hares -modify MNT-psprd1-rdo1 Enabled 1
hares -modify MNT-psprd1-rdo1 MountPoint /psprd1-rdo1
hares -modify MNT-psprd1-rdo1 BlockDevice /dev/vx/dsk/oradg/psprd1-rdo1
hares -modify MNT-psprd1-rdo1 FSType vxfs
hares add VOL-psprd1-rdo1 Mount oracle-ha-grp
hares -modify VOL-psprd1-rdo1 Enabled 1
hares -modify VOL-psprd1-rdo1 Volume psprd1-rdo1
hares -modify VOL-psprd1-rdo1 DiskGroup oradg
haconf dump -makero