Replacing a SCSI disk in a mirrored rootvg and datavg

Prerequisite:

1.Problem ticket needs to created.

2.Change ticket needs to be created and get approved.

3.Take a snapshot of the following..

a) lscfg –vl Bad_PV_name
b) lsvg –l VG_name
c) lspv –l PV_name
d) lsvg –p rootvg

Note:
Please check all LVs in the VG except sysdump device for rootvg needs to be
mirrored before we proceed to unmirror the VG. However, if any of the LVs
(application LVs / log device) in a VG is not mirrored, kindly encage CDT team to do
the needful.

Most importantly never try to do mirror a LV from the bad disk to the new disk if any
LV is not mirrored.

a) Procedure to unmirror the rootvg if all the LVs in a rootvg are mirrored.
b) Procedure to unmirror the rootvg (assumes all the LVs in a rootvg are mirrored
and we are having user defined dump devices)
c) Procedure to unmirror the rootvg if all the LVs in a rootvg are not mirrored .
(ex: log device is not mirrored)
d) Procedure to unmirror the datavg if all the LVs in a datavg are mirrored.
e) Procedure to unmirror the datavg if all the LVs in a datavg are not mirrored.
(ex: log device is not mirrored)
Procedure to unmirror the rootvg (assumes all the LVs in a rootvg are
mirrored and assumes default paging device is a dump device)

1. Complete the prerequisite (page one) before preceding the next step.

2. Check the lsvg –l rootvg to see all the LVs are mirrored,

lsvg –l rootvg

3. Break the mirror in a rootvg

unmirrorvg rootvg

4. Clear the boot record on the bad disk.

chpv –c Bad_PV_Name

5. Remove the bad disk from the volume group

reducevg rootvg Bad_PV_Name

Note: Kindly make sure BLV is present in the good disk. You can run “lslv –m hd5” to
make sure it is present. Moreover, the good disk should be available in the first in
bootlist order as well.

6. Remove the bad disk from OS level

rmdev –dl Bad_PV_Name

Once you have removed a bad disk from OS successfully, kindly inform our RSM team
member who is on call to replace the faulty disk. Please provide the following
information as well.

Hostname / rackname / bad_PV_name / bad_PV_hardware_location

7. Run “cfgmgr” command for the OS to recognize the new disk. Once the new disk is
available to the system, you can run “lscfg –vl PV_name” and take a output. Now, you
can compare the serial number of the new disk with old one and it should be
different.

Once the faulty disk is replaced and found, run a log repair action against the new
device

diag  Advanced Diagnostics Routines  Problem Determination 

select the correct device  esc+7

8. Add the new disk to the volume group.

extendvg rootvg PV_name

9. Now mirror the volume group.

 mirrorvg rootvg

It may take some time to do a mirroring with the new disk. During the time, volume
group will be in a locked state.

10. Once mirroring is successfully completed, create a boot record on the newly
added disk.

bosboot -ad /dev/PV_name

11. Alter the bootlist order with the new disk.

bootlist -m normal existing_PV new_PV entX

12. Run “lslv –m hd5” to make sure that BLV is present on both good disks in a
volume group.

lslv –m hd5
Procedure to unmirror the rootvg (assumes all the LVs in a rootvg are
mirrored and we are having user defined dump devices)

1. Complete the prerequisite (page one) before preceding the next step.

2. Check the lsvg –l rootvg to see all the LVs are mirrored and not down the dump
devie names and on which PV it is located.

lsvg –l rootvg

Note: Dump devices will not be mirrored.

3. Create a new dump device with different one from the existing one on a good PV

mklv –y dumpX –t sysdump rootvg 4 hdiskX

4. Make the new dump device as primary/secondary on a good PV

sysdumpdev –s /dev/dumpX  to make it as a secondary dump device

sysdumpdev –p /dev/dumpX  to make it as a primary dump device

5. Remove the dump device from the bad PV

rmlv dump_lv_name

6. Break the mirror in a rootvg

unmirrorvg rootvg

7. Clear the boot record on the bad disk.

chpv –c Bad_PV_Name

8. Remove the bad disk from the volume group

reducevg rootvg Bad_PV_Name

Note: Kindly make sure BLV is present in the good disk. You can run “lslv –m hd5” to
make sure it is present. Moreover, the good disk should be available in the first in
bootlist order as well.

9. Remove the bad disk from OS level

rmdev –dl Bad_PV_Name

Once you have removed a bad disk from OS successfully, kindly inform our RSM team
member who is on call to replace the faulty disk. Please provide the following
information as well.
 Hostname / rackname / bad_PV_name / bad_PV_hardware_location
10. Run “cfgmgr” command for the OS to recognize the new disk. Once the new disk
is available to the system, you can run “lscfg –vl PV_name” and take a output. Now,
you can compare the serial number of the new disk with old one and it should be
different.

Once the faulty disk is replaced and found, run a log repair action against the new
device

diag  Advanced Diagnostics Routines  Problem Determination 

select the correct device  esc+7

11. Add the new disk to the volume group.

extendvg rootvg PV_name

12. Now mirror the volume group.

mirrorvg rootvg

It may take some time to do a mirroring with the new disk. During the time, volume
group will be in a locked state.

13. Once mirroring is successfully completed, create a boot record on the newly
added disk.

bosboot -ad /dev/PV_name

14. Alter the bootlist order with the new disk.

bootlist -m normal existing_PV new_PV entX

15. Create a new dump device on a newly added PV in the VG

repeat the steps 3,4,5 appropriately

16. Run “lslv –m hd5” to make sure that BLV is present on both good disks in a
volume group.

lslv –m hd5
Procedure to unmirror the rootvg if all the LVs in a rootvg are not mirrored
properly. ( ex. log device is not mirrored)

1. Complete the prerequisite (page one) before preceding the next step.

2. Check the lsvg –l rootvg to see all the LVs are mirrored,

lsvg –l rootvg

Note:

Kindly encage CDT team if all the LVs (application related) are not mirrored. In this
case, the loglv is not mirrored, however application LVs are mirrored and the loglv is
present on a bad disk, hence, we require a application downtime to proceed further.

3. Once we have got a downtime (application will be shutdown by the CDT team), we
need to unmount the filesystems related to the log device.

unmount /fs_name

4. Once unmounting the filesystems is completed, kindly remove a log lv from the
bad disk.

rmlv log_lv_name

5. Now, create a new log lv with the same name on a good PV

mklv –y logX -t jfs2log rootvg 1 hdiskX

6. Format the newly created log device.

logform /dev/logX

7. Mount all the filesystems which are unmounted on step 3.

mount /fs_name

8. Break the mirror in a rootvg

unmirrorvg rootvg

9. Clear the boot record on the bad disk.

chpv –c Bad_PV_Name

10 Remove the bad disk from the volume group

reducevg rootvg Bad_PV_Name

Note: Kindly make sure BLV is present in the good disk. You can run “lslv –m hd5” to
make sure it is present. Moreover, the good disk should be available in the first in
bootlist order as well.

11. Remove the bad disk from OS level

rmdev –dl Bad_PV_Name

Once you have removed a bad disk from OS successfully, kindly inform our RSM team
member who is on call to replace the faulty disk. Please provide the following
information as well.

Hostname / rackname / bad_PV_name / bad_PV_hardware_location

12. Run “cfgmgr” command for the OS to recognize the new disk. Once the new disk
is available to the system, you can run “lscfg –vl PV_name” and take a output. Now,
you can compare the serial number of the new disk with old one and it should be
different.

Once the faulty disk is replaced and found, run a log repair action against the new
device

diag  Advanced Diagnostics Routines  Problem Determination 

select the correct device  esc+7

13. Add the new disk to the volume group.

extendvg rootvg PV_name

14. Now mirror the volume group.

mirrorvg rootvg

It may take some time to do a mirroring with the new disk. During the time, volume
group will be in a locked state.

15. Once mirroring is successfully completed, create a boot record on the newly
added disk.

bosboot -ad /dev/PV_name

16. Alter the bootlist order with the new disk.

bootlist -m normal existing_PV new_PV entX

17. Run “lslv –m hd5” to make sure that BLV is present on both good disks in a
volume group.

lslv –m hd5
Procedure to unmirror the datavg if all the LVs in a datavg are mirrored
properly

1. Complete the prerequisite (page one) before preceding the next step.

2. Check the lsvg –l datavg to see all the LVs are mirrored,

lsvg –l datavg

3. Break the mirror in a datavg

unmirrorvg datavg

4. Clear the boot record on the bad disk.

chpv –c Bad_PV_Name

5. Remove the bad disk from the volume group

reducevg datavg Bad_PV_Name

6. Remove the bad disk from OS level

rmdev –dl Bad_PV_Name

Once you have removed a bad disk from OS successfully, kindly inform our RSM team
member who is on call to replace the faulty disk. Please provide the following
information as well.

Hostname / rackname / bad_PV_name / bad_PV_hardware_location

7. Run “cfgmgr” command for the OS to recognize the new disk. Once the new disk is
available to the system, you can run “lscfg –vl PV_name” and take a output. Now, you
can compare the serial number of the new disk with old one and it should be
different.

Once the faulty disk is replaced and found, run a log repair action against the new
device

diag  Advanced Diagnostics Routines  Problem Determination 

select the correct device  esc+7

8. Add the new disk to the volume group.

extendvg datavg PV_name

9. Now mirror the volume group.

 mirrorvg datavg

It may take some time to do a mirroring with the new disk. During the time, volume
group will be in a locked state.

Procedure to unmirror the datavg if all the LVs in a datavg are not mirrored
properly. ( ex. log device is not mirrored)

1. Complete the prerequisite (page one) before preceding the next step.

2. Check the lsvg –l datavg to see all the LVs are mirrored,

lsvg –l datavg

Note:
Kindly encage CDT team if all the LVs (application related) are not mirrored. In this
case, the loglv is not mirrored, however application LVs are mirrored and the loglv is
present on a bad disk, hence, we require a application downtime to proceed further.

3. Once we have got a downtime (application will be shutdown by the CDT team), we
need to unmount the filesystems related to the log device.

unmount /fs_name

4. Once unmounting the filesystems is completed, kindly remove a log lv from the
bad disk.

rmlv log_lv_name

5. Now, create a new log lv with the same name on a good PV

mklv –y logX -t jfs2log datavg 1 hdiskX

6. Format the newly created log device.

logform /dev/logX

7. Mount all the filesystems which are unmounted on step 3.

mount /fs_name

8. Break the mirror in a rootvg

unmirrorvg datavg

9. Remove the bad disk from the volume group

reducevg datavg Bad_PV_Name

11. Remove the bad disk from OS level

 rmdev –dl Bad_PV_Name

Once you have removed a bad disk from OS successfully, kindly inform our RSM team
member who is on call to replace the faulty disk. Please provide the following
information as well.
Hostname / rackname / bad_PV_name / bad_PV_hardware_location

12. Run “cfgmgr” command for the OS to recognize the new disk. Once the new disk
is available to the system, you can run “lscfg –vl PV_name” and take a output. Now,
you can compare the serial number of the new disk with old one and it should be
different.

Once the faulty disk is replaced and found, run a log repair action against the new
device

diag  Advanced Diagnostics Routines  Problem Determination 

select the correct device  esc+7

13. Add the new disk to the volume group.

extendvg datavg PV_name

14. Now mirror the volume group.

mirrorvg datavg

It may take some time to do a mirroring with the new disk. During the time, volume
group will be in a locked state.