Adding a Node to a Cluster

Product: Oracle Server - Enterprise Edition Product Version: 10gR2

This document was authored by an Oracle customer and has not been subject to an independent technical
review.

Abstract

This document depicts the steps for adding the new node to the existing two node
cluster. We have two node RAC build system linux1 & linux2. We need to add the new
node to this cluster setup to make it as 3Node-RAC. The new node which is going to be
added will be linux3. Assumption that the new node has got all the OS pre-requisites
satisfied like IP-Bonding (Private, public & virtual), kernel parameter's, OS rpm's,
making the shared ocr,voting,asm disks visible on the new node.

Document History

Author : SRAVAN DALAVAI

Title : Mr.
Date Created: 10-Apr-2009
Company : Sierra Atlantic s/w services Ltd. Hyderabad.

Title:- Adding a RAC node to a 10gR2 Cluster

Summary

This presentation will provide detail steps & screenshots for an Oracle DBA to
add new rac node to existing 10gR2 RAC database.

The most critical steps that need to be followed are:-

I. Pre-installation checks.
II. Adding an Oracle clusterware home to new nodes using OUI in interactive mode.
III. Adding an Oracle home to new nodes using OUI in interactive mode.
IV. Reconfigure listener on new node using NETCA.
V. Adding ASM instance to new nodes manually.

Part I: Pre-installation checks

1
 ü Verify cluster healthy on existing nodes (ocrcheck, cluvfy)
ü Check OS version, Kernel parameters, /etc/hosts file and ensure they are
identical on all nodes
ü Cross check ssh login on all nodes
ü Cross ping hostname on all nodes
ü Set the same environment on the new node as that of the existing nodes (Copy
ENV/bash_profile file from existing nodes to new one).
ü Ensure ocr, css files and shared ASM disk are visible from new node and have
right permissions.
ü Backup crs, ocr and oraInventory on existing nodes
ü If the CRS Home is owned by root and you are on a version < 10.1.0.4, change
the ownership of the CRS Home directories on all nodes to the Oracle user so
that OUI can read and write to these directories.
ü Comment out two lines below in $ORA_CRS_HOME/bin/srvctl on existing nodes
o #LD_ASSUME_KERNEL=2.4.19
o #export LD_ASSUME_KERNEL
Add "unset LD_ASSUME_KERNEL" in $ORA_CRS_HOME/bin/vipca line 126

Step9 and 10 are 10.2.0.3 specific. The issue is fixed on 10.2.0.4.

Please reference Oracle metalink Doc ID: Note:414163.1 -- 10gR2 RAC
Install issues on Oracle EL5 or RHEL5 or SLES10 (VIPCA Failures)
ü Ensure $ORACLE_BASE directory exist on new node with correct permission.

Part II: Adding an Oracle Clusterware Home to a New Node Using OUI in
Interactive Mode

Ø Set the DISPLAY, other environment variables and run the addNode.sh script
from $ORA_CRS_HOME/oui/bin on one of the existing nodes as the oracle user
in VNC session.
Example:
. ./.bash_profile
DISPLAY=ipaddress:0.0; export DISPLAY
cd $ORA_CRS_HOME/oui/bin
./addNode.sh

2
Ø The Oracle Universal Installer (OUI) displays the Node Selection Page

Ø Enter the node that you want to add and verify the entries that OUI displays on
the Summary Page click Next.

3
Ø Monitor the progress of the copy crs home to new node and verify the total size
of the CRS directory

Ø Execute configuration scripts by root users

4
a. Run the orainstRoot.sh script on the new node if OUI prompts you to do so.

linux3:/u01/app/oracle/oraInventory # ./orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770.
Changing groupname of /u01/app/oracle/oraInventory to oinstall.
The execution of the script is complete

b. Run the rootaddNode.sh script from the $ORA_CRS_HOME/install/ directory

on the node from which you are running OUI.

linux1:/u01/app/oracle/product/10.2.0/crs_1/install # ./rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 3: linux3 linux3-priv linux3
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/u01/app/oracle/product/10.2.0/crs_1/bin/srvctl add nodeapps -n linux3 -A
linux3-vip.us.dell.com/255.255.240.0/eth0 -o
/u01/app/oracle/product/10.2.0/crs_1

c. Run the root.sh script on the new node from $ORA_CRS_HOME to start Oracle
Clusterware on the new node.

5
linux3:/u01/app/oracle/product/10.2.0/crs_1 # ./root.sh
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
OCR LOCATIONS =/u02/oradata/ocr1,/u02/oradata/ocr2
OCR backup directory
'/u01/app/oracle/product/10.2.0/crs_1/cdata/dfsddeamercrs' does not exist.
Creating now
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: linux1 linux1-priv linux1
node 2: linux2 linux2-priv linux2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster configuration.Oracle
Cluster Registry for cluster has already been initialized Startup will be queued to
init within 30 seconds.
Adding daemons to inittab Expecting the CRS daemons to be up within 600
seconds.
CSS is active on these nodes.
linux1
linux2
linux3
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps

Creating VIP application resource on (0) nodes.

Creating GSD application resource on (0) nodes.
Creating ONS application resource on (0) nodes.
Starting VIP application resource on (2) nodes.........

6
 Starting GSD application resource on (2) nodes.........
Starting ONS application resource on (2) nodes.........
Done.

Ø Verify crs is started on new node and nodeapps are started except for listener,
and then exit OUI.

Ø Obtain the remote port identifier, which you need to know for the next step, by
running the following command on the existing node from the
$ORA_CRS_HOME/opmn/conf directory:
$ cat $ORA_CRS_HOME/opmn/conf/ons.config
localport=6113
remoteport=6201
loglevel=3
useocr=on

Ø From the $ORA_CRS_HOME/bin directory on an existing node, run the Oracle

Notification Service (RACGONS) utility as in the following example where
remote_port is the port number from the previous step and node2 is the name of
the node that you are adding:
./racgons add_config New_Node:Remote_Port
Ex:
$ ./racgons add_config linux3:6201

7
Part III: Adding an Oracle Home to a New Node Using OUI in Interactive
Mode

Ensure that you have successfully installed Oracle with the Oracle RAC software on at
least one node in your cluster environment. To use these procedures as shown, your
$ORACLE_HOME environment variable must identify your successfully installed Oracle
home.

Ø Go to $ORACLE_HOME/oui/bin and run the addNode.sh script on node1.

. ./ .profile
DISPLAY=ipaddress:0.0; export DISPLAY
cd $ORACLE_HOME/oui/bin
./addNode.sh

Ø When OUI displays the Node Selection Page, select the node to be added and
click Next.

Ø Verify the entries that OUI displays on the Summary Page and click Next.

8
Ø Run the root.sh script on the new node from Oracle_home when OUI prompts
you to do so.

linux3:/u01/app/oracle/product/10.2.0/db_1 # ./root.sh

9
 Running Oracle10 root.sh script...

The following environment variables are set as:

ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/10.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.

Part IV: Reconfigure listener on new node

On the new node, run the Oracle Net Configuration Assistant (NETCA) to add a
Listener.
export DISPLAY=ipaddress:0.0
./netca &

10
11
 Waiting for listener configuration to complete. Verify the listener status using
srvctl utility.

Part V: Add ASM instance on new node manually

12
 a) Add ASM entry in /etc/oratab on new node
+ASM3:/u01/app/oracle/product/10.2.0/db_1:N

b) Create init+ASM3.ora and password file for new ASM3

cd /u01/app/oracle/product/10.2.0/db_1/dbs
mv init+ASM1.ora init+ASM3.ora
mv orapw+ASM1 orapw+ASM3

c) Create admin directories for ASM instance

mkdir /u01/app/oracle/admin/+ASM/udump
cdump
hdump
bdump
pfile
d) Copy init.ora to new node

cd /u01/app/oracle/admin/+ASM/pfile
scp linux1:/u01/app/oracle/admin/+ASM/pfile/init.ora .

e) Add new ASM instance in

/u01/app/oracle/admin/+ASM/pfile/init+ASM*.ora on all RAC nodes
+ASM3.instance_number=3
f) Add ASM to cluster
srvctl add asm -n linux3 -i +ASM3 –o
/u01/app/oracle/product/10.2.0/db_1
g) Start ASM instance
srvctl start asm -n linux3

After bringing up the asm instance you can bring up the database instance on the
newly added node.

References:
Note ID: 270512.1 Adding a Node to a 10g RAC Cluster

By Sravan Dalavai,
Oracle Apps DBA,
Sierra Atlantic s/w services
Sravan.dalavai@sierraatlantic.com

13
Disclaimer: The information in this document is the opinion of the author, not of Oracle Corporation. Any content, materials,
information or software downloaded or otherwise obtained through the use of the site is done at your own discretion and
risk. Oracle shall have no responsibility for any damage to your computer system or loss of data that results from the
download of any content, materials, information or software. This document is provided for information purposes only and
the contents hereof are subject to change without notice. Oracle does not warrant that this document is error-free, nor
does it provide any other warranties or conditions, whether expressed orally or implied in law, including implied warranties
and conditions of merchantability or fitness for a particular purpose. Oracle specifically disclaims any liability with respect
to this document and no contractual obligations are formed either directly or indirectly by this document.

14