Escolar Documentos
Profissional Documentos
Cultura Documentos
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireWire (Continued)
For development and testing only; production deployments will not be supported!
1 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
# cat /proc/sys/kernel/shmmax 33554432 The default value for SHMMAX is 32MB. This size is often too small to configure the Oracle SGA. I generally set the SHMMAX parameter to 2GB using the following methods: You can alter the default setting for SHMMAX without rebooting the machine by making the changes directly to the /proc file system (/proc/sys/kernel/shmmax ) by using the following command: # sysctl -w kernel.shmmax=2147483648 You should then make this change permanent by inserting the kernel parameter in the /etc/sysctl.conf startup file: # echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf Setting SHMMNI We now look at the SHMMNI parameters. This kernel parameter is used to set the maximum number of shared memory segments system wide. The default value for this parameter is 4096. You can determine the value of SHMMNI by performing the following: # cat /proc/sys/kernel/shmmni 4096 The default setting for SHMMNI should be adequate for your Oracle RAC 10g Release 2 installation. Setting SHMALL Finally, we look at the SHMALL shared memory kernel parameter. This parameter controls the total amount of shared memory (in pages) that can be used at one time on the system. In short, the value of this parameter should always be at least: ceil(SHMMAX/PAGE_SIZE) The default size of SHMALL is 2097152 and can be queried using the following command: # cat /proc/sys/kernel/shmall 2097152 The default setting for SHMALL should be adequate for our Oracle RAC 10g Release 2 installation. (Note: The page size in Red Hat Linux on the i386 platform is 4,096 bytes. You can, however, use bigpages which supports the configuration of larger memory page sizes.) Setting Semaphores Now that you have configured our shared memory settings, it is time to configure your semaphores. The best way to describe a "semaphore" is as a counter that is used to provide synchronization between processes (or threads within a process) for shared resources like shared memory. Semaphore sets are supported in UNIX System V where each one is a counting semaphore. When an application requests semaphores, it does so using "sets". To determine all semaphore limits, use the following: # ipcs -ls ------ Semaphore Limits -------max number of arrays = 128 max semaphores per array = 250 max semaphores system wide = 32000 max ops per semop call = 32 semaphore max value = 32767 You can also use the following command: # cat /proc/sys/kernel/sem 250 32000 32 128 Setting SEMMSL The SEMMSL kernel parameter is used to control the maximum number of semaphores per semaphore set. Oracle recommends setting SEMMSL to the largest PROCESS instance parameter setting in the init.ora file for all databases on the Linux system plus 10. Also, Oracle recommends setting the SEMMSL to a value of no less than 100. Setting SEMMNI The SEMMNI kernel parameter is used to control the maximum number of semaphore sets in the entire Linux system. Oracle recommends setting the SEMMNI to a value of no less than 100. Setting SEMMNS The SEMMNS kernel parameter is used to control the maximum number of semaphores (not semaphore sets) in the entire Linux system.
2 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
Oracle recommends setting the SEMMNS to the sum of the PROCESSES instance parameter setting for each database on the system, adding the largest PROCESSES twice, and then finally adding 10 for each Oracle database on the system. Use the following calculation to determine the maximum number of semaphores that can be allocated on a Linux system. It will be the lesser of: SEMMNS -or(SEMMSL * SEMMNI)
Setting SEMOPM The SEMOPM kernel parameter is used to control the number of semaphore operations that can be performed per semop system call. The semop system call (function) provides the ability to do operations for multiple semaphores with one semop system call. A semaphore set can have the maximum number of SEMMSL semaphores per semaphore set and is therefore recommended to set SEMOPM equal to SEMMSL. Oracle recommends setting the SEMOPM to a value of no less than 100. Setting Semaphore Kernel Parameters Finally, we see how to set all semaphore parameters using several methods. In the following, the only parameter I care about changing (raising) is SEMOPM. All other default settings should be sufficient for our example installation. You can alter the default setting for all semaphore settings without rebooting the machine by making the changes directly to the /proc file system (/proc/sys/kernel/sem ) by using the following command: # sysctl -w kernel.sem="250 32000 100 128" You should then make this change permanent by inserting the kernel parameter in the /etc/sysctl.conf startup file: # echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf Setting File Handles When configuring our Red Hat Linux server, it is critical to ensure that the maximum number of file handles is sufficiently large. The setting for file handles denotes the number of open files that you can have on the Linux system. Use the following command to determine the maximum number of file handles for the entire system: # cat /proc/sys/fs/file-max 102563 Oracle recommends that the file handles for the entire system be set to at least 65536. You can alter the default setting for the maximum number of file handles without rebooting the machine by making the changes directly to the /proc file system (/proc/sys/fs/file-max) using the following: # sysctl -w fs.file-max=65536 You should then make this change permanent by inserting the kernel parameter in the /etc/sysctl.conf startup file: # echo "fs.file-max=65536" >> /etc/sysctl.conf You can query the current usage of file handles by using the following: # cat /proc/sys/fs/file-nr 825 0 65536 The file-nr file displays three parameters: total allocated file handles, currently used file handles, and maximum file handles that can be allocated. (Note: If you need to increase the value in /proc/sys/fs/file-max, then make sure that the ulimit is set properly. Usually for 2.4.20 it is set to unlimited. Verify the ulimit setting my issuing the ulimit command: # ulimit unlimited
3 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
health of the system. If the system hangs or pauses, the timer resets the node. The hangcheck-timer module uses the Time Stamp Counter (TSC) CPU register, which is incremented at each clock signal. The TCS offers much more accurate time measurements because this register is updated by the hardware automatically. Much more information about the hangcheck-timer project can be found here. Installing the hangcheck-timer.ko Module The hangcheck-timer was originally shipped only by Oracle; however, this module is now included with Red Hat Linux starting with kernel versions 2.4.9-e.12 and higher. If you followed the steps in Section 8 ("Obtain & Install New Linux Kernel / FireWire Modules"), then the hangcheck-timer is already included for you. Use the following to confirm: # find /lib/modules -name "hangcheck-timer.ko" /lib/modules/2.6.9-11.0.0.10.3.EL/kernel/drivers/char/hangcheck-timer.ko /lib/modules/2.6.9-22.EL/kernel/drivers/char/hangcheck-timer.ko In the above output, we care about the hangcheck timer object (hangcheck-timer.ko) in the /lib/modules/2.6.9-11.0.0.10.3.EL/kernel/drivers/char directory. Configuring and Loading the hangcheck-timer Module There are two key parameters to the hangcheck-timer module: hangcheck-tick: This parameter defines the period of time between checks of system health. The default value is 60 seconds; Oracle recommends setting it to 30 seconds. hangcheck-margin: This parameter defines the maximum hang delay that should be tolerated before hangcheck-timer resets the RAC node. It defines the margin of error in seconds. The default value is 180 seconds; Oracle recommends setting it to 180 seconds. NOTE: The two hangcheck-timer module parameters indicate how long a RAC node must hang before it will reset the system. A node reset will occur when the following is true: system hang time > (hangcheck_tick + hangcheck_margin) Configuring Hangcheck Kernel Module Parameters Each time the hangcheck-timer kernel module is loaded (manually or by Oracle), it needs to know what value to use for each of the two parameters we just discussed: (hangcheck-tick and hangcheck-margin). These values need to be available after each reboot of the Linux server. To do that, make an entry with the correct values to the /etc/modprobe.conf file as follows: # su # echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf Each time the hangcheck-timer kernel module gets loaded, it will use the values defined by the entry I made in the /etc/modprobe.conf file. Manually Loading the Hangcheck Kernel Module for Testing Oracle is responsible for loading the hangcheck-timer kernel module when required. For that reason, it is not required to perform a modprobe or insmod of the hangcheck-timer kernel module in any of the startup files (i.e. /etc/rc.local). It is only out of pure habit that I continue to include a modprobe of the hangcheck-timer kernel module in the /etc/rc.local file. Someday I will get over it, but realize that it does not hurt to include a modprobe of the hangcheck-timer kernel module during startup. So to keep myself sane and able to sleep at night, I always configure the loading of the hangcheck-timer kernel module on each startup as follows: # echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local (Note: You don't have to manually load the hangcheck-timer kernel module using modprobe or insmod after each reboot. The hangcheck-timer module will be loaded by Oracle automatically when needed.) Now, to test the hangcheck-timer kernel module to verify it is picking up the correct parameters we defined in the /etc/modprobe.conf file, use the modprobe command. Although you could load the hangcheck-timer kernel module by passing it the appropriate parameters (e.g. insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180 ), we want to verify that it is picking up the options we set in the /etc/modprobe.conf file. To manually load the hangcheck-timer kernel module and verify it is using the correct values defined in the /etc/modprobe.conf file, run the following command:
# su # modprobe hangcheck-timer # grep Hangcheck /var/log/messages | tail -2 Sep 27 23:11:51 linux2 kernel: Hangcheck: starting hangcheck timer 0.5.0 (tick is 30 seconds, margin is 180 seconds)
4 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
cluster. Therefore you should be able to run r* commands like rsh, rcp, and rlogin on the Linux server you will be running the Oracle installer from, against all other Linux servers in the cluster without a password. The rsh daemon validates users using the /etc/hosts.equiv file or the .rhosts file found in the user's (oracle's) home directory. (The use of rcp and rsh are not required for normal RAC operation. However rcp and rsh should be enabled for RAC and patchset installation.) Oracle added support in Oracle RAC 10g Release 1 for using the Secure Shell (SSH) tool suite for setting up user equivalence. This article, however, uses the older method of rcp for copying the Oracle software to the other nodes in the cluster. When using the SSH tool suite, the scp (as opposed to the rcp) command would be used to copy the software in a very secure manner. First, let's make sure that we have the rsh RPMs installed on each node in the RAC cluster: # rpm -q rsh rsh-server rsh-0.17-25.3 rsh-server-0.17-25.3 From the above, we can see that we have the rsh and rsh-server installed. Were rsh not installed, we would run the following command from the CD where the RPM is located: # su # rpm -ivh rsh-0.17-25.3.i386.rpm rsh-server-0.17-25.3.i386.rpm To enable the "rsh" and "rlogin" services, the "disable" attribute in the /etc/xinetd.d/rsh file must be set to "no" and xinetd must be reloaded. Do that by running the following commands on all nodes in the cluster: # su # chkconfig rsh on # chkconfig rlogin on # service xinetd reload Reloading configuration: [
OK
To allow the "oracle" UNIX user account to be trusted among the RAC nodes, create the /etc/hosts.equiv file on all nodes in the cluster: # # # # su touch /etc/hosts.equiv chmod 600 /etc/hosts.equiv chown root.root /etc/hosts.equiv
Now add all RAC nodes to the /etc/hosts.equiv file similar to the following example for all nodes in the cluster: # cat /etc/hosts.equiv +linux1 oracle +linux2 oracle +int-linux1 oracle +int-linux2 oracle Note: In the above example, the second field permits only the oracle user account to run rsh commands on the specified nodes. For security reasons, the /etc/hosts.equiv file should be owned by root and the permissions should be set to 600. In fact, some systems will only honor the content of this file if the owner is root and the permissions are set to 600. Before attempting to test your rsh command, ensure that you are using the correct version of rsh. By default, Red Hat Linux puts /usr/kerberos/sbin at the head of the $PATH variable. This will cause the Kerberos version of rsh to be executed. I will typically rename the Kerberos version of rsh so that the normal rsh command is being used. Use the following: # su # which rsh /usr/kerberos/bin/rsh # mv /usr/kerberos/bin/rsh /usr/kerberos/bin/rsh.original # mv /usr/kerberos/bin/rcp /usr/kerberos/bin/rcp.original # mv /usr/kerberos/bin/rlogin /usr/kerberos/bin/rlogin.original # which rsh /usr/bin/rsh You should now test your connections and run the rsh command from the node that will be performing the Oracle Clusterware and 10g RAC installation. I will be using the node linux1 to perform all installs so this is where I will run the following commands from: # su - oracle $ rsh linux1 ls -l /etc/hosts.equiv -rw------- 1 root root 68 Sep 27 23:37 /etc/hosts.equiv $ rsh int-linux1 ls -l /etc/hosts.equiv -rw------- 1 root root 68 Sep 27 23:37 /etc/hosts.equiv $ rsh linux2 ls -l /etc/hosts.equiv -rw------- 1 root root 68 Sep 27 23:37 /etc/hosts.equiv
5 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
6 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
# +---------------------------------------------------------+ # | FILE HANDLES | # ----------------------------------------------------------+ fs.file-max=65536 /etc/hosts (All machine/IP entries for nodes in our RAC cluster.) # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost # Public Network - (eth0) 192.168.1.100 linux1 192.168.1.101 linux2 # Private Interconnect - (eth1) 192.168.2.100 int-linux1 192.168.2.101 int-linux2 # Public Virtual IP (VIP) addresses for - (eth0) 192.168.1.200 vip-linux1 192.168.1.201 vip-linux2 192.168.1.106 melody 192.168.1.102 alex 192.168.1.105 bartman /etc/hosts.equiv (Allow logins to each node as the oracle user account without the need for a password.) +linux1 oracle +linux2 oracle +int-linux1 oracle +int-linux2 oracle /etc/rc.local (Loading the hangcheck-timer kernel module.) #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local # # # # +---------------------------------------------------------+ | HANGCHECK TIMER | | (I do not believe this is required, but doesn't hurt) | ----------------------------------------------------------+
/sbin/modprobe hangcheck-timer
7 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
compat-libstdc++-296-2.96-132.7.2 openmotif-2.2.3-9.RHEL4.1 setarch-1.6-1 To query package information (gcc and glibc-devel for example), use the "rpm -q <PackageName> [, <PackageName>] " command as follows: # rpm -q gcc glibc-devel gcc-3.4.3-22.1 glibc-devel-2.3.4-2.9 If you need to install any of the above packages, use "rpm -Uvh <PackageName.rpm> ". For example, to install the GCC 3.2.3-24 package, use: # rpm -Uvh gcc-3.4.3-22.1.i386.rpm Reboot the System If you made any changes to the O/S, reboot all nodes in the cluster before attempting to install any of the Oracle components!!! # init 6
8 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
1:ocfs2-tools ########################################### [ 33%] 2:ocfs2-2.6.9-11.0.0.10.3########################################### [ 67%] 3:ocfs2console ########################################### [100%] Disable SELinux (RHEL4 U2 Only) RHEL4 U2 users (CentOS 4.2 is based on RHEL4 U2) are advised that OCFS2 currently does not work with SELinux enabled. If you are using RHEL4 U2 (which includes you, since you are using CentOS 4.2 here) you will need to disable SELinux (using tool system-config-securitylevel ) to get the O2CB service to execute. To disable SELinux, run the "Security Level Configuration" GUI utility: # /usr/bin/system-config-securitylevel &
Figure 6 Security Level Configuration Opening Screen Now, click the SELinux tab and check off the "Enabled" checkbox. After clicking [OK], you will be presented with a warning dialog. Simply acknowledge this warning by clicking "Yes". Your screen should now look like the following after disabling the SELinux option:
9 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
Figure 7: SELinux Disabled After making this change on both nodes in the cluster, each node will need to be rebooted to implement the change: # init 6 Configure OCFS2 The next step is to generate and configure the /etc/ocfs2/cluster.conf file on each node in the cluster. The easiest way to accomplish this is to run the GUI tool ocfs2console. In this section, we will not only create and configure the /etc/ocfs2/cluster.conf file using ocfs2console, but will also create and start the cluster stack O2CB. When the /etc/ocfs2/cluster.conf file is not present, (as will be the case in our example), the ocfs2console tool will create this file along with a new cluster stack service (O2CB) with a default cluster name of ocfs2. This will need to be done on all nodes in the cluster as the root user account: $ su # ocfs2console & This will bring up the GUI as shown below:
10 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
Figure 8 ocfs2console GUI Using the ocfs2console GUI tool, perform the following steps: 1. Select [Cluster] -> [Configure Nodes...] . This will start the OCFS Cluster Stack (Figure 9) and bring up the "Node Configuration" dialog. 2. On the "Node Configuration" dialog, click the [Add] button. This will bring up the "Add Node" dialog. In the "Add Node" dialog, enter the Host name and IP address for the first node in the cluster. Leave the IP Port set to its default value of 7777. In my example, I added both nodes using linux1 / 192.168.1.100 for the first node and linux2 / 192.168.1.101 for the second node. Click [Apply] on the "Node Configuration" dialog - All nodes should now be "Active" as shown in Figure 10. Click [Close] on the "Node Configuration" dialog. 3. After verifying all values are correct, exit the application using [File] -> [Quit]. This needs to be performed on all nodes in the cluster.
Figure 9. Starting the OCFS2 Cluster Stack The following dialog show the OCFS2 settings I used for the node linux1 and linux2:
11 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
Figure 10 Configuring Nodes for OCFS2 After exiting the ocfs2console, you will have a /etc/ocfs2/cluster.conf similar to the following. This process needs to be completed on all nodes in the cluster and the OCFS2 configuration file should be exactly the same for all of the nodes: node: ip_port = 7777 ip_address = 192.168.1.100 number = 0 name = linux1 cluster = ocfs2 node: ip_port = 7777 ip_address = 192.168.1.101 number = 1 name = linux2 cluster = ocfs2 cluster: node_count = 2 name = ocfs2 O2CB Cluster Service Before we can do anything with OCFS2 like formatting or mounting the file system, we need to first have OCFS2's cluster stack, O2CB, running (which it will be as a result of the configuration process performed above). The stack includes the following services: NM: Node Manager that keep track of all the nodes in the cluster.conf HB: Heart beat service that issues up/down notifications when nodes join or leave the cluster TCP: Handles communication between the nodes DLM: Distributed lock manager that keeps track of all locks, its owners and status CONFIGFS: User space driven configuration file system mounted at /config DLMFS: User space interface to the kernel space DLM All of the above cluster services have been packaged in the o2cb system service (/etc/init.d/o2cb). Here is a short listing of some of the more useful commands and options for the o2cb system service. /etc/init.d/o2cb status Module "configfs": Not loaded Filesystem "configfs": Not mounted Module "ocfs2_nodemanager": Not loaded Module "ocfs2_dlm": Not loaded Module "ocfs2_dlmfs": Not loaded Filesystem "ocfs2_dlmfs": Not mounted Note that with this example, all of the services are not loaded. I did an "unload" right before executing the "status" option. If you were to check the status of the o2cb service immediately after configuring OCFS using ocfs2console utility, they would all be loaded.
12 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
/etc/init.d/o2cb load Loading module "configfs": OK Mounting configfs filesystem at /config: OK Loading module "ocfs2_nodemanager": OK Loading module "ocfs2_dlm": OK Loading module "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Loads all OCFS modules. /etc/init.d/o2cb online ocfs2 Starting cluster ocfs2: OK The above command will online the cluster we created, ocfs2. /etc/init.d/o2cb offline ocfs2 Unmounting ocfs2_dlmfs filesystem: OK Unloading module "ocfs2_dlmfs": OK Unmounting configfs filesystem: OK Unloading module "configfs": OK The above command will offline the cluster we created, ocfs2. /etc/init.d/o2cb unload Cleaning heartbeat on ocfs2: OK Stopping cluster ocfs2: OK The above command will unload all OCFS modules. Configure O2CB to Start on Boot You now need to configure the on-boot properties of the OC2B driver so that the cluster stack services will start on each boot. All the tasks within this section will need to be performed on both nodes in the cluster. Note: At the time of writing this guide, OCFS2 contains a bug wherein the driver does not get loaded on each boot even after configuring the on-boot properties to do so. After attempting to configure the on-boot properties to start on each boot according to the official OCFS2 documentation, you will still get the following error on each boot: ... Mounting other filesystems: mount.ocfs2: Unable to access cluster service Cannot initialize cluster mount.ocfs2: Unable to access cluster service Cannot initialize cluster [FAILED] ... Red Hat changed the way the service is registered between chkconfig-1.3.11.2-1 and chkconfig-1.3.13.2-1. The O2CB script used to work with the former. Before attempting to configure the on-boot properties: REMOVE the following lines in /etc/init.d/o2cb ### BEGIN INIT INFO # Provides: o2cb # Required-Start: # Should-Start: # Required-Stop: # Default-Start: 2 3 5 # Default-Stop: # Description: Load O2CB cluster services at system boot. ### END INIT INFO
Re-register the o2cb service. # chkconfig --del o2cb # chkconfig --add o2cb # chkconfig --list o2cb o2cb 0:off 1:off
2:on
3:on
4:on
5:on
6:off
# ll /etc/rc3.d/*o2cb* lrwxrwxrwx 1 root root 14 Sep 29 11:56 /etc/rc3.d/S24o2cb -> ../init.d/o2cb The service should be S24o2cb in the default runlevel. After resolving this bug, you can continue to set the on-boot properties as follows:
13 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
# /etc/init.d/o2cb offline ocfs2 # /etc/init.d/o2cb unload # /etc/init.d/o2cb configure Configuring the O2CB driver. This will configure the on-boot properties of the O2CB driver. The following questions will determine whether the driver is loaded on boot. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Load O2CB driver on boot (y/n) [n]: y Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2 Writing O2CB configuration: OK Loading module "configfs": OK Mounting configfs filesystem at /config: OK Loading module "ocfs2_nodemanager": OK Loading module "ocfs2_dlm": OK Loading module "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting cluster ocfs2: OK Format the OCFS2 Filesystem You can now start to make use of the partitions created in the section Create Partitions on the Shared FireWire Storage Device. (Well, at least the first partition!) If the O2CB cluster is offline, start it. The format operation needs the cluster to be online, as it needs to ensure that the volume is not mounted on some node in the cluster. Earlier in this document, we created the directory /u02/oradata/orcl under the section Create Mount Point for OCFS / Clusterware. This section contains the commands to create and mount the file system to be used for the Cluster Manager - /u02/oradata/orcl. Create the OCFS2 Filesystem Unlike the other tasks in this section, creating the OCFS2 filesystem should only be executed on one node in the RAC cluster. You will be executing all commands in this section from linux1 only. Note that it is possible to create and mount the OCFS2 file system using either the GUI tool ocfs2console or the command-line tool mkfs.ocfs2. From the ocfs2console utility, use the menu [Tasks] - [Format]. See the instructions below on how to create the OCFS2 file system using the command-line tool mkfs.ocfs2. To create the filesystem, use the Oracle executable mkfs.ocfs2. For the purpose of this example, I run the following command only from linux1 as the root user account: $ su # mkfs.ocfs2 -b 4K -C 32K -N 4 -L oradatafiles /dev/sda1 mkfs.ocfs2 1.0.2 Filesystem label=oradatafiles Block size=4096 (bits=12) Cluster size=32768 (bits=15) Volume size=1011675136 (30873 clusters) (246984 blocks) 1 cluster groups (tail covers 30873 clusters, rest cover 30873 clusters) Journal size=16777216 Initial number of node slots: 4 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing lost+found: done mkfs.ocfs2 successful Mount the OCFS2 Filesystem Now that the file system is created, you can mount it. Let's first do it using the command-line, then I'll show how to include it in the /etc/fstab to have it mount on each boot. Mounting the filesystem will need to be performed on all nodes in the Oracle RAC cluster as the root user account. First, here is how to manually mount the OCFS2 file system from the command line. Remember, this needs to be performed as the root user account: $ su # mount -t ocfs2 -o datavolume /dev/sda1 /u02/oradata/orcl If the mount was successful, you will simply got your prompt back. You should, however, run the following checks to ensure the fil system is mounted correctly. Let's use the mount command to ensure that the new filesystem is really mounted. This should be performed on all nodes in the RAC cluster: # mount /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) none on /proc type proc (rw)
14 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) /dev/hda1 on /boot type ext3 (rw) none on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) cartman:SHARE2 on /cartman type nfs (rw,addr=192.168.1.120) configfs on /config type configfs (rw) ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) /dev/sda1 on /u02/oradata/orcl type ocfs2 (rw,_netdev,datavolume) Note: You are using the datavolume option to mount the new filesystem here. Oracle database users must mount any volume that will contain the Voting Disk file, Cluster Registry (OCR), Data files, Redo logs, Archive logs, and Control files with the datavolume mount option so as to ensure that the Oracle processes open the files with the o_direct flag. Any other type of volume, including an Oracle home (not used in this guide), should not be mounted with this mount option. The volume will mount after a short delay, usually around five seconds. It does so to let the heartbeat thread stabilize. In a future release, Oracle plans to add support for a global heartbeat, which will make most mounts instantaneous. Configure OCFS to Mount Automatically at Startup Let's review what you've done so far. You downloaded and installed OCFS2, which will be used to store the files needed by Cluster Manager files. After going through the install, you loaded the OCFS2 module into the kernel and then formatted the clustered filesystem. Finally, you mounted the newly created filesystem. This section walks through the steps responsible for mounting the new OCFS2 file system each time the machine(s) are booted. Start by adding the following line to the /etc/fstab file on all nodes in the RAC cluster: /dev/sda1 /u02/oradata/orcl ocfs2 _netdev,datavolume 0 0
Notice the _netdev option for mounting this filesystem. The _netdev mount option is a must for OCFS2 volumes; it indicates that the volume is to be mounted after the network is started and dismounted before the network is shutdown. Now, let's make sure that the ocfs2.ko kernel module is being loaded and that the file system will be mounted during the boot process. If you have been following along with the examples in this article, the actions to load the kernel module and mount the OCFS2 file system should already be enabled. However, you should still check those options by running the following on all nodes in the RAC cluster as the root user account: $ su # chkconfig --list o2cb o2cb 0:off 1:off
2:on
3: on
4:on
5:on
6:off
The flags that I have marked in bold should be set to "on". Check Permissions on New OCFS2 Filesystem Use the ls command to check ownership. The permissions should be set to 0775 with owner "oracle" and group "dba". If this is not the case for all nodes in the cluster (which was the case for me), then it is very possible that the "oracle" UID (175 in this example) and/or the "dba" GID (115 in this example) are not the same across all nodes. Let's first check the permissions: # ls -ld /u02/oradata/orcl drwxr-xr-x 3 root root 4096 Sep 29 12:11 /u02/oradata/orcl As you can see from the listing above, the oracle user account (and the dba group) will not be able to write to this directory. Let's fix that: # chown oracle.dba /u02/oradata/orcl # chmod 775 /u02/oradata/orcl Let's now go back and re-check that the permissions are correct for each node in the cluster: # ls -ld /u02/oradata/orcl drwxrwxr-x 3 oracle dba 4096 Sep 29 12:11 /u02/oradata/orcl Adjust the O2CB Heartbeat Threshold This is a very important section when configuring OCFS2 for use by Oracle Clusterware's two shared files on our FireWire drive. During testing, I was able to install and configure OCFS2, format the new volume, and finally install Oracle Clusterware (with its two required shared files; the voting disk and OCR file), located on the new OCFS2 volume. I was able to install Oracle Clusterware and see the shared drive, however, during my evaluation I was receiving many lock-ups and hanging after about 15 minutes when the Clusterware software was running on both nodes. It always varied on which node would hang (either linux1 or linux2 in my example). It also didn't matter whether there was a high I/O load or none at all for it to crash (hang). Keep in mind that the configuration you are creating is a rather low-end setup being configured with slow disk access with regards to the FireWire drive. This is by no means a high-end setup and susceptible to bogus timeouts. After looking through the trace files for OCFS2, it was apparent that access to the voting disk was too slow (exceeding the O2CB heartbeat threshold) and causing the Oracle Clusterware software (and the node) to crash. The solution I used was to simply increase the O2CB heartbeat threshold from its default setting of 7, to 301 (and in some cases as high as 900). This
15 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
is a configurable parameter that is used to compute the time it takes for a node to "fence" itself. First, let's see how to determine what the O2CB heartbeat threshold is currently set to. This can be done by querying the /proc file system as follows: # cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold 7 The value is 7, but what does this value represent? Well, it is used in the formula below to determine the fence time (in seconds): [fence time in seconds] = (O2CB_HEARTBEAT_THRESHOLD - 1) * 2 So, with a O2CB heartbeat threshold of 7, you would have a fence time of: (7 - 1) * 2 = 12 seconds You need a much larger threshold (600 seconds to be exact) given your slower FireWire disks. For 600 seconds, you will want a O2CB_HEARTBEAT_THRESHOLD of 301 as shown below: (301 - 1) * 2 = 600 seconds Let's see now how to increase the O2CB heartbeat threshold from 7 to 301. This will need to be performed on both nodes in the cluster. You first need to modify the file /etc/sysconfig/o2cb and set O2CB_HEARTBEAT_THRESHOLD to 301: # O2CB_ENABELED: 'true' means to load the driver on boot. O2CB_ENABLED=true # O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start. O2CB_BOOTCLUSTER=ocfs2 # O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead. O2CB_HEARTBEAT_THRESHOLD=301 After modifying the file /etc/sysconfig/o2cb , you need to alter the o2cb configuration. Again, this should be performed on all nodes in the cluster. # umount /u02/oradata/orcl/ # /etc/init.d/o2cb unload # /etc/init.d/o2cb configure Load O2CB driver on boot (y/n) [y]: y Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2 Writing O2CB configuration: OK Loading module "configfs": OK Mounting configfs filesystem at /config: OK Loading module "ocfs2_nodemanager": OK Loading module "ocfs2_dlm": OK Loading module "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting cluster ocfs2: OK You can now check again to make sure the settings took place in for the o2cb cluster stack: # cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold 301 Important Note: The value of 301 used for the O2CB heartbeat threshold will not work for all the FireWire drives listed in this guide. Use the following chart to determine the O2CB heartbeat threshold value that should be used.
FireWire Drive Maxtor OneTouch II 300GB USB 2.0 / IEEE 1394a External Hard Drive - (E01G300) Maxtor OneTouch II 250GB USB 2.0 / IEEE 1394a External Hard Drive - (E01G250) Maxtor OneTouch II 200GB USB 2.0 / IEEE 1394a External Hard Drive - (E01A200) LaCie Hard Drive, Design by F.A. Porsche 250GB, FireWire 400 - (300703U) LaCie Hard Drive, Design by F.A. Porsche 160GB, FireWire 400 - (300702U) LaCie Hard Drive, Design by F.A. Porsche 80GB, FireWire 400 - (300699U) Dual Link Drive Kit, FireWire Enclosure, ADS Technologies - (DLX185) Maxtor OneTouch 250GB USB 2.0 / IEEE 1394a External Hard Drive - (A01A250) Maxtor OneTouch 200GB USB 2.0 / IEEE 1394a External Hard Drive - (A01A200) O2CB Heartbeat Threshold Value 301 301 301 600 600 600 901 600 600
Reboot Both Nodes Before starting the next section, this would be a good place to reboot all of the nodes in the RAC cluster. When the machines come up, ensure that the cluster stack services are being loaded and the new OCFS2 file system is being mounted: # mount
16 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) /dev/hda1 on /boot type ext3 (rw) none on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) cartman:SHARE2 on /cartman type nfs (rw,addr=192.168.1.120) configfs on /config type configfs (rw) ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) /dev/sda1 on /u02/oradata/orcl type ocfs2 (rw,_netdev,datavolume) You should also verify that the O2CB heartbeat threshold is set correctly (to our new value of 301): # cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold 301 How to Determine OCFS2 Version To determine which version of OCFS2 is running, use: # cat /proc/fs/ocfs2/version OCFS2 1.0.4 Fri Aug 26 12:31:58 PDT 2005 (build 0a22e88ab648dc8d2a1f9d7796ad101c)
17 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
# chown oracle:dba /dev/raw/raw2; chmod 660 /dev/raw/raw2 # chown oracle:dba /dev/raw/raw3; chmod 660 /dev/raw/raw3 # chown oracle:dba /dev/raw/raw4; chmod 660 /dev/raw/raw4 3. The last step is to reboot the server to bind the devices or simply restart the rawdevices service: # service rawdevices restart As I mentioned earlier, the above example was just to demonstrate that there is more than one method for using ASM with Linux. Now let's move on to the method that will be used for this article, "ASM with ASMLib I/O." Download the ASMLib 2.0 Packages First download the ASMLib 2.0 libraries (from OTN) and the driver (from my web site). Like OCFS, you need to download the version for the Linux kernel and number of processors on the machine. You are using kernel 2.6.9-11.0.0.10.3.EL #1 on single-processor machines: # uname -a Linux linux1 2.6.9-11.0.0.10.3.EL #1 Tue Jul 5 12:20:09 PDT 2005 i686 i686 i386 GNU/Linux Oracle ASMLib Downloads for Red Hat Enterprise Linux 4 AS oracleasm-2.6.9-11.0.0.10.3.EL-2.0.0-1.i686.rpm - (Driver for "up" kernels) -ORoracleasm-2.6.9-11.0.0.10.3.ELsmp-2.0.0-1.i686.rpm - (Driver for "smp" kernels) oracleasmlib-2.0.0-1.i386.rpm - (Userspace library) oracleasm-support-2.0.0-1.i386.rpm - (Driver support files) Install ASMLib 2.0 Packages This installation needs to be performed on all nodes as the root user account: $ su # rpm -Uvh oracleasm-2.6.9-11.0.0.10.3.EL-2.0.0-1.i686.rpm \ oracleasmlib-2.0.0-1.i386.rpm \ oracleasm-support-2.0.0-1.i386.rpm Preparing... ########################################### 1:oracleasm-support ########################################### 2:oracleasm-2.6.9-11.0.0.########################################### 3:oracleasmlib ########################################### Configure and Loading the ASMLib 2.0 Packages Now that you downloaded and installed the ASMLib Packages for Linux, you need to configure and load the ASM kernel module. This task needs to be run on all nodes as root: $ su # /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracle Default group to own the driver interface []: dba Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: [ OK ] Creating /dev/oracleasm mount point: [ OK ] Loading module "oracleasm": [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ] Create ASM Disks for Oracle In Section 10, you created three Linux partitions to be used for storing Oracle database files like online redo logs, database files, control files, archived redo log files, and a flash recovery area. Here is a list of those partitions we created for use by ASM: Oracle ASM Partitions Created Filesystem Type Partition ASM ASM ASM /dev/sda2 /dev/sda3 Size Mount Point File Types 50GB ORCL:VOL1 Oracle Database Files 50GB ORCL:VOL2 Oracle Database Files
18 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
Total
200GB
The last task in this section it to create the ASM Disks. Creating the ASM disks only needs to be done on one node as the root user account. I will be running these commands on linux1. On the other nodes, you will need to perform a scandisk to recognize the new volumes. When that is complete, you should then run the oracleasm listdisks command on all nodes to verify that all ASM disks were created and available. $ su # /etc/init.d/oracleasm createdisk VOL1 /dev/sda2 Marking disk "/dev/sda2" as an ASM disk [ OK ] # /etc/init.d/oracleasm createdisk VOL2 /dev/sda3 Marking disk "/dev/sda3" as an ASM disk [ OK ] # /etc/init.d/oracleasm createdisk VOL3 /dev/sda4 Marking disk "/dev/sda4" as an ASM disk [ OK ] Note: If you are repeating this guide using the same hardware (actually, the same shared drive), you may get a failure when attempting to create the ASM disks. If you do receive a failure, try listing all ASM disks using: # /etc/init.d/oracleasm listdisks VOL1 VOL2 VOL3 As you can see, the results show that I have three volumes already defined. If you have the three volumes already defined from a previous run, go ahead and remove them using the following commands and then creating them again using the above (oracleasm createdisk) commands: # /etc/init.d/oracleasm deletedisk VOL1 Removing ASM disk "VOL1" [ OK ] # /etc/init.d/oracleasm deletedisk VOL2 Removing ASM disk "VOL2" [ OK ] # /etc/init.d/oracleasm deletedisk VOL3 Removing ASM disk "VOL3" [ OK ] On all other nodes in the cluster, you must perform a scandisk to recognize the new volumes: # /etc/init.d/oracleasm scandisks Scanning system for ASM disks [ OK ]
You can now test that the ASM disks were successfully created by using the following command on all nodes as the root user account: # /etc/init.d/oracleasm listdisks VOL1 VOL2 VOL3
19 of 20
1/4/2006 8:46 AM
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireW...
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_2.html...
$ cd ~oracle/orainstall $ unzip 10201_database_linux32.zip Finally, extract the Oracle Companion CD Software: $ cd ~oracle/orainstall $ unzip 10201_companion_linux32.zip
20 of 20
1/4/2006 8:46 AM