https://support.oracle.com/CSP/main/article?cmd=show&type=NO...

RAC Assurance Support Team: RAC Starter Kit and Best Practices (AIX) [ID 811293.1]

Modified 28-OCT-2010 Type BULLETIN Status PUBLISHED

In this Document
Purpose
Scope and Application
RAC Assurance Support Team: RAC Starter Kit and Best Practices (AIX)
RAC Assurance Support Team: RAC and Oracle Clusterware Starter Kit and Best Practices (Generic)
RAC Platform Specific Starter Kits and Best Practices
RAC on AIX Step by Step Installation Instructions
RAC on AIX Highlighted Recommendations
RAC on AIX Best Practices
Storage
Networking
Configuration
Install
Mandatory Patches
Recommended Patches
RAC on AIX References
References

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.2.0.1 - Release: 10.2 to 11.2
IBM AIX on POWER Systems (64-bit)
IBM AIX Based Systems (64-bit)

Purpose
Rate this document
The goal of the Oracle Real Application Clusters (RAC) Starter Kit is to provide you with the latest information on generic and platform specific best practices for implementing an Oracle RAC cluster. This docume
provided based on Oracle’s experience with its global RAC customer base.

This Starter Kit is not meant to replace or supplant the Oracle Documentation set, but rather, it is meant as a supplement to the same. It is imperative that the Oracle Documentation be read, understood, and refe
answers to any questions that may not be clearly addressed by this Starter Kit.

All recommendations should be carefully reviewed by your own operations group and should only be implemented if the potential gain as measured against the associated risk warrants implementation. Risk asses
made with a detailed knowledge of the system, application, and business environment.

As every customer environment is unique, the success of any Oracle Database implementation, including implementations of Oracle RAC, is predicated on a successful test environment. It is thus imperative that a
recommendations from this Starter Kit are thoroughly tested and validated using a testing environment that is a replica of the target production environment before being implemented in the production environment
is no negative impact associated with the recommendations that are made.

Scope and Application

This article is intended for use by all new (and existing) Oracle RAC implementers.

RAC Assurance Support Team: RAC Starter Kit and Best Practices (AIX)

RAC Assurance Support Team: RAC and Oracle Clusterware Starter Kit and Best Practices (Generic)

The following note focuses on Generic RAC Best Practices that are applicable to all platforms including a white paper on available RAC System Load Testing Tools and RAC System Test Plan outlines for 10gR2 &

Note:810394.1 RAC Assurance Support Team: RAC and Oracle Clusterware Starter Kit and Best Practices (Generic)

RAC Platform Specific Starter Kits and Best Practices

The following notes contain detailed platform specific best practices including Step-By-Step installation cookbooks.

Note 811306.1 RAC Assurance Support Team: RAC Starter Kit and Best Practices (Linux)

Note 811280.1 RAC Assurance Support Team: RAC Starter Kit and Best Practices (Solaris)

Note 811271.1 RAC Assurance Support Team: RAC Starter Kit and Best Practices (Windows)

Note 811293.1 RAC Assurance Support Team: RAC Starter Kit and Best Practices (AIX)

Note 811303.1 RAC Assurance Support Team: RAC Starter Kit and Best Practices (HP-UX)

RAC on AIX Step by Step Installation Instructions

Click here for a Step By Step guide for installing Oracle RAC 10gR2 on AIX.
Click here for a Step By Step guide for installing Oracle RAC 11gR1 on AIX.
Click here for a Step By Step guide for installing Oracle RAC 11gR2 on AIX.

RAC on AIX Highlighted Recommendations

This is a list of the core best practices and references identified by the RAC Assurance Team relevant to AIX specific RAC implementations. The rest of this chapter provides additional best practices and referenc

Ensure APAR IZ71987 (for AIX 6.1) or APAR IZ67445 (for AIX 5.3) has been applied on the OS otherwise the related bug may cause the whole system hang. Please reference to <<1088076.1>> for mor
Tune Virtual memory parameters. IBM recommended numbers are:

minperm%=3
maxperm%=90
maxclient%=90
lru_file_repage=0
strict_maxperm=0
strict_maxclient=1
page_steal_method=1

Example script for setting these parameters:

#!/usr/bin/ksh
vmo -p -o maxperm%=90;
vmo -p -o minperm%=3;
vmo -p -o maxclient%=90;
vmo -p -o strict maxperm=0;
vmo -p -o strict maxclient=1;
vmo -p -o lru_file_repage=0;

1 de 3 08/12/2010 12:05
 https://support.oracle.com/CSP/main/article?cmd=show&type=NO...

vmo -r -o page_steal_method=1; (need to reboot to take into effect)

vmo -p -o strict_maxclient=1
vmo -p -o strict_maxperm=0;

On AIX 5.3, apply APAR IY84780 to fix a known kernel issue with per-cpu freelists.
This fix is also included in Technology Level 4 (TL4) and higher. If necessary, check with IBM for any superceding fixes.
For details on this APAR, refer to this note: http://www-01.ibm.com/support/docview.wss?uid=isg1IY84780
set AIXTHREAD_SCOPE=S in the environment: export AIXTHREAD_SCOPE=S – Reference Note 458403.1 Why AIXTHREAD_SCOPE should be set to 'S' on AIX
When implementing RAC, you must take care with the SAN storage chosen. The SAN Storage must be capable of read/write concurrency (writing at the same time from any member of the RAC cluster) th
This means that “reserve_policy” attribute from the discovered disks (hdisk, hdiskpower, dlmfdrv, etc …) must be capable of handling settings with: “no_reserve” or “no_lock” values.
When using the default Processor Folding feature, please be sure that the following minimum Service Pack or Technology Level packages from IBM are in place:
http://www-01.ibm.com/support/docview.wss?uid=isg1fixinfo105201
On AIX 6.1 please follow Note 1056896.1 to workaround the VIP startup issue.
Refer to the Joint IBM/OracleWhite Paper for detailed instructions on tuning AIX for Oracle RAC:
http://www.oracle.com/technetwork/database/clusterware/overview/rac-aix-system-stability-131022.pdf
For versions 10gR2 and 11gR1, it is a best practice on all platforms to set the CSS diagwait parameter to 13 in order to provide time for dumping diagnostics in case of node evictions. Setting the diagwait
recommended without explicit instruction from Support. This setting is no longer required in Oracle Clusterware 11g Release 2. Reference Document 559365.1 for more details on diagwait.

RAC on AIX Best Practices

Storage

Do not assign PVIDs (Physical Volume IDs) to disks or volumes that are being used for ASM Diskgroups. PVIDs should be cleared on all nodes from any candidate disks or volumes prior to being added to
Once a disk or volume is added to an ASM Diskgroup, PVIDs should never be assigned after the fact, from any node in the cluster, including nodes that are being added to an existing cluster. Reference
Note:750016.1 for more details on this issue.
Set FSCSI Device Attribute FC_ERR_RECOV to FAST_FAIL for Voting Disk and ASM storage. This setting has been shown to avoid reboots in situations where a SAN storage outage of the volumes hosti
disks caused reboots to occur. - See knowledge document Note 560077.1 ASM Hangs After Loss Of Failgroup.
When implementing RAC, you must take care with the SAN storage chosen. The SAN Storage must be capable of read/write concurrency (writing at the same time from any member of the RAC cluster) th
This means that “reserve_policy” attribute from the discovered disks (hdisk, hdiskpower, dlmfdrv, etc …) must be capable of handling settings with: “no_reserve” or “no_lock” values.
When using EMC disk devices ensure that the reserve policy is set correctly. Change the “reserve_lock” attribute to “no” to prevent failures during CRS installations: # chdev -l hdisk? -a reserve_lock=no
When implementing GPFS, please refer to knowledge document Note 302806.1 for recommendations on LUN configuration, filesystem blocksize, AIO configuration, inodes, and implementation examples.
By default, Oracle uses the asynchronous I/O (AIO) and direct I/O (DIO) features of the AIX 5L operating system to do its own scheduling of I/O directly to disks, bypassing most of the GPFS caching and
facilities.
Do not use the DIO mount option for the GPFS file system or change the DIO attribute for any Oracle files.
The Oracle init.ora parameter filesystemio_options setting will be ignored for Oracle files on GPFS
Users of AIX occasionally encounter long interactive-application response times when other applications in the system are running large writes to disk. Configuring I/O pacing limits the number of outstanding
against a file. AIX 6.1 enables I/O pacing by default and the default value: "minpout=4096 and maxpout=8193" is good for AIX6.1. However, in AIX 5.3, you need to explicitly enable this feature. Oracle's te
that starting values of 8 for minpout and 12 for maxpout are a good baseline for most Oracle customers. However, every environment is different, and therefore different values may very well be acceptable
been properly tuned and shown to perform with differing values. To configure I/O pacing on the system via SMIT, using Oracle's recommended baseline values, enter the following at the command line as ro

# smitty chgsys
# chdev -l sys0 -a minpout=8 -a maxpout=12

Set FSCSI Device Attribute FC_ERR_RECOV to FAST_FAIL for Voting Disk and ASM storage. This setting has been shown to avoid reboots in situations where a SAN storage outage of the volumes hosti
disks caused reboots to occur.
On AIX ASM can use concurrent RAW logical volumes or RAW partitions. ASM must use the pseudo device /dev/rhdiskpwerx in environments using multi-pathing with EMC Powerpath.
When using SDDPCM on AIX use the logical device /dev/rhdiskx to build ASM disk groups.
Architecture disk storage management Note 132670.1
When using ASM failure groups with AIX5L, make the following change to ensure ASM detects the IO failure of the failure group/storage to prevent the database from hanging for 10 minutes before I/O on
storage is resumed. Reference Note 560077.1:

Set the Fibre Channel SCSI I/O Controller Protocol Device event error recovery policy to fast_fail for each fibre-channel adapter:
chdev -l fscsi0 -a fc_err_recov=fast_fail -P

Networking

The value of sb_max, which specifies the maximum socket buffer size for any socket buffer, should be at least twice the size of the largest of the UDP and TCP send and receive buffers. Mostly on AIX ud
64K for Oracle Server. A suggested starting value for udp_recvspace is 10 times the value of udp_sendspace, because UDP may not be able to pass a packet to the application before another one arrives
be 2 times of udp_recvspace.
Oracle clusterware VIP’s IP address and corresponding nodes names must not be used on the network prior to Oracle Clusterware installation. Don’t make any AIX alias on the public network interface, the
installation will do it. Just reserve 1 VIP and it’s hostname per RAC node. Oracle Clusterware VIP’s IP and corresponding nodes names can be declared in the DNS, or at minimum in the local hosts file.

Configuration

Ensure that Maximum number of PROCESSES allowed per user(maxuproc) has been set to 16384. It's already documented at 11gR1 and 11gR2 installation guide and the detail way to check and set plea
392006.1
Ensure that the GI and ORACLE owner account has the capabilities:
CAP_NUMA_ATTACH, CAP_BYPASS_RAC_VMM, and CAP_PROPAGATE. It is required per the 11gR2 installation guide and it is also required by 10.2.0.4 and 11.1.0.6 or later version Note 603489.1

Check and Set example for GRID user:

#/usr/bin/lsuser -a capabilities grid

#/usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid

If using RAID devices, configure a single logical unit number (LUN) for each RAID device. Do not create LUNs across RAID devices for use by GPFS as this will ultimately result in a significant loss in perfor
Operating system must be installed the same way on each node, with the same maintenance level, same apar and filesets levels.
Hostname must be the same as the cluster public node name. That is to say, the public node name must match what is returned by “hostname” command.
If not using HACMP then HACMP filesets must not be installed.

Install

Patches: when using the IBM Logical Host Ethernet Adapter (LHEA) for the Oracle RAC Public or VIP interfaces, it's required to apply the fix for Bug 6782569 which is included in the 11.1.0.7 CRS bundle#
other version please contact Oracle Support for the one-off patch or workaround for this issue.
Parameters: As mentioned here above in the 'Highlighted Recommendations' section of this article, AIX Kernel Settings: set AIXTHREAD_SCOPE=S in the environment: export AIXTHREAD_SCOPE=S
Parameters: Having an ORACLE_HOME on a filesystem mounted with "cio" option is not supported. Such a configuration will cause, installation, relinking and other unexpected .
Configuration: SSH implementation: The home for the default user(s): ORACLE, ASM, CRS, RDBMS must be in one directory in (such as /home), otherwise you may have trouble with ssh setup.
Patches:

Mandatory Patches

OS Patch Number Description Recommendation

IY89080 I/O reading problems after installing Required for technology level 5.
ibm technology level 5 (5300-05)

2 de 3 08/12/2010 12:05
 https://support.oracle.com/CSP/main/article?cmd=show&type=NO...

IY84780 Kernel memory garbage collector fails Required for environments with more
to free lists than 4 CPU’s

IY68874 Application core dumps frequently Required for large page data
with large page option turned to application
mandatory

IY70031 Corruption from Simultaneous CIO Required if using Journal File System
Writes with O_DSYNC ON JFS2 Version 2 (JFS2) for Oracle Database
files

Recommended Patches

OS Patch Number Description Recommendation

IY65481 TCP Retransmission Reference Note 478229.1

IY98287 rsct.basic.rte, rsct.compat.clients.rte For RAC configurations only.

RAC on AIX References

http://www.oracle.com/technology/software/products/database/oracle10g/htdocs/10201aixsoft.html
Oracle® Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide10g Release 2 (10.2) for AIX: http://download.oracle.com/docs/cd/B19306_01/install.102/b14201/toc.htm
Oracle® Clusterware Installation Guide 11g Release 1 (11.1) for AIX Based Systems
http://download.oracle.com/docs/cd/B28359_01/install.111/b28258/toc.htm
Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for IBM AIX Based Systems http://download.oracle.com/docs/cd/E11882_01/install.112/e10814/toc.htm
Note 567286.1 RAC on AIX: With Virtual Interfaces Racgvip Fails Even Though Public Interface is Up
Note 302806.1 IBM General Parallel File System (GPFS) and Oracle RAC on AIX 5L and IBM eServer pSeries
Note 551658.1 CSS errors out with clssnm_skgxninit: Compatible vendor clusterware not in use
Note 404474.1 Status of Certification of Oracle Clusterware with HACMP 5.3 and 5.4
Note.282036.1 Minimum Software Versions and Patches Required to Support Oracle Products on IBM pSeries
Note 458403.1 Why AIXTHREAD_SCOPE should be set to 'S' on AIX
Note 560077.1 ASM Hangs After Loss of Failgroup on AIX

Modification History
[11-Aug-2009] created this Modification History section
[21-Apr-2010] added links to generic Starter Kit as well as other platform specific starter kits
[03-Jun-2010] updated versions to include 11.2
[21-Jul-2010] processor folding recommendation modified from 'disable' to relevant link reference
[30-Jul-2010] processor folding recommendation modified for clarity
[30-Jul-2010] added PVID recommendation to the storage section
[17-Sep-2010] uploaded 11gR2 Step by Step Cookbook
[21-Oct-2010] modified diagwait recommendation details
[27-Oct-2010] replaced reference to 805536.1 with reference to 1056896.1

References

NOTE:810394.1 - RAC Assurance Support Team: RAC and Oracle Clusterware Starter Kit and Best Practices (Generic)
NOTE:869644.1 - Having an ORACLE_HOME on a filesystem mounted with "cio" option is not supported and will have issues

Attachments

RACGuides_Rac10gR2OnAIX.pdf (2,212.31 KB)

RACGuides_Rac11gR1OnAIX.pdf (12,525.08 KB)
RACGuides_Rac11gR2OnAIX.pdf (1,749.82 KB)

Related

Products

Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition

Keywords

RAC; KIT; AIX; COOKBOOK; REAL APPLICATION CLUSTERS

Back to top

3 de 3 08/12/2010 12:05