Subject: Implementing Address Windowing Extensions (AWE) or VLM on Windows Platforms Doc ID: Note:225 !

"#$ Type: %&LLE'IN Last Revision Date: Status: $$()&L(2**+ P&%LI,-E.

PURPOSE ------To address the growing need for use of more memory on 32-Bit Windows platforms, and explain how AWE is implemented by Oracle on Windows.

SCOPE & APPLICATION ------------------Oracle DBA's running on the Microsoft Windows platform. Oracle Support Analysts, Field Engineers troubleshooting problems related to AWE and/or memory issues on Windows. AWE Memory implementation on Windows 2000 -----------------------------------------A common question on the Windows NT/Windows 2000 platform revolves around how to take advantage of systems with more than 4 GB of RAM. As discussed in Metalink Note 46001.1 and Note 46053.1, the 32-Bit process address space for any process on Windows equates to a total of 4GB of addressable RAM. Of this, by default, 2GB is reserved for the process itself, and 2GB for the kernel. On systems running either Windows 2000 Advanced Server, or Windows NT 4.0 Enterprise Edition, this ratio can be changed by adding the /3GB switch to the boot.ini, allowing a process to address 3GB and reserving 1GB for the kernel. However, the total addressable memory for a single process is still only 4GB. See also Note 1036312.6 : Utilizing Up to 3GB Virtual Memory on Windows NT Server 4.0

__________________________________________________________________ What can be done to address memory beyond 4GB?: ===============================================

The answer is to take advantage of Physical Address Extensions (PAE), or Address Windowing Extensions (AWE)(These two terms are used interchangeably, so the rest of this document will refer to this simply as AWE). AWE support is available if you are running on a machine with more than 4GB of physical RAM which is running any of the below Windows operating systems: * * * * Windows Windows Windows Windows 2000 2000 2003 2003 Datacenter Server Advanced Server Data Center Edition (32-Bit) Enterprise Edition (32-Bit)

On the above operating systems, AWE support is built into the OS. No special drivers are needed to take advantage of the additional memory.

AWE CANNOT be used on the following Operating Systems: * * * * * * Windows Windows Windows Windows Windows Windows 2000 Server (Standard) 2000 Professional XP Home Edition XP Professional 2003 Standard Edition 2003 Web Edition

NOTE Also that on 64-Bit Windows operating systems, there is no need for AWE implementation support, because the directly addressable memory for a single process on 64-Bit Windows is 8 Terabytes. __________________________________________________________________ Oracle versions that can use AWE: ================================= Oracle can take advantage of AWE in the following 32-Bit RDBMS releases: * * * * * Oracle Oracle Oracle Oracle Oracle 8.1.6.x 8.1.7.x 9.2.x 10.1.x 10.2.x

Oracle does NOT implement AWE support in release 9.0.1.x

AWE support is available on both the Enterprise Edition of Oracle and the Standard Edition of Oracle. However, on Standard Edition of 9.2.0.1, you may receive the following error if trying to start the database with USE_INDIRECT_DATA_BUFFERS=TRUE: ORA-439 - FEATURE NOT ENABLED: VERY LARGE MEMORY In Standard Edition 9.2.0.2 and 9.2.0.3, you will not receive the above errors, but VLM functionality is still not enabled. Refer to BUG#2945011 for more detail. This BUG is fixed in 9.2.0.3 Patch 2, and will be fixed in 9.2.0.4 as well. __________________________________________________________________ Enabling support at the OS level: ================================== AWE can be enabled at the OS by adding the /PAE switch to the boot.ini as such: multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /PAE It IS possible to have BOTH the /PAE and /3GB switch in place on the same machine, as such: multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /3GB /PAE

However, be aware that if BOTH switches are in place, the server will only be able to recognize up to 16GB of RAM. If you are working with a server with more than 16GB of RAM, you will need to choose between the two. It is important to note that once either or both of these switches are in place in the boot.ini, ALL processes running can take advantage of these switches. Thus, in a case where multiple Oracle instances are running on the same server, ALL instances can take advantage of the additional memory afforded by these switches, up to the amount of physical memory on the box.

Operating System Privileges Needed at the OS Level: ==================================================== In order to take advantage of the additional memory afforded through PAE, the operating system user account which is used to start the OracleService<SID> must be granted the 'Lock Pages in Memory' system privilege at the operating system level. By default, the OracleService<SID> starts as the LocalSystem account. The LocalSystem account has the privilege to Lock Pages in Memory granted to it by default. However, if you change the OracleService<SID> to logon as a user OTHER than LocalSystem, you may see the following errors when attempting to start the database with USE_INDIRECT_DATA_BUFFERS set to TRUE :

SQL> startup pfile=c:\temp\initscott.ora ORA-27102: out of memory OSD-00010: Message 10 not found; product=RDBMS; facility=SOSD O/S-Error: (OS 1300) Not all privileges referenced are assigned to the caller.

To rectify this, you must grant the 'Lock pages in memory' privilege to the user that the OracleService<SID> starts as. To do this, click on: Start -> Programs -> Administrative Tools -> Local Security Policy (on a Domain Controller, click on 'Domain Security Policy' instead of 'Local Security Policy') Double-click on the 'Lock Pages in memory' policy. Add the appropriate user and click 'Ok'. Restart the OracleService<SID>

__________________________________________________________________ Understanding the Oracle implementation of AWE support: ======================================================= What the PAE switch allows you to do from the Oracle perspective is to increase the amount of memory that can be used for the Oracle Database Block Buffer Cache. It is important to note that this additional memory can ONLY be used by Oracle in the form of an increased value for DB_BLOCK_BUFFERS.

There is still confusion on the old style of VLM versus AWE on Windows 2000. With VLM on Windows NT 4.0, there was the concept of pointers pointing to the extended memory area, but that is no longer the case on Windows 2000. Instead, the windowing technology as described in these articles is being used. For more information on AWE/PAE implementation on the Windows platform, refer to Microsoft's website. As mentioned previously, with AWE enabled, this allows the process(es) (in this case ORACLE.EXE) to use memory above and beyond the 4GB mark defined by a 32-Bit Process Address space. The physical location of these blocks does not matter. However, the database blocks must still be accessed from within a window, which exists (logically) in that regular 3GB process address space. The size of this window is defined by a registry setting in the HOME key for Oracle (HKLM\Software\Oracle\Homex) called AWE_WINDOW_MEMORY. By default, this value is 1GB, so if this value is not set in the registry, AWE_WINDOW_MEMORY will be 1GB. If you add the registry key yourself, the datatype should be a string value, or a REG_SZ. The value for AWE_WINDOW_MEMORY must be specified in BYTES. It is important to realize that any database blocks accessed by Oracle (or any user/background thread within Oracle.exe) must first be mapped into the 'window' defined by AWE_WINDOW_MEMORY. In this scenario, it does not matter where the blocks are physically located - there is no need to be concerned with where the blocks are physically residing. The window will be drawn around the block (i.e. the block will be mapped) wherever it is located in memory. If the block is in memory but has not been mapped into the window, then it may be necessary to unmapped another block that IS in the window, in order to accommodate the new block. While this mapping and unmapping of blocks does add some cost, it is still faster than incurring an I/O operation to read the block from disk. This will be discussed further down in the section on troubleshooting. Note: Keep in mind that if there are multiple instances on a machine with the /PAE switch enabled, ALL instances can take advantage of the additional memory. However, AWE_WINDOW_MEMORY cannot be set on a per-instance basis, so all databases that are running out of the HOMEx key where AWE_WINDOW_MEMORY is set will inherit the same value.

__________________________________________________________________ Enabling AWE Support at the Database/Instance Level: ==================================================== To enable the AWE implementation on Oracle, you must set the following parameter in the init file (or spfile) used to start the instance: USE_INDIRECT_DATA_BUFFERS=TRUE

Note again that the buffer cache MUST be defined using the parameter DB_BLOCK_BUFFERS, no matter what version of the RDBMS you are running. The 9.2 feature allowing for Multiple block sizes in a database will be disabled if you set USE_INDIRECT_DATA_BUFFERS=TRUE, and you cannot specify the DB_CACHE_SIZE parameter to define the size of the buffer cache.

On 9.2, if you attempt to startup a database with this combination of parameters: USE_INDIRECT_DATA_BUFFERS=TRUE DB_CACHE_SIZE=xxxxx (Any number) The startup will fail with the following error:

SQL> startup ORA-00385: cannot enable Very Large Memory with new buffer cache parameters You must change DB_CACHE_SIZE to use DB_BLOCK_BUFFERS instead, as was the syntax under Oracle8i and earlier.

__________________________________________________________________ AWE_WINDOW_MEMORY Within the 3GB Process Address Space: ======================================================= If you are using /PAE and the /3GB switch together, the address space for ORACLE.EXE will be 3GB. The value for AWE_WINDOW_MEMORY must come from the normal address space used by the ORACLE.EXE process. Memory that comes from that 3GB address space addressable by the oracle.exe process includes the following:

The Value for AWE_WINDOW_MEMORY The rest of the SGA (shared_pool, large_pool, java_pool, log_buffers, etc) Overhead for Oracle.exe and DLLs (65-100M depends on version & options) Stack space for all threads (Defaults to 1MB/thread, unless orastack is used) PGA and UGA memory for all user sessions Therefore, the value for AWE_WINDOW_MEMORY should be tuned such that mapping and unmapping operations are avoided as much as possible, while still allowing enough memory within the 3GB address space for the rest of the process memory that MUST fit within the 3GB (i.e. overhead, remaining SGA components and all user connection memory (stack + uga + pga) noted above). The total size of the buffer cache can then be set to the amount of physical memory remaining above the 4GB barrier, plus AWE_WINDOW_MEMORY. On a machine with 12GB of RAM, using the default value of 1GB for AWE_WINDOW_MEMORY, your total buffer cache could theoretically be as high as 9GB: (Total RAM - 4GB + AWE_WINDOW_MEMORY) = 12GB - 4GB + 1GB = 9GB

In reality, your maximum buffer cache size will be somewhat less than this, allowing for some overhead and additional processes running on the system. Attempting to startup the database with a buffer cache larger than the maximum value as calculated above may result in the following errors: ORA-27102 out of memory OSD-00034 Message 34 not found; Product=RDBMS;facility =SOSD O/S Error: (OS 8) Not enough storage is available to process this command (Note - If you are on Release 9.2, another possible cause for these errors is noted further down, in the troubleshooting section) As mentioned above, the buffer cache must be specified using DB_BLOCK_BUFFERS rather than DB_CACHE_SIZE, so assuming an 8K block size (8192), to get a 9GB buffer cache, you would set the following init parameters: DB_BLOCK_BUFFERS = 1179648 DB_BLOCK_SIZE = 8192

__________________________________________________________________ Troubleshooting AWE_WINDOW_MEMORY implementation: ========================= ========================= Minimum Value Required for AWE_WINDOW_MEMORY in 9.2 and Above: ============================================================== Here are key points to understand when using AWE_WINDOW_MEMORY: 1. 2. Under Oracle 8.1.7 we do NOT enforce a minimum value for AWE_WINDOW_MEMORY to be able to start the database. This was changed under Oracle9i Release 2, such that we DO enforce a minimum value for AWE_WINDOW_MEMORY. This change was done to help improve performance by enforcing a larger window size. You can alter the minimum required value for AWE_WINDOW_MEMORY under 9.2 by changing/setting the value of the parameter _DB_BLOCK_LRU_LATCHES. Under 8.1.7, this parameter was named DB_BLOCK_LRU_LATCHES. However, under 9.x, this parameter was changed to be a hidden parameter.

3.

The minimum value for AWE_WINDOW_MEMORY starting with 9.2 is calculated as such: MIN(AWE_WINDOW_MEMORY)=(4096 * DB_BLOCK_SIZE * _DB_BLOCK_LRU_LATCHES)/8 Starting with 9.2, to calculate the value for _DB_BLOCK_LRU_LATCHES, we need this formula: _DB_BLOCK_LRU_LATCHES = (Max buffer pools * SETS_PER_POOL) Max Buffer Pools is a constant = 8 SETS_PER_POOL is variable, and depends on whether or not VLM is enabled.

SETS_PER_POOL = 2* CPU_COUNT (if VLM is enabled) SETS_PER_POOL= CPU Count /2 (If VLM is NOT enabled) /* Recall that VLM is enabled by setting USE_INDIRECT_DATA_BUFFERS=TRUE So, as you can see, the value for _DB_BLOCK_LRU_LATCHES in 9.2 and above is dependent on the number of CPU's in the box, and therefore MIN(AWE_WINDOW_MEMORY) is dependent on the # of CPU's as well as the DB_BLOCK_SIZE. The larger the Block Size, and the more CPU's in a system, the higher the value for MIN(AWE_WINDOW_MEMORY). Here are a couple of example configurations and caclulations showing MIN(AWE_WINDOW_MEMORY).

Example #1: ---------------# of CPU's = 8 DB_BLOCK_SIZE = 8192 Total RAM = 8GB SETS_PER_POOL = 2 * CPU_COUNT = 16 _DB_BLOCK_LRU_LATCHES = (max buffer Pools * sets_per_pool) = 8*16 = 128 MIN(AWE_WINDOW_MEMORY) =(4096*DB_BLOCK_SIZE*_DB_BLOCK_LRU_LATCHES)/8 = ( 4096 * 8192 * 128) / 8 = 536870912 bytes = 512 MB

Example #2: --------------# of CPU's = 16 DB_BLOCK_SIZE = 8192 Total RAM = 16 GB SETS_PER_POOL = 2 * CPU_COUNT = 32 _DB_BLOCK_LRU_LATCHES = (max buffer Pools * sets_per_pool) = 8*32 = 256 MIN(AWE_WINDOW_MEMORY) =(4096*DB_BLOCK_SIZE*_DB_BLOCK_LRU_LATCHES)/8 = ( 4096 * 8192 * 256) / 8 = 1073741824 bytes = 1024 MB

These values above are the minimum values required for AWE_WINDOW_MEMORY to be set to, UNLESS you explicitly set _DB_BLOCK_LRU_LATCHES to a lower value. If AWE_WINDOW_MEMORY is not set to the minimum value, you will receive the following errors: ORA-27102 out of memory OSD-00034 Message 34 not found; Product=RDBMS;facility =SOSD O/S Error: (OS 8) Not enough storage is available to process this command If you receive these errors when trying to start the database under 9.2 or 10g, this may be because the AWE_WINDOW_MEMORY value in the registry is set too low for the calculated minimum value. If you cannot increase the value for AWE_WINDOW_MEMORY, then you can explicitly set _DB_BLOCK_LRU_LATCHES to a value lower than the calculated value, and retry the startup. _DB_BLOCK_LRU_LATCHES must be at least 8 (Equal to the maximum number of buffer pools)

Note #1 - Recall from the earlier section that these errors may also occur if you are trying to start up with a buffer cache that is too large for the physical memory available. Note #2 - The same errors above have also been observed with a buffer cache that is too small. When USE_INDIRECT_DATA_BUFFERS is set to TRUE the value for DB_BLOCK_BUFFERS should equate to a buffer cache that is AT LEAST equal to AWE_WINDOW_MEMORY. In most cases, the total buffer cache size will be greater than AWE_WINDOW_MEMORY. If you attempt to start up with a buffer cache that is too small (i.e. < AWE_WINDOW_MEMORY) that may also result in the ORA-27102 error. Note#3 - It has been observed on some systems that you may need to add a few additional meg to AWE_WINDOW_MEMORY to calculate for overhead. if you go through the above calculations, and the instance still does not start, try adding an additional 10 Meg or so to the calculated value. Note#4 - Also, keep in mind that when calculating system, you have to take hyperthreading into account. On the OS will think that you have double the # of CPU's in actually have, and this is the number that must be used in the # of CPU's in the a hyperthreaded system, the system over what you the calculations. Therefore,

How to calculate the maximum used memory ========================================= With respect to awe_window_memory the following maximum amount of memory can be used from physical memory: The SGA size is composed from: ((db_block_buffers * block size) + (shared_pool_size + large_pool_size + java_pool_size + log_buffers) + 1MB The size of SGA + Oracle's overhead must not exceed the available virtual memory. The size of buffer cache depends on the available virual memory and can be calculated with buffer cache = db_block_buffer * db_block_size

CPU Spins Possible When Using AWE Implementation: ================================================= Use caution when setting _DB_BLOCK_LRU_LATCHES or AWE_WINDOW_MEMORY too low. If we are unable to map a requested buffer into the window because all of the space defined by AWE_WINDOW_MEMORY is in use with buffers already actively being accessed, then we spin and wait, checking every so often until an existing buffer in the window can be unmapped, and a new buffer can be mapped in. This spin will consume CPU cycles until enough buffers can be Mapped/Unmapped to satisfy the request. In some cases, there may be so

many buffers needing to be mapped into the window, that DBWR will consume 100% of cycles on all CPUs, effectively locking up the machine. This is normal behavior under some circumstances, and is simply an indication that AWE_WINDOW_MEMORY is too small. Monitoring Mapping Operations in 9.2 and later releases: ======================================================== Starting with 9.2, we have added additional statistics which can be measured in v$sesstat (sesssion-level stats) and v$sysstat (system-wide stats): STATISTIC# ---------154 number 155 number NAME -----------------------------of map operations of map misses

This query below will give you system-wide information on map operations and map misses: SQL> select * from v$sysstat where statistic# in (154, 155); If the # of Map misses is relatively high, or particularly of the # of map misses increases consistently over time, this may be an indication that the value for AWE_WINDOW_MEMORY is set too low.

Note that the statistic#'s change from version to version, so the below query will allow you to determine the statistic# for your particular DB version. this example is from a 10gR2 database: SQL> select statistic#, name from v$sysstat where name like '%map %'; STATISTIC# ---------168 169 NAME ---------------------------------------------------------------number of map operations number of map misses

So simply substitute in the correct statistic#, depending on your DB version. Dynamic Memory Management/Automatic Memory Management with AWE Enabled ============================================================= Oracle10g introduces the concept of Automatic Memory Management, whereby the Oracle RDBMS will dynamically adjust SGA parameters such as SHARED_POOL_SIZE, JAVA_POOL_SIZE, DB_CACHE_SIZE, etc. This is enabled by setting the parameter SGA_TARGET to a non-zero value. However, in order for this to work properly, you must use DB_CACHE_SIZE for the buffer cache. When setting USE_INDIRECT_DATA_BUFFERS, you cannot set DB_CACHE_SIZE, as noted above. Therefore, SGA_TARGET should not be set when using AWE - these two features are mutally exclusive. When setting USE_INDIRECT_DATA_BUFFERS=TRUE on Oracle10g, you should also set SGA_TARGET to 0.

Diagnosing Spins Associated With AWE in 8.1.x:

============================================== The above stats are not available in 8.1.7, so if you are encountering problems with CPU spins, with AWE_WINDOW_MEMORY enabled, it is more difficult to diagnose. You can start by identifying and monitoring the thread associated with DBWR via the following query: SQL> select b.name, p.spid where p.addr=b.paddr; NAME ----PMON DBW0 LGWR CKPT SMON RECO SNP0 SNP1 SNP2 SNP3 SPID --------1900 1956 572 1908 1808 920 1784 1892 1896 1844 from v$process p, v$bgprocess b

10 rows selected. As you can see, DBWR has an SPID of 1956, which will equate to the Thread ID of that thread within the Oracle executable. This thread can then be monitored using Performance Monitor and/or the PSLIST utility, which is available as a free download from http://www.sysinternals.com If your monitoring shows that DBWR is consuming excessive CPU, you can attempt to get an errorstack from that thread using oradebug: SQL> oradebug setospid 1956 Oracle pid: 3, Windows thread id: 1956, image: ORACLE.EXE SQL> oradebug unlimit Statement processed. SQL> oradebug dump errorstack 3 Statement processed. This should dump the errorstack to the DBWR trace file, found in BDUMP. If the errorstack contains the function SKGMMAP, this is an indication that DBWR is working to map/unmap database block buffers. Note: In 8.1.7 of the RDBMS, you cannot use DBWR_IO_SLAVES in combination with USE_INDIRECT_DATA_BUFFERS, due to BUG#3042660/BUG#2215894. You must leave DBWR_IO_SLAVES at its default value - otherwise, buffers are not unmapped and eventually a spin of the process will result. This problem is resolved in 9.2.0.1 - the fix is NOT backported to 8.1.7 KNOWN ISSUES -------------------BUG#2461474 - SHOW SGA DOES NOT SHOW CORRECT # OF DB BUFFERS412485 - LONG SHUTDOWN TIME WITH AWE_WINDOW_MEMORY: FIXED IN 8.1.7.1 BUG#1406194 - AWE_WINDOW_MEMORY NOT RELEASED WHEN DB SHUTDOWN: FIXED IN 8.1.7.1 BUG#2520796 - ORA-439 TRYING TO ENABLE VLM IN STANDARD EDITION OF ORACLE FIXED IN 9.2.0.4

BUG#2945011 - VLM DOES NOT WORK ON STANDARD EDITION ORACLE 9.2.0.2 ON WINDOWS - FIXED IN 9.2.0.4 BUG#3120033 - ORA-600[KCBVMAP] may occur with AWE, or DBWR may crash with ORA471 on 9.2.0.4 - FIXED 9.2.0.4 PATCH 2 BUG#3042660 / BUG#2215894 - IO SLAVES DON'T UNMAP BUFFERS ON LINUX IN VLM MODE (APPLIES TO WINDOWS AS WELL) RELATED DOCUMENTS ----------------u! 1"033## - WINDOWS AWE MEMORY AND CPU - CAN'T MAP THE BUFFER Note 46001.1 - Oracle Database and the Windows NT memory architecture Note 46053.1 - Windows NT Memory Architecture Overview Oracle 8.1.7 Release Notes for Windows: http://download-west.oracle.com/docs/pdf/A85305_01.pdf Oracle9i Database Getting Started Release 2 (9.2) for Windows Part Number A95490-01 Chapter 4: Oracle9i Architecture on Windows http://downloadwest.oracle.com/docs/cd/B10501_01/win.920/a95490/architec.htm#1005809 Oracle Database Platform Guide 10g Release 1 (10.1) for Windows Part Number B10113-01 Chapter 1: Oracle Database Architecture on Windows http://otn.oracle.com/pls/db10g/portal.portal_ <<End_of_Artic