Oracle Database

Acceleration
Discovery Day
Oracle on Solaris:
Tips, Tricks &
Best Practices
Andreas Neuhold
Commercial Products Sales Lead
Sun Microsystems
1
Agenda

> Orac
le & S o
laris : #
1
> Orac
le D B o
n Solar
is
> Orac
le DB i n
Contain
> Orac ers
le in C o
ntainer
> Res
C lusters
ources

2
Source: Oracle + Sun Strategy Webcast 3
 Number of Unique Customers Entering Tars for
Mainstream Unix Platforms –
2H FY2007, 2H FY2008, 2H FY2009

8000
7000
6000
5000
4000 2H2007
3000 2H2008
2000 2H2009

1000
0
Solaris SPARC Solaris x64 HP-UX ItaniumHP-UX PA-RISC AIX 64
64

4
Percentage of UNIX Customers for mainstream UNIX
platforms per Service Request – 2HFY2009

Solaris SPARC

Solaris x64
27%
36%
HP-UX Itanium

14% HP-UX PA-

10%
RISC
13%
AIX 5L 64-bit

• Total Percentage of Customers per UNIX O/S

• 46% Solaris
• 27% HP-UX
• 27% AIX

5
Agenda

> Orac
le & S o
laris : #
1
> Orac
le D B o
n Solar
is
> Orac
le DB i n
Contain
> Orac ers
le in C o
ntainer
> Res
C lusters
ources

6
• Oracle DB on Solaris – based on

> “Best Practices for Optimal Configuration

of Oracle Databases on Sun Hardware”
by Allan Packer and Bob Sneed

> Oracle Openworld 2009, San Francisco,

CA

http://wikis.sun.com/download/attachments/128484865/OOW09 Optimal Oracle DB

Perf 1_1.pdf

7
Oracle on Solaris
• Solaris 10 is the #1 UNIX platform for Oracle
deployments
• Solaris benefits from many years of performance
enhancements designed to benefit Oracle
• Solaris offers outstanding -

> Scalability, to hundreds of CPUs

> Observability (e.g. Dtrace)
> Virtualization (container / zones, @ no additional cost)
> File System technology (ZFS – no limits)
> Robustness, Security, Application support,...

8
More than 15 years of joint
Solaris/Oracle DBMS Optimizations
• Intimate Shared Memory (1993) /
Dynamic Intimate Shared Memory (2002)
• Scheduler control optimizations (1997)
• 64-bit memory space (1998)
• /dev/poll (1998)
• Reconfiguration Coordination Manager (2002)
• Locality Groups (2002)
• Large Page Support (2002, 2005)
• FX – Fixed Priority Class (2002)
• Direct IO with concurrent writes (2002)
• Sun Cluster optimizations for Oracle RAC (multiple)

http://wikis.sun.com/display/Performance/Key+Sun+technologies+for+running+Oracle
9
 The Dark Underbelly of DB Workloads:
Data Stalls
• Data stalls dominate execution time
> 46%-64% of execution time wasted on data stalls
> True for both OLTP and DSS database workloads
> Even more execution time wasted on instruction stalls
> Especially for OLTP, where you waste a further 10%-15%

• This problem applies to typical multi-core chips

> Designed with sophisticated out-of-order cores and aggressive speculation

Source: “Database Servers on Chip Multiprocessors: Limitations and Opportunities”

September 2006, Carnegie Mellon University

10
 Everything Changes with Sun's
CoolThreads CMT Systems
• Data stalls drop to 13%
> 76%-80% of execution time spent on usefull computation
> True for both OLTP and DSS database workloads
> Instruction stalls are negligible
> END RESULT: 70% higher throughput than traditional systems

• BUT: not good for unsaturated workloads

> Single-threaded workloads perform 41% worse
> So CMT is a win, provided you keep the CPUs busy

Source: “Database Servers on Chip Multiprocessors: Limitations and Opportunities”

September 2006, Carnegie Mellon University

11
A Simple Analogy: Cars are Good!

12
Buses are good, too

13
Oracle Performance Expectations
• Most OLTP environments run very well with CMT
> Understand response time components
> DB CPU usually around 20% of response time
> IO, Network, and application tier makes up the rest
• Bad code is a bottleneck on any server
> CPU-bound bad code bites you sooner on CMT servers
• Serial Batch jobs run slower with CMT
> Check the usual suspects
> Execution Plans, Indexes, Business Logic
> Divide and Conquer
> Increase Concurrency and Parallelism

14
 Oracle on SPARC Servers – Summary
• Sun's CMT Server Advantages
> Low power consumption
> Maximum rack computational density
> Logical Domains (LDoms) powerful and bundled (by use of Solaris 10)
• Sun's M-Series Server Advantages
> Outstanding scalability
> Enterprise-class RAS features
> Flexible domaining
• Solaris Containers run on both CMT and M-Series
> Secure virtual OS instances, negligible perf overhead
> Capped Containers recognized for licensing purposes

15
 Best Practices Categories
• Avoiding known performance bugs
> Some are surprisingly high-impact
> Architecturally-dependent factors are not widely-known
• Copying proven patterns, avoiding known antipatterns
> Filesystem and network choices
> “Big knob” tunables
> Platform- or application-specific settings (some not widely-known)
• In general, being disciplined about performance
> Holistically, including the application code, solution architectures, operational
plans, backup and recovery, business continuance
> Methods, tools, interdisciplinary and interdepartmental alignment, frameworks,
metrics, processes

NOTE: While there is no clear objective line between “Best Practices” and “tuning” - the distinction is much
like how “just do it” relates to “optimization”. Both may be needed to meet system performance objectives.
16
Oracle Performance Bugs & Patches
• 10.2.0.4 is a Good Place to be, with these fixes (among others) ...
> Bug #6814520 - Compute efficiency of block checksum code on SPARC was
very poor
> Bug #5752399 - For enabling direct I/O on certain filesytems,
filesystemio_options=setall was broken

• 10.2.0.4 has point patches available for these ...

> Excessive client attach latency & high cost of session initiation
> Bug #8737425 - Excessive close() calls on new session creation
> Scalability/consolidation issue: kstat overhead excessive with many instances
running
> Patch #8531434 – "KSTAT CALLS BY MMNL/CJQ0 INCUR HIGH
SYS ...”
> RE: Oracle NUMA features, see Metalink #759565.1
> Patch #8199533 - Disables NUMA features as new default

17
Filesystem/Feature Selection
• Performance Best Practice: use an unbuffered, concurrent I/O stack and
consistent contiguous ondisk data placement

> Solutions with these characteristics perform similarly

> RAW
> UFS direct I/O
> QFS with samaio or qwrites
> ASM
> VxFS with ODM, QIO, or direct with 'cio' (VxFS has new 'cio' option; ODM no longer an
extra fee)
> NFS with direct I/O (Of course, performance depends greatly with NFS server factors and
network factors)

• Each solution has its own footnotes

> See: http://blogs.sun.com/bobs

18
Control LGWR Priority on Purpose
• Run LGWR at Fixed Priority (FX) 60, even when FSS is in use
> Otherwise, TS scheduling will reduce its priority when it gets busy -
exactly opposite of what one would want!
• Other background processes may also benefit from special
scheduling, but LGWR is uniquely critical path to all
transactions
• This is especially important on systems with high aggregate
CPU utilization or systems with slower single-thread speeds

19
RAC: Use Jumbo Frames!
• Jumbo frames delivers multiple benefits
> Reduces GC latency
> Reduces CPU utilization, %sys, and hardware interrupts
> Avoids scaling challenges of IP packet reassembly
> Radically improves scaling of RAC on CMT CPUs
• Implementation notes
> Oracle specifies dedicated interconnects for RAC; jumbo frames should not be
an 'infrastructure project'

• See also
> http://blogs.sun.com/glennf/entry/jumbo_frames_with_oracle_rac

20
For more information
Check out the Sun Performance Wiki!
> For Sun Performance information generally

> Servers, Storage, and Software

> For Oracle on Sun Performance information in particular

http://wikis.sun.com/display/performance

21
Agenda

> Orac
le & S o
laris : #
1
> Orac
le D B o
n Solar
is
> Orac
le DB i n
Contain
> Orac ers
le in C o
ntainer
> Res
C lusters
ources

22
Whitepaper

23
Solaris Containers
• Provide a virtual OS environment for applications; ideal for
consolidation, e.g. Oracle Dbs

• Secure execution environment with priviliges granted from the

global zone
> No way of reaching out of the zone except via the network

• Many controls for resource management

> CPU sets, dedicated CPUs, max CPUs

• Most resources must be mapped into a container explicitely

> e.g. filesystems, devices, etc.

24
Solaris 10 Container
• OS Virtualization
> One single OS installation
> Multiple runtime environments, appear
Database
System
App as “regular” Solaris OS to applications
Appl.Srv.
and users
Container Container Container
• Flexible, software-defined boundaries
OS > “Can't see outside zone”
> “Can't affect activity outside zone”

• Simple isolation
Server > Network, disks, memory, CPU,
processes, environment
• Each Container has it's own identity
(/etc/...)

What is a Solaris 10 Container ??

It is NOT an additional OS installation. But it is a separated runtime environment, within an
existing Solaris 10 OS installation.
Analogy with Databases: One single RDBMS installation, providing multiple separated
database instances.
25
Solaris Container – Technical view
(Whole-root Zone)
cpu-shares = 30 cpu-shares = 20 cpu-shares = 50

memory_target = 3G memory_target = 2G em/initial_size_MB = 4096

abap/heap_area_total = 2048

Global Zone Local Zone Local Zone Local Zone

Database Database ERP System
/ / / /
/etc /etc /etc /etc
/lib /lib /lib /lib
/opt /opt /opt /opt
/platform /platform /platform /platform
/usr /usr /usr /usr
/var /var /var /var
/u01/ora/product/... /u01/ora/product/... /usr/sap/P01
/u01/ora/fra/CRM1 /u01/ora/fra/CRM2 /sapmnt/P01
/u02/ora/data /u02/ora/data /oracle/P01

/dev → lofs mount ...

Solaris 10 Kernel
26
Solaris Containers
A convenient virtual platform for Oracle
Solaris 10 Containers have unique features, valuable especially in
Oracle environments !!
• Flexibility
> Fast creation (takes only couple of minutes, no new OS Installation !),
Reboot, Migrate, Clone.
• Consolidation
> Isolate applications software stacks and services.
> "Security island" (EAL4+ certification).
> Resource Management. Limit or assure resources (CPU, Memory, IO
bandwidth) for Containers. Online and dynamically.
• Standardization & Administration
> Reuse one quality-assured Solaris OS implementation for the system
landscape.
> Only one OS installation for multiple separated runtime environments. Thus
providing not only a server consolidation but also an OS consolidation.
• Built-in
> Contained in Solaris 10. No additional (3rd party) software installation. No
additional license fees or support costs.

27
Solaris Containers
A convenient virtual platform for Oracle Part2
Solaris 10 Containers have unique features, valuable especially in
Oracle environments !!

• Scalable
> Up to 8192 Container. Nearly no performance degradation. (SAP Benchmark:
1,25% for 4 Containers).
• Platform Support
> Available for all Solaris 10 platforms: Intel, AMD, UltraSPARC on Sun, Dell,
HP, IBM, Fujitsu/Siemens and more ...
• Oracle Support
> Oracle does support “single instance” in Solaris 10 Containers.
> New:
New Oracle will certify and support RAC in Solaris 10 Containers. (Status:
QA).
> Container (CPU pool or CPU cap) is accepted as Oracle license limitation.
• Proven
> Many customer references

28
Highly Available Solaris Containers
• Containers can be moved between Servers
• Containers can be made HA using Oracle Solaris Cluster

> HA Container agent

> Blackbox approach
> Zone nodes
> Switch Applications between “static” containers
> Container Clusters
> Full virtual cluster

29
Agenda

> Orac
le & S o
laris : #
1
> Orac
le D B o
n Solar
is
> Orac
le DB i n
Contain
> Orac ers
le in C o
ntainer
> Res
C lusters
ources

30
Blueprint

31
Container Clusters - why

• A logical step forward – a virtual cluster for a

virtual OS
• Delegation of cluster administration
• Encapsulated cluster
> Global components only manageable from the global zone
• Primary target application:
> Oracle Real Application Cluster

32
Oracle Solaris Container Cluster
Best Environment For Multi-tier Deployments
Node 1 Node 2 Node 3 Node 4

Finance
Finance Finance Container
Cluster with
RAC 10g Application fault isolation
Security isolation
Sales
Container Sales Sales Consolidation use cases
Cluster with
RAC 10g • Multiple Oracle RAC
databases
Engineering Engineering • Test & Development
Engineering Container Engineering Engineering
databases
Cluster with
RAC 11g • Multiple data center tier /
multiple application

Finance Database Engineering Database

Sales Database

33
Agenda

> Orac
le & S o
laris : #
1
> Orac
le D B o
n Solar
is
> Orac
le DB i n
Contain
> Orac ers
le in C o
ntainer
> Res
C lusters
ources

34
Resources
• Cool Tools
> http://cooltools.sunsource.net/
• Blueprints
> http://wikis.sun.com/display/BluePrints/Main
• CoolThreads Experts Q&A Portal
> http://www.sun.com/secure/servers/coolthreads/tnb/contact.jsp
• Performance Wiki
> http://wikis.sun.com/display/performance
• Blogroll
> Glenn Fawcett - http://blogs.sun.com/glennf
> Denis Sheahan - http://blogs.sun.com/deniss
> Tanel Poder - http://blog.tanelpoder.com
> Greg Rahn - http://structureddata.org
> Bob Sneed - http://blogs.sun.com/bobs (paused)
> Allan Packer - http://blogs.sun.com/allanp

35
Oracle Database
Acceleration
Discovery Day
Oracle on Solaris:
Tips, Tricks &
Best Practices

Andreas Neuhold
andreas.neuhold@sun.com

36