Oracle E-Business Suite 12.

2
Administration

Prepared by:
Ivo Dujmovic
Ivo@patchivo.com
Entrepreneur
Patchivo, Interana

REMINDER
Check in on the
COLLABORATE mobile app

About the Presenter

Ivo Dujmovic
Patchivo : 12.2 PatchStatus free iPhone app

Specialized tools delivering insight into IT

ivo@patchivo.com

Interana : Hadoop accelerator for customer behavior analytics

Stealth VC-backed startup

ivo@interana.com

Formerly Oracle Snr Director of Development

Managed development organization for EBS Administrative

functionality: Install, Cloning, Configuration, Patching, EM plugins,
OAM, iSetup, TechStack

Wrapped up 12.2 to pursue entrepreneurship

Holds Online Patching patents

Main Changes from 12.0/1 to 12.2

Re-platforming to FMW 11 , DB 11.2
same WLS, webtier, DB as used by Fusion Applications

Fusion applications use a different FND core version

Native consumption of technology

Install: bootstraps then defers to native tech installs

Configuration: bootstraps then defers to native tech config
Cloning: wraps native tech cloning
10.1.2 Forms & Reports considered EBS internal

Online Patching
Uses EBR from 11.2 of DB
New concepts to EBS: dual FS, FS cloning
Cheat sheet for Online Patching

Fusion Middleware (FMW) 11g

WebLogic Server (WLS) 11g
BEA acquisition, first partially integrated version
Key concepts

Admin Server manages Managed Servers (farm)

Managed Servers have local copy of config, apps if Admin Server is

unavailable

Industry leadership due to great administrative functionality

Deploy new version of staged App from AS to select MSs

Run two versions side-by-side or preview new version with secret url

Roll-back to previous version if needed

Other Technology
WebTier delivers Oracle HTTP Server, aka Apache 2.2
Needed for mod_security

Database
11.2 was the latest at development time, as well as the first
delivering EBR, the key feature for Online Patching
Grid Install concept:

Lay down grid infrastructure (storage management, fast network

interconnects) prior to installing the database binaries on top

Forms & Reports 10.1.2

Same as in other R12 releases
Forms servlet deployed to WLS instead of OC4J

12.2 Install
Preparation
Virtualization makes sense virtualbox with 64b Linux
9GB mem for the install; otherwise hard to troubleshoot bugs
Platform install doc has gotchas listed

Hostname/network sensitivity even more pronounced

OS patching and settings: install 32b libs before 64b ones

Staging script
When downloading zip files

You do not need all

You should not unzip them your self

Start with startCD and follow readme

See blog on patchivo.com for more tips & details

Dual File System

File System: 1

R2

File System: 2

Oracle HTTP Server

Oracle HTTP Server

WebLogic Server

WebLogic Server

Developer 10.1.2
APPL_TOP

COMMON_TOP
INST_TOP

Edition-Based
Redefinition

Developer 10.1.2

Synchronization managed
by patching tools

COMMON_TOP

File System: Non Editioned

APPL_TOP_NE
PATCH_TOP
LOGS

APPL_TOP

INST_TOP

12.2 File System for Install

$ ls /oracle
ebs122

oraInventory

stage

$ ls /oracle/stage/
zipMedia
client

Disk1

EBSInstallMedia

misc

startCD

current

Disk2

examples

patches

TechInstallMedia

database

Disk3

gateways

readme.htm

TechPatches

deinstall

Documents

grid

README.txt

wls1036_generic.jar

12.2 File System: FS1 , FS2 , FS_NE

$ cd /oracle/ebs122/
$ ls
EBSapps.env

fs1

fs2

fs_ne

VIS

$ . EBSapps.env

E-Business Suite Environment Information

---------------------------------------RUN File System : /oracle/ebs122/fs1/EBSapps/appl

PATCH File System : /oracle/ebs122/fs2/EBSapps/appl

Non-Editioned File System : /oracle/ebs122/fs_ne
DB Host: ip-10-28-88-207.ec2.internal

Service/SID: VIS

12.2 Key Environment Variables

FILE_EDITION=run
RUN_BASE=/oracle/ebs122/fs1
PATCH_BASE=/oracle/ebs122/fs2
CONTEXT_FILE=/oracle/ebs122/fs1/inst/apps/VIS_i
p-10-28-88-207/appl/admin/VIS_ip-10-28-88207.xml
ADMIN_SCRIPTS_HOME=/oracle/ebs122/fs1/inst/apps
/VIS_ip-10-28-88-207/admin/scripts

FS1 - File System One

$ ls fs1
EBSapps

FMW_Home

inst

$ ls fs1/EBSapps/
10.1.2

appl

comn

$ ls fs1/inst/
apps

$ ls fs1/inst/apps/VIS_ip-10-28-88-207/
admin

appltmp

logs

out

portal

soa

appl

conf_VIS.txt

ora

pids

rgf

temp

Instance Top of FS1

$ ls fs1/inst/apps/VIS_ip-10-28-88-207/appl
admin

fnd

$ ls fs1/inst/apps/VIS_ip-10-28-88-207/appl/admin/
adovars_VIS_ip-10-28-88-207.env

oamextensions.xml

cutover/

ojspCompile.conf

forms-c4ws_wls.properties

ojspCompile.properties

forms_wls.properties

VIS_ip-10-28-88-207_patch.txt

fsclone_config.txt

VIS_ip-10-28-88-207_run.txt

oacore_wls.properties

VIS_ip-10-28-88-207.xml

oafm_wls.properties

FS_NE Non-Editioned File System

$ ls fs_ne
EBSapps inst
$ ls fs_ne/EBSapps/
appl log patch
$ ls fs_ne/EBSapps/appl/
ad
$ ls fs_ne/EBSapps/log
adop adopreports
$ ls fs_ne/inst/
VIS_ip-10-28-88-207
$ ls fs_ne/inst/VIS_ip-10-28-88-207/
certs logs oracle_common soa

12.2 Database File System

$ ls
EBSapps.env

fs1

fs2

fs_ne

VIS

$ ls VIS/
11.2.0 checkpoints data diag
$ ls VIS/checkpoints/
$ ls VIS/diag/
asm clients crs diagtool lsnrctl
ofm rdbms tnslsnr

VIS/11.2.0 is the database ORACLE_HOME

netcman

12.2 Admin Scripts

$ ls $ADMIN_SCRIPTS_HOME
01041419.log

adexecsql.pl

adstpall.sh

jtffmctl.sh

adadminsrvctl.sh

adformsrvctl.sh

adstrtal.sh

msc

adalnctl.sh

admanagedsrvctl.sh

cz64bitengine.pl

mwactl.sh

adapcctl.sh

adnodemgrctl.sh

gsmstart.sh

mwactlwrpr.sh

adautocfg.sh

adopmnctl.sh

ieo

sqlnet.log

adcmctl.sh

adpreclone.pl

java.sh

$ ls /oracle/ebs122/VIS/11.2.0/appsutil/scripts/VIS_ip-10-28-88-207/

adautocfg.sh

addbctl.sh

adexecsql.pl

adpreclone.pl

adchknls.pl

addlnctl.sh

adlsnodes.sh

adstopdb.sql

adstrtdb.sql

New Cloning Type: FS Clone

Clone an Instance: Prod to Copy-of-Prod, Test1 to Test2
Clone a Node: scale-out, rolling upgrades
New: Clone a File System
Replicate one File System on a node, from the other File
System on that node

Instance and Node Cloning still copy bits for only one FS
At target bits are recycled for creating both FSs

ADOP AD Online Patching

$ which adop
/oracle/ebs122/fs_ne/EBSapps/appl/ad/bin/adop

$ adop -help
Applications DBA Online Patching Tool (adop)

Usage: adop [phase=<phase,phase,...>] [patches=<patch#,patch#,...>]

[<parameter>=<value> ...] [input_file=<filename>]

Enter adop -examples for a detailed list of parameters and their usage.

See Oracle E-Business Suite Maintenance Guide for a full description of

adop features, operation, and usage.

Online Patching Affects DBAs Life

Traditional Patching involved:
scheduling downtime, kicking off users, stopping services,
applying patch, restarting services

Online Patching allows for patch application to happen during

online use:
Schedule quiet time to apply patch, bounce services
Downtime slims to 5-20mins range, easier to schedule
If applying patchsets, load can be non-negligible

Target 1 RAC node (similar to reporting node)

Apply patches during quiet periods

Adworker count logic is improved, but can still be manually set

No workshifts/ramp-up

ADOP Phases
Standard phases:
prepare

- Prepare the instance for patch application.

apply

- Apply patches (to the patch edition).

finalize

- Ready the instance for cutover.

cutover

- Make the patch edition the new run edition.

cleanup

- Drop obsolete objects and data from old editions.

Special phases:
abort

- Abort the current patching cycle.

actualize_all - Create new copies of all code objects in the patch

edition.
fs_clone

- Copy the run file system to the patch file system.

ADOP Phase Parameters

Apply parameters:
patches=<patch#>[,<patch#>...]
A single patch or comma-separated list of patches to apply.
This parameter is required when executing the apply phase.
For language patches, you must also specify the driver file:
patches=10124646_AR:10124646.drv,10124646_KO:10124645.drv

restart=(yes|no)

[default: no]

Resume a failed apply action where processing left off.

abandon=(yes|no)

[default: no]

Re-apply a failed patch from the beginning.

Finalize parameters:
finalize_mode=(full|quick)

[default: quick]

Quick mode will provide the shortest execution time, by

More ADOP Phase Parameters

Finalize parameters:
finalize_mode=(full|quick)

[default: quick]

Quick mode will provide the shortest execution time, by

skipping non-essential additional actions.
Full mode performs additional actions such as gathering
statistics, which may improve performance after cutover.

Cutover parameters:
mtrestart=(yes|no)

[default: yes].

Specifies whether to restart application tier servers after

cutover. Leave at default unless performing manual steps
during downtime.

Cleanup parameters:

More ADOP Phase Parameters

Cleanup parameters:
cleanup_mode=(full|quick)

[default: quick]

Quick mode provides the shortest execution time, by

skipping non-essential additional actions.
Full mode performs additional processing to remove all
unused code, data, and old editions. May take a long time.

General parameters apply to all phases:

workers=<number> [default: computed]

Number of parallel workers used to execute tasks.

Default value is computed principally according to number of
available CPU cores.

All ADOP Phases: Shared Parameters

input_file=<file_name>
As well as being entered directly on the command line,
adop parameters can be specified in a text file, with
one <parameter>=<value> on each line of the file.
Examples:
phase=prepare,apply,finalize,cutover,cleanup

patches=123456
workers=4
Command line parameters override input file parameters.

loglevel=(statement|procedure|event|warning|error|unexpected)
[default: event]
Controls the level of diagnostic log detail displayed.

More Shared Parameters

allnodes=(yes|no) [default: yes]
Specifies whether actions should be executed on all
application tier nodes of a multi-node system.

action=(db|nodb)

[default: db]

Specifies whether to execute database actions (as well as

file system actions).

-status [<session_id>]

Display status of the latest adop session, or a specified session.

-help
-examples

Adop Examples
Complete patching cycle, running each phase separately:
adop phase=prepare
adop phase=apply patches=12345,67890 workers=4
adop phase=finalize workers=4
adop phase=cutover workers=4
adop phase=cleanup

Complete patching cycle, running all phases in a single adop command:

adop phase=prepare,apply,finalize,cutover,cleanup patches=12345

Complete patching cycle, specifying all parameters in an input_file:

adop input_file=adop2013_05_13.txt

Online Patching Cycle - Cutover

Cutover Requires a Brief Downtime
Middle-tier processes stopped

End Users are disconnected

File system roles swapped

Patched (FS-2) promoted to Run

FS-1 available for next patching cycle

Database Patch Edition

promoted to Run Edition
Middle-tier processes restarted

Users reconnect

Great Presentation on Online Patching

Slides with red corner from Kevin Hudsons:
https://oracleus.activeevents.com/2013/connect/sessionDetail.w
w?SESSION_ID=8432

ADOP Special Phases

abort

- Abort the current patching cycle.

actualize_all - Create new copies of all code objects in the patch

edition.

fs_clone

- Copy the run file system to the patch file system.

ADOP Behind The Curtain: Prepare

Prepare
Checks if cleanup was run after last cutover

If not, runs it now

Checks what was done to the other FS (currently run)

Patches applied
Configuration changed (both EBS and WLS,Webtier)

Redoes those on this FS: patches applied and configuration cloned

Checks if there was an aborted session on this FS (current

patch)

Losses confidence in the patchFS, so calls fs_clone to recreate the

patchFS from the runFS

ADOP Behind the Curtain: Apply

Apply
Applying many patches within a single list merges them
To apply patches without merging, use separate adop calls
Applying many patches unassisted in test systems

Use abandon = yes

Cutover
Does a finalize first if it was not run as last action before cutover

Run finalize before cutover

Runs certain (deferred) db actions from patch apply

Duration of cutover can be impacted by patches applied in cycle

Hotpatching
Apply a patch to runFS with your historical hotpatching
processes
adop phase=apply hotpatch=yes patches=
Must be done outside normal patching cycle, so if you already
did a prepare, you need to do a cutover (empty cycle)

Empty cycle does not require an fs_clone unlike the other option:
abort current cycle

Not recommended unless you are really sure in what you are
doing: review the patch, read all files delivered and actions listed
in driver (some actions are not really documented)
Know which users will be connected
Bounce processes

Patching Many Nodes

Adop automatically synchronizes patching across nodes
FS_NE is shared across nodes, with patches and logs
Sharing the file system amongst many nodes is possible to
the extent that the underlying technologies allow it
DB has Oracle Home sharing
Webtier / Apache 2.2 is easy to share
WLS has some competing infrastructure

Rolling patching is possible, but has no benefit

Applying Technology Stack Changes

[worse] Outside a patching cycle
it should be done to the Run file system
Followed by an FS_Clone to separate propagation of change
from patching

[better] Within the patching cycle

After prepare finishes, perform on target Patch file system

Applies to techstack config as well, although those changes

should be automatically detected and propagated

Custom Code Patching

During ADOP cycle
Apply to Patch FS

Register files in
FS_NE/EBSapps/appl/ad/custom/adop_sync.drv
Will propagate from Run to Patch FS

Special Cutover Processing

See doc or code for pre-cutover and post-cutover hooks
You can hang your special cutover processing functionality

Q&A

Oracle E-Business Suite 12.2

Administration
Session ID#:

This is a subtitle for the presentation

that can be extended to three lines

Prepared by:
Joseph Josephine
Head of Oracle Product Usage
Acme Manufacturing

REMINDER
Check in on the
COLLABORATE mobile app