12.

2 Ebusiness Suite - Collecting Fusion Middleware Log Files

In this Document

Goal
Solution
Section 1: Steps to collect log files
Alternative...
Section 2: Weblogic Log Viewer tool

Section 3: Steps to configure higher debug levels and collect log files

Section 4: Collecting configuration Information

Appendix A: Script for automatically clearing contents of log files

References

APPLIES TO:

Oracle Applications Technology Stack - Version 12.2 to 12.2 [Release 12.2]

Oracle EBS Applications Performance - Version 12.2 to 12.2 [Release 12.2]
Information in this document applies to any platform.

GOAL

This note is intended to be used as a debugging tool for the Fusion Middleware 11g components of
E-Business Suite R12.2 which include OPMN, OHS and WebLogic components

These include:

Oracle Process Manager and Notifications Server which manages the HTTP Server

Oracle HTTP Server (Apache) which acts as a single Entry Point Proxy Server

WebLogic Server which contains Oracle E-Business Suite Application Server Deployments

This note provides steps to enable extra debug and automate collection of log files required for
debugging issues.

Any changes made to increase log levels should be reverted once the information is collected. Oracle
does not recommend detailed logging be turned on for extended periods, especially in production.

SOLUTION
Section 1: Steps to collect log files

This section should be followed by customers who are creating service requests for issues that have
perhaps already occurred or which are not easily reproducible. This section does not enable extra
debug - it simply collects log files at whatever debug levels are currently configured.
In some cases the debug collected here may not suffice and in that case you may be directed by
support to follow the steps in Section 2 to enable extra debug.

a) OPMN and OHS log files

Run this script as the owner of the applications file system to gather the log files for OPMN and
OHS services:
zip -r /tmp/`uname -n`_`date +%m%d%y.%H
%M`_OPMN_OHS.zip $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/*/*log*
$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/*

b) WebLogic Server

Run this script as the owner of the applications file system to gather the log files for Fusion
Middleware components such as NodeManager, AdminServer, forms, oacore and oafm services:
zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_FMW.zip $IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager
$EBS_DOMAIN_HOME/servers/oa*/logs/*out* $EBS_DOMAIN_HOME/servers/oa*/logs/*log*
$EBS_DOMAIN_HOME/servers/forms*/logs/*out* $EBS_DOMAIN_HOME/servers/forms*/logs/*log*
$EBS_DOMAIN_HOME/servers/AdminServer/logs/*out* $EBS_DOMAIN_HOME/servers/AdminServer/logs/*log*
$EBS_DOMAIN_HOME/sysman/log/* $EBS_DOMAIN_HOME/servers/oac*/adr/diag/ofm/EBS_domain_*/oac*/incident/*
$EBS_DOMAIN_HOME/servers/forms_s*/adr/diag/ofm/EBS_domain_*/forms_s*/incident/*

These 2 commands will create zip files in the /tmp directory with names including server name and
date. e.g. server.customer.com_092811.2005_OPMN_OHS.zip. Please collect these files from the
/tmp directory and upload them to your service request.

Alternative...

Use the following script to gather the Apache, WLS and Apps log files which have been modified in
the last 1 day

#################################################################################
####
#### Start of script
#################################################################################
####
(
# pick up files which have been modified in the last 1 day only
HowManyDaysOld=1
echo "Picking up files which have been modified in the last ${HowManyDaysOld} days"
set -x
find $LOG_HOME -type f -mtime -${HowManyDaysOld} > m.tmp
find $FMW_HOME/webtier/instances/*/diagnostics/logs -type f -mtime -${HowManyDaysOld} >>
m.tmp
find $FMW_HOME/wlserver_10.3/common/nodemanager/nmHome*/*.log -type f -mtime -$
{HowManyDaysOld} >> m.tmp
## Get log files for only the WLS servers needed. Valid server names are one or more of:
## AdminServer forms-c4ws_server forms_server oacore_server oaea_server oafm_server
for SERVERNAME in AdminServer oacore_server forms_server oafm_server
do
find $EBS_DOMAIN_HOME/servers/${SERVERNAME}*/logs -type f -mtime -$
{HowManyDaysOld} >> m.tmp
find $EBS_DOMAIN_HOME/servers/${SERVERNAME}*/adr/diag/ofm/EBS_domain_*/$
{SERVERNAME}*/incident -type f -mtime -${HowManyDaysOld} >> m.tmp
done
zip -r mzAppsLogFiles_`hostname`_`date '+%m%d%y'`.zip -@ < m.tmp
rm m.tmp
) 2>&1 | tee mzLogZip.out
#################################################################################
####
#### End of script
#################################################################################
####

Section 2: Weblogic Log Viewer tool

The Weblogic Log Viewer tool can be useful for traversing log files and aid in quickly finding error
messages within large log files.

You can download this tool from here

Section 3: Steps to configure higher debug levels and collect log files

This section explains the steps to enable higher logging levels to capture debug information required
for resolving problems. You may be directed to this section by Oracle Support. Follow the steps
below to enable debug and collect the log files generated after reproducing the issue. Generally
speaking, increasing the log levels as per below should have only a minimal impact on performance
(max 5% degradation)

a) OPMN
To enable debug for opmn, edit
$IAS_ORACLE_HOME/instances/<instance>/config/OPMN/opmn/opmn.xml

Change the following line

<debug comp="" rotation-size="1500000"/>

to
<debug comp="ons[all];pm[all]" rotation-size="1500000"/>

alternatively you can increase opmn logging for the current session by issuing command
$IAS_ORACLE_HOME/instances/<yourinstance>/bin/opmnctl set target=debug comp="ons[all];pm[all]"

This would enable debug for all components running on the opmn process...however this may not be
practical in most situations

b) Increasing OHS Logging

To adjust HTTP logging levels access the Enterprise Manager console as follows
http://server.domain:wls_admin_port/em
login as weblogic/<weblogic admin password>

Expand the Web Tier menu

Right click EBS_web_component and choose administration - log configuration
Here you have the option to set the Log level. For debug purposes you can set this to "Trace:32"

c) Node Manager
To adjust log levels for Node Manager edit
$IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager/nodemanager.properties

adjust LogLevel=INFO to LogLevel=FINEST

d) Start and Stop Logging

To adjust start/stop log levels access the Enterprise Manager console as follows
http://server.domain:wls_admin_port/em
login as weblogic/welcome1

Expand WebLogic Domain then expand EBS_domain_X

Right click AdminServer and select logs - log configuration
Change the logging level to Trace:32

e) Web Logic Logging

Step 1: Enable log recording levels

To adjust WebLogic log levels access the WebLogic console as follows
http://server.domain:wls_admin_port/console
login as weblogic/<password>

Expand environment - servers

Select the servers to enable debug for. In most cases this will be the oacore servers. If you have more
than one server defined in a cluster you will need to enable debug for each server individually. Only
enable logging for the managed servers that are having issues.

Choose the logging tab, select the advanced hyperlink

Change severity level to Trace
Change Severity levels for standard out and domain log broadcaster to Debug

On this same configuration page you will see the setting "limit number of retained files". You may
need to increase this value as extra logging may result in log files being overwritten due to excessive
log file rotation.

Step 2: Enable debug areas

Continuing from step 1 above, select the debug tab
Expand the Weblogic section
Select the Servlet group which you wish to enable debug for (see below)
Choose enable to activate debug for these components.

Servlet Enable this for all types of issues

JDBC Enable logging for this component if your is
connection problems
Classloader Enable logging for this component for issues
classloading. Such as missing classes on star

Note: If support or the customer is directed to enable any other component besides JDBC, servlet or
classloader please inform the note author so these can be considered for addition to this note.

f) Restart Application Tier Services and clear log files

Stop application tier services using the scripts located in $ADMIN_SCRIPTS_HOME. Clear the log
files and then restart services
Note: Ideally at this stage it is a good idea to clear log files so that the problem can be pinpointed
easily in the log files.This is especially true of production systems where logs can be large and
almost un-manageable to debug. Log files should be cleared when services are down. You can do this
manually or use the script provided in Appendix A

g) Reproduce issue and upload log files

Reproduce the issue and use the scripts in Section 1 to gather the log files. Upload the log files to
your service request

Section 4: Collecting configuration Information

a) Run the technology stack inventory report to collect a list of patches applied to all middle tier
homes (besides Weblogic). This script will also report the patch level for each oracle home:
$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP
-contextfile=$CONTEXT_FILE -appspass=<appspassword>
-outfile=$APPLTMP/Report_App_Inventory.html

b) Provide the following files which contain managed server configuration information:

$IAS_ORACLE_HOME/instances/EBS_web_<SID>_OHS1/config/OHS/EBS_web_<SID> (ZIP
all files)
$INST_TOP/appl/admin/*.properties
$EBS_DOMAIN_HOME/config/config.xml (contains managed server start / stop options and
memory settings)
$EBS_APPS_DEPLOYMENT_DIR/[forms forms-c4ws oacore oafm] /APP-INF/node_info.txt

Appendix A: Script for automatically clearing contents of log files

The following script can be run as the applmgr user with applications environment sourced. This
script will automatically clear the contents of the log files so that fresh logs can be captured. All
application tier services should be shutdown prior to running this script.
for files in $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn $EBS_DOMAIN_HOME/sysman/log
do
for file in `ls $files/*`
do
echo "Clearing file - " $file
cat /dev/null > $file
done
done
for files in $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/EBS_web_component
$EBS_DOMAIN_HOME/servers/*/logs $IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager
do
for file in `ls $files/*log* $files/*out*| grep -v lck`
do
echo "Clearing file - " $file
cat /dev/null > $file
done
done

Note: one can comment the following line

#cat /dev/null > $file
so that the command identifies the files without making any changes. Once comfortable with the
changes the comment can be removed and the script run again to clear the files