Você está na página 1de 35
<Insert Picture Here>
<Insert Picture Here>
<Insert Picture Here> Demantra Diagnostic Tools VCP PMO Feb 2012

Demantra Diagnostic Tools

VCP PMO Feb 2012

Agenda

This presentation summarizes some of the key diagnostic tools that can be utilized when dealing with various Demantra errors

and problems.

Focus on the following areas:

Installation Data model Data loading (EP_Load and Integration Interfaces) DB deadlocks and internal errors

Web Application troubleshooting:

  • Application memory errors

  • WS performance Engine errors

© 2009 Oracle Corporation Proprietary and Confidential

Agenda • This presentation summarizes some of the key diagnostic tools that can be utilized when

2

Installation

The installer includes a configurable log file that can trace the install process.

The installer needs to be configured (first panel of the installer), otherwise no log will be written at all (in the past by default the log was written to the $temp$ directory this is not the case any more).

When having installation errors there are few traces that can be analyzed:

Installer log Screenshots of the selections in the different Installer panels.

Review db_audit_log table for object changes made by the upgrade

DB_Exception_Log table

© 2009 Oracle Corporation Proprietary and Confidential

Installation The installer includes a configurable log file that can trace the install process. The installer

3

Data Model:

When having data model errors there are few traces that can be

analyzed:

An Export of the Data model

Review db_audit_log table for object changes made by the upgrade.

LOG_DATA_MODEL table (an automatic LOG_IT log).

© 2009 Oracle Corporation Proprietary and Confidential

Data Model: When having data model errors there are few traces that can be analyzed: •

4

Data Loading:

Data loading includes 2 aspects Ep_Load stored procedures and Integration Interface.

Each one of them can be monitored both for tracking errors as well as tracking performance problems.

© 2009 Oracle Corporation Proprietary and Confidential

Data Loading: Data loading includes 2 aspects – Ep_Load stored procedures and Integration Interface. Each one

5

Data Loading Ep_Load:

Ep_load Errors:

Enable LOG_IT process on relevant Demantra stored procedures as defined in LOG_IT_PARAMS table

DB_Exception_Log.

Ep_Load performance:

Count of data from the staging tables. DB_Params table Ep_Load performance related parameters

Execute “Call_DM_Build_Procedures” after changing

parallelization parameters in DB_Params

DB HW definitions.

System Memory Disk IO and CPU Utilization monitoring- Either system performance monitor or Linux top events

AWR /ADDM reports.

© 2009 Oracle Corporation Proprietary and Confidential

Data Loading – Ep_Load: Ep_load Errors: • Enable LOG_IT process on relevant Demantra stored procedures as

6

Data Loading Integration:

Integration Errors:

Collaborator & Integration log (Enable sql logging: appserver.sql, appserver.update.sql)

Concurrent request logs (collaborator.login.user) apscheck.sql (Note: 246150.1)

Integration Performance:

Count of data from the staging tables. APS_Params Integration Interface related parameters DB HW definitions.

system memory disk IO and CPU utilization monitoring- either system performance monitor or Linux top events.

AWR /ADDM reports.

© 2009 Oracle Corporation Proprietary and Confidential

Data Loading – Integration: Integration Errors: • Collaborator & Integration log (Enable sql logging: appserver.sql, appserver.update.sql)

7

DB deadlock/ internal database error:

Collaborator Log

Database \udump logs (USER_DUMP_DEST specifies the pathname for a directory where the DB server will write debugging trace files on behalf of a user process.)

Enable LOG_IT process on selected Demantra stored procedures as defined in LOG_IT_Params table.

© 2009 Oracle Corporation Proprietary and Confidential

DB deadlock/ internal database error: • Collaborator Log • Database \udump logs ( USER_DUMP_DEST specifies the

8

Web Application troubleshooting:

Demantra Web application includes 2 components:

Application Server Demantra Web application deployment inside a Web server.

Client User side browser and resources (CPU, Memory) running SUN Java Plug-in.

They are connected by a Network (LAN, WAN…).

For better troubleshooting we need to understand where we have a problem is it on the Server side/

Client side / Both ?

© 2009 Oracle Corporation Proprietary and Confidential

Web Application troubleshooting: • Demantra Web application includes 2 components: • Application Server – Demantra Web

9

Application out of memory errors:

Identify where the problem is by using logs:

Collaborator log server side log. Plug-in log Client side log.

Server side issues:

Heap dump from the webserver

Thread dump from the Webserver (in case of application freeze or slow down)

Application server memory configuration (-Xmx) Count of Level members (need level member caching?)

Client side issues:

Heap dump from the Java plug-in. Thread dump from the Java plug-in Plug-in memory configuration (-Xmx)

© 2009 Oracle Corporation Proprietary and Confidential

Application out of memory errors: • Identify where the problem is by using logs: • Collaborator

10

WS performance and errors:

Identify where the problem is:

Server side AWR /ADDM reports, DB side CPU and Memory. Application server CPU and configuration, APS_PARAMS relevant

parameters. Client side - CPU utilization (and for how long).

Server side issues:

Demantra Index Advisor report. Thread dump (in case of application freeze or slow down). System Memory Disk IO and CPU Utilization monitoring- either system performance monitor or Linux top events. DB HW definitions. Number of concurrent users at the time (concurrent user log).

Client side issues:

Plug-in Xmx setting WS wizard panels screen shots.

Client side log If requested (Number of combinations in the WS).

© 2009 Oracle Corporation Proprietary and Confidential

WS performance and errors: • Identify where the problem is: • Server side – AWR /ADDM

11

Engine errors:

Always provide a full Engine log (manager and engine) with relevant logging option.

DB errors:

Engine log with Sql logging depends on where the DB error is Engine manager or engine2k.

Processor error:

Engine log with DebugPro logging’ groups 100/110.

Engine Out of memory errors:

No of Tasks

Count of Active Combinations (Select Count(1) from MDP_Matrix where prediction_status = 1).

© 2009 Oracle Corporation Proprietary and Confidential

Engine errors: Always provide a full Engine log (manager and engine) with relevant logging option. •

12

Engine errors cont’:

Engine not running:

On Linux Application server log

Engine performance:

AWR report

Check whether CPU Hyper threading enabled on Engine servers Disable if its enabled

Engine Blades/Server + DB HW definitions.

DB/Engine Blades Memory Disk IO and CPU Utilization Monitoring - Either

system performance monitor or Linux top events.

For more information check Oracle Support note 800030.1 Demantra Engine Troubleshooting” Presentation (April 6 th

2011).

© 2009 Oracle Corporation Proprietary and Confidential

Engine errors cont’: • Engine not running: • On Linux – Application server log • Engine

13

Appedix

Configure auditing capabilities for BM configuration changes:

Client expression Evaluator

© 2009 Oracle Corporation Proprietary and Confidential

Appedix • Configure auditing capabilities for BM configuration changes: • Client expression Evaluator © 2009 Oracle

14

Configure auditing capabilities for BM

configuration changes:

This is a customization enhancement introduced by DEV application team in a white paper (“Implementing Demantra Audit

log”)

Currently captures Inserts/ Updates on Sessions/ Computed_fields tables by a trigger based process

The mechanism and idea can easily be enhanced for further Metadata audit opportunities

Difference from BAL diff this mechanism captures all the changes, not just how the latest change differs between

schemas

© 2009 Oracle Corporation Proprietary and Confidential

Configure auditing capabilities for BM configuration changes: • This is a customization enhancement introduced by DEV

15

Configure auditing capabilities for BM

configuration changes (cont’):

The logs are kept in a separate table (AUDIT_LOG ), so no impact on the out of the box schema.

AUDIT_LOG table includes the following information (columns):

log_date session_id os_user host_name module user_id object_type object_name old_value new_value Message

© 2009 Oracle Corporation Proprietary and Confidential

Configure auditing capabilities for BM configuration changes (cont’): • The logs are kept in a separate

16

Client expression Evaluator:

Can be used for troubleshooting client expressions that did not pass verification in the BM

Supported from 7.3 Version and above only.

Client expression Evaluator: • Can be used for troubleshooting client expressions that did not pass verification

© 2009 Oracle Corporation Proprietary and Confidential

Client expression Evaluator: • Can be used for troubleshooting client expressions that did not pass verification

17

Client expression Evaluator:

New file created
New file
created

© 2009 Oracle Corporation Proprietary and Confidential

Client expression Evaluator: New file created © 2009 Oracle Corporation – Proprietary and Confidential

18

Client expression Evaluator:

SQL Parser Statement For computed_name

'perc_trade_spend' :

SELECT 0 as total_plan_spend, 0 as ttl_plan_spend_ex_slot, 0 as total_event_dollars, CAST(SYSTIMESTAMP as DATE) as sdate From Dual

Evaluating Client Expression

Non Translated expression before evaluate:

if(isnull(( Evt Spend exS / Ttl Evt Values )),0, ( Evt Spend exS /

Ttl Evt Value ) )

Translated expression for evaluate:

if(isnull((ttl_plan_spend_ex_slot/ttlevtvalues)),0,(ttl_plan_spen

d_ex_slot/total_event_dollars))

Invalid Client Expression

© 2009 Oracle Corporation Proprietary and Confidential

Client expression Evaluator: SQL Parser Statement For computed_name 'perc_trade_spend' : SELECT 0 as total_plan_spend, 0 as

19

Key Oracle Demantra Support Notes/Web Cast Materials

Information Center: Oracle Demantra Demand Management (Doc ID 1381213.2)

Demantra Development Suggested Performance Advice plus Refernece doc (Doc ID 1157173.1)

470852.1)

Information Center: Oracle Value Chain Planning Demantra (Doc ID 1157065.1)

Oracle Demantra Documentation Library (Doc ID 443969.1)

Trouble Shooting the Demantra Worksheet Performance 7.3.0.1 and Beyond! UPDATED! (Doc ID:

Demantra Development Suggested Performance Advice Plus Reference Docs (Doc ID: 1157173.1)

860576.1)

Rebuild_Schema procedure - One of the Keys to Improved Demantra performance (Doc ID:

Troubleshooting EBS - DEMANTRA : User Synchronization does not work (Doc ID: 602530.1)

1064995.1)

Step by Step Troubleshooting of the Silent Installer (Doc ID: 947322.1)

Additional Configurations needed when Running Demantra on Oracle 11g Databases (Doc ID:

ORA-24247 During Shipment and Booking collections (Doc ID 1070104.1)

Worksheet Design & Best Practices

How to Analyze Demantra Forecast Engine Performance (Processing Time) Issues / Demantra

Engine is Slow (Doc ID 863025.1) Improving Demantra Engine Performance of Resetting Previous Forecast For Inactive

Combinations (Doc ID 1270690.1) How to check Demantra Performance. Troubleshooting. Database. Maintenance. Parameters.

Java. Worksheet. Engine (Doc ID 1356886.1) Web Cast Materials (Doc ID: 800030.1)

Demantra Post Production Support Common Issues, Troubleshooting Tips, and Maintaining Your Instance

Demantra Engine Troubleshooting

© 2009 Oracle Corporation Proprietary and Confidential

Key Oracle Demantra Support Notes/Web Cast Materials • Information Center: Oracle Demantra Demand Management (Doc ID

20

© 2009 Oracle Corporation – Proprietary and Confidential

© 2009 Oracle Corporation Proprietary and Confidential

© 2009 Oracle Corporation – Proprietary and Confidential

21

Reviewing an AWR

Introduction

Introduced in Oracle 10g Similar to a Statspack report used in prior versions Runs as a background process Takes snapshot of statistics and SQLs every hour. You can manually take AWR snapshots for any operation (before and after).

Reviewing an AWR Introduction • Introduced in Oracle 10g • Similar to a Statspack report used

© 2009 Oracle Corporation Proprietary and Confidential

Reviewing an AWR Introduction • Introduced in Oracle 10g • Similar to a Statspack report used

22

Reviewing an AWR

Important report sections

Report has a DB overview section

Lists the Top 5 waits Important for troubleshooting performance bottlenecks

 

DB File type waits Physical IO related Buffer type waits Logical IO related

Log Type waits Redo log related

PX Parallel query related

GC Global Cache related in a RAC environment

Longest wait times can be used for troubleshooting

Watch for DB sequential read related waits as these are single threaded reads of indexes

Reviewing an AWR Important report sections • • Report has a DB overview section • Lists

© 2009 Oracle Corporation Proprietary and Confidential

Reviewing an AWR Important report sections • • Report has a DB overview section • Lists

23

Reviewing an AWR

SQL related sections

SQL Time Model statistics are important for SQL tuning

Total Elapsed Time Includes all IO waits

Total CPU time Type of joins (which depend on PK, Index usage)

Total Buffer Gets Logical IO

Total Disk reads Physical IO

Total Executions Indicates SQL executions and size of worksheet

Total Parse calls SQL, Bind variable issues

For better performance, you want DB CPU time to be high compared to the total SQL processing time

Time may be spent in parsing and IO, reducing these would typically be the primary focus of the tuning exercise, which indicate IO issues

Efficiencies listed are typically close to 100%, with decent memory settings

Reviewing an AWR SQL related sections • SQL Time Model statistics are important for SQL tuning
Reviewing an AWR SQL related sections • SQL Time Model statistics are important for SQL tuning

© 2009 Oracle Corporation Proprietary and Confidential

24

Reviewing an AWR

Verify Memory Settings

PGA_Aggregate_Target SGA_Target, SGA_MAX_SIZE

Efficiencies listed are typically close to 100%, with decent memory settings

Buffer Pool statistics (default, 16k) show, how efficiently buffer cache is being used - Can be used for sizing

Reviewing an AWR Verify Memory Settings • • PGA_Aggregate_Target SGA_Target, SGA_MAX_SIZE • Efficiencies listed are typically
•
Low Hits indicate relevant memory settings need to be increased
Low Hits indicate relevant memory settings need to be increased
Reviewing an AWR Verify Memory Settings • • PGA_Aggregate_Target SGA_Target, SGA_MAX_SIZE • Efficiencies listed are typically
Reviewing an AWR Verify Memory Settings • • PGA_Aggregate_Target SGA_Target, SGA_MAX_SIZE • Efficiencies listed are typically
Reviewing an AWR Verify Memory Settings • • PGA_Aggregate_Target SGA_Target, SGA_MAX_SIZE • Efficiencies listed are typically

© 2009 Oracle Corporation Proprietary and Confidential

25

Configure Installer log:

Configure Installer log: Configure log © 2009 Oracle Corporation – Proprietary and Confidential

Configure log

Configure Installer log: Configure log © 2009 Oracle Corporation – Proprietary and Confidential
Configure Installer log: Configure log © 2009 Oracle Corporation – Proprietary and Confidential

© 2009 Oracle Corporation Proprietary and Confidential

26

Exporting a Data model

Export
Export

© 2009 Oracle Corporation Proprietary and Confidential

Exporting a Data model Export © 2009 Oracle Corporation – Proprietary and Confidential

27

LOG_IT:

LOG_IT is a logging mechanism for Demantra PL/SQL database

procedure code. It is analogous to log4j in Java.

LOG_IT can be used to trace a procedure flow, show variable

values, and record performance timing without having to run a

debugger.

LOG_IT is only available for a limited number of procedures (like

CHAINING , PROPORT,SIMULATION and more) but the list

keeps growing.

The available procedures are listed in the LOG_IT_PARAMS table.

© 2009 Oracle Corporation Proprietary and Confidential

LOG_IT: LOG_IT is a logging mechanism for Demantra PL/SQL database procedure code. It is analogous to

28

LOG_IT (cont’):

How to use LOG_IT:

The customer / support can enable the logging and the contents of the log table.

Some important points to remember:

It uses a sequential key to write logs so you know exactly the order of the log.

Each main procedure writes to one log table. ie PROPORT writes to the

LOG_PROPORT table. Sub procedures can be set up to write to the same log table.

INSERT_UNITS and DYN_INSERT_UNITS_BRANCH_ID both write to LOG_INSERT_UNITS. Log tables are usually overwritting each run (truncated).

Logging is enabled by setting the LOGGING_LEVEL to be greater than 0 for

a specific procedure in the LOG_IT_PARAMS table.

© 2009 Oracle Corporation Proprietary and Confidential

LOG_IT (cont’): How to use LOG_IT: • The customer / support can enable the logging and

29

LOG_IT (cont’):

Logging levels:

0 = No logging

1 = Procedure and sub procedure 'start' and 'end' logs. (SP,EP) 2 = Procedure progress markers (labels) (M) 3 = Procedure data , values held in specific variables (D) (also temp procedures and tables are not dropped) 4 = Sub Procedure progress markers (labels) (M) 5 = Sub Procedure data , values held in specific variables (D) 6 = Procedure Data, values from with in loops (L) 7 = Sub procedure progress markers inside loops (M) 8 = Sub procedure data from inside loops (L)

© 2009 Oracle Corporation Proprietary and Confidential

LOG_IT (cont’): Logging levels: • 0 = No logging • 1 = Procedure and sub procedure

30

LOG_IT (cont’):

LOG_IT (cont’): © 2009 Oracle Corporation – Proprietary and Confidential

© 2009 Oracle Corporation Proprietary and Confidential

LOG_IT (cont’): © 2009 Oracle Corporation – Proprietary and Confidential

31

LOG_IT (cont’):

The logging is enabled by setting the LOGGING_LEVEL above 0

for a specific procedure in the LOG_IT_PARAMS table. When

you commit the change to LOG_IT_PARAMS a trigger re-writes and compiles the procedure LOG_IT which then contains a list of procedures that it will process.

Note: before 7.3.1.1 -Only one LOG_IT_PARAMS row should be changed between each commit because of the way the trigger works.

Note- LOG_IT has some performance impact on the procedures it logs, it should not be used at a normal production mode (all log levels should be 0 for runtime operations).

LOG_IT (cont’): The logging is enabled by setting the LOGGING_LEVEL above 0 for a specific procedure
LOG_IT (cont’): The logging is enabled by setting the LOGGING_LEVEL above 0 for a specific procedure

© 2009 Oracle Corporation Proprietary and Confidential

32

Enabling client side logs:

Traditionally, client side logs are configured via the clientLogConf.lcf file.

Some Drawbacks:

User would have to restart the client (logout/login) in order for changes to take effect.

All clients which would be connected to that specific Demantra server would

be exposed to the same level of verbosity.

From 7.3.0, the Client Log Categories Editor was introduced to enable client side logging for a specific client session.

The Client Log Categories Editor loads the ClientLogConf.lcf upon the initialization of the PartnerApplet.

The editor dialog receives the properties object as parameter to display all the log categories listed currently in the configuration file.

© 2009 Oracle Corporation Proprietary and Confidential

Enabling client side logs: • Traditionally, client side logs are configured via the clientLogConf.lcf file. Some

33

Enabling client side logs (cont’):

Enabling client side logs (cont’): © 2009 Oracle Corporation – Proprietary and Confidential

© 2009 Oracle Corporation Proprietary and Confidential

Enabling client side logs (cont’): © 2009 Oracle Corporation – Proprietary and Confidential

34

Enabling client side logs (cont’):

To activate the Editor user needs to log to the WS applet (Partner) and press the CTRL+ALT+L button combination in the main application screen.

If a WS is open in the Applet click on the 'Open' button to select a

worksheet and close the "Open Worksheet" dialog without choosing any worksheet.

After that press the CTRL+ALT+L .

The editor has 4 buttons which are I18N ready:

Apply - sets the selected Log categories to the selected Log Levels and close the dialog

Cancel - closes the dialog without doing anything

Undo Changes - resets the state of selections in the dialog to the

last saved state Config file Settings - resets all the selections in the Dialog to the original settings of the configuration file

Enabling client side logs (cont’): • To activate the Editor user needs to log to the
Enabling client side logs (cont’): • To activate the Editor user needs to log to the

© 2009 Oracle Corporation Proprietary and Confidential

35