Oracle GoldenGate

Performance Tuning and Monitoring

Ease of Manageability

Volker Kuhr
Senior Principal Product Manager

Jiahui Shao
Product Manager

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracles products remains at the sole discretion of Oracle.

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 2

Oracle GoldenGate and Data Integration
Over 10K Customers Worldwide
Communications Finance / Banking Media Services

Energy/Industrial Insurance / Health Retail Other

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |

Oracle GoldenGate
Architecture

Capture Trail Files Trail Files Delivery

Source
Network
Firewall Target
Oracle / Non-Oracle
Oracle / Non-Oracle
Database Cloud Database

Bi-directional

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 4

Program Agenda

1 Automatic Heartbeat Functionality

2 Oracle GoldenGate Diagnosis

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 5

New 12.2 Feature - Automatic Heartbeats
Built-in Functionality to Monitor End-to-End Replication Lag

Simple Automatic Real-time

Enable via a single ADD Automatically discovers Database views to query
HEARTTBEATTABLE replication flows for all end-to-end real-time
command at each site topologies latency
Unidirectional Lag information on
Bidirectional incoming flow to and
N-way outgoing flow from a
database

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 6

Simple Bidirectional Replication Example
Enable Heartbeat functionality by executing GGSCI command ADD
HEARTBEATTABLE at each database
Creates required heartbeat tables, views and jobs
Updates heartbeat every 60s by default
In Database A: GG_LAG View contents
INCOMING_ OUTGOING_
LOCAL_ REMOTE_ INCOMING_ OUTGOING_
INCOMING_PATH HEARTBEAT_ OUTGOING_PATH HEARTBEAT_
DATABASE DATABASE LAG LAG
AGE AGE

A B ExtB==>PmpBA==>RepBA 1.066182 14.150614 ExtA==>PmpAB==>RepAB 1.391718 25.387458

ExtA PmpAB RepAB

RepBA PmpBA ExtB

Database A Database B

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 7

Monitoring Lag using GG_LAG View
Column Name Data type Description
LOCAL_DATABASE VARCHAR2(30) Local database name
CURRENT_LOCAL_TS TIMESTAMP(6) Current timestamp in UTC time zone
REMOTE_DATABASE VARCHAR2(30) Remote database name
INCOMING_PATH VARCHAR2(4000) Group names on the incoming flow
Period of time between remote database generating
INCOMING_LAG NUMBER heartbeat and local database receiving heartbeat
CURRENT_LOCAL_TS minus most recent heartbeat
INCOMING_HEARTBEAT_AGE NUMBER timestamp of remote database

OUTGOING_PATH VARCHAR2(4000) Group names on the outgoing flow

Period of time between local database generating

OUTGOING_LAG NUMBER heartbeat and remote database receiving heartbeat
CURRENT_LOCAL_TS minus recent heartbeat timestamp
OUTGOING_HEARTBEAT_AGE NUMBER of local database

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 8

Additional Tables and Views
GG_LAG_HISTORY View GG_HEARTBEAT and
Historical heartbeat lag GG_HEARTBEAT_HISTORY Tables
Underlying tables for the views
Column Name
Get lag for each process on the path
LOCAL_DATABASE Column Name
CURRENT_LOCAL_TS INCOMING_HEARTBEAT_TS
REMOTE_DATABASE INCOMING_EXTRACT_TS
} Extract lag
INCOMING_PATH INCOMING_ROUTING_TS
} Pump lag
INCOMING_LAG INCOMING_REPLICAT_TS } Replicat lag
INCOMING_HEARTBEAT_AGE OUTGOING_HEARTBEAT_TS
OUTGOING_EXTRACT_TS
OUTGOING_PATH
OUTGOING_ROUTING_TS
OUTGOING_LAG
OUTGOING_REPLICAT_TS
OUTGOING_HEARTBEAT_AGE

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 9

Discovers New Replication Flows Automatically
Example: Add new site Database C

Simply execute ADD HEARTBEATTABLE for new site

Database C (Newly added site)
In Database A: GG_LAG View contents
LOCAL_ REMOTE_
INCOMING_PATH OUTGOING_PATH
DATABASE DATABASE

A B ExtB==>PmpBA==>RepBA ExtA==>PmpAB==>RepAB

A C ExtC==>PmpCA==>RepCA ExtA==>PmpAC==>RepAC

ExtA==>PmpAB==>RepAB

RepBA<==PmpBA<==ExtB

Database A Database B

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 10

Diagnosing Replication Issues using Heartbeat Table
Diagnosis Cause Action
Replication processes on the
INCOMING_HEARTBEAT_AGE Check processes on incoming
incoming path have stopped
is high path and the remote heartbeat
Network issues
scheduler job
Heartbeat scheduler job has stopped

INCOMING_LAG is high Use GG_HEARTBEAT table to

Replication processing is slow on the
determine slow process on the
incoming path
incoming path

Replication processes on the

OUTGOING_HEARTBEAT_AGE Check processes on outgoing path
outgoing path have stopped
is high and the local heartbeat scheduler
Network issues
job
Heartbeat scheduler job has stopped
OUTGOING_LAG is high Use GG_HEARTBEAT table to
Replication processing is slow on the
determine slow process on the
outgoing path
outgoing path

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 11

Program Agenda

1 Heartbeat Functionality
2 Oracle GoldenGate Diagnosis
GoldenGate Extended Metrics
Healthcheck

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 12

Potential Sources of Lag
System Resources
Memory
CPU
IO and Network
Configuration
Database
GoldenGate

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 13

Tools for Performance Analysis
OS Utilities OS Utilities OS Utilities
(top, vmstat, iostat) (netstat) (top, vmstat, iostat)

LogMining Extract Capture Extract Pump Replicat Inbound Server

Server client client

HealthCheck GoldenGate HealthCheck

UTL_SPADV UTL_SPADV
AWR & ASH Extended Metrics AWR & ASH

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 14

New GoldenGate Extended Metrics
Fine-grained Performance Monitoring

Access to Monitoring Point through Restful Web Services

http://<hostname>:<mgr_port>/mpointsx
Real-time insight into GoldenGate client programs

Requires ENABLEMONITORING in GLOBALS

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 15

New Extended Metrics
Status and Configuration Information
Process Level Metrics for Extract, Pump and Replicat
Thread Level Metrics for Extract and Replicat
Database Statistics for Extract and Replicat
In-flight-transactions and Cache- & Queue statistics for Extract
Table statistics for Replicat
Network Statistics

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 16

Graphical real-time instance monitoring
Utility available on https://java.net/projects/oracledi

Replication
Components

Monitoring
Metrics
Process
Status
Change

Recording
Function

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 17

Integrated Healthcheck Report
Snapshot of all GoldenGate related Information within the Database

The Healthcheck reports a snapshot of the database information for

Integrated Extract and Apply.
Recommendation: Run periodically for differential analysis
Used for diagnostic purposes
Configuration Issues
Performance
Analyze at component and sub-component level

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 18

Healthcheck Script

1. Database 2. Summary and 3. Extract/Replicat 4. Extract/Replicat

configuration Notification Configuration Statistics

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 19

Notification Section
Heartbeat Table

Pre-Analysis with INFO, WARNING & ERROR Messages

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 20

Lag within the Integrated Extract

Integrated
Extract
Record Capture LCR1

changes changes LCR2

LCR3 Trail
Extract
. Files
.
.

Redo Logical Change

Oracle Database Records

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 21

GoldenGate Logmining Server Architecture
Oracle Database
Logmining Server

Preparer N

Reader Preparer 2 Builder Capture

Integrated Trail
Extract Files
Preparer 1

Logmining Server (Database Capture Process) Extract

Reader: Reads logfile and splits into regions Requests LCRs from logmining
server
Preparer: Scans regions of logfiles and prefilters based on extract parameters
Performs Mapping and
Builder: Merges prepared records in SCN order Transformations
Capture: Formats Logical Change Records(LCRs) and passes to GoldenGate Extract Writes Trail File

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 22

Monitoring Integrated Extract
Streams Performance Advisor (UTL_SPADV)
PATH 2 RUN_ID 15 RUN_TIME 2014-FEB-21 22:32:17 CCA N
<C> OGG$CAP_EXT_1A 305260 305245 790813
LMR 0.0% 53.3% 33.3% "CPU + Wait for CPU" proc %idle %flowctrl %TopWait Event
LMP(2) 0.0% 0.0% 146.7% "CPU + Wait for CPU"
LMB 73.3% 0.0% 6.7% " "
CAP 20% 0% 73.3% "CPU + Wait for CPU"
<Q> "OGGADMIN"."OGG$Q_EXT_1A" 0.01 0.01 0
<E> EXT_1A 304027 80076708 790813 20% 6.7% 73.3% "CPU + Wait for CPU"
<B> NO BOTTLENECK IDENTIFIED

% of Idle, Flow Control and major Wait Event for each sub component
Check output for multiple occurrences over time
Increase Parallelism of Capture if CPU is available
(LMR in flow control, and LMP is 0% idle with high CPU)
Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 23
Lag between Database and Extract client

State and "Ready to send Chunks" as one Indicator for an Extract program issue
If "Ready to send Chunks" >> " Delivered Chunks" and Capture State indicates "
Waiting for Client Requests", Extract program needs to checked.
Check TRANLOGOPTIONS for BUFFER_SIZE , _READAHEADCOUNT &
ASYNCTRANSPROCESSING.

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 24

Integrated Replicat Oracle Database
Replicat: Apply section
Inbound Server

Applier N
Lightweight
Trail Receiver Preparer Coordinator Applier 2
Replicat Streaming
Files API Applier 1

Waiting for memory might indicate Memory issues

Replicat IntegratedParams MAX_SGA_SIZE
Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 25
Integrated Replicat
Transaction Dependencies

20000 20000

Total LCRs with Dependencies indicates transaction dependencies

One reason for Total LCRs with WM Dependency might be a large transaction

Increasing EAGER_SIZE from Default to a moderately higher number

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 26

Apply Server Statistics Wait Event Analysis
Apply Server Processes as common Database sessions
Wait Event Analysis of Apply Server Processes
Classical Waits: Enqueues, IO Waits,
Replication related Wait Events: Waiting for Dependency

REPL Apply: dependency

(CPU or WAIT on CPU)

Check IO Subsystem, Classic Database Tuning (e.g.: INITRANS, Index Rebuild, )

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 27

Additional Database Resources
AWR & UTL_SPADV
.

UTL_SPADV Package to monitor database background

processes for Extract and Replicat
Two ways of displaying information: single line or HTML
Start and Stop monitoring
Shows bottleneck based on database wait events for the interval

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 28

Performance Parameter
Database
STREAMS_POOL_SIZE,
Integrated Extract Integrated REPLICAT
TRANLOGOPTIONS DBOPTIONS
ASYNCTRANSPROCESSING BATCHSQL
_READAHEADCOUNT GROUPTRANSOPS (if PARALLELISM=1)

TRANLOGOPTIONS (INTEGRATEDPARAMS ) DBOPTIONS (INTEGRATEDPARAMS )

MAX_SIZE_SIZE MAX_SGA_SIZE
PARALLELISM PARALLELISM
BUFSIZE MAX_PARALLELISM
COMMIT_SERIALIZATION
EAGER_SIZE

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 29
Summary
Monitoring:
Use Automatic Heartbeat Table
Run periodically Healthcheck Reports
Analysis
Use Heartbeat Table to drill down to the component
Monitor OGG extended Metrics
Healthcheck Reports
UTL_SPADV
Check Database Performance (AWR & ASH) and OS Performance

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 30
References
MOS Notes

Note 1557031.1 Recommended patches for Integrated Capture/Replicat

Note 1448324.1 OGG Integrated Healthcheck Script
Note 1488668.1 GoldenGate Performance Data Gathering
Note 1596493.1 GoldenGate Integration with AWR
Note 1485620.1 Best Practices Downstream Capture
MAA white paper Oracle GoldenGate Performance Best Practices
http://www.oracle.com/technetwork/database/availability/maa-gg-performance-
1969630.pdf

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 31

 Keep Learning with Oracle University

Classroom Training Cloud

Learning Subscription Technology

Live Virtual Class Applications

Training On Demand Industries

education.oracle.com
Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 32
 Session Surveys

Help us help you!!

The organizing committee would like to invite you to take a moment to give
us your session feedback. Your feedback will help us to improve your
conference.
Please be sure to add your feedback for your attended sessions by using
the Mobile Survey or in Schedule Builder.

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 33