IBM - SAP Java Monitoring & Tuning Guide

IBM Americas Advanced Technical Support
IBM SAP Technical Brief
IBM - SAP Java Monitoring & Tuning Guide
Damir Rubic

SAP & Oracle Solutions
Version: 1.0
Date: December 21, 2007
© 2007 International Business Machines, Inc.

1. PREFACE ............................................................................................................................................. 3

1.1 ACKNOWLEDGEMENTS.................................................................................................................... 3
1.2 DISCLAIMERS .................................................................................................................................. 3
1.3 COPYRIGHTS ................................................................................................................................... 3
1.4 FEEDBACK ...................................................................................................................................... 3
1.5 VERSION UPDATES.......................................................................................................................... 3
1.6 WHO SHOULD READ THIS DOCUMENT.............................................................................................. 4
2. JAVA TECHNOLOGY IN THE SAP APPLICATION SERVER ................................................. 5
2.1. SAP J2EE ENGINE SYSTEM ACHITECTURE ..................................................................................... 5
2.2. JAVA CLUSTER ACHITECTURE ...................................................................................................... 10
3. SAP JAVA SERVER ADMINISTRATION.................................................................................... 13
3.1. SAP J2EE ENGINE ADMINISTRATION TOOLS................................................................................ 13
3.2. SOFTWARE DEPLOYMENT MANAGER (SDM) ............................................................................... 19
4. SAP JAVA SERVER MONITORING............................................................................................. 21
4.1. TECHNIQUES FOR PROBLEM DETECTION ....................................................................................... 22
4.1.1 Monitoring of the Log and Trace files .................................................................................... 22
4.1.2 Availability Monitoring of SAP Java engine and Java based applications (GRMG)............. 26
4.1.3 Distributed Statistics Records (DSRs) .................................................................................... 28
4.1.4 State Monitoring ..................................................................................................................... 31
4.2. TECHNIQUES FOR PROBLEM ANALYSIS ......................................................................................... 34
4.2.1 Performance Trace ................................................................................................................. 34
4.2.2 Single Activity Trace / JARM .................................................................................................. 37
4.2.3 Single Activity Trace ............................................................................................................... 40
4.2.4 SQL Trace ............................................................................................................................... 42
4.2.5 Application Trace.................................................................................................................... 44
4.2.6 Conclusion .............................................................................................................................. 47
5. IBM JAVA TECHNOLOGY FOR SAP .......................................................................................... 49
5.1 IBM JVM GARBAGE COLLECTION ............................................................................................... 50
5.2 IBM JVM PARAMETERS ............................................................................................................... 51
5.3 MONITORING THE JVM ACTIVITY ................................................................................................. 51
PUBLICATIONS: ...................................................................................................................................... 54
Page 2
1. Preface
1.1 Acknowledgements
I would like to thank Mark Gordon and Jim Dilley for all their help regarding this and many other Oracle
& SAP & pSeries performance-related subjects.
1.2 Disclaimers
IBM has not formally reviewed this paper. While effort has been made to verify the information, this
paper may contain errors. IBM makes no warranties or representations with respect to the content hereof
and specifically disclaim any implied warranties of merchantability or fitness for any particular purpose.
IBM assumes no responsibility for any errors that may appear in this document. The information
contained in this document is subject to change without any notice. IBM reserves the right to make any
such changes without obligation to notify any person of such revision or changes. IBM makes no
commitment to keep the information contained herein up to date.
1.3 Copyrights
SAP and R/3 are copyrights of SAP A.G.

RS/6000, pSeries, and AIX are copyrights of IBM Corporation.
Oracle is a copyright of Oracle Corporation.
1.4 Feedback
Please send comments or suggestions for changes to drubic@us.ibm.com.
1.5 Version Updates
• Version 1.0 – initial version
Page 3
1.6 Who should read this document
This paper is primarily intended for pSeries Sales Specialists who would like to know more about the new
SAP Technology Application Framework – NetWeaver 2004s.
The goal of this paper is to provide the audience with material that will help them understand the most
important elements of the SAP NetWeaver Application Framework, and how to combine these elements
with the strengths of the IBM pSeries platform.
The SAP NetWeaver Application Platform presents a new framework consisting of several different
elements. This paper will focus on the J2EE engine, which is the central and most interesting element of
the latest SAP Java announcement.
This paper will provide the reader with an overview of the most important features of the J2EE engine.
The J2EE engine also presents opportunities for extension into multiple areas. I would be happy to accept
suggestions regarding exploring these additional areas for development.
Page 4
2. Java Technology in the SAP Application Server
2.1. SAP J2EE Engine System Achitecture
As a system, the J2EE Engine consists of three logical layers:
1. Java Enterprise Runtime – comprised of low-level subsystems that provide functions such as class
loading, cluster communication, persistent configuration data management, etc.
2. J2EE Engine components – consists of interfaces, libraries and services components that provide
various runtime functions and programming APIs
3. Applications – refers to the applications that are deployed and run on the J2EE Engine
The J2EE Engine architecture is based on the following general rule: components from the higher level
can use components from lower level; whereas components from the lower level are not aware of the APIs
of the components from the higher level and therefore cannot use them.
This rule is reflected by the starting order of the system modules: runtime is first, then services (libraries
are loaded, and interfaces resolved at this phase), and lastly, applications. The system is considered
functional when all runtime managers and core services components are started in the proper order.
Components from the higher level use a set of defined APIs to utilize the functions of the components
from the lower level. The J2EE Engine components use the Framework APIs to connect to the SAP Java
Enterprise Runtime. Applications use the J2EE Engine components by the APIs that are defined by J2EE
1.3 Specification (and supporting specifications) and SAP-proprietary APIs.
Page 5
Figure 1-1 SAP J2EE Engine Architecture
• SAP Java Enterprise Runtime
The Java Enterprise Runtime provides the core functions of the system. It is built of several low-level
subsystems called Managers. These are:
• Log Manager – manages the process of logging system events

• Ports Manager – manages the process of opening various communication ports on the system
• Pool Manager – manages the process of pooling Java objects (this does not refer to pooling JDBC
connections)
• Application Thread Manager – handles threads in which client applications’ source code is
executed
• Thread Manager – handles threads in which the J2EE Engine system operations are executed
• Connections Manipulator Manager – manages the process of opening TCP/IP sockets from the
client to the J2EE Engine
• Locking Manager – handles locking as interface to the Enqueue Service
• Configuration Manager – manages the process of storing and reading persistent configuration
data (such as properties and deployment data) to and from a relational database
Page 6
• Class-loading Manager – controls activities for registering and unregistering of loaders and
references between them, as well as for accessing loaders referenced by another loader. This is
necessary for locating and loading Java classes
• Licensing Manager – handles SAP licenses
• Cluster Manager – manages communication (in terms of cluster elements status and modules
available on it at runtime) between J2EE Engine cluster elements
• Service Manager – represents a container of pluggable components. It provides runtime lifecycle
management of those components
The Service Manager is the container of all J2EE Engine components of the system. Using the functions
of other runtime subsystems, it loads and starts the components when the J2EE Engine starts up. This is
why the Service Manager is the last runtime subsystem to be started. It also manages components at
runtime, and stops them when the J2EE Engine is being stopped.
Figure 1-2 Service Manager
Page 7
The process of starting (loading) the J2EE Engine components is strictly defined. It is conducted
exclusively by the Service Manager. It performs the following steps:
1. Reads the components persistent data from the database

2. Performs checks for data integrity. This includes checks for duplication of component names, for
availability and consistency of components deployment descriptors, and so on
3. Resolves components for which the previous steps are successfully completed
4. Checks whether all core components are resolved. If not, the Service Manager stops the entire
system
5. Creates the class loaders of the components and loads them
6. Starts service components
7. Performs a check on whether all core service components are started. If not, the Service Manager
stops all started components, unloads them and stops the whole system
• SAP J2EE Engine Components
The J2EE Engine components build the second level of the system. Built on top of the runtime and able to
communicate and use each other, these components form the complete system infrastructure to run both
J2EE and SAP proprietary applications.
Three types of J2EE Engine components are defined:
1. Interfaces – “contracts” that define how different components of the system work together. These
components do not provide any runtime functions. At runtime, they provide the system with their
name and classes (no objects). They are used by services components that provide their
implementation. Interface components can also be implemented directly by applications.
2. Libraries – provide name, classes and objects to the system. These objects are created by the
system when it loads the library, or when an object is first requested. Other library components or
services components typically access those using static methods.
3. Services – more powerful than the other two types of components. They provide the system with
their name, classes and runtime objects. The runtime objects are registered on the system once the
components classes have been loaded. Service components can access and utilize functions of the
runtime through the Framework API.
Page 8
• Applications
The applications form the third level of the J2EE Engine architecture. The boundary between the
applications level and the J2EE Engine components level is defined by the J2EE 1.3 APIs along with a
few SAP proprietary APIs. Applications use these APIs to utilize the functions of the different types of
components on the lower level in the system.
An enterprise application consists of several types of application components. These components reside in
different containers on the J2EE Engine as shown in Figure 1-3. Each of these containers provides runtime
services and life cycle management for the application components.
Browser
Web Server J2EE Application Server
Presentation Layer J2EE Services

Web Container
JDBC JNDI JAAS
Servlet Java
Server
Page
JMS JTA JCA
Business Layer
...
EJB Container
Session Entity
Bean Bean
mySAP EJB Server Legacy

DB DB
component
Server Server
Figure 1-3 SAP J2EE Engine Containers
Page 9
2.2. Java Cluster Achitecture
A Java cluster installation consists of:

• One or more instances of the AS Java
• The Central Services, which also form an instance
• One or more databases
The dispatchers and servers can be split up among different physical servers. The Central Services
(Message Service and Enqueue Service) are installed on a host that meets the requirements for high
availability.
• Java Instance
A Java instance is a unit in the AS Java cluster which can be individually started, stopped, and monitored.
The cluster elements that form an instance run on one physical machine. It is also possible to run several
instances on one physical machine. An instance is identified by the system ID (SID) and the instance
number. Central Services are a special example of a Java instance.
Another special instance is one that runs the SDM (Software Deployment Manager). This instance usually
runs with the database and Central Services on the same machine, and is then indicated as the central
instance.
A Java instance normally is comprised of one Java Dispatcher and several server processes. A Java
instance is started and stopped by the Java Startup and Control Framework. An HTTP request to execute a
servlet runs through several layers in the J2EE engine. The Java dispatcher receives the request, selects a
server process for the processing and establishes the connection from the client to the server.
Figure 1-4 Layers of the SAP J2EE Engine
Page 10
• Minimum Cluster Installation
The following graphic shows the simplest installation of usage type AS Java. A system installed in this
way can only process requests to Java applications.
Figure 1-5 Minimum Cluster Installation
This minimal AS Java installation consists of:

• A Java central instance with a dispatcher, a server process, and Software Deployment
Manager (SDM)
• The Central Services instance
• The Database
A Java Instance consists of (with the exception of Central Services):

• a Java Dispatcher
• One or several server processes
Page 11
• Large Cluster Installation
The graphic below shows a larger Java cluster installation. It consists of four instances, each of which has
an instance number and can be started, stopped, and monitored separately.
Figure 1-6 Large Cluster Installation
Page 12
3. SAP Java Server Administration
3.1. SAP J2EE Engine Administration Tools
The SAP J2EE Engine provides the following basic administration options:
o offline – enables the configuration and administration of J2EE Engine cluster elements
when the server is not running
o online – enables the administration of J2EE Engine cluster elements at runtime
o remote – enables configuration of the J2EE Engine from a distance; that is, the
administration tool runs on one machine and the server that must be managed is running on
a different machine
These options are implemented in the administration tools for the J2EE Engine: Visual Administrator,
Shell Console Administrator, Config Tool and SAP Netweaver Administrator.
• Visual Administrator
SAP J2EE Engine Visual Administrator is a graphical user interface (GUI) that enables administration of
the entire cluster, cluster elements, and modules running on them. It provides remote monitoring and
management of managers, services, libraries, and interfaces working on each element in a single GUI.
Visual Administrator enables:

o Obtaining general information about a service, manager, interface or library (for example, its
name, group, and so on)
o Logging on to the SAP J2EE Engine via the Visual Administrator tool
o Administrating and changing common properties or those specific to a service or manager
o Configuring the global properties
o Runtime administration and control
Page 13
Figure 1-7 Visual Administrator
To start the Visual Administrator, enter the following command:
“/usr/sap/<SID>/<Instance Name>/j2ee/admin/go.sh” (or go.bat for Win based environment).
• Shell Console Administrator
The J2EE Engine Console Administrator is an alternative to the Visual Administrator. Unlike the Visual
Administrator, it is not a GUI and the runtime control and administration is done using specific commands
in the shell language. The commands are entered on the command line of the console where the cluster
element is running. This tool enables monitoring of the processes started on the different elements of the
cluster and provides an opportunity for prompt and adequate reaction whenever problems occur. The
Console Administrator enables remote administration through Telnet clients or applets that simulate a
Telnet client, as well as continuous deployment of applications and additional libraries.
By default, the Telnet shell is opened on the dispatcher. Once connected, the user can access and use all
shell commands available on the different J2EE Engine cluster elements. Use the LSC command to
Page 14
display all server components with their ID, component name, host, port, and type. The first component
displayed is the current one.
To pass over from one component to another, use the JUMP command and specify the ID of the target
element. For example, executing jump 4001 enables the remote administration of a cluster element with
ID 4001.
The JUMP command is available for Telnet administration only. To close the Telnet connection type exit
on the command line.
Figure 1-8 Shell (Telnet) Administration
To connect to the J2EE dispatcher, enter the following command:
“telnet hostname telnet_provider_service_port” or, as an example: “telnet ibm_node_01 50008”.
Page 15
• ConfigTool
The Config Tool enables offline configuration of the J2EE Engine cluster elements. It is XML-based and
allows the user to export for later use. When working offline, it is not required to have a SAP J2EE Engine
running, as the Config Tool connects directly to database and applies the changes. When running the J2EE
Engine Config Tool via a GUI or text-only interface, it connects to the database and scans the server
configuration. The information provided is passed to the corresponding interface that is used for
configuration.
The J2EE Engine GUI Config Tool enables configuration of J2EE Engine cluster elements concerning
properties, adding new elements, and exporting the system configuration to an XML file.
After installing the J2EE Engine a configtool directory is created, containing a configtool script file.
To start the GUI ConfigTool enter the following command:
“/usr/sap/<SID>/<Instance Name>/j2ee/configtool/configtool.sh” (or configtool.bat for the Win based

environment).
Figure 1-9 The GUI ConfigTool
Page 16
• SAP Netweaver Administrator
SAP NetWeaver Administrator (NWA) is a new Web Dynpro-based tool for administration and
monitoring, offering a central point of entry point into the SAP NetWeaver system landscape. The SAP
NetWeaver Administrator can be used in a central scenario, where it is capable of operating an entire
system landscape containing ABAP and Java systems as the application platform of SAP NetWeaver.
The NWA unifies the most important administration and monitoring tools both for Java and ABAP
systems. The most important advantages of the NWA are:
• No longer need to switch between different tools for administration, troubleshooting and problem
analysis of SAP NetWeaver system landscape
• Central administration tool available landscape-wide for both Java and ABAP systems, for starting
and stopping instances, checking configuration settings and logs, and monitoring error-free
functioning of components
• Interface follows the current guidelines for interface design, is easy-to-use, task-oriented and
complete. Web Dynpro runs in a normal web browser
• Interface allows seamless navigation to other SAP NetWeaver administration tools (User
Management Engine, System Landscape Directory)
• For Java, the NWA represents a crossover from various expert tools to an integrated, simple and
clear solution. The NWA also completes the integration of the data sources for monitoring
• For ABAP, the NWA represents a crossover from many different expert transactions, which are
difficult to use and integrate
Page 17
Figure 1-10 Netweaver Administrator
To connect to the Web Administrator, open a following http page:
“http://<fully qualified hostname>:50000+(100 * instance number)/nwa” or in the following example
“http://ibm_node_01:50010/nwa”.
Page 18
3.2. Software Deployment Manager (SDM)
The Software Deployment Manager (SDM) is a tool to deploy and manage software packages from SAP.
SDM resides in the SAP Web Application Server. The SDM is responsible for installing and updating
software components into the SAP J2EE Engine. It also provides modification support on demand. Each
time a new component version is deployed, the CBS consults the SDM on the target system. This process
can be initiated via the SAP NetWeaver Developer Studio.
• Software Deployment Archive (SDA)
The Software Deployment Archive (SDA) is the delivery format for SAP applications in programming
languages other than ABAP. It is a ZIP-compatible archive format that can be used as a container for other
archives. The SDA contains the manifest information - that is, package-related data - of its archives (such
as jar, war) and an SAP manifest, which contains additional software logistics information.
The EAR archive is a special case in the J2EE context. If an EAR archive contains a SAP manifest, it is
also a SDA. The SDM recognizes the EAR archive as a SDA, but does not rename the archive extension as
<archive_name>.sda.
SDA is the smallest unit that can be deployed. Furthermore, the SDA is the smallest unit for which
patches can be created and delivered.
• Software Component Archive (SCA)
A Software Component Archive (SCA) is the physical representation of a version of a software

component. It contains a specific number of SDAs, whose quantity describes a precisely-defined version
level.
A SCA update always results in a new version level of the software component.
• Deployment
The deployment is the final step in the software delivery process; it involves the deployment of available
software packages - SDAs or SCAs - in the runtime environment of the SAP Systems.
When deploying SDAs/SCAs, the Software Deployment Manager stores the data in the SDM Repository,
from where it then manages the installed archives. The SDM recognizes dependencies between archives
and provides support when installing and maintaining shared applications.
The SDM uses the Deployment Manager to control the deployment of SDAs/SCAs for the following
o J2EE applications
o File system content
o Database content
o SAP J2EE Engine Additional Libs
Page 19
Figure 1-11 Software Deployment Manager (SDM)
To start the SDM GUI tool, enter the following commands:
“/usr/sap/<SID>/<Instance Name>/SDM/program/StartServer.sh” (if server not started),
“/usr/sap/<SID>/<Instance Name>/SDM/program/sdm.sh remotegui”.
Page 20
4. SAP Java Server Monitoring
The previous chapters have provided information about the basic architectural elements of the SAP Java
landscape. This chapter will cover the different aspects of the Java monitoring techniques. The following
tools and techniques should be utilized to achieve a robust and scalable environment through higher
availability and optimized performance of the monitored applications.
The basic Java monitoring architecture is developed and organized centrally, and in most of cases, will
take advantage of the SAP CCMS monitoring architecture. Instead of investigating potential configuration
problems or performance bottlenecks through individual system logging, the data are collected and
presented in a central monitoring system (CEN). Problems will be displayed as soon as they occur, while
log files and performance data can be accessed from one location to simplify administration.
Figure 1-12 Central Monitoring of the SAP NW04s System Landscape
To address the different aspects of Java monitoring, two different task groups have been defined:
• Problem detection
• Problem analysis
Page 21
4.1. Techniques for Problem detection
4.1.1 Monitoring of the Log and Trace files
The first step in the problem detection phase is the search for information contained in the log files
generated by the Java engine. All the important events that describe either the startup or typical workload
of the engine are contained in this set of log files. SAP has done an excellent job of separating the log files
in two groups, providing different levels of the information that support the needs of two different types of
users: administrators and developers.
The log files for administrators contain only the information needed for a broad overview of typical Java
engine related activities. System administrators need to scan a wide range of engine related elements to
determine current status (overall system availability, CPU and memory utilization, performance related to
the DB, etc.)
On the other hand, developers require detailed information about engine-related activities and applications
they are working on. Typical elements of interest include status of the heap memory and program stack.
Thus, during typical Java engine activity, two types of the files are created:
• Log files for the administrators
• Trace files for the developers
The Visual Administrator contains all the necessary tools needed to configure and review Java engine
generated files. Those three configuration elements are:
• Log Manager – manages the logging of system events (Figure 1-13)
Figure 1-13 Log Manager in Visual Administrator
Page 22
Each of the generated log files are positioned into a separate “home” directory of the specific node
(dispatcher or server), for example: “/usr/sap/SID/JC<InstNo>/j2ee/cluster/dispatcher<No> or
server<No>”. The location of the generated trace file, however, is defined by the values of the variables
circled in the Figure 1-13.
The functionality of the Log Manager is also provided in the Configtool. The benefit is that the Java
engine doesn’t have to be started to use this tool.
Once the specific set of the parameters has been changed and applied, the engine has to be restarted.
• Log Configurator – used to configure individual logging and tracing settings (Figure 1-14)
This element enables the user to manage logs and traces for individual Java components. Even though it is
possible to create customized elements (category objects for logs and location objects for traces), all SAP
applications come with predefined categories and locations. There are two modes of work on this panel
(circled on Figure 1-14): “simple mode” for use on production systems and “advanced mode” for use on
development systems.
Figure 1-14 Log Configurator in Visual Administrator
Page 23
• Log Viewer – used to review Java engine generated logs and traces, the “Log Viewer”
functionality is also provided by other tools:
o as a standalone Java application contained in the directory:
“/usr/sap/SID/JC<InstNo>/j2ee/admin/logviewer-standalone” (Figure 1-15)
o Command line log viewer
o Log Viewer functionality integrated in the SAP Netweaver (Figure 1-16)
Figure 1-15 Standalone Log Viewer
Attached is a list of the most important log & trace files to be reviewed in the problem resolution process.
They are located in the: “/usr/sap/SID/JC<InstNo>/work” directory.
Files providing basic information:
• INSTSTAT:
o Information about J2EE instance startup
• log_bootstrap_ID_<InstNo>:
o Information about the files that are synchronized during the engine startup
o Every server process creates a separate log file
• std_dispatcher.out:
o Information about the J2EE dispatcher process startup and shutdown
• std_server<ServNo>.out:
o Information about the specific J2EE server process startup and shutdown
• std_sdm.out:
o Information about the SDM server process startup and shutdown, exists only on the J2EE
central instance
Page 24
Files providing more detailed information:
• dev_bootstrap_ID<InstNo>
o Detailed information about the J2EE bootstrap process including the corresponding
JLaunch processes and startup parameters
• dev_dispatcher
o Detailed information about the startup and shutdown of the J2EE dispatcher process
• dev_server<ServNo>
o Detailed information about the startup and shutdown of the J2EE server process and
startup parameters
• dev_sdm
o Detailed information about the SDM server process startup and shutdown, exists only on
the J2EE central instance
• jvm_bootstrap_ID<InstNo>.out
o Detailed information about the J2EE bootstrap process, files that are synchronized and
synchronization statistics
Figure 1-16 Log Viewer in SAP Netweaver Administrator
Page 25
4.1.2 Availability Monitoring of SAP Java engine and Java based

applications (GRMG)
This technique is based on the so-called “Heartbeat” (GRMG) central infrastructure, which is used to
check the availability of the Java engine and Java based applications.
GRMG contains two parts:
• GRMG Infrastructure, to send, receive and display GRMG communication elements in the alert
monitor
• GRMG application performs the actual application monitoring
o Central monitoring system periodically calls a URLs – the GRMG applications
o GRMG checks its component or scenario
o GRMG application send a XML message back to the CEN
o Message received is transferred into a CCMS monitoring tree and displays availability and
messages of the component
Figure 1-17 Availability Monitoring using GRMG
Most current Java based SAP applications (EP, XI, BI etc) use the GRMG infrastructure.
The following figures show how to configure and activate the GRMG for the monitoring of the Enterprise
Portal infrastructure.
Page 26
Required set of the steps are to be executed:
• Maintain the GRMG customizing file in the Visual Administrator
• Transfer the GRMG customizing file into the CCMS agent (use the “Upload” button, Figure 1-18)
Figure 1-18 GRMG customizing in Visual Administrator
• Customize GRMG information in the CEN system using the GRMG transaction (Figure 1-19)
• Display the availability of the activated scenarios in the RZ20 sub-tree
Figure 1-19 GRMG customizing in CEN
Page 27
4.1.3 Distributed Statistics Records (DSRs)
SAP Performance Statistics Records are generated to monitor the performance of an ABAP system and its
components. Generated statistics records provide information about the system workload and the
resources used by a specific action. Recently, this functionality was extended by using Distributed
Statistics Records to monitor the non-ABAP components, primarily the SAP J2EE engine.
Figure 1-20 provides an overview of the elements and communication paths required for performance data
collection, aggregation and display (Java Standalone System and Double-Stack System):
Figure 1-20 Collection, Aggregation and Display of the DSR’s
To activate the J2EE service to generate statistics records, a certain set of configuration activities must be
executed in the Visual Administrator. Detailed explanation of these configuration steps can be found on
the SAP Global Help site (http://help.sap.com).
Once the data collection agents are successfully activated (SAPCCMSR with the –j2ee option,) the
generated statistics will be transferred to the CEN to which that agent was registered.
Page 28
There are several steps that DSRs go through before they are displayed in the CCMS of the CEN:
• Activated DSR service in the J2EE engine starts generating the records
• Generated records are written to the specific location of the file system containing J2EE relevant
objects
• Activated agent reads the records and sends them to the CEN
• Raw transferred data can be collected and aggregated by the DSR collector (runs on CEN) and
then presented in the Global Workload Monitor (ST03G) (Figure 1-21)
Figure 1-21 Aggregated DSR presentation in ST03G
• Raw transferred data can be presented directly (without aggregation) in the CEN using the
STATTRACE transaction; this single record data overview will be used as a functional trace to
provide a finer granularity of statistic files (Figure 1-22)
• In the case of a double-stack system, the agent has to be registered twice, with the local ABAP
system and the CEN
• In both types of activation, single or double J2EE stack, the SAPCCMSR agent is launched
automatically after the J2EE engine is started successfully
Page 29
Figure 1-22 Detailed DSR presentation in STATTRACE
Page 30
4.1.4 State Monitoring
The most important elements of a system management discipline are the diagnostic tools and procedures
that monitor system performance. Within a global corporate IT infrastructure, the monitored systems can
include computers, networks, storage, operating systems, databases and applications.
This chapter will focus on the JMX (Java Management Extensions) monitoring infrastructure
implemented in the J2EE engine. JMX is an open Java standard that can be used to monitor all Java-based
web applications, in addition to SAP Netweaver applications. This infrastructure provides a consolidated
and automated process to monitor the system state, along with the important parameters of SAP
Netweaver and Java applications running on those systems. Using JMX will decrease the frequency and
granularity of the tasks required to retrieve necessary information about the system status.
Figure 1-23 JMX Implementation in SAP Netweaver
Figure 1-23) provides an overview of the steps that are executed during the runtime data collection:
o Monitored resource (Java application, service, etc.) creates a monitoring object (MBeans)
o Developers are responsible for creating and registering MBeans in the application code
o Data collection can be passive (polled by monitor) or active (pushed by resource)
o Monitoring objects are registered with the JMX Agent (MBean Server)
o MBean server in the SAP Java engine is presented through the JMX Adapter Service
o MBean Server manages the MBeans information and makes it available to the JMX clients
through the monitor Bean
o JMX clients are Monitoring Services in the Visual Administrator and RZ20 in CCMS
Page 31
The JMX Adapter service links the data providers implemented in the shape of MBeans and the JMX
clients used to monitor the system (or Java applications). To control the behavior of this service, two
different tools are available: Java engine ConfigTool and Visual Administrator (Figure 1-24). Properties
of the service may be changed on the global and local level. If the properties need to be replicated across
all nodes, the global configuration should be applied. If specific nodes need to be tested, the local
configuration should be applied. The local change has a priority over the global change. Every time the
property is changed, the affected nodes have to be restarted.
Figure 1-24 JMX Adapter Service Global Properties in the Visual Administrator
The data generated by the link established between the system (or Java application) and the MBeans are
grouped, consolidated and presented in one of the administrator tools (Visual Administrator or CCMS).
The monitoring tree consists of five separate “nodes:” Applications, Kernel, Performance, Services and
System. Each of these nodes has a predefined group of sub-nodes that consolidates a monitoring structure
for a specific application or system resource (MBean). Every resource has a predefined threshold which
can be changed using the “Configuration” button (Figure 1-25). A color-coded scheme (red = problem;
yellow = warning, green = OK) signals down to the root node once a certain threshold is reached.
Page 32
Figure 1-25 Visual Administrator Monitoring Service
Besides the Visual Administrator, the CCMS transaction RZ20 (Alert Monitor) can also be used to present
data submitted by the source Java-related elements. Before this is established, agent registration and
configuration has to be executed. One of the benefits of the Alert Monitor is that like Java, it can present
the data generated by the ABAP instances.
The unique value presented through the state monitoring infrastructure is a simplification of the anomoly
detection generated in the system and Java applications.
Page 33
4.2. Techniques for Problem Analysis
The previous chapter presented tools and procedures used to detect Java-based system performance and/or
configuration issues. Once the cause of the problem is determined, the problem analysis phase will start.
This section contains a set of tools that will provide detailed information about any system anomaly. At
this point in time, since it presents such an important prerequisite, I will assume that the basic knowledge
about the system architecture of the monitored environment and the Java programming language is
already present.
4.2.1 Performance Trace
Performance tracing is the “advanced” version of the activities discussed in the description of the DSR
collection. The same performance data collection elements will be used. The source (monitored) system
must have the Java engine “Distributed Statistics” service enabled, and the CCMS agent SAPCCMSR
registered with the Central Monitoring System (CEN). The elements are presented in the Figure 1-26.
Figure 1-26 Data presentation in the STATTRACE and ST03G
The functional trace tool (transaction STATTRACE) is used to display raw statistical data (individual
records) and traces from the non-ABAP systems.
Page 34
It is an extension of the following two transactions:
o STAD, displaying only raw statistical data from the ABAP systems
o ST03G (Global Workload Monitor), displaying aggregated statistics data collected by the
specific collector
Data path (C) in Figure 1-26 represents a single record display option for the data, read directly by the
functional trace tool from the statistics files accessed by the remote CCMS agent. The granularity of the
information in the functional trace is finer than that of the data in the Global Workload Monitor.
DSRs can only measure the duration of the processing between the entry points of the “local” J2EE
service presented on Figure 1-27.
Figure 1-27 DSR’s across Multiple Components
It is possible to trace the actions of the business process across several components and system boundaries
using the communication element called Passport. Every time the LUW is executed on the host, a new
DSR with a GUID is generated and sent to the SAPCCMSR agent for presentation in CEN. If the business
process spans multiple hosts with multiple LUWs, each time a remote call is made, a passport is sent with
the previously generated GUID. Figure 1-28 provides an overview of the Passport function.
To activate the performance trace, the following Visual Administrator menu is selected: Performance
Tracing Service -> Runtime -> Trace Config. The chosen page will present a set of the functional modules
that can be activated to generate a trace for a specific user. If no user is specified, a trace is activated for
all users.
Page 35
Figure 1-28 DSR’s across Multiple Components
Page 36
4.2.2 Single Activity Trace / JARM
The Java Application Response Time Measurement (JARM) is a method for collecting response time data
from Java applications. The data delivers an overview of components passed by a thread, along with the
time remaining in the corresponding component. The JARM displays not only the response time, but also
the user who sent the request and the amount of data that was transferred. This function is designed to
allow performance measurement in productive systems and to identify areas causing problems. JARM
data is shown in the Performance Tracing service of the J2EE Engine Visual Administrator tool.
Figure 1-29 JARM Data in Visual Administrator
The JARM data are additionally available in the CCMS and can be displayed using the RZ20 transaction.
JARM monitors must be implemented by the developer into the coding of the application. They are very
low overhead and can be used during runtime without any restrictions.
The purpose of this monitoring is to track the status of the Java system. The data are presented in several
different views:
o Request summary
This tab displays data accumulated since system start or data reset. The most
important counters are: Average Response Time of a Request and Number of
Requests. If system is permanently under load, attention should be paid to the
Requests per Second counter.
Figure 1-30 JARM Data – Request Summary
Page 37
o Request Overview
Most important requests and components may be selected by different criteria such
as response time or amount of transferred data. The slowest response time gives an
indication of where performance analysis may be necessary. If one of the requests is
chosen, the detailed information in the pane below is displayed. The net time metric
identifies the slowest components and the levels indicated how deep component is
nested.
Figure 1-31 JARM Data – Request Summary
o Component Overview
Shows the data collected for each component in a condensed way. The data can be
sorted by average gross time, outbound data, average net time or the number of
calls.
Figure 1-32 JARM Data – Component Overview
Page 38
o User Overview
Shows the username with average response time, number of requests and time of
the first and last request.
Figure 1-33 JARM Data – User Overview
o Thread Overview
Lists all the threads currently running in applications instrumented with JARM. The
number of threads indicates the current load on the server. This tab also tracks the
time a request is running and the active time of components. This helps in locating
long running or blocked requests.
Figure 1-34 JARM Data – Thread Overview

Page 39
4.2.3 Single Activity Trace
The Single Activity Trace (SAT) is one usage of Java Application Response time Measurement (JARM).
To write a Single Activity Trace, every component has to be instrumented with JARM. Use JARM to
analyze a single user’s request to find performance problems in an application, or to understand the logical
processing steps inside an application. The performance data collected are automatically written in a trace
file using the SAP Logging API. This file can be viewed in the Log Viewer. The difference between SAT
and JARM is that JARM only provides the top value data, while SAT provides all the trace data generated
for the selected request.
Figure 1-35 SAT Functionality
To activate the Single Activity Trace choose: Server -> Services -> Performance Tracing -> Runtime ->
Trace Config tab.
Figure 1-36 SAT Activation
Page 40
SAT log files are displayed in a format different from general logs:
o Steps are displayed in hierarchical manner

o Username indicates the beginning of a request or an action
o Displays within the structure how deep a component is nested (first column shows the
level)
Figure 1-37 SAT log files
Page 41
4.2.4 SQL Trace
SQL trace is a convenient tool for detecting problems when working with the persistent layer. It provides
traces for JDBC methods that access the database or are otherwise important for database connectivity.
SQL can trace the execution of SQL statements in both system and application threads. The function
provides detailed information about traced statements such as a thread, application, method and session in
which the statement was executed. The log format is database independent. Besides the SQL statement
text, each log record contains information about the time when the statement was executed, its duration,
input parameters and results, as well as context information.
To launch the SQL Trace enter the URL of the application in your browser:
“http://<hostname>:<http_port>/OpenSQLMonitors”
Only an administrator has authorization to use this tool.
Figure 1-38 Accessing the SQL Trace
SQL trace can be switched on and off dynamically. Follow the Web browser path to turn the SQL Trace
on and off (it is turned off by default): “Main Page” -> “Switch on and off SQL Trace” -> “SQL Trace on
or off”. SQL trace is more suitable tool for developers than for detecting general performance issues. It is
not recommended to keep constantly activated as it consumes additional resources.
The SQL Trace can also be activated or deactivated using the Visual Administrator. Besides using the
“OpenSQLMonitors” Web application, the results of the trace can be reviewed using the Log Viewer.
To evaluate the SQL Trace follow the Web path: “Main Page” -> “Trace Evaluation” -> Evaluate the
selected trace for the specific node. The typical output is presented in the Figure 1-39.
Page 42
Figure 1-39 SQL Trace output
For a detailed description of the data columns in the presented sample output, please use the following
link:
http://help.sap.com/saphelp_nw2004s/helpdata/en/24/d4ff0eccf33a49a5c26a280a119b14/content.htm
If additional information is needed, click on the specific SQL statement and the detailed output will be
presented.
Figure 1-40 Detailed SQL Trace output
At this time, SQL Trace does not offer the functionality and flexibility of the equivalent ABAP ST05
transaction, but further improvements are expected, especially with the upcoming 7.10 kernel based NW.
Page 43
4.2.5 Application Trace
Application Tracing is aimed primarily for the developers. It allows the user to see all method invocations
during a process flow. For every method, the used time is measured in microseconds. The application
therefore must be started in the “bytecode-modified” mode. This can be done without putting the VM into
the debug mode or redeploying the application. When the application is setup in the “bytecode-modified”
mode, it is restarted and its new class loader knows there are changes to be made in the bytecodes. Once
the debugging is over, the application can be started again in the “normal” mode. To use this tracing
method, no instrumentation is required from the application developer. The runtime control provides
everything needed for proper application tracing.
To access the runtime control in the Visual Administrator, follow the path: Server -> Services ->
Performance Tracing -> Runtime -> Application tracing tab (Figure 1-41).
Figure 1-41 Application Trace Menu in Visual Administrator
Sub-tabs “Application”, “Classes”, “Invocations”, “Objects” and “Plug-ins” are used to control the tracing
activities for the application.
“Application” sub-tab – area provides a list of the applications that are deployed on the server. Right
mouse button click on the specific application will provide an option for the apps restart in the “bytecode-
modified” or “normal” mode (Figure 1-42).
Page 44
Figure 1-42 Application Sub-tab
“Classes” sub-tab presents the Java bytecode-modified classes loaded. To check the content of the loaded
classes, the action from the Figure 1-43 has to be executed (“View Source”).
Figure 1-43 Classes Sub-tab
“Invocations” sub-tab presents the methods invoked, combined with the length of the execution timed in
microseconds, Figure 1-44.
Figure 1-44 Invocations Sub-tab
Page 45
“Objects” sub-tab provides a more detailed overview of the Java object graph on the target virtual
machine. It can display the values of the fields of every object, show the inheritance of classes, and
support paged views of arrays. Figure 1-45.
Figure 1-45 Objects Sub-tab
“Plug-ins” sub-tab provides the information about the “bytecode-modified” plug-ins, Figure 1-46.
Figure 1-46 Plug-ins Sub-tab
Page 46
4.2.6 Conclusion
Figure 1-47 When to use different performance tools (1)
After reviewing the various problem detection and problem analysis techniques, this paper will examine
the following:
When, how and which set of performance techniques should be utilized to properly locate and repair
application anomalies on the system?
A daily review of the system can be conducted using the following set of the tools:
o Application availability information is provided using the GRMG instrumentation (RZ20)
o DSR statistics in the ST03G (CEN) provide a basic set of the system performance related
data
o State Monitors in Visual Administrator (or CCMS in CEN, RZ20) present the state of the
Java engines,
o Properly configured Log Monitoring is another valuable source of the system status related
information.
Page 47
If problem is detected using one of the previously mentioned steps, an initial set of the problem analysis
techniques will be executed:
o Performance trace using the STATTRACE transaction
o Single Activity Trace using the Visual Administrator and Log Viewer
o SQL Trace if anomaly was discovered in the JDBC layer
Once the problem has been properly discovered and described, the application developers will determine
the problem solution path primarily using the Application Trace and Application Debugging functionality
in the Visual Administrator and SAP Netweaver Developer Studio.
An overview of the previously described steps is presented in Figure 1-48.
Figure 1-48 When to use different performance tools (2)
Page 48
5. IBM Java Technology for SAP
As mentioned before, SAP J2EE Engine utilizes IBM JDK 1.4.2 implementation as an underlying JVM
mechanism. Following paragraphs provide detailed description of the IBM JDK prerequisites and
configuration details for the successful Java based SAP Netweaver implementation (from the SAP Web
AS perspective).
It is important to state that the application of the recommendations from the following chapters should
only be executed after careful review of the specific group of the SAP OSS notes. Most of these notes will
be mentioned in this chapter, but the good starting point is the OSS note 716927.
IBM recently exchanged the JDK version 1.4.2 for the application stacks of SAP Netweaver 2004 and
SAP Netweaver 2004s. The new JDK version is called 1.4.2 Hybrid JVM and it is available for NW2004
and NW2004s on AIX 5L. It provides more consistent runtime behavior, enhances memory management
and improves debugging capability. It is also 100% upward compatible for all SAP business applications.
To check the details of the announcement, check the SAP OSS Note 1039948.
Additionally, there is a set of the OSS notes that should be reviewed before the recommendations are
applied:
o If planning to a new installation or an upgrade of SAP NW2004 SR1 or SAP NW2004s
SR1, follow the instructions in SAP Note 1024700
o For new installations of SAP NW2004s SR2, follow the instructions in SAP Note 1008311
o For SAP NW2004 SP14 and above or SAP NW2004s SP6, it is recommended to use the
new IBM 1.4.2 Hybrid JVM with J9 technology, follow the instructions in SAP Note
1024539
Before describing the IBM JVM parameters, we will provide the overview for one of the most important
activities executed inside the JVM working area. It is called a garbage collection and, in the simple
words, it provides a “heap memory recycling” mechanism.
Page 49
5.1 IBM JVM Garbage Collection
When the JVM cannot allocate an object from the current heap because of lack of space, a memory
allocation fault occurs, and the Garbage Collector is invoked. The first task of the Garbage Collector is to
collect all the garbage that is in the heap. This process starts when any thread calls the Garbage Collector,
either indirectly as a result of allocation failure, or directly by a specific call to System.gc(). The first step
is to retrieve all the locks the garbage collection process needs. This step ensures that other threads are not
suspended while they are holding critical locks. All the other threads are then suspended. Garbage
collection can then begin. It occurs in three phases:
o Mark
o Sweep
o Compaction (optional)
o Mark phase
In the mark phase, all objects that are referenced from the thread stacks, statics, interned strings and JNI
references are identified. This action creates the root set of objects the JVM references. Each of those
objects can, in turn, reference others. Therefore, the second part of the process is to scan each object for
other references it makes. These two processes together generate a vector that defines live objects.
o Sweep phase
After the mark phase, the mark vector contains a bit for every reachable object that is in the heap. The
mark vector must be a subset of the allocbits vector. The task of the sweep phase is to identify the
intersection of these vectors; that is, objects that have been allocated but are no longer referenced.
The original technique for this sweep phase was to start a scan from the bottom of the heap, and visit each
object in turn. The length of each object was held in the text that immediately preceded it on the heap. At
each object, the appropriate allocbit and markbit was tested to locate the garbage.
Now, the bitsweep technique avoids the need to scan objects that are in the heap and therefore avoids the
associated overhead cost for paging. In the bitsweep technique, the mark vector is examined directly to
look for long sequences of zeros (not marked), which probably identify free space.
When such a long sequence is found, the length of the object at the start of the sequence is examined to
determine the amount of free space to be released. Objects are not normally allocated from the heap itself
but from thread local heap, which is allocated from the heap and later used by an individual thread to meet
any allocation requirements.
o Compaction phase
When the garbage has been removed from the heap, the Garbage Collector can consider compacting the
resulting set of objects to remove the spaces between them. Because compaction can take a long time, the
Garbage Collector tries to avoid it if possible. Compaction is, therefore, a rare event.
Page 50
5.2 IBM JVM parameters
The following is an overview of the Hybrid JVM related parameters used in the previously mentioned
OSS notes. This list is a subset of all parameters available. No parameters for the “old” JVM version will
be discussed. If the default values of the presented parameters need to be changed, it must be done under
the close supervision of either the SAP and/or IBM support service.
o -Xms<size>, sets the initial memory size.

o -Xmx<size>, sets the maximum memory size.
o -Xmn<size>, sets both the values of the parameters –Xmns (the initial size of the new
space) and –Xmnx (the max size of the new space.)
o -Xgcpolicy:gencon, one of the four garbage collection policies. Handles short-lived objects
differently than objects that are long-lived. Applications that have many short-lived objects
can experience shorter pause times with this policy while still producing good throughput.
o -Xj9, switch that tells the JVM not to run in the “classic” mode.
o -Xdisableexplicitgc, disables the explicit calls to System.gc to request garbage collection.
In some cases, these calls can degrade performance time through premature garbage
collection and compactions.
o -Verbose:gc, this option is used to obtain information about the Java Object heap in real
time while running Java applications.
5.3 Monitoring the JVM activity
The initial set of the steps suggested to be used to check the status of the JVM (or the SAP JLaunch
process) are based on AIX system performance tools. Topas, nmon, vmstat, ps and svmon will provide a
basic overview of process related activity.
To view detailed information about the JLaunch process linked with the IBM JVM, the previously
explained Verbose:gc JDK option must be utilized.
When this option is used, all Garbage Collection activities are logged into a file named standard server out
(or std_serverX.out,) where X is the number of the corresponding server process (Jlaunch ID.) Every
server process has its own std_output file that resides in the “/usr/sap/<SID>/JCXX/work” directory.
The content of the simplified sample std_output file is presented in Figure 1-49. This is a sample file
simulating the content generated using a System.gc call. This call is used to manually initiate a Garbage
Collection.
Figure 1-49 Verbose:gc content as a result of the System.gc call
Another example is included to show how the content of the std_output file is changed if heap memory
allocation happens, Figure 1-50.
Page 51
Figure 1-50 Verbose:gc content as a result of the Heap memory allocation failure
Browsing the content of the sample file is a complex and time consuming process. Several different free
tools exist that can help present the GC generated data in a much more understandable way.
The tool that I use most often is called PMAT, “Pattern Modeling and Analysis Tools for Java Garbage
Collector”. PMAT can be downloaded from the IBM AlphaWorks Web site:
http://www.alphaworks.ibm.com/tech/pmat.
PMAT is used for parsing verbose GC traces, analysis of Java heap usage, and for recommending key
configurations based on pattern modeling of Java heap usage.
Figure 1-51 Initial screen of the PMAT tool
Page 52
One of the best options in the PMAT is the graphical overview of the JVM Garbage Collection related
activities, presented in the Figure 1-52:
Figure 1-52 PMAT tool (Graphical overview of the GC activities)
This chapter provided a basic overview of the IBM JVM tuning steps around the Garbage Collection
process.
I didn’t cover the specific tuning challenges around the most popular SAP solution, Enterprise Portal since
a document about it already exists and is mentioned in the Publications Appendix.
Page 53
Publications:
The publications listed in this section are considered particularly suitable for a more detailed discussion of
the topics covered in this white paper.
1. Publications
• “SAP Enterprise Portal on AIX 5.3 and Power5”, Walter Orb (IBM)
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100740
• “Tuning SAP R/3 with Oracle on pSeries”, Mark Gordon (IBM)
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100377
2. Online Resources
• SAP Service Marketplace

http://service.sap.com
• SAP Help Portal

http://help.sap.com
• Developer Works (IBM’s Resource for Developers)

http://www-130.ibm.com/developerworks/java/
• AlphaWorks (Java Technology)

http://www.alphaworks.ibm.com/java
• IBM Infocenter for AIX:

http://publib.boulder.ibm.com/infocenter/pseries/index.jsp
Page 54

IBM - SAP Java Monitoring &amp; Tuning Guide

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

IBM - SAP Java Monitoring &amp; Tuning Guide

Enviado por

Direitos autorais:

Formatos disponíveis

IBM Americas Advanced Technical Support

IBM SAP Technical Brief