TUNING THE IBM WEBSPHERE APPLICATION SERVER Tuning IBM's WebSphere product.

White paper: "Methodology for Production Performance Tuning". Only non-product specific Java tips have been extracted here. (Page last updated September 2000, Added 2001-01-19, Author Gennaro (Jerry) Cuomo, Publisher IBM). Tips:

A size restricted queue (closed queue) allows system resources to be more tightly managed than an open queue. The network provides a front-end queue. A server should be configured to use the network queue as its bottleneck, i.e. only accept a request from the network when there are sufficient resources to process the request. This reduces the load on an app server. However, sufficient requests should be accepted to ensure that the app server is working at maximum capacity, i.e. try not to let a component sit idle while there are still requests that can be accepted even if other components are fully worked. Try to balance the workload of the various components. [Paper shows a nice throughput curve giving recommended scaling behavior for an server] The desirable target bottleneck is the CPU, i.e. a server should be tuned until the CPU is the remaining bottleneck. Adding CPUs is a simple remedy to this. Use connection pools and cached prepared statements for database access. Object memory management is particularly important for server applications. Typically garbage collection could take between 5% and 20% of the server execution time. Garbage collection statistics provide a useful monitor to determine the server's "health". Use the verbosegc flag to collect basic GC statistics. GC statistcs to monitor are: total time spent in GC (target less than 15% of execution time); average time per GC; average memory collected per GC; average objects collected per GC. For long lived server processes it is particularly important to eliminate memory leaks (references retained to objects and never released). Use -ms and -mx to tune the JVM heap. Bigger means more space but GC takes longer. Use the GC statistics to determine the optimal setting, i.e the setting which provides the minimum average overhead from GC. The ability to reload classes is typically achieved by testing a filesystem timestamp. This check should be done at set

intermediate periods, and not on every request as the filesystem check is an expensive operation.

Using IBM WebSphere Application Server Performance Tools

When it comes to e-business, performance is not merely a good thing to have. Far from it. Optimal performance is critical to the success of an e-business. In cognizance of this fact, the IBM WebSphere Application Server ships, as part of the product, several tools that can be used to effectively fine-tune the performance of an enterprise system by optimizing resources. With their different perspectives of the system, these tools, if used in conjunction with each other, can help customers gain a comprehensive insight into performance problem areas in their systems. This article presents concrete strategies to tackle typical performance problems encountered in enterprise applications, using the IBM WebSphere Application Server performance-monitoring tools. Performance Tools Overview Performance monitoring entails collecting and analyzing data pertinent to the performance of enterprise applications and their environment. (See Figure 1.)

The WebSphere Application Server collects data on run-time and applications through the Performance Monitoring Infrastructure (PMI). This infrastructure is compatible with and extends the Java 2 Platform, Enterprise Edition (J2EE) specification. Performance data can be monitored and analyzed with the Tivoli Performance Viewer (TPV) and

performance advisors. The TPV uses PMI data to provide graphical displays and summary reports of collected data. There are two performance advisors that offer advice to tune the application server configuration for improved performance: the Runtime Performance Advisor (RPA), which runs in the background when enabled; and the Tivoli Performance Advisor (TPA), which gives advice upon request. Further, Request Metrics collects data by timing requests as they travel through the system and logging time spent in major components, such as Web server, Web container, enterprise bean container, Web services, asynchronous beans, messaging and backend database. This article assumes prior knowledge of the WebSphere Application Server performance tools, and presents a set of techniques, hints, tips and tricks for experienced IT professionals to get the best out of the WebSphere Application Server. Strategies for Problem Determination When faced with a system that needs optimization, there are several approaches that can be taken, as shown in Figure 2, depending on who is involved in the system optimization effort. Typically, this task is performed by either an application developer or a system administrator.

System Administrators Viewpoint System administrators might begin analysis of the hosted infrastructure at the operating system level. They may use tools such as vmstat and top to gather system information. This information may be analyzed to detect bottlenecks that might exist at the OS level, such as insufficient memory, slow processor, etc. Before moving on to analyze the Java Virtual Machine (JVM), PMI should be enabled with an appropriate set of counters. Since the monitoring overhead is low for most PMI data except for JVMPI metrics and EJB per-method metrics (about 2.5 percent from the basic set and 3.7 percent from the extended set of counters), PMI data can be incorporated into a 24x7 monitoring tool (like IBM Tivoli Monitor) in a production environment. PMI has more than 100 different performance metrics. When monitoring PMI data, it might be useful to start with the top 10 metrics

and focus on the data that is most relevant to your infrastructure to detect problems and improve performance. For typical metrics that might be monitored as the top 10 high-priority metrics, see Figure 3.

The data collected by PMI can be analyzed via the TPV. This view provides insight into resource usage statistics like thread pools, connection pools, etc. Both programmers and system administrators can use TPV to monitor the overall health of the system. TPV also can be used to monitor system performance in a production environment. Once the JVM has been analyzed for potential performance bottlenecks, the system administrator may move to the hosted application. At this stage, Request Metrics may be used to get a granular view into the performance state of the system. Request Metrics helps track individual transactions by recording the process time in each of the major WebSphere Application Server components. This information can be used to determine the most time-intensive components. Customers can then focus on improving the configuration of those resources that contribute most to responsetime deterioration. Finally, performance advisors (both Runtime Performance Advisor and TPV Performance Advisor) can be used to view and analyze PMI metrics, understand the run-time state of the application server and provide tuning advice to optimize resource utilization in the application server. Application Developers Viewpoint Subtle changes to the preceding plan lead to an optimal strategy for application developers. Since a developer might be more comfortable working in the application domain, he may start his analysis with the application and adopt an inward-out approach, moving toward the operating system and hardware configuration. Thus, a developer may approach performance tuning as follows: 1. Detect resource constraints and performance bottlenecks in the system using Request Metrics, and take corrective measures to optimize these.

2. Make use of PMI data, as viewed through the Tivoli Performance Viewer, to understand average system resource usage statistics related to the JVM, and optimize these. 3. Analyze the advice given by the Runtime Performance Advisor, and apply as appropriate. 4. Move down the stack to analyze the OS resources and hardware configuration using the various system tools and optimize these. A thread dump, which should be used as a last resort, pinpoints problem areas to a significant degree of granularity (e.g., which SQL statement is causing a particular problem). A thread dump, however, is not recommended for production scenarios. Diagnosing Real-World Problems With an understanding of the typical strategies for using the WebSphere Application Server performance tools, lets look at some performance bottleneck scenarios that commonly occur in the real world, and how these can be monitored and detected. We used the trade application for our experiments: WebSphere Application Server Node:

Intel XEON @ 2.7 GHz x 2 Two-way 2.8GHz CPU 3.86 GB Memory 100 MB Ethernet Running Microsoft Windows 2003 Enterprise Edition

DB2 Node:

Intel XEON @ 3.2GHz x 2 Two-way 2.8GHz CPU 3.86 GB Memory 100 MB Ethernet Running Microsoft Windows 2000 Service Pack4 Version: IBM DB2 V 8.2, DB2 Universal JCC Type 4 (XA) Driver

Number of concurrent clients: 50 The performance bottleneck due to connection pool:

Trade DataSource Connection Pool Size: 5/5 WebContainer Thread Pool Size: 50/50 Prepare Statement Cache Size: 60

Tuning the database connection pool size based on the application and system configuration is important to improve overall performance. PMI and performance advisors can help to do this effectively. Upon enabling RPA, alerts to increase the connection pool size were received. However, the alerts provided by the performance advisors are based on the context, and you can get counterproductive results if the alerts are applied out of context. Consider the scenario where the pool size is set to too low. In this case, the application waits to get a database conn