Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

Tivoli Distributed Monitoring and Application Management

Tivoli Composite Application Manager for Transactions Best

Practices for Web Server Monitoring
Added by obriend, last edited by obriend on Jul 27, 2009 (view change)
Labels: (None)

Distributed Monitoring and Application Management

Home > Best Practices > Tivoli Composite Application Manager For Transactions Best Practices for Web Server Monitoring

Tivoli Composite Application Manager For Transactions Best

Practices for Web Server Monitoring
This white paper includes best practices for Web Server Monitoring using IBM Tivoli Composite Application Manager for
Transactions.

1 Introduction
1.1 Deployment considerations
2 Operational modes
2.1 Local traffic monitoring only
2.2 Remote traffic monitoring
2.3 General operational Requirements
3 Recommended Deployment Scenarios
3.1 Local Server Deployment
3.2 Appliance Mode Deployment
4 Capacity Planning
4.1 Application Management Console Editor (AMCE) Scale and Performance Guidelines
4.2 Aggregate only collection
4.3 Aggregate with Instance data Collection
5 Conclusion
6 About the Author

Introduction
Monitoring the performance and availability of your enterprise Web servers is a very important and challenging task. The
ability to collect statistics and to report and document the performance availability of your enterprise is critical to
achieving and documenting compliance with contracted Service Level Agreements (SLAs). There are many ways to
monitor your Web servers; however, there are essentially two main approaches in use today.

Simulated Monitoring, which is often referred to as Robotic Monitoring, is based on deploying prerecorded scripts
to different data centers across the world and having those scripts played back on a periodic schedule.
Real User Transaction Monitoring, sometimes referred to as passive monitoring, is usually implemented as a server
side monitoring solution that either analyzes log files or network traffic to determine the availability of servers and
the response time of user transactions. This type of monitoring gives you data on all, or alternatively a filtered set,
of your real user transactions.

Each of these approaches has advantages and disadvantages. This paper describes best practices for Real User

1 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

Transaction Monitoring, more specifically Real User Transaction Monitoring using the ITCAM for Transactions Web
Response Time Agent (ITCAM for WRT). ITCAM for Transactions 7.1 with Fix Pack 2 is required to perform all of the
functions described in this paper. You can find more information about ITCAM for Transactions 7.1 at the following Web
sites.

Product Web site: http://www-01.ibm.com/software/tivoli/products/composite-application-mgr-transactions/

Product information center: http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/index.jsp?topic=
/com.ibm.itcamt.doc/welcome.htm

Deployment considerations

You can use the ITCAM for Web Response Time (WRT) agent to monitor your Web servers by using network packet
analysis. The WRT Agent is not just an IP, UDP, or TCP analysis tool. The WRT Agent provides deep packet analysis for
application protocols such as HTTP and HTTPS. The WRT Agent has a couple of different modes of operation, depending
on your deployment needs.

Operational modes

Local traffic monitoring only

In this mode of operation, the WRT Agent only monitors traffic that has an origin, or destination, that is the same system
that the WRT agent is running on. This mode of operation is targeted at users who want to share the Web server
resources with the monitoring solution. The WRT agent has configurable processor and memory usage limits so that you
can limit its resource consumption when deployed, using this operating mode.

Remote traffic monitoring

In this mode of operation, the WRT Agent monitors all traffic that is visible from the network adapter. Thus, you can
deploy your monitoring solution using a dedicated physical system that is deployed such that it has access to all of the
network traffic for all, or a large subset, of the Web servers that you want to monitor. This results in keeping your
monitoring resources separate from your IT infrastructure resources, which prevents any monitoring impact on your
services, and possibly allowing different teams to manage the resources.

General operational Requirements

In either mode of operation, the WRT Agent needs minimal configuration to isolate and interpret your enterprise network
traffic. The common information that the agent needs in both operational modes is the HTTP or HTTPS ports that you are
using within your Web servers. When monitoring HTTPS traffic, you can use a Web server plug-in to gain access to the
Web server decrypted traffic and communicate the necessary information to a locally deployed WRT Agent. A
recommended alternative method is to use the Remote Traffic Monitoring function with a local or remote installation, and
provide the WRT Agent with a secure CMS key store containing the server certificates that the WRT Agent can use to
decrypt your Web servers HTTPS traffic for analysis. Essentially what you are doing in this preferred mode is installing the
WRT Agent on a remote system, giving it access to the Web servers SSL keys through the secure CMS key store, and
then instructing it to monitor all traffic to or from the host. This is a much simpler and reliable solution than using a Web
server plug-in cooperating with the WRT Agent.

Recommended Deployment Scenarios

There are several different deployment scenarios that you can use to monitor your Web servers. This paper details the
recommended methods of deploying the WRT Agent but briefly mentions the alternatives, if you have specific needs that
the recommended method cannot address. The deployment scenarios are organized into several different categories,
based on your preferred deployment method and monitoring requirements.

2 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

There are two supported deployment models:

Local Server Deployment

This involves installation on the physical Web server.
Remote Deployment (also referred to as Appliance Mode)
This involves a dedicated system installation of WRT, using a Network Tap or Switched Port Mirroring to
gain access to the network traffic of the Web servers.

There are two different monitoring requirements:

HTTP monitoring
HTTPS monitoring

Local Server Deployment

Local server deployment is the simplest deployment scenario. With this deployment, you can share and use unused,
resources with your Web server. This deployment model is only recommended for small deployments to a few servers and
to servers with moderate to low traffic. Large installations should use the Appliance Mode Deployment model to minimize
Web server impact of dropping data.

In this scenario you also want to configure the WRT Agent with processor and memory utilization limits to cap the amount
of resources it uses. When configuring the WRT Agent for local only monitoring, this defaults to a 20% maximum
processor and 1 GB of memory usage. If you enable Remote Monitoring, for example to enable HTTPS monitoring, these
limits are disabled and you must manually configure them to impose appropriate limits.
To reenable these limits, complete the following steps:

1. Open the kfcmenv file:

For Windows systems:

C:\IBM\ITM\TMAITM6\wrm\Analyzer\kfcmenv

For UNIX and Linux systems:

/opt/IBM/ITM/tmaitm6/wrm/kfcmenv

2. Modify the following two lines in bold text as appropriate:

# Target maximum CPU usage is 20% by default

KFC_CPU_ENFORCE_MAX_LIMIT=Y; KFC_CPU_TARGET_THRESHOLD=20;
# KFC_MEMORY_LIMIT enforces a maximum memory usage (1 GB by default)
The value should be in the following format:
# <number>(K|M|G), where the K, M or G suffix specifies the units as being
# Kilobytes, Megabytes or Gigabytes respectively.
KFC_MEMORY_LIMIT=1G;

HTTP Monitoring Only

Local deployment with HTTP only monitoring is the simplest deployment and configuration model that the WRT Agent
supports. In this model, you install the WRT Agent on the physical Web server and configure the Ports that the agent
should listen to for HTTP traffic.

The following screen capture shows configuring basic HTTP monitoring:

3 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

HTTP and HTTPS Monitoring

In the scenario in which you must also monitor HTTPS traffic, things become slightly more involved. It is strongly
recommended that you monitor HTTPS traffic using the Remote HTTPS monitoring capability even when you install the
WRT Agent using the Local Server Deployment model^ ^. The reason for this is that it doesn't require installation of a
plug-in to your Web server and the performance is much better because there is minimal Web Server to WRT Agent
cooperation occurring. In this deployment scenario, configure your HTTP monitoring normally, but in addition, complete
the following steps outlined in "Configuring WRT to Monitor Remote HTTPS Traffic" to enable HTTPS monitoring.

In this mode of operation it is also recommended you configure processor and memory limits.

Appliance Mode Deployment

Appliance mode deployment is where the WRT agent is installed on a dedicated system that is then deployed into the
data center and hooked into the network such that it has access to all of the data center Web server traffic. This is the
highest complexity installation, but it is also the highest performing, the easiest to maintain, and is likely to be the
cheapest overall solution for large data centers.

Network Access

Before the methods of deployment are described, the general concept of a network tap must be explained. In general, a
network tap is a method of having all traffic duplicated down a secondary path for monitoring purposes. This can be
accomplished in many ways, but the two most common methods are literal network taps and switch/router port
mirroring. When using these modes of operation, the appliance device must have a dedicated Network Interface Card
(NIC) for listening to network traffic and an additional NIC for communicating with the Tivoli Enterprise Monitoring Server
and other network resources.

Network Taps

A literal network tap is analogous to a phone tap. It is essentially a physical device that you insert in the middle of a

4 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

network connection, which duplicates traffic that is seen on the connection onto additional ports that it provides. Most
networks use full-duplex operation mode, meaning that if you have a 100 MB connection, you can send 100 MB/sec and
receive 100 MB/sec. Therefore, you have 200 MB/sec of aggregate data occurring within your network. Because of this,
most network taps have two output ports, one for the send channel and one for the receive channel that it is listening on.
These two ports then must be channel bonded by the monitoring appliance. Alternatively, there are aggregating network
taps, which combine the send and receive channels onto a single channel, which is then sent to your appliance device.

To use an appliance device with a network tap, you must use either an aggregating network tap or have a dual port
Ethernet adapter that supports link aggregation or link bonding on the appliance device, to receive the two different
channels as a single connection^ ^.

Figure 1

Figure 1 shows an example of a typical network tap. It is important when selecting network taps to choose a network tap
that allows traffic to continue to flow in the advent of a power failure. This prevents the network tap from becoming a
single point of failure in your system.

Port Mirroring

Another form of network monitoring is using a switch to perform Port Mirroring. Port mirroring is the duplication of traffic
from a specific port on the switch to another port for monitoring purposes. With this method you can use existing
networking infrastructure for monitoring purposes. The primary drawback is that you must have a free port on every
switch that you need to access to listen to network traffic. Additionally, because the switch is multiplexing the send and
receive channels onto a single send channel, there can be packet loss when the data flow exceeds the capacity of the
output port being used to monitor the traffic. If more than one port is mirrored onto a single port, it is likely that
significant packet loss will occur and an equivalent (or worse in the case of HTTPS) transaction data record loss. When
using port mirroring, it is best to do so with a one-to-one relationship.

Phases of Deployment

There are several phases involved in appliance mode deployment and we will discuss each phase separately.

Phase 1: Capacity Planning

This phase accomplishes the following high-level goals:

1. Identifies the number of Web Servers that you want to monitor and whether they service HTTP, HTTPS, or both
types of traffic
2. Identifies the number and location of your network taps, or routers for port mirroring, using the information
gathered in step 1
3. Identifies, approximately, the volume of traffic that each network tap, or mirrored port, must process.

5 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

Step 1 is the most important step because you need to identify what you want to monitor. Step 2 is often the most
difficult, given the size of today's intranets. If you only want to monitor traffic originating from customers on the external
Internet, then this step can be simplified by identifying the entry points into your data center and installing a network tap
at this location, or monitoring a port on the primary router or switch that governs this entry point. Step 3 is important in
identifying the hardware requirements for the appliance device hosting the WRT Agent.

Phase 2: Configuration requirements gathering

This phase is only important if you plan to monitor HTTPS traffic. To monitor HTTPS traffic, you must obtain the following
information for each Web server that you want to monitor:

Server IP Address
HTTPS Ports
Server Certificates exported as a PKCS file

These three pieces of information are required when configuring the WRT Agent for monitoring HTTPS traffic. This
information is required because you use it to associate network traffic for specific Web servers with the Web servers SSL
certificate, so that you can decrypt and process the network traffic.

Phase 3: Deployment and Configuration

During this phase you complete the following steps:

1. Install the physical network taps or configure the switches for port mirroring.
2. Hook up the system to the network tap or switch and validate that it is seeing SSL traffic.
3. Install and configure the ITCAM for WRT Agents on the physical appliance and configure it for monitoring the SSL
traffic.

If you are using network taps, this phase requires a brief outage of the network while the network tap is installed. If you
are using port mirroring, a network outage is probably not required.

Step 1 is rather straight forward but you probably need help from the network team. In step 2, I recommend
downloading and using the Wireshark analyzer^ ^ to validate that step 1 was done correctly. This is validated by
configuring this analyzer to capture the same traffic that ITCAM for WRT Agents will monitor and validating that it sees
this traffic. During step 3 you install and configure the WRT Agent using the information gathered during Phase 2.

Capacity Planning
Enterprises typically service a lot of Web traffic. Because of this, monitoring each and every Web transaction can require
significant resources. The amount of resources that are required varies, depending on your monitoring needs. The
primary components are as follows:

1. WRT Agent
1. Processor for processing the transactions
2. Disk space for storing the transaction data
3. Network bandwidth for accessing and transporting the data to permanent central storage
2. IBM Tivoli Monitoring Warehouse Proxy
1. Processor cycles for processing the data
2. Bandwidth for receiving the network data
3. Bandwidth for storing the data in the database
3. IBM Tivoli Monitoring Tivoli Enterprise Monitoring Server
1. Processor for transporting data from the WRT Agent to the Warehouse Proxy
2. Bandwidth for transporting the WRT Agent Data
4. Enterprise Database
1. Processor for processing inserts and queries

6 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

2. Disk space for storing the data

5. Tivoli Enterprise Portal Server
1. Processor for servicing Workspace requests
2. Bandwidth for Servicing Workspace requests
6. Monitoring and Reporting Configuration
7. How the data is configured to be aggregated and reported upon

The amount of resources that is used by these resources can be significantly affected by the configuration of the system.
This section includes recommended deployment configurations for large scale environments. Typical large scale
environments monitor Web sites that receive traffic from 1 - 30 million hits in a 24 hour period.

Whether you want to collect, store, and report on, raw instance data is the primary element that affects performance. If
you do not need to collect and report on individual instance records, your deployment considerations for large scale
deployments are much simpler. However, if you require the warehousing of instance data, for customized reports for
example, then you require significant customization of your deployment and configuration. Collection and warehousing of
instance data is not recommended and you will probably need to create your own custom reports to sift through the
volume of data that is collected.

Application Management Console Editor (AMCE) Scale and Performance

Guidelines

The AMCE is the primary method of configuring what and how transactions are monitored by ITCAM for Transactions. You
use this editor to define filters to include and exclude transactions and to customize how monitored transactions are
reported. This configuration can dramatically affect performance. The AMCE Configuration defines how transactions are
aggregated through the creation of Transaction Definitions and their Respective Reporting properties. The Reporting
properties directly define how transactions are aggregated, and thus directly control the number of unique aggregates
created from the instance data. A very poorly configured aggregation rule set can cause a one-to-one ratio of aggregate-
to-instance data and defeat all the benefits of aggregation.

In general, it is important to only aggregate the data based on your specific reporting needs. You do this by defining
Transaction Definitions that filter based on how you want to report on it and then only include the minimum required
information to report on in the Transaction Reporting Properties. Use of the QueryString value in the reporting properties
tab is a clear indication of potential performance and scale issues, unless your application has a very limited range of
possible QueryString properties and values. It is not recommended that you use this value in your reporting properties.
Instead use the $HTTP.GET:<name>$ variable to select specific Query String properties that you want to use in your
reports.

Aggregate only collection

Typical installations only save aggregate records for all the transactions being monitored. In this scenario, the
deployment considerations are significantly simplified because the amount of data being stored, transferred, and queried
by the monitoring infrastructure is significantly reduced. In this scenario the primary components to consider for
performance are:

WRT Agent Resources

Processor for Aggregation and workspace queries
AMCE Configuration
Application and Transaction definitions that affect the number of unique aggregate records that are created

For these large scale deployments, it is recommended that the WRT Agent have the following minimum hardware
requirements:

Linux is the recommended operating system.

4 processors: Either physical processors or Processing Cores
4 GB of RAM

7 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

25 GB of disk space

The Windows network driver tends to drop large amounts of packets when under high load and is not recommended for
monitoring very large throughput of Web servers in appliance mode. While dropping packets is usually not an issue
because the client system just resends the packet, in the situation where you are not the destination, but rather a
mirrored port or network tap, the fact that your network drops the packet causes permanent packet loss from the
perspective of the WRT Agent but not the actual Web server. This is because the packet being lost does not trigger a
resend by the TCP layer from the client because the client does still receive the receipt acknowledgement from the Web
server, which did receive the packet. This causes respective data loss in the transactions monitored, and in the case of
HTTPS monitoring causes the loss of all future transactions for that HTTPS session (not to be confused with the Web
session).

Aggregate with Instance data Collection

Collection of instance data for large scale deployments can significantly increase the deployment complexity and
resources that are required. The following overview lists the elements that must be customized and configured to handle
the large amount of data that will be generated. For large scale deployments it is important that during your planning
phase you include time to test and reconfigure the system, based on unforeseen bottlenecks due to the large amount of
data volume you will see in this scenario.

WRT Agent Requirements

Linux is the recommended operating system.
4 processors: Either physical processors or Processing Cores.
8 GB of RAM
25 GB of disk space
WRT Agent Configuration Requirements:
WRT TEMA Configuration: In the Data Analysis tab, set the Number of hours to save data = 1
KHD_RETENTION=-1
Triggers immediate warehousing of all historical data
Limitation: Instance data history will not be available for viewing in the Tivoli Enterprise
Portal until after 24 hours.
As an alternative, set =1 hour to see 1 hour of history
KHD_DEBUG=1
Enables warehouse at the same time as historical collection which should a minimum of (5
minutes)
3 Network Interface Cards (NICs)
Bandwidth must be 2x maximum bandwidth of monitored device
1 dedicated NIC to monitor IP traffic (connected to tap)
1 dedicated NIC for IBM Tivoli Monitoring communication
1 dedicated NIC for Warehouse Proxy Agent (WPA)+DB inserts
Alternative: You can use just 2 NICs and share IBM Tivoli Monitoring +WPA

IBM Tivoli Monitoring Warehouse Database Requirements

IBM Tivoli Monitoring Warehouse Database must also be configured to be able to store up to 30 million records into the
WRT Instance data table and be able to handle a very large size database.

Configuration Requirements

Configure the IBM Tivoli Monitoring infrastructure to warehouse the following tables, using the following
recommendations:

The WRT_Transaction_Instance Table contains all of the WRT instance data:

Collect and Warehouse the data every 5 minutes.

8 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

Do not summarize this table.

Prune every at least every 7 days.
WRT_Transaction_Status and WRT_Application_Status:
5 minute aggregate tables (configureable).
Collect & warehouse on whatever schedule you need.
Summarize & Prune on whatever schedule you need.
These are the main tables for all historical aggregated reports (including Tivoli Common Reports).

IBM Tivoli Monitoring Infrastructure Deployment Requirements

The IBM Tivoli Monitoring infrastructure must also be deployed and configured in a certain manner to support this large
volume of data.

1. There must be a dedicated Remote Tivoli Enterprise Monitoring Server for each of the WRT appliance installations,
installed locally on the WRT appliance.
2. There must be a dedicated Warehouse Proxy Agent for each of the WRT appliance installations:
1. Installed locally on the WRT appliance
2. Configured to be talking only to the local Remote Tivoli Enterprise Monitoring Server
3. Ability to talk directly with the IBM Tivoli Monitoring Warehouse Database

You install the Remote Tivoli Enterprise Monitoring Server and the Warehouse Proxy on the same system as the WRT
Agent to avoid the primary bottleneck of transferring the data from the WRT Agent through the monitoring server and to
the Warehouse Proxy Agent.

Conclusion
The ITCAM for Transactions Web Response Time Monitoring component can be used in very simple environments all the
way up to very large and complex environments. A variety of installation and deployment modes are supported so that
you can determine what best fits into your IT infrastructure.

About the Author

This white paper was written by Bret Patterson.

9 of 10 11/7/2012 11:57 AM
Tivoli Composite Application Manager for Transactions Best Practices f... https://www.ibm.com/developerworks/wikis/display/tivolimonitoring/Tiv...

10 of 10 11/7/2012 11:57 AM