Testing and Monitoring Citrix-based Applications

Executive Summary
The evolution of advanced computing has enabled the growth of the virtual workplace. Today, companies compete in global markets while supporting communications and information sharing among globally dispersed individuals and work teams. Tying people and business applications together via desktop computers, laptops, and handheld devices is no easy task. Enterprises need an application infrastructure that is easy to maintain, is scalable, and is easily managed. Thin-client/server-based architectures serve the needs of large enterprises because application processing is centralized. End user machines only need a thin application client component that results in a high degree of control, flexibility, and maintainability over client applications. This reduces IT overhead costs by reducing the time spent maintaining client PCs and simplifies the software upgrade process. Citrix virtual workplace solutions offer a cost effective, scalable and flexible way to keep IT costs down while supporting a continually evolving workplace. Their flagship product, MetaFrame, is a server-based computing solution in which client applications are executed on MetaFrame servers, typically in a load-balanced data center environment. This architecture allows all client applications to be installed and maintained centrally, and the only application installed on client PCs is the Citrix thin-client connection protocol, known as the independent computing architecture, or Citrix ICA. MetaFrame supports a wide variety of client devices, including PCs, PDAs, and cell phones, and the architecture lends itself to a wide range of network connections ranging from low-bandwidth connections such as wireless to high-speed LAN connections. Citrix virtual workplace solutions enable the enterprise to efficiently deploy, maintain, and continuously provide access to mission-critical applications with minimal workstation overhead. As with other components in the IT environment, deployment of applications running on MetaFrame servers must be carefully planned, monitored and managed. Until recently, the main testing facility available to MetaFrame customers was a free utility, downloadable from the Citrix Developer Network Web site. Known as the Citrix Server Test Kit, this utility provides basic script-based load testing on single MetaFrame servers. As Citrix customers began to implement more and more applications on more and more MetaFrame servers, they began to request an enterprise-class testing and monitoring facility capable of supporting these large-scale, multipleserver Citrix rollouts. Citrix and Mercury Interactive have responded to this need by forming a partnership with the goal of providing Mercurys enterprise-class application testing and monitoring solutions, optimized for the Citrix platform. Mercurys new products, LoadRunner for Citrix and Topaz for Citrix, are designed to accelerate MetaFrame application deployments and enable rapid detection and resolution of application performance problems. This paper will discuss the challenges of managing Citrix deployments, new MetaFrame testing and monitoring products, describe a case study of a Citrix customer that is using these products, and provide EMAs perspective on these new developments.

2002 Enterprise Management Associates, Inc.

PRODUCT PROFILE

C I T R I X

S Y S T E M S
technology allows new client devices to be adopted rapidly as business needs change and technology evolves. As MetaFrame has gained acceptance in the marketplace, Citrix customers began to realize that rollouts of client applications beyond a single MetaFrame server required more than basic testing tools. The addition of new applications to MetaFrame server farms must include thorough performance testing and capacity planning to ensure acceptable application availability and response times. Testing identifies potential performance problems before applications go live, and production performance monitoring enables IT organizations to quickly pinpoint and fix application bottlenecks. This combination of deployment testing and production monitoring reduces the application maintenance cycle and costs. The challenges of managing Citrix deployments must be addressed with enterprise-class management solutions. This paper will describe these challenges, new products for addressing these challenges, and how one Citrix customer is leveraging these products in their environment.

Introduction
The emergence of the Internet and wireless technologies as mediums for conducting business, coupled with lower cost computing devices, has spurred a rise in the number of virtual users accessing corporate applications. A workers physical location is no longer important, as emerging technologies provide access to corporate applications anytime, from anywhere, using a variety of devices. As people have moved from working in corporate offices to remote or home offices, the same application performance expectations exist, regardless of physical location. Virtual users expect application response times that are the same as working in the corporate office. In order to meet the needs of virtual and on-site users, IT organizations have turned to thin-client/serverbased computing as a means of simplifying application upgrades and maintenance. Server-based applications also require a fraction of the network bandwidth that traditional client/server architectures need, which is ideal for bandwidth-challenged remote workers. This enables the delivery of best-in-breed applications to users using a variety of platforms, including wireless technologies. Another advantage of thin-client/server-based computing environments is that the end user device only needs a lightweight client component to provide access to corporate applications. Thin clients display the application user interface on the end users device, leaving the processing and back-end data access to dedicated servers located in the corporate data center. Cost savings are realized because application changes take place on the server and not on the individual end-user machines. Cost savings are also realized through reduced application deployment times. For example, an application hosted on a ten server farm supporting 1,000 users, only needs updating in ten placesnot 1,000. This update process may be further automated using Citrix Installation Management Services to carry out the application provisioning across the server farm. Today, there are a number of application-serving solutions available, but the Citrix MetaFrame serverbased computing solution is unique because it offers an independent computing architecture (ICA) and remote presentation services protocol that can convert virtually any client device into a Citrix client. ICA
2002 Enterprise Management Associates, Inc.

Challenges of Managing Citrix Deployments

Thin-client/server-based architectures like Citrix reduce the complexity of application deployment and support, but the complexity and necessity of application testing prior to being released into production are increased. Traditional client/server applications are load-tested to ensure the performance of the back-end application and database server architecture; applications hosted on Citrix platforms also require testing of the MetaFrame architecture, adding another testing dimension to application deployments. Think about it a MetaFrame implementation supporting 10,000 users, each running 10 to 15 unique applications per workstation, results in a seemingly overwhelming number of testing possibilities. One Citrix customer has over 4,000 different applications that IT must support! Application lifecycles are decreasing due to a host of business and competitive reasons. Enterprise Management Associates (EMA) has discovered
October 2002 Page 2

through interviews of IT managers that extension of the coding portion of the application development schedule generally results in a corresponding decrease in the testing phase before deploymentnot due to decreased application defects, but because of increased pressure to deploy on schedule. Applications that are deployed without adequate production-quality testing may contain many serious defects that are undetected until the application is actually placed under actual production loads. Developers and IT staff must then use tools at hand to diagnose and repair problems, resulting in lengthy times-to-fix and counterproductive finger pointing. The lack of off-the-shelf production quality testing and monitoring solutions for MetaFrame applications have resulted in the Citrix environment or the network being blamed for problems that occur. Delays in troubleshooting and repairing production systems rapidly translate into escalated costs. Employees with inaccessible applications cannot perform their jobs, and minutes of application downtime translate into thousands or even millions of dollars of opportunity cost while people wait for the system to be fixed.

interaction. The value of the CSTK is that it provides basic load testing for individual MetaFrame servers, aiding IT in planning how many servers to buy. However, increasing scale and complexity of MetaFrame deployments dictate the need for a wholly different type of testing solution that can simulate actual end-user demand in a real-world environment that includes multiple applications running on many MetaFrame servers simultaneously.

Traditional MetaFrame Performance Testing Techniques

EMA discovered a number of makeshift performance testing techniques employed by Citrix customers because of the lack of enterprise-class testing products, including custom scripting, overprovisioning, and controlled rollouts.
Custom Scripting

Testing and Performance Monitoring in the Real-World

Enterprise Management Associates interviewed a number of Citrix MetaFrame users to learn about specific challenges in managing these deployments. All interviewees stated that they are continually asked to do more with less, and despite expanding application deployments, IT headcounts were unchanged or shrinking. The key to achieving best practices in this type of scenario is to leverage products that provide a stretched IT team with preventive diagnostic solutions and ongoing application performance monitoring that scales as the IT infrastructure expands.

A large global manufacturing firm with a 13-server MetaFrame server farm relies on an informal application testing process. Application stakeholders participate in a test plan meeting, after which the IT department creates homegrown test scripts and configures a test environment for a 30-day controlled test regimen. Writing unique test scripts for each new application rollout is a time consuming process. The lack of Citrix-specific application testing products and the fact that over 60 unique applications are deployed means that the IT group has to create new test plans and test scripts whenever a new application is added or upgraded.
Over-provisioning

The Citrix Server Test Kit (CSTK)

As mentioned in the Executive Summary, Citrix QA personnel have access to a free downloadable utility known as the Citrix Server Test Kit (CSTK), available on the Citrix Developer Web site (http:// www.citrix.com/cdn). The CSTK is a basic, semiautomated tool that allows administrators to perform limited user load testing against a single MetaFrame server. Simulation test scripts require manual coding and once created may be executed without user

The Director of IT for a facility ser vices firm, supporting more than 300 concurrent users, reports that it was less expensive to over-provision their systems rather than go through the process of writing custom test scripts. In this case, all application development was outsourced to a thirdparty vendor. The IT Director said, We talk to the application vendor first to

According to the IT Manager, We had to start [testing] from scratch each time because 70% of our applications are homegrown. We cant just use the same test scripts over and over. The homegrown applications have really burned us. The risk is that we do not know how applications can affect other applications or the operating system.
October 2002 Page 3

PRODUCT PROFILE

C I T R I X

S Y S T E M S
Regardless of how large or small the Citrix architecture is, unresolved application performance issues negatively impact end-user productivity, add to the cost of IT overhead, and negatively impact the business. Proactive application monitoring is the key to enabling rapid problem resolution and minimizing application downtime. In the absence of a MetaFrame troubleshooting solution, one IT department uses a homegrown threestep approach. When the help desk receives a call from an end user reporting slow application performance, support personnel look at the Citrix Management Console for clues to the problem. Then they check the status of server event logs, followed by checking the application itself. This process frequently leads to problem resolution, but only after a lengthy process where support personnel must manually correlate disparate data to pinpoint the cause of the problem. The lack of monitoring information forced one IT department to troubleshoot their application performance problems by rebuilding or restoring the server. This does not permanently repair the problem, which continues to reoccur. One corporation in the manufacturing sector, faced with a problem server, restores the server by shifting user loads to other servers in the server farm, taking the problem server One IT Director said, We offline, rebooting it, and support, on average, 3,000 then re-balancing the load on the server farma clients worldwide. Since we dont process that takes four have written ser vice level hours. The cost of this agreements or formal procedure in terms of IT resources, possibly reduced performance guarantees, it is performance levels and sometimes better to just let the increased risk of total outage results in application slow down instead of unacceptable risks to the taking it offline to troubleshoot business. the problem. When the A Citrix MetaFrame application is not available, we application testing and get a landslide of complaints. If performance-monitoring solution, used proactively in we had a tool to help us easily combination with IT best see the end-to-end application practices, could provide diagnostic information performance, we would know required to prevent what to fix and could address application outages entirely. slow downs immediately.
October 2002 Page 4

be sure the solution is Citrix compatible. Then we ask the vendor how many user connections each Citrix server can handle running this application. Then we triple the number of recommended servers. A large telecommunications companys global infrastructure group supports an IT architecture with MetaFrame servers located in Asia, Europe, and North America. Testing is part of this customers IT best practices, and each time a new application is rolled out or hardware is updated, the system is tested in production to ensure that it can handle the change. Test scripts are written to simulate 60 users per MetaFrame server, even though, by design, the maximum number of concurrent users on any of the Telcos MetaFrame systems never exceeds 40 users. The Automated Testing group is responsible for ensuring stability and performance of all
Controlled Rollouts

EMA spoke with a user who rolls out applications in a controlled and measured way. First, they deploy the data center applications. The IT application to a limited set Director said, We need to test of users for a 30-day period. During that time, the our Oracle forms application. We IT team closely monitors the write our own test scripts to MetaFrame servers, checkmeasure performance from the ing for any indication of performance bottlenecks. point of the Citrix server After trouble spots are reoutward. What we really need solved, the application is rolled out to another set of is something that will let us test users, and so on, until over a true end-to-end simulated user a period of months all end beginning at the end users PC, users have access to the application. This limited reall the way through to the Citrix lease testing is not ideal, server. particularly if an application must be simultaneously deployed to the entire user community.

Performance Monitoring Issues

Based on EMAs interviews, the size of Citrix deployments are increasing in terms of the quantity of end users accessing applications, the number of applications being accessed, and the number of MetaFrame servers within a given server farm. Enterprises continue to leverage their Citrix investments by scaling them upward to handle thousands of end users.
2002 Enterprise Management Associates, Inc.

New MetaFrame Application Testing and Monitoring Tools

Clearly, an enterprise-class applications testing and monitoring solution for the Citrix MetaFrame environment is needed. Mercury Interactive and Citrix Systems have formed an alliance to provide native load testing and performance monitoring products for MetaFrame applications. Mercurys LoadRunner for Citrix allows IT personnel to create real end-user test scripts that fully exploit Citrix ICA protocols, and Topaz for Citrix allows scripts generated by LoadRunner to be used for real-time monitoring of application performance and availability.

against the MetaFrame server farm, testing the application and server farm under load to assure stability and scalability. For example, a business process might include entering and posting a new customer via a Citrix client application. To record the transaction, the tester invokes VUGen and uses the application normally. Once the transaction has completed, the VUGen automatically generates a test script. This process requires minimal training and programming in order to execute performance tests. Without LoadRunner for Citrix, customers must devise their own scripts, tools, and methodologies. Mercury Interactive also added a new capability to LoadRunner for Citrix known as WAN emulation that provides the ability to simulate and test the effects of Wide Area Networks on end-user response-times and performance prior to deployment. Testing support capabilities can determine if the specific implementation of the MetaFrame platform is at risk from network behavior and provides follow-on alerts that prompt proactive steps to assure high performance. This WAN emulation applicationnetwork testing is critical to narrowing the responsetime gap between local and remotely located users. It is also instrumental in setting application performance and network deployment expectations and requirements.

LoadRunner for Citrix

LoadRunner for Citrix is Mercury Interactives classic LoadRunner utility, optimized for the Citrix MetaFrame environment using native ICA protocols. It simulates hundreds or thousands of users using a Virtual User Generator (VUGen) that allows testers to easily record, save, and replay test scripts. Testers use the keyboard and mouse to perform actual application transactions and the VUGen automatically generates a test script that reproduces the actions. The value of this functionality is that IT quality assurance testers can record real business transactions from within the application and then replay them repeatedly

FIGURE 1: LoadRunner Architecture PRODUCT PROFILE

October 2002

Page 5

C I T R I X

S Y S T E M S

FIGURE 2: Topaz Architecture

Topaz for Citrix

Topaz for Citrix is Mercury Interactives application performance and availability monitoring utility, optimized for the Citrix platform. Test scripts generated by LoadRunners Virtual User Generator may be re-used to monitor the performance of the application once it is placed into production. Topaz for Citrix automates monitoring, saving time over manual correlation methods, resulting in decreased time spent in problem resolution. Topaz allows administrators to proactively identify performance problems, often correcting the problem before endusers notice any impact on application response times. The effectiveness of LoadRunner and Topaz for Citrix products will be proven with time, but given Mercurys record of delivering stable and robust testing and monitoring solutions, EMA believes that Citrix and Mercury customers will embrace these products. For a smooth integration, LoadRunner and Topaz for Citrix licenses include Citrix professional services to ensure the success of testing and monitoring projects.

LoadRunner for Citrix Customer Case Study

Information Network Corporation (INC) is the information technology subsidiary of AmeriChoice Corporation. INC is an application service provider that processes medical and dental claims, maintains member and provider databases, and offers a variety of other healthcare business solutions for AmeriChoice as well as other companies throughout the United States.

The Decision to Move to a Citrix MetaFrame Solution

During a recent project, INC was faced with the task of implementing Perot Systems DIAMOND 950 medical claims processing and adjudication application. DIAMOND 950 had originally been implemented by INC via classic two-tier, fat client/ server architecture with an Oracle database on the back-end. INCs IT personnel were concerned that the DIAMOND 950 client/server architecture would not scale very well. They were also concerned about rolling out a fat client to end-users, as the bulk of the users are geographically dispersed at various locations in
October 2002 Page 6

2002 Enterprise Management Associates, Inc.

New York, New Jersey, and Pennsylvania. INC made the decision to implement a Citrix MetaFrame solution in order to drastically reduce bandwidth usage and to simplify the application rollout process.

Installing and configuring LoadRunner was as simple as picking up the manual.

According to Cantor, We originally investigated other test products, but they couldnt scale to the number of virtual users we needed to test. I was familiar with LoadRunner from previous engagements. As soon as we got LoadRunner in-house, we realized its value because it could scale to simulate enough virtual users for our needs. Our testers wanted to be sure there were no hotspots and wanted to be sure that the system could handle at least 250 concurrent users.

INCs Application
EMA spoke with Michael Cantor, Director of INCs Distributed Systems, to learn why the Citrix MetaFrame solution was selected. Cantors sixteenmember organization is responsible for all of the operational maintenance and support of AIX, Windows NT, and Linux operating systems; Oracle databases; and the overall LAN and WAN network infrastructure. INC processes 20,000 medical claims each day. The claims are entered into the adjudication application and the software determines the amount to pay out, automatically sending checks to the medical insurance providers.

Additional Uses for LoadRunner for Citrix

LoadRunner for Citrix is already slated for use in future application deployments. INC is preparing to rollout CarePlanner, which is a medical management package. CarePlanner is used by nurses to enter vital patient information, and the program then recommends the length of the hospital stay. CarePlanner must be available 24x7x365.

Potential Impact on End Users

Changes or updates to the claims application directly impact end-user productivity, particularly if an enmasse update is required. INC considered deploying DIAMOND 950 in a classic two-tiered environment. We were concerned about end-users being impacted by slow application response times if a fat client was used, said Cantor. Instead, INC decided to implement the application across 37 Citrix MetaFrame servers running Windows 2000. The existing DIAMOND 950 application supports 250 concurrent users, and plans call for it to scale upward to 750 concurrent users.

Cantor is looking forward LoadRunner let us identify to using LoadRunner for Citrix in the future problems in the Perot Systems CarePlanner rollout. It DIAMOND 950 application gives us a professional that needed to be fixed. evaluation of the systems much more professional than what was possible before. It allows us to also have better utilization of our servers and shows us where we do and do not have capacity.

ROI
While INC has not performed a formal ROI on their LoadRunner for Citrix investment, it is already paying off in terms of user satisfaction. Cantor notes the reason for buying LoadRunner for Citrix, We bought it to present a more professional appearance to our end-users, so we look good when the new application is launched. High system availability translates directly to the bottom line, in addition to bolstering ITs credibility.

LoadRunner for Citrix

At the onset of testing, INC made the decision to use Mercury Interactive testing products to ensure that the application would perform and scale according to its business requirements. The biggest goal that INC had was to turn on the new system, knowing that it would perform according to specifications without crashes or performance problems. Cantor utilized LoadRunner for Citrix as insurance that the system and application would perform as expected. Installation and configuration went smoothly, and Cantors team had LoadRunner up and running in three days. According to Cantor,

EMAs Perspective
Through customer interviews, EMA discovered that there is a real need for Citrix MetaFrame application performance testing and monitoring solutions in the marketplace, and Mercury Interactives LoadRunner PRODUCT PROFILE
October 2002 Page 7

C I T R I X

S Y S T E M S

for Citrix and Topaz for Citrix easily satisfy this requirement. Citrix and Mercury Interactive are the dominant players in their respective markets. Both firms have wellestablished customer bases and a record of proven financial success, including millions of deployed licenses. The partnership between these two companies to jointly provide a solution to previously unsolved performance testing and monitoring issues on the Citrix platform is a winning combination. LoadRunner for Citrix fully supports the Citrix native ICA architecture, allowing IT quality assurance personnel to create test scripts that simulate upward to tens of thousands of virtual Citrix end-users. The mechanism used to create and execute scripts is userfriendly and intuitive. LoadRunner provides more accurate, detailed and scalable end-user tests than the Citrix Server Test Kit does, because users can create actual end-user application traffic against multiple MetaFrame servers, helping to identify potential application performance bottlenecks before deployment. Test results provide IT personnel with the ability to deploy applications that have been optimized for top performance and scalability. Topaz for Citrix can share the same test scripts that LoadRunner for Citrix creates, supporting production application performance monitoring. This product gives IT operations personnel visibility into the performance of their applications from the end-user perspective, in real-time. Topaz can be used to ensure application, end user, and infrastructure service levels, and its end-to-end perspective provides a true picture of the end-user Quality of Experience (QoE). No other vendor offers production monitoring of this caliber for the Citrix MetaFrame environment. Citrix customers can now expand their deployments with confidence because Mercury Interactives LoadRunner for Citrix and Topaz for Citrix provide essential application testing and monitoring capabilities that were previously unavailable. IT managers concerned about performance, reliability, and scalability of applications running on their Citrix MetaFrame servers should seriously consider adding LoadRunner for Citrix and Topaz for Citrix to their arsenal of testing and monitoring products. The endresult will be rapid time-to-value, satisfied end-users, high return on investment, not to mention a high degree of IT credibility.
2002 Enterprise Management Associates, Inc. October 2002 Page 8

About Citrix Systems, Inc.

Citrix Systems, Inc. is a global leader in virtual workplace software and services that provide access to applications, information, processes and people on any device, over any network, anywhere, anytime. Citrix makes todays digital office completely virtual - instead of having to go to your office, the office follows you. Working with the worlds top industry, integration and consulting partners, Citrix delivers enterpriseready solutions to business challenges including application deployment, remote office connectivity, workforce mobility and business continuity. More than 120,000 companies worldwide use Citrix Solutions for the Virtual Workplace to give users more mobility, give IT organizations more flexibility and reduce the cost of computing.

851 West Cypress Creek Road Ft. Lauderdale, FL 33309 Phone: 954.267.3000 Fax: 954.267.3100 Email: info@citrix.com Web: www.citrix.com

Corporate Headquarters

About Mercury Interactive

Mercury Interactive, the global leader in business technology optimization (BTO), delivers Optane(tm), a suite of integrated products for enterprise testing, production tuning and performance management, that enables customers to optimize business processes and maximize business results. Customers worldwideincluding 75% of the Fortune 500use Mercury Interactive solutions across their application and technology infrastructures to continuously measure, maximize and manage performance at every level of the business process and each stage of the application lifecycle to improve quality, reduce costs, and align IT with business goals.

1325 Borregas Avenue Sunnyvale, CA 94089 Phone: 408.822.5200 Fax: 408.822.5300 Email: info@mercuryinteractive.com Web: www.mercuryinteractive.com

Corporate Headquarters

About Enterprise Management Associates, Inc.

Enterprise Management Associates, Inc. is the fastest growing analyst firm focused on the management software and services market. EMA brings strategic insights to both vendors and IT professionals seeking to leverage areas of growth across ebusiness, network, systems and application management. Enterprise Managements vision and insights draw from its ongoing research and the perspectives of an experienced team with diverse, real-world backgrounds in the IT, service provider, ISV and publishing communities. This report in whole or in part may not be duplicated, reproduced, stored in a retrieval system or retransmitted without prior written permission of Enterprise Management Associates, Inc. All opinions and estimates herein constitute our judgement as of this date and are subject to change without notice. Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. 2002 Enterprise Management Associates, Inc. All Rights Reserved.

Corporate Headquarters
2108 55th Street Suite 110 Boulder, CO 80301 Phone: 303.543.9500 Fax: 303.543.7687 Email: info@enterprisemanagement.com Web: www.enterprisemanagement.com
619.102802