Performance Testing Tool Evaluation for Web Performance Suite 3.

599

Performance Testing Tool Evaluation for Web Performance Suite 3.599

By Arkadyuti Roy (169655) Performance Testing CoE

White Paper

Page 1 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

Abstract
This paper discusses the details of the evaluation of Web Performance Suite 3.599, a performance testing tool targeted to test web sites that use standard technologies such as web forms, J2EE, ASP, .NET/C#, PHP, Ruby, ColdFusion, Java, etc. A comparative study of the tool has been done with respect to other well-known performance testing tools like LoadRunner 7.8, Silk Performer 6.0 and IBM Rational Robot 7 satisfying criteria used by Cognizant for evaluation of performance testing tools.

White Paper

Page 2 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

Table of Contents
Abstract................................................................................................................2 1.0 2.0 Introduction ................................................................................................4 Cognizant Evaluation: Criteria used by Cognizant for Evaluation.........4
2.1 2.2 Key Factors considered ........................................................................................ 4 Other factors (to be considered) ........................................................................... 5

3.0 4.0 5.0 6.0 7.0 8.0 9.0

Load and Stress Test Capabilities ............................................................5 Tool Reporting Capability ..........................................................................7 Tools Ranking.............................................................................................7 Tools / Supported Protocols .....................................................................8 Performance Monitor Test Features .........................................................8 Consulting Requirements ..........................................................................9 Vendor Qualifications ..............................................................................10

10.0 Vendor Support ........................................................................................10 11.0 Conclusion ................................................................................................11

11.1 Potential Benefits ................................................................................................ 11 11.2 Potential Limitations............................................................................................ 11 11.3 Constraints on Current Evaluation ..................................................................... 12

12.0 References ................................................................................................12

White Paper

Page 3 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

1.0 Introduction
The Web Performance Suite is web testing software targeted at web sites that use standard technologies such as web forms, J2EE, ASP, .NET/C#, PHP, Ruby, ColdFusion, Java, etc. Its growing list of modules currently can tackle everything from baseline performance analysis to a load test with tens of thousands of simulated users. The evaluation of web performer began with the need to evaluate the tool as a load generator tool in place of the Web Load tool that is currently being used for the Web Re-launch project. To evaluate the tool a Cognizant benchmark criteria is followed and certain functionalities of the tool (Web Performance Suite 3.5xx Demo Version) have been evaluated and compared with other standard tools, like LoadRunner 7.8 (the most popular Performance testing tool in the market), Silk Performer 6.0 ( the most powerful performance testing tool in the market) and IBM Rational Robot 7. The LoadRunner, Silk Performer and Rational Robot are already evaluated tools. The problem with the most commonly used Load Testing Tools (like LoadRunner, Silk Performer or Rational) is that they are costly. Generally Licenses are given on basis of the number of VUsers to be used for the tests, or in a timeline basis. Thats why the search for a cheaper tool like web Performer is encouraged. The license of Web Performance Suite is cheaper and its for Life time. Scripting is simpler in case of web performer in comparison to the other tools. But the tool has got some limitations while modifying the test cases as there is no customized scripting as we get in case of LR, Silk or Rational.

2.0 Cognizant Evaluation: Criteria used by Cognizant for Evaluation

While evaluating any load testing tool, following evaluation criteria are used by Cognizant. The team has reviewed the product based on these key factors:

2.1 Key Factors considered

o Load and Stress Test Capabilities o Performance Monitor Test Features o Tool Reporting Capability In addition certain additional factors will also need to be considered to ensure a comprehensive review of the product and the vendor (for due diligence):

White Paper

Page 4 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

2.2 Other factors (to be considered)

o o o o o Consulting Requirements Maturity of vendor Market share of vendor Vendor Qualifications (e.g. Financial stability) Vendor support

3.0 Load and Stress Test Capabilities

WEB PERFORMLOADRATIONAL ER 3.5 RUNNER 7 Scripting and Debugging Capabilities SILK PERFOR-MER

FEATURE

Allows for reusing scripts from functional test suite

Yes

Yes

Yes

Yes Yes Yes

Tool Customization No Yes Yes Test Language Features (Full Debugging Capabilities) No Yes Yes Ability to have data-driven scripts that can use a stored pool of data Yes Yes Yes Actions in a script can be iterated any specified number of times without programming or rerecording of the script Yes Yes Yes Cookies and sessions IDs automatically correlated during recording and playback for dynamically changing Web environments Yes Yes Yes Different modem connection speeds and browser types can be applied to a script without any rerecording Yes Yes Yes Automatic correlation or handling dynamic data No Yes Yes Load Test Scenario Creation Capabilities All users can be queued to execute a specified action at the same time. No Yes Yes

Yes

Yes

Yes

Yes Yes

Yes

White Paper

Page 5 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

Automatic load scenario generation based on load testing goals: hits/second, number of concurrent users before specified performance degradation, and so on Yes Yes Yes Load runs and groups of users within load runs can be scheduled to execute at different times Yes Yes Yes Scales to 5001,000 virtual users Yes Yes Yes Thread-based virtual user simulation Yes Yes Yes Process-based virtual user simulation No Yes Yes Centralized load test controller Yes Yes Yes Allows for variable access methods and ability to mix access methods in a single scenario: modem simulation or various line speed simulation Yes Yes Yes Allows for throttle control for dynamic load generation No Yes No Allows for automatic servicelevel violation (boundary value) checks No Yes Yes Allows for variable recording levels (network, Web, API, and so on) Yes Yes Yes Load Test Analysis Capabilities Allows for transaction breakdown/drilldown capabilities for integrity verification at the per client, per session, and per instance level for virtual users Yes Yes Yes Load Test Execution Capabilities Adding or removing users No Yes No during test run Snapshot on error Distributing load on across
White Paper

Yes

Yes Yes Yes Yes Yes

Yes Yes

Yes

Yes

Yes Yes

Yes Yes

Yes yes

Yes yes

Yes yes
Page 6 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

agents Load Test Feedback Allows for Web application server integration: ColdFusion, NetDynamics, Yes Dynamo, and so on Yes Can run tests on various hardware and software configurations Yes Yes Compatible with SSL recording Yes Yes Compatible with one or more of the relevant technologies: Linux, UNIX, NT, XWindows, Windows CE, Win3.1, Win95, Win98, in2000, WinME Yes Yes CITRIX / Microsoft Terminal Services Compatibility Yes Yes

Yes

Yes

Yes Yes

Yes Yes

Yes Yes

Yes Yes

4.0 Tool Reporting Capability

FEATURE WEB PERFORME R 3.5 Yes Yes Yes Load Runner 7.8 Yes Yes Yes Rational 7 Silk Perform er 6.0 Yes Yes Yes

Summary Level Reporting Test Report Presentation , Customization Of Reports Automatic HTML Report Generation Settings Web page/component breakdown with component size Merging graphs Zoom in/out in the graphs, along with granularity definition Filtering graphs in different timestamps.

Yes Yes Yes

Yes No

Yes Yes

Yes No

Yes Yes

No No

Yes Yes

Yes No

Yes Yes

White Paper

Page 7 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

5.0 Tools / Supported Protocols

Tools/ Supported Protocol HTTP HTTPS COM/DCOM RMI/IIOP Corba/IIOP Winsock ODBC People soft Siebel Web WEB PERFORMER 3.5 X X X X X X X Load Runner 7.8 Rational 7 X X Silk Performer 6.0

The symbol means that the testing tool supports the corresponding protocol.

6.0 Performance Monitor Test Features

FEATURE WEB PERFORME R 3.5 No LOAD- RATI RUNNE ONAL R Yes Yes SILK PERFOR -MER Yes

Monitors various tiers: Web server, database server, and app server separately Supports monitoring for one or more of ColdFusion, Broadvision, BEA WebLogic, Silverstream, ATG Dynamo, Apache, IBM Websphere, Oracle RDBMS, MS SQL Server, Real Media Server, IIS, Netscape Web Server

No

Yes

Yes

No

White Paper

Page 8 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

Supports monitoring for one or more of the relevant technologies: Linux, NT, UNIX, XWindows, Windows CE, Win3.1, Win95/98, Win2000 Monitors network segments Supports resource monitoring Synchronization ability in order to determine locking, deadlock conditions, and concurrency control problems Ability to correlate any metrics from all monitors to identify performance bottlenecks Ability to detect when events have completed in a reliable fashion Ability to provide client-to-server response times Ability to provide graphical results and export them to common formats

Yes

Yes

Yes

No

Yes Yes Yes

Yes Yes Yes

Yes Yes Yes

No Yes Yes

No

Yes

Yes

Yes

Yes No Yes

Yes Yes Yes

Yes Yes Yes

Yes Yes Yes

7.0 Consulting Requirements

FEATURE WEB PERFORMER 3.5 Tool Has been supported since 2004 Load Runner 7.8 Tools have been supported since early 1990s. Considerable Rational 7 Various testing products have been on the market since early 1980s Sustaining Silk Performer 6.0 Silk Test/QA Partner products on the market since early 1990s. Sustaining

Maturity of vendor

Market share of vendor

Sustaining

White Paper

Page 9 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

8.0 Vendor Qualifications

FEATURE WEB Load PERFORMER Runner 3.5 7.8 Continued/consistent Web growth pattern over Performance last several years Incorporated 1999 1989 Rational 7 Stable / Profitable Company 1980 Silk Performer 6.0 Stable / Profitable Company 1990

Financial stability of vendor Time existed

9.0 Vendor Support

FEATURE WEB PERFORMER 3.5 Yes No Load Runner 7.8 Yes Yes SSPA STAR Award in the last 2 years for excellence in Support Rational 7 Yes Yes Service has received many awards Silk Performer 6.0 Yes Yes

Support Onsite and Public Training around the world Awards Received

Good Tech Support

10.0 Tools Ranking

Criteria WEB PERFORME R 3.5 2 3 3 2 Load Runner 7.8 4 4 4 4 Rational 7 4 4 3 3 Silk Perfor mer 6.0 4 4 4 4
Page 10 of 12

Scripting Load Test Scenario Creation Load Test Analysis Load Test Execution
White Paper

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

Load Test Feedback Reporting

3 2

4 3

3 4

4 3

Tools Ranking (1 5, 1 = Least, 2 = Low, 3= Average, 4= Good, 5=Best) is evaluated from section 3.0 and 4.0.

11.0

Conclusion

From the evaluation of the tool, the following potential benefits, and limitations of Web Performer can be concluded.

11.1 Potential Benefits

Automatic generation of summary load testing analysis reports, including automatic HTML report generation. Ability to provide graphical results and export them to common formats Cookies and sessions IDs automatically correlated during recording and playback for dynamically changing Web environments Allows for variable access methods and ability to mix access methods in a single scenario. Ability to have data-driven scripts that can use a stored pool of data. Allows for transaction breakdown/drilldown capabilities for integrity verification at the per client, per session, and per instance level for virtual users Supports workload, resource, and/or performance modeling Centralized load test controller Compatible with SSL recording Monitors various tiers: Web server, database server, and app server separately Supports monitoring for one or more of Oracle RDBMS, IIS Ability to provide client-to-server response times

11.2 Potential Limitations

Does not facilitate customized reports (such that 90th % or 95th% of response time, or page time) to compare results.

White Paper

Page 11 of 12

PDF created with pdfFactory trial version www.pdffactory.com

Performance Testing Tool Evaluation for Web Performance Suite 3.599

There is not customized scripting. On the contrary, it uses a set of screens / dialogue boxes to construct the scripts. Test cases cannot be modified to take randomized query parameter values. It uses a table of pre-defined parameter sets to vary the request query string. Does not allow adding or removing a virtual user during execution. Does not support protocols like COM/DCOM, RMI/IIOP, Corba/IIOP, Winsock, ODBC, People soft, Siebel Web. Does not support monitoring for one or more of ColdFusion, Broadvision, BEA WebLogic, Silverstream, ATG Dynamo, Apache, IBM Websphere, Oracle RDBMS, MS SQL Server, Real Media Server, IIS, Netscape Web Server. Does not allow all users to be queued to do a specific action at a specific time (inability to produce Rendezvous point).

11.3 Constraints on Current Evaluation

The evaluation copy only allows a limited number of users. This is not suitable to scale up the number of users to test under high load. Several features as documented in the website are unavailable on this evaluation version.

12.0 References
http://www.webperformanceinc.com/

White Paper

Page 12 of 12

PDF created with pdfFactory trial version www.pdffactory.com