Escolar Documentos
Profissional Documentos
Cultura Documentos
Proof of Concept Load Testing in the Linux Cloud using Open Source Tools
PROPRIETARY NOTICE This document contains confidential information. In consideration of receipt of this document, the recipient agrees to maintain such information in confidence and to not disclose this information to any person outside of Wipro Technologies without the written permission of the author. This Document contains total number of < > pages inclusive cover page.
Page 1 of 39
TABLE OF CONTENTS
1. EXECUTIVE SUMMARY..............................................................................................3 2. POC DETAILS...........................................................................................................5 2.1 Scope and Objective of the POC...........................................................................5 2.2 About TSung, Grinder and Jmeter in Load Testing Perspective............................5 <<WIP>>.....................................................................................................................7 1.1 2.3 Platform Details................................................................................................8 3. TEST DETAILS..........................................................................................................16 3.1 Test Approach & Procedure................................................................................16 Installing Apache JMeter............................................................................................22 3.2 CHALLENGES AND RESOLUTION STEPS...........................................................24 3.3 OBSERVATIONS AND ACHIEVEMENTS...............................................................30 4. CONCLUSION...........................................................................................................37 5. GLOSSARY...............................................................................................................38 6. POC TEAM................................................................................................................39
Page 2 of 39
1. EXECUTIVE SUMMARY
This Proof of Concept (PoC) is an initiative taken by Wipro Cloud Testing Practice and Performance Engineering Practice in-order to evaluate the capability of the existing open-source load testing tools that can be used to inject or simulate heavy load (as much as 1 million simultaneous users) on web-based applications. This PoC had to be carried out in Linux Based Amazons Cloud Environment. The tools that were selected were TSung, Grinder and Jmeter. As this PoC was first of its kind, it is appropriate to mention that team assigned had to proceed on Hit-and-Try mode at every step of this PoC execution as documentation available was poor. There were humungous challenges that the team had come across in carrying out this successfully. Most of the information is captured in this document so that it can be referred and used in future assignments. The outcome of the PoC is that TSung is able to inject up to 50,000 simultaneous users per agent machine, whereas Grinder and Jmeter is able to simulate up to ____ and 1000 request/sec (throughput) respectively. TSung majorly showed positive results in achieving the objective of this PoC. Below is the TSung Graph that was showcases the simulation of 0.9 million Simultaneous Users with 1 Controller and 11 Agent machines in the Amazons Ubuntu Cloud.
As a result of this PoC, Wipro Testing Services declares that using TSung tool, Wipro customers will be able to avail On-demand Performance Verification services, pay-as-you-go service using a flexible, affordable, hourly / per test usage model.
Page 3 of 39 HIGHLY CONF IDENTIAL
Page 4 of 39
2.
POC DETAILS
The purpose of this PoC is to present the Observations and Conclusions arrived at the Proof of Concept using various Open Source Tools: Grinder, TSung and Jmeter. The intention is to validate each Tools capability to ramp up 1m simultaneous users or simultaneous transactions on a Linuxbased Cloud Environment.
Benchmark the Scalability per server Benchmark the Horizontal Scalability Arrive at Peak Elasticity and Benchmark the same
scenarios Tsung enables developers to monitor the target server. The tool produces detailed statistics on performance, errors and target cluster behavior, which can be viewed in graph format. All statistics are produced in real time, so developers and systems administrators can view the results as the test progresses
GRINDER
The Grinder is a load testing framework that makes it easy to run a distributed test across a number of load injector machines. Test scripts are written in Jython, and HTTP scripts can be recorded easily from a browser session. This is a GUI based performance testing tool. The framework is comprised of three types of process or program (that can run test scripts across a number of machines): worker processes, agent processes, and the console. The responsibilities of each of the process types are: Worker processes o Interpret Jython test scripts and perform tests using a number of worker threads Agent processes o Manage worker processes The console o Coordinate the other processes o Collate and display statistics o Script editing and distribution As The Grinder is written in Java, each of these processes is a Java Virtual Machine (JVM).
Page 6 of 39
For heavy duty testing, agent process has to be started on each of several load injector machines. The worker processes they launch can be controlled and monitored using the console. Note that it is recommended to run one agent on each load injector for optimum results. JMETER Apache JMeter could be used to test functional and performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It is primarily used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. Also, it can be used to perform a functional test on websites, databases, LDAPs, webservices etc.
<<WIP>>
Page 7 of 39
1.1
TSUNG
Ubuntu Operating System 9.1 JDK 1.6 Erlang R13B01 TSung 1.3.2 Apache Tomcat Server 6.0 Sample Application (web page of 25KB, 50KB, 100 KB data) Amazon Cloud Watch and HyperHQ - for monitoring Gnuplot and perl5 - for HTML reports Python and mathplotlib - for graphical output with tsung-plotter SSH access to remote machines without password (use a RSA/DSA key without pass-phrase or ssh-agent) - for distributed tests Bash WinSCP to transfer local files to cloud machine and vice versa Note: Tsung is protocol independent and can be used to test Jabber, XMPP, HTTP, SOAP, LDAP and PostgreSQL servers. The solution can also be easily extended to provide support for new protocols.
GRINDER
Java5 Install the packages: sudo apt-get -y install openjdk-6-jre zip apache2 Test the Java installation: java -version Test Apache: curl http Jython
Page 8 of 39
JMETER
Java Environment A fully compliant Java 1.2 environment is required for Apache JMeter to execute.
(Note: Hopefully this will be changed back to 1.1 in the final release. Since Apache JMeter uses only standard Java APIs (java.*), please be informed that the JRE might fail to execute because of a class implementation could be missing, or not fully compliant. Java JFC (Swing components)
JSSE is used to provide an SSL implementation to enable testing of https protocol. Java Compiler [optional]
A Java compiler is not needed since the distribution includes a precompiled java binary archive. Note: A compiler is required if you plan to recompile the source code or if you plan to build your own plugins.
Page 9 of 39
TSUNG Controller VM Configuration #Instances Ag ent AUT 8 ECU, 4 Core, 15 GB 4 AUT 8 ECU, 4 Core, 15 GB 4 AUT 8 ECU, 4 Core, 15 GB 4 Load Balancer Am azon Load Balancer 1 Load Balancer Am azon Load Balancer 1 Load Balancer Am azon Load Balancer 1
26 ECU, 8 Core, 8 ECU, 4 Core, 15 68 GB GB 1 Controller 12 GRINDER Ag ent 8 ECU, 4 Core, 15 GB 4 JMETER Controller Ag ent 20 ECU, 8 Core, 7 GB 3
VM Configuration #Instances
8 ECU, 4 Core, 15 GB 1
VM Configuration #Instances
The following are the machine details that were used for each tools PoC. TSUNG - MACHINE DETAILS: Details VM Configuration Controller 26 ECU, 8 Core, 68 GB Agent 8 ECU, 4 Core, 15 GB AUT 8 ECU, 4 Core, 15 GB
Page 10 of 39
# of Instances
12
GRINDER - MACHINE DETAILS: Details VM Configuration # of Instances Controller 8 ECU, 4 Core, 15 GB 1 Agent 8 ECU, 4 Core, 15 GB 4 AUT 8 ECU, 4 Core, 15 GB 4
JMETER - MACHINE DETAILS: Details VM Configuration # of Instances Controller 4 ECU, 2 Core, 7.5 GB 1 Agent 20 ECU, 8 Core, 7 GB 3 AUT 8 ECU, 4 Core, 15 GB 4
Page 11 of 39
As per these details, each tool was set up as per the figure below and was connected to the cloud machines through the connection steps explained further.
Page 12 of 39
Go to Auth and locate the .ppk file (through the browse button) as shown below.
Under the session tab, provide the IP and the port and click save and then open
Page 13 of 39
Enter login as root and the Cloud machine is ready to start the tests. (Incase,
if it prompts for the password type in Wipro@123)
Page 14 of 39
Page 15 of 39
3. TEST DETAILS
3.1 Test Approach & Procedure 3.1.1 Tsung: Test Approach
Inorder to carry out a successful load scenario execution, the following are the approach steps that were required to be executed.
Set-up the pre-requisites Record the Business Scenarios Update main configuration file with recorded session Ex ecute the TSung script
i n c d l y v r o f s t a p e R
Conclude
Figure: Test Approach As per the above process, it is required to start the client and the server side machines listed in the Tsung Configuration file. To increase the number of users, it is possible to add more agent machines to the client tag of the Tsung configuration file.
Tsung: End-to-End Test Procedure Once the pre-requisite is set-up, follow the installation instructions for the first time and proceed towards configuring, recording, executing and generating reports for analysis. Installation of Tsung: Download the installation file: TSung-1.3.2.tar.gz Untar the TSung-1.3.2.tar.gz by typing the command tar -zxvf
TSung-1.3.2.tar.gz
Page 16 of 39
Recording with https protocol: Set Proxy setting to locahost:8090 in browser Start recording by typing command as:
> TSung-recorder start Open a browser and type http://www.abcabc.com (while login it opens https site) when we click on Signin It displays the address as http://-www.abcabc.com/login.asp
Steps to record the Tsung Script: Set the proxy settings to localhost:8090 Run the command from terminal > tsung-recorder p <protocol>
start. It takes http as default protocol. (Note: Various protocols that tsung supports are webdav, mysql, postgresql etc) Open a browser and access the URL, do transactions once you are done with the transactions logout Run the command from terminal> tsung-recorder stop To check the status of the recording Run the command from terminal> tsung-recorder status Once you stop the recording it will create a .xml file in /root/.tsung/xxx.xml floder
Below is the format of tags that is used in the main configuration file of tsung:
<tsung loglevel="emergency" version="1.0"> <clients> <Agents that are used for generation of load are listed here> <client host="<hostname>" weight="1" maxusers="<number of Users>" cpu="<number>"/> </clients> <servers> <Server Details are given here either one or amazon load balancer machine > <server host="Apache-SUT-Ravi-85998336.us-east-1.elb.amazonaws.com" port="80" type="tcp"/> </servers> <load> <Generation of load is defined in this tag> <arrivalphase phase="1" duration="10" unit="minute"> <users arrivalrate="50000" unit="second"/> </arrivalphase> </load> <options>
Page 17 of 39 HIGHLY CONF IDENTIAL
<option type="ts_http" name="user_agent"> <user_agent probability="80">Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Galeon/1.3.21</user_agent> <user_agent probability="20">Mozilla/5.0 (Windows; U; Windows NT 5.2; frFR; rv:1.7.8) Gecko/20050511 Firefox/1.0.4</user_agent> </option> </options> <sessions> <Insert the script that is generated while recording using tsung> </sessions> </tsung>
For this PoC scenario, please refer the TSung Main Configuration file (in xml format) below:
200request6Agents. xm l.xm l
Running Tsung Script: After recording, goto /home/Ubuntu/.tsung folder Type this command to run the tsung Script: tsung -f xxx.xml start Report Generation: Now, Goto /root/root/.tsung/log/log_timestamp folder Run the script by typing this command
Note: It takes time to run the script (as per the time specified in the script)
/usr/lib/tsung/bin/tsung_stats.pl It will generate the graph.html, reports.html and images folder Copy these folder and .html files to your system using WinSCP Open the graph.html and reports.html to analyze the reports To get a comparison report, run the following command tsplot <Text1 to be displayed in graph> <Location of first tsung.log> <Text2 to be displayed><Location of second tsung.log> -d <Destination folder>
Example: tsplot "scenario1" 20100818-05:00/tsung.log "scenario2" 2010081805:13/tsung.log -d /home/ubuntu/.tsung/report Note: tsung.log files will be generated for each scenario in the log folder located at /root/root/.tsung/log/log_timestamp folder as shown in below figure.
Page 18 of 39
Figure: Comparison report Note: In some cases it might take a long time to run the script and to check the processes type
ps -aef |grep erl (Controller M/c) kill -9 <ProcessID> ps -aef|grep beam(Agent Machine) kill -9 <ProcessID>
Note: The Grinder console is the user interface which provides the below features for the grinder. The Grinder console provides an easy way to control multiple testclient machines, display test results, and control test runs. Using the grinder console one can start and restart the test The console is used to coordinate the actions of the worker processes by sending them start, reset, and stop commands. Used to visualize test results in real-time.
To Start Grinder Agent: Use putty.exe grinder agent system, if you already logged into different machine then do ssh -I /home/ravi.pem root@agent IP Goto root folder of Grinder tool for ex: cd /usr/local/bin/grinder3.4/examples Update the console IP in Grinder agents "grinder.properties" file Now start the Agent by typing this command: java net.grinder.Grinder Eg.
Page 20 of 39 HIGHLY CONF IDENTIAL
The settings in the grinder.properties file are: grinder.processes=5 grinder.threads=3 grinder.runs=4 grinder.logDirectory=log grinder.numberOfOldLogs=2 #grinder.initialSleepTime=500 #grinder.sleepTimeFactor=0.01 #grinder.sleepTimeVariation=0.005
Figure: Settings in the grinder.properties file To Record in Grinder: Set the proxy settings to localhost: 8001 in the browser in the controller machine only. (access control m/c through VNC) Run the below mentioned command in the controller machine (access controller m/c through VNC) Type the below command from Controller Machine in its terminal. Java net.grinder.TCPProxy console http > Filename.py and hit enter. Open the browser from the Controller machine and start recording; for eg. start browsing through the website it will capture the script. Once you have done with the recording click on Stop button from the window. Now, copy or move the file to >mv Groc_Login.py/home/grinder3.4/examples/Groc_Login.py (where the grinder is installed to view this file in Grinder Console window). Set Grinder.properties file according to the requirements Executing the test scripts
Page 21 of 39 HIGHLY CONF IDENTIAL
To run the grinder test scripts; In the same directory where grinder.properties file is there, you will find your recorded scripts files, if not copy those script files to this directory which you would like to run. For example: httpscript.py- run java net.grinder.Grinder from that directory From the Console Click File > Start Collection and then start processes.
Below is the report screen on the Grinder Console that is shown as an output after execution of the test(s).
binary application and the system wrappers you need to run Apache JMeter. You may want to go the running instructions to continue. URL: http://www.jajakarta.org/jmeter/1.5.1/en/installing.html
Page 23 of 39
3.2 CHALLENGES
AND
RESOLUTION STEPS
The following table depicts the high level issues and challenges faced by the PoC team during this assignment.
Installing Tsung Tool on the linux machine - local Linux 4 RedHat machine 5 Installing Erlang on the linux First installation of Erlang software, the version used was machine Amazon Cloud 64bit "otp_src_R14A.tar.gz") Linux Fedora machine *we encountered errors for the command: "./configure" configure: error: No curses library functions found configure: error: /bin/sh '/root/TsungFiles/otp_src_R14A/erts/configure' failed for erts Second we tried to install Erlang s/w again ( the version used "otp_src_R13B.tar.gz") *we encountered errors for the command: "./configure" configure: error: No curses library functions found
Page 24 of 39
configure: error: /bin/sh '/root/TsungFiles/otp_src_R13B/erts/configure' failed for erts Finally we got the Fedora version of the Erlang s/w "otp_src_R12B...". We could successfully install Erlang.
Installing Tsung on the linux Tried to install Tsung 1.3.2 initially. Installation was not happening at all. Tried machine - cloud 64bit to find a 64bit compatible version of Tsung. But, the team couldnt find one. Hence we moved to the 32bit machine in cloud. 6 Linux Fedora machine Installing Erlang and Tsung on Team was successfully able to install Erlang (version 12B.) on the 32bit Fedora the linux machine - Cloud machine. But again the team was not able to install or find Fedora 32bit Linux Fedora machine compatible version of TSung. Hence it was moved to an Ubuntu machine on 7 the cloud. Installing of Erlang & TSung on We were successfully able to install the dependencies on the Ubuntu cloud Ubuntu machine in the cloud machine followed by Erlang & TSung tool 8 environment Exploring the recording With the help of the user manual & self exploration of the tool we were able to options in TSung identify the way in which the recording can happen successfully. 9 Recording the TSung script on The Ubuntu machine on the cloud does not have a GUI/Console to start a the cloud Ubuntu machine TSung recording. Hence we planned to record the TSung script on the local Linux machine copy the same to the cloud machine to run. 10 Recording the TSung script on TSung Tool listens to all the communication through a listening port (8090). the local Linux machine for an After this was set in the Browser connection settings. Session xml files got 11 intranet site. generated successfully. Replaying the TSung script User manual, sample xmls & other documentation on the internet helped to get after enhancements on the the TSung replay xml file in working condition. The replay script should be in a local Linux machine particular format of xml file. 12
Page 25 of 39
Replaying the new TSung script for sample page after enhancements on Ubuntu cloud machine/Local Linux 13 machine Recording TSung script for a sample page deployed outside the Wipro intranet on the local Linux machine 14 Replaying the new TSung script for sample page after enhancements on Ubuntu cloud machine/Local Linux 15 machine. 16 Enhancing the TSung script to accommodate multiple agents configuration, multiple option setting and other settings. 17 https issue while recording 18 https issue while execution 19 Inability to scale-up the number of simultaneous users 20 TS plot errors: a) While generating a comparison report, faced errors as matplotlib is not installed. b) DISPLAY environment
We are facing the error on trying to replay the script: - Permission denied (public key). (Solution : Need the SSH to be set as passwordless for client/server) We know that the local machine needs proxy setting in the browser to access the external pages, outside the firewall. Hence we faced the limitation of the local machine to record only the scripts for the sites within the firewall. We had to make modification in the TSung configuration file to get it work for the external sites as well. We were facing the error on trying to replay the script as - Kernel pid terminated (application controller).and more such errors These issues was resolved when we reinstalled TSung again on both the machines. This a very challenging task of enhancements because of the complexity in understanding the TSung xml file and adding enhancements to the file. Its also very challenging to make sure the enhancements are working as expected. Involves lot of trial & error method. In config file, parent_proxy attribute is set to TRUE to proceed with recording and instead of https:// we gave http://- Due to above step, it wasnt allowing to execute from backbone and so had to reset the parent_proxy attribute to FALSE and recorded without proxy in the cloud machine directly Load Balancer was used to distribute load to the additional servers and handle the peak requests & get maximum responses a) Installed matplotlib b) resolved by editing the tsplot file located at /usr/bin folder with the following text: import matplotlib matplotlib.use(Agg)
Page 26 of 39
As we are using Remote Connection to access the machine, need to make the drawing done virtually on the file instead of server.
Page 27 of 39
Page 28 of 39
Recording the script using badboy software and then 11 convert to Jmeter script file Exploring the recording options in both badboy & Jmeter 12 tool Replaying the Jmeter script after enhancements on the 13 local Linux machine 14 Tuning the JVM stack in agents machine
Installed the software in client machine and created the script for jmeter test execution With the help of the user manual & self exploration of the tool we were able to identify the way in with a recording can happen successfully. User manual, sample xmls & other documentation on the internet helped to get the Jmeter script in working condition. Added the thread stack size argument in the Jmeter file which is available in the Jmeter software bin folder
Page 29 of 39
3.3 OBSERVATIONS
AND
ACHIEVEMENTS
Page 30 of 39
Response Time:
Network Throughput:
Page 31 of 39
Simultaneous Users:
Observations: One Agent Server is able to scale upto 45,000 to 60,000 simultaneous users
Page 32 of 39
Amazons load balancer enabled in increasing the number of users to hit the server by balancing the load across the servers Number of simultaneous users has increased consistently with respect to the increase in the number of agents (from 1 agent to 11 agents) When the agents are increased beyond 11, the number of simultaneous users start to degrade in numbers
Achievements: Successfully utilized the Amazon Cloud Services to install and run tsung scripts even with poor documentation and forum support By using an open-source tool and the Ubuntu environment, huge cost savings is achieved by the organization as this POC requires high-end machines to generate such heavy loads Significantly, TSung was able to reach the number of simultaneous users to 0.9 million simultaneous users
Page 33 of 39
Grinder Readings: Work Proces ses 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 3 3 3 3 Respon se Time(m s) 2.5 2.8 3.06 3.33 3.97 2.54 2.65 2.89 2.82 3.15 3.12 3.63 4.03 4.39 4.65 6.68 7.64 10.4 52.04
Ap pl. 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2
Agen ts 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 3 3
Threa ds 1 2 3 4 5 6 7 8 9 10 11 12 13 15 10 10 10 10 10
TPS (mean) 375 692 955 1090 1210 1990 2290 2460 2830 2850 3150 3020 2980 3150 3760 4160 7300 8220 1710
TPS (Peak) 468 1130 1850 2660 2810 3820 5270 6330 4800 5220 6780 4790 8030 5250 5920 10200 11000 12100 2210
Errors 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Page 34 of 39
Sampl es 100 1000 3000 4500 5500 7500 9000 12000 13000 15000
Page 35 of 39
Page 36 of 39
4. CONCLUSION
TSung is successfully able to scale up to 0.9 m concurrent users on websites. It is yet to be proved to be working on web-applications. This will be taken up as PoC Phase II. .. . <<WIP>>
The below table gives more information on various tools: TSUNG GRINDER JMETER Per Server Upto 50,000 Only upto 1,000 Scalability Simultaneous Requests Users generated/sec
Using TSung tool, Wipro will be able to provide On-demand Performance Verification services, pay-as-you-go service using a flexible, affordable, hourly / per test usage model.
Page 37 of 39
5. GLOSSARY
Amazon Web Services (AWS): The Amazon Web Services are a collection of remote computing services (also called web services) that together make up a cloud computing platform, offered over the Internet by Amazon.com. The most central and well-known of these services are Amazon EC2 and Amazon S3. Scalability Testing: This test is to measure an applications capability to scale up or scale out in terms of any of its non-functional capability - be it the user load supported, the number of transactions, etc. TSung Tool: A robust benchmarking tool Tsung is an open-source distributed load testing tool. It is protocolindependent and can be used to stress and benchmark http and jabber servers. It simulates user behaviour using an xml description file, reports many measurements in real time (statistics can be customized with transactions, and graphics generated using gnuplot). For http, it supports 1.0 and 1.1, has a proxy mode to record sessions, supports get and post methods, cookies, and basic www-authentication. It also has support for ssl.
Page 38 of 39
6. POC TEAM
Initiative Drivers:
Sharad Kumar Engineering Practice Aditya Kittur Laksminarayana Prasad Praveen P Practice Krishna K Mariyanna Practice : : : : : Principal Consultant - Performance Technical Manager Cloud Testing Services Technical Manager Cloud Testing Services Sr. Consultant Performance Engineering Delivery Head - Performance Engineering at Wipro Testing Services
Contributors:
Jayasree T Nazeer A Pavana N S Dharmendra Soni Ramanathan G A Ravi Krishna N Kasturi Narra : : : : : : : Performance Technical Consultant Sr. Performance Technical Consultant Sr. Project Engineer Sr. Project Engineer Sr. Technical Consultant Project Engineer Project Engineer at Wipro Technologies
Authors:
Harsha R P, Praveen P, Jayasree T, Nazeer A at Wipro Technologies
Page 39 of 39