Ten Tips of Web App Testing

How to test and launch a world-class web application

WHITEPAPER: March, 2010

Table of Contents

Introduction: Testing on a Moving Target…………………………………………………………………………………………….……………………….…….… 2

New Challenges, Old Solutions …………………………….………………………………………………………………..…………………….………………..……….. 2
The Crowdsourcing Advantage………………………………….………………………………………………………………………………………..……………………. 2
Testing Types………………………………………………………………………………………………………………………………………………………………….……………………….. 3
Functional Testing………………………………………………..……………………………………………..…………………………………………………………..……. 3
Usability Testing……………………………………..…………………………………………………………….…………………………………………………………..……. 3
Load Testing ……………………………………………………………..………………………………………………………..………………………………………………..……. 4
Web Testing Checklist ………………………………………………………………………………………………………………………………………………………………..……. 4

Web Testing Tips: Launching a Higher Quality Web Application……………..……………………………………………..….……….. 5

1. Don’t Be a Slave To Your Metrics ………………………………………………………………………………………………………………………………..……. 5
2. Know Thy Third Party Apps ……………………………………………………………………………………………………………………………………………..……. 5
3. Understand The "Testing Managers Dilemma" ……………………………………………………………………………………………..……. 5
4. Invest In Testers, Then Tools…………………………………………………………...……………………………………………………………..…………………… 6
5. Protect Users, Save Yourself ……………………………………………………….…………………………………..………………………………………………….. 6
6. Multimedia, Multi-Problems ……………………………………………………………………………………………………….…….………………………….……. 7
7. Forfeit The Blame Game ……………..………………………………………………………………………………………….………………………………...…………. 7
8. Be Everywhere …………………………………….……………………………………………..……………………………………………………………………………………..…… 7
9. Plugins: Proceed With Caution …………………………………………………………………..………………………………………………………………..…… 8
10. Respect The Dead (IE6), Make Way For The New (HTML 5)……………………………………………….……………………. 8

About uTest……………………………………………………………………………………………………………………………………………………………………………………………………. 9

"Why is there never time to do it right, but always time to do it over?"

- Colonel Mike Mullane

Retired NASA Astronaut

WHITEPAPER: 10 Tips of Web Testing 1

Testing on a Moving Target (the Web)
New Challenges, Old Solutions for Testing Your Web Application
Establishing procedure amidst a constant state of change may seem like an absurdity (and a losing
battle), but this is what today’s software companies face when it comes to testing their web
applications. Aside from an expanding matrix of new
browsers, plugins, third party apps, programming “Complexity kills. It sucks the life out of
languages and more, there are now thousands, if not developers, it makes products difficult to
millions, of new users from all around the globe. Of plan, build and test, it introduces security
course, none of this was in the manual! challenges, and it causes end-user and
administrator frustration.”
Thus, the purpose of this whitepaper is twofold. While
our ‘10 Tips’ section will help you stay on top on of the - Ray Ozzie
latest trends in web app testing, our introduction will
Chief Software Architect, Microsoft
outline the basics of functional, usability and load
testing for web-based applications. Along the way,
we’ll cover topics such as:

 Third Party Apps: Your product is only part of the equation - what about the satellite apps that
are orbiting your web application?
 Cloud Computing: With opportunity comes risk. There’s a ton of upside to cloud computing, but
what happens to your web app when it fails to functional properly?
 Multimedia: How will your web app perform for users with different versions of Flash, Acrobat,
Java, etc.? Have you thought about HTML5 and the challenges of managing H.264 vs. Theora?
 Plugins: While the use of plugins may be slowing, the use of extensions are not. Learn why
plugins and their extensions should be a testing focal point.
 Localization: Obviously, your application needs to work where your users reside. Discover why
most software companies underestimate the importance of localization in their web testing.
 Deadlines: Your web application will never be perfect, so how will you know when it’s safe to
launch? It’s called the “Testing Managers Dilemma” and we’ll show you how to deal with it.
 Metrics: As one testing expert put it, “When your car is about to go off a cliff, it’s a weird time to
be thinking about gas mileage and drag coefficients.” Find out how metrics can mislead.

The Crowdsourcing Advantage

While the following material is suited for companies that rely on traditional staffing models for testing
(i.e. internal QA teams or offshoring firms), this whitepaper will be especially useful for those who
leverage some aspect of crowdsourced (or community-based) testing.

To learn more about the growing trend of crowdsourcing - and how companies of all sizes can use it to
complement their in-house efforts – check out the 8 Essentials of Crowdsourcing whitepaper.

WHITEPAPER: 10 Tips of Web Testing 2

Web Applications: Testing Types
Functional Testing
No surprises here. The most frequent and critical task that QA takes on before launch is to perform
comprehensive functional testing. This can include - but is not limited to - testing of:

 Web Forms: This includes sign-up, sign-in, contact and purchase forms; database queries,
checking for cross-site scripting/AJAX vulnerabilities and other issues related to incorrect inputs.
 Links: Ensuring that all outgoing and internal links
“For a successful technology, reality
are functional and accurate.
 GUI: Despite back-end test automation, front end must take precedence over public
problems with web applications are almost always relations, for nature cannot be fooled.”
discovered by human testers, including issues
related to browser compatibility or CSS flaws. - Richard Feynman
 Language: Does your web app properly support Renowned Scientist
Unicode? Can it accept and handle foreign
characters? This is a growing concern for developers and should be considered as an essential
part of your testing efforts.
 Security: Security testing – including checks for open redirects, cookie tampering, SQL injections
and other vulnerabilities that could harm your users (and reputation) – should also be
performed prior to launch.

Bottom line: Does your web application work when, where and how your users need it to work?

Usability Testing
Beta testing may be suitable for some large companies (mainly those with name recognition) who want
to solicit user feedback, but it is NOT an effective substitute for usability testing, which should include
professional analysis of:

 Graphics: Have you identified and addressed the major GUI bugs in your web app? Are the color
schemes easy to understand (in every country of your user base)? Many of these flaws will not
be reported by beta users, but they should be known about and fixed before launch.
 Feature Set: This can include the intuitiveness of shopping lists, online checkouts and other e-
commerce actions; the accuracy of search results, data entry, sign-up forms and other features.
 Accessibility: Is your application accessible for all users, including the visually impaired? Does
your app involve parental or administrator restrictions? Be aware of your legal obligations.
 Benchmarking: How does your web application stand up to that of your closest competitors?
What about your own previous versions? What areas does it compare favorably or unfavorably?
 Overall Usability: Beta testers - if they report bugs at all - are likely to respond with “this-site-
sucks” or “this-site-is-great” type of analysis. When launching a web application, professional,
non-biased analysis is essential.

Bottom line: Is your web application intuitive and highly usable for your target audience?

WHITEPAPER: 10 Tips of Web Testing 3

Load Testing
How will your web application hold up when it needs to most? Synthetic load tools are helpful in
simulating these types of scenarios, but with the assistance of real testers, you can obtain a complete
picture of how your application performs under stress.

Load testing should, at the very least, involve one of the following methods:

 Live Load Testing: A team of live testers (preferably from where your users reside – whether it
be North America or a global audience) will test your
“The real value of tests is not that
application simultaneously, enabling you to see how
your app performs under real-world, non-simulated they detect bugs in the code, but
load. This process, as its name would indicate, requires that they detect inadequacies in
NO automated tools. Ideal for web apps that contain the methods, concentration and
Flash, streaming video or other types of multimedia skill of those who design and
that are difficult to evaluate with simulated load. produce the code.”

 Simulated Load Testing: Using the simulated load - Tony Hoare

testing tool of your choice (more on this below) you can Software Testing Expert
obtain a realistic snapshot of your web application's
performance under peak synthetic usage.

 Hybrid Load Testing: A combination of the aforementioned methods, this process involves
having live testers perform functional testing on your web application while under automated
load. This lets you see bugs or performance issues that only show themselves under peak loads.

Bottom line: Will your web application perform as expected under maximum pressure? Will it crash?
Will the performance degrade?

Web Testing Checklist: A Summary

Regardless of whether you are performing functional, usability or load testing, it is imperative that each
of these practices achieve maximum testing coverage that matches your user base across:

 Location: If you have a global user base, why would you only test your web application in your
office? It’s therefore critical to extend your testing coverage beyond your borders if needed.
 Language: If your application is available in multiple languages, you must verify that nothing is
lost in translation (like the intuitiveness of your content, error messages and core features).
 Operating Systems: Your app needs to work seamlessly across all the different flavors and
versions of Windows, Mac and Linux.
 Browsers: The days of Internet Explorer hegemony are over. Today’s web applications must be
tested thoroughly against multiple versions of IE, Firefox, Chrome, Safari and others.

And now, the Ten Tips of Web Testing….

WHITEPAPER: 10 Tips of Web Testing 4

The 10 Tips of Web Testing
1. Don’t Be a Slave to Your Metrics
When testing a web application, it’s easy to become overwhelmed by the data that that’s available
to you. It can drag you in all sorts of directions and distract
you from more important matters – if you allow it to. “When your car is about to go off a
Don’t. Said testing expert Michael Bolton of DevelopSense: cliff, it’s a weird time to be thinking
about gas mileage and drag
“Some people enslave numbers. They make numbers work coefficients; better to take the right
too hard, and too often. I’ve seen organizations collect piles
control action—look out the
of data about defect escape ratios and defect detection
percentages. They hire market research firms and calculate window and steer or use the brake
the ratio of happy customers to unhappy customers. But until you’re back on course.”
the aggregated data doesn’t tell you anything specific on
how to make things better for the unhappy customers.” - Michael Bolton
CEO, DevelopSense
In other words, while comprehensive testing will present
you with a ton of raw data, the most important information will still come from your testers,
managers and customers.

2. Know Thy Third-Party Apps

When launching a web application, remember that it won’t operate in a vacuum. There are many
third-party applications that could interfere with its performance, apps like:

 Live chat
 Checkout processes
 Search plug-ins
 RSS Feeds
 Embedded videos or audio players
 Ad servers or embedded ads
 Web analytics packages
 Blogs, forums and message boards
 Social networking modules or toolbars (like Tweet streams or Digg)

Do any of these apply to your web application? To your users, these are part of your app.

3. Understand the Testing Manager’s Dilemma

As any honest test manager will tell you, it is NOT his or her responsibility to make the final decision
about when the product is completed – that is clearly the role of the product owner. Testing
managers have enough to worry about. Between time limits and budget constraints, there’s no
shortage of obstacles to must deal with – so don’t make them the final arbiter of product readiness.

WHITEPAPER: 10 Tips of Web Testing 5

 Besides, to a good testing manager, their job is never complete. In theory, software testing could go
on forever! Despite knowing better, software executives regularly pressure their testing managers
into providing an answer, and end up being told what exactly they want to hear.

Also, it is okay to fall off the agile wagon every once and awhile. With the short sprint cycles and
constant deadlines, it’s amazing that more companies don’t wind up there more often. But just
because you can’t maintain a truly agile schedule 24/7, doesn’t mean that certain aspects of your
development (like testing consistently) should ever be abandoned.

4. Invest In Testers, Then Tools

While bug-tracking systems and automated test tools are an essential part of any test team, there’s
a price to be paid for relying on them too heavily – literally and figuratively. Consider automated
tools, for instance. Automation will never find bugs for
use cases that haven’t been conceived and “I urge you not to use expensive tools,
documented. It simply helps to ensure that old bugs even if they work. Never let your
don’t resurface and that the main test cases are clear. manager buy them. Because
expensive tools become something
Conversely, live testers will find bugs that are new or you MUST use, even if they don’t
unique to your app - which is probably most of them. In
work. A free tool may be freely
short, ignoring real-world testing at the expense of tools
is a recipe for disaster. abandoned. This gives you flexibility.”

- James Bach
Speaking of expense, if you dole out big money for
elaborate test tools (and expect an even bigger ROI), Testing Author and Expert
you will feel compelled to keep using it, regardless of its
effectiveness. In this regard, software companies would be wise to follow the advice James Bach
gave to individual testers (see sidebar).

5. Protect Your Users, Save Yourself

To have a safe, secure web application is obviously a no-brainer – nobody launches with anything
less in mind. What’s not so obvious is the way to achieve this type safety for your web app. To point
you in the right direction, here are a few questions to ask yourself before your next launch:

 Is your web app behaving the way your privacy policy claims it does?
 Can a tester easily get an app to cough up the private data of another user? For example, if
they see “user_id=232” in the URL, what happens if they change it to “user_id=231”? Do
they get to see someone else’s personal data?
 Are you sharing personally identifiable information about your users with third-parties like
Salesforce.com or Google Analytics? What about the company that’s hosting your app?
 What happens if your web app is cached when it shouldn’t be? Does it share the wrong
data with people? What happens if you actually want caching? Is it sending out the right
things to be cached? Are end users actually seeing a benefit?

WHITEPAPER: 10 Tips of Web Testing 6

  Is your site vulnerable to common security exploits like XSS, injection flaws, broken
authentication, flawed session management, unvalidated redirects and forwards?

Since these areas are sometimes overlooked by in-house teams, it can be helpful to leverage testers
who are skilled in testing, but new to your application.

6. Multimedia, Multi-Problems
How much does your web application rely on Flash, HTML 5, Java and other rich-media tools? Even if
you answered “a little bit”, multimedia testing - including load testing - should become a critical
component of your testing from this day forward.

Of course, many in-house teams will find they have neither the time nor the headcount to achieve
testing across all versions of multimedia players. If this is the case, you should strongly consider
leveraging a community of professional testers to quickly and easily find the versions you need.

7. Forfeit the Blame Game

Reproducing defects, filing bug reports and “verifying” fixes are tasks commonly believed to be
bottlenecks in the testing process. They are not.
While they can comprise close to 20% of a test team’s “Testers need to really keep reminding
duties - and while testing teams are often blamed for the developers that it’s not personal.
delays in these processes - it is often actually a On the other hand, it’s important to
bottleneck on the development side (where the state that not every functionality issue
engineers are competing with their own challenges, is a bug. It’s important to have a tester
obstacles and deadlines).
that can tell the difference between an
If test teams can work with development (rather than enhancement and a true bug. I know, it
blame them) to improve the quality of the software sounds so ridiculously trivial but finding
prior to code completion, then it will improve the quality testers who can also understand
speed of the whole system. Buggy software can seem the nuances of business is key.”
like the test team’s fault, but it is important realize
that other factors are contributing the pace of each - Jack Margo
development cycle - so don’t take it out on your SVP of Web Operations, DevShed
testing team.

Making developers better, helping them understand failures and the factors that cause them will
mean fewer bugs to find in the future. Testers are quality gurus and that means teaching those
responsible for anti-quality what they are doing wrong and where they could improve.

8. Be Everywhere

Does your web application work in New Zealand? What about Singapore? If your web app is
localized – that is, if your users are dispersed throughout the globe – then you’ll need to test

WHITEPAPER: 10 Tips of Web Testing 7

 language translation, currency conversion and other location-specific features. Aside from
functionality, it is equally important to run usability testing with users from around the world.

Neither in-house QA teams nor outsourcing firms can efficiently complete such a task. It would be
(and indeed, has been) time-consuming, costly and impractical. And thus, it has been ignored.
However, crowdsourcing enables you to easily leverage a global community of professional testers,
letting you handpick the testers you need to fill out your testing coverage matrix.

9. Plugins: Proceed With Caution

Plugins remain a significant problem for web application developers. In fact, most of the security
issues on the web today are a direct result of Flash and Acrobat bugs. In many cases, people end up
with ancient versions of Acrobat reader that can run random exploits simply because they forgot to
make the needed updates. Does this sound familiar?

Also, while the sheer number of browser plugins has declined in recent years, extensions have
gained in popularity. While less common (and usually only found on more advanced users’
browsers), extensions can dramatically impact how an app will work if they are buggy. As such, they
too will require testing prior to launch.

10. Respect The Dead – Make Way For The New

It’s no secret that IE6 is on its way out, but large organizations can’t seem to kill it off just yet. As
frustrating as it may be, support for older browsers can be critical, especially for B2B applications.

Conversely, the next wave of HTML5 is going to introduce a flurry of competing multimedia
standards. “HTML5 vs. Flash” is going to be a big debate in the years ahead, and companies will have
to make a very important choice: Either they deal with Flash’s security issues and lack of support on
mobile, or they opt for HTML5’s confusing vendor specific standards (H.264 vs. Theora).

Also, CSS3 is slowly making its presence felt. In fact, many web apps already use it for additional
layout effects. If you go down this road, how will non-CSS3 browsers deal with this?

Cloud computing is another major trend to keep in mind when testing your web app. How will your
application behave when different parts of the cloud
are misbehaving? What happens to your lead “The Internet? Is that thing still around?”
generation efforts, for example, if your CRM system
malfunctions? Or if Omniture goes down for an - Homer Simpson
hour? What effect will this have on your users? Typical Web User
Consider these cloud issues when testing.

While some trends will die before their time, others will persist far beyond what’s anticipated. For
web application companies, it is critical to be mindful of these trends and remain on the offensive.

Then again, you don’t have much of a choice.

WHITEPAPER: 10 Tips of Web Testing 8

About uTest
Headquartered near Boston, uTest is the world's largest marketplace for software testing services. The
company provides real-world QA services through its community of 20,000+ professional testers from
over 160 countries around the world. To date, more than 500 companies - from web start-ups to
enterprise software firms - have signed up to get their apps tested by the uTest community.

uTest enables companies to launch higher quality products; get their desktop, web and mobile
applications to market faster; and control the cost of testing. Customers specify their requirements for
tester experience, location, language, OS and browser, and uTest selects the right testers for each
project. And because uTest is on-demand, companies only pay for completed test cycles.

A brief online demo is available at www.utest.com/demo. uTest can be contacted at:

uTest, Inc.
153 Cordaville Road
Southborough, MA 01772
p: 1.800.445.3914
e: info@utest.com
w: www.utest.com

WHITEPAPER: 10 Tips of Web Testing 9