Test Data:
What You Need and
How to Get It
by Linda Hayes
In most test cases, the data is key to
the result. Not just entering or verify-
ing data values, but knowing what
the state of the data is supposed to be
so you can predict expected results.
Getting control of the test data is fun-
damental for any test effort, because
a basic tenet of software testing is
that you must know both the input
conditions of the data and the
required output results to perform a
valid test. If you don't know either of
these, it's not a test; it's an experi-
ment, because you don't know what
will happen. This predictability is
important for manual testing, but for
automated testing it's essential.
For many systems you can't even get
started until you have enough test
data to make it meaningful...and if you
need thousands or millions of data
records, you've got a whole new prob-
lem. In an extreme case, testing an air-
line fare pricing application required
tens of thousands of setup transactions
to create the cities, flights, passengers
and fares needed to exercise all of the
requirements. The actual test itself took
less time than the data setup.
There are four basic strategies for
assembling a test data environment:
production sampling, starting from
scratch, seeding data, or generating it.
Let's look at each strategy and consider
the advantages and disadvantages of
June 2003 http://www.testinginstitute.com
There are four basic strategies
for assembling a test data envi-
ronment: production sampling,
starting from scratch, seeding
data, or generating it.
Approach Advantages Disadvantages
Sample production Represents reality Complex to extract sample
Large random volume Fractional coverage
Already exists Refresh maintenance
Unavailable for new application
Start from scratch Complete control
Easily recreated Requires automation
Targeted Not all files can be created
Lacks reality, randomness
Seed data Expands on production Effort to define, add
Targeted Maintenance
Generate data Large defined volume Lacks reality, randomness
Multiple file types Referential integrity
Automated Locating specific data
each. First, the test platform seldom replicates
production capacity, and so a subset
Sample Production must be extracted. Acquiring this subset
If your application is already in produc- is not as easy as taking every Nth record
tion and you need data for regression or some flat percentage of the data: the
testing, the most common test-data complex interrelationships between
acquisition technique is to take it from files means that the subset must be
production. This approach seems both internally cohesive. For example, the
logical and practical: production repre- selected transactions must reference
sents reality, in that it contains the actu- valid selected master accounts, and the
al situations the software must deal with totals must coincide with balances and
and it offers both depth and breadth histories. Simply identifying these rela-
while ostensibly saving the time tionships and tracing through all of the
required to create new data. files to assure that the subset makes
sense can be a major undertaking in and
There are other drawbacks to note. of itself.
Journal of Software Testing Professionals 1
Further, it is difficult to know how large
of a sample is necessary to achieve cov-
erage of all critical states and combina-
tions, and whether the complete set
even exists in production at any given
point. A snapshot may reflect only half
or even less of all possible data types, as
one financial services firm learned.
Even after exercising production data
for five different customers, their code
coverage tool showed only fractional
results. Obviously missing from pro-
duction are invalid data and transaction
types that exercise exception and error
handling functionality, but less obvious
might be conditions that are time or
event specific. So, even complete pro-
duction data does not represent code
complete coverage. Another drawback
of this approach is that the tests them-
selves and the extracted data must be
constantly modified to work together,
and may even require scrubbing to
remove sensitive data such as social
security numbers or names.
Going back to our basic tenet, we must
know the input conditions for a valid
test: in this case, the data contents. Each
fresh extraction starts everything over.
If a payroll tax test requires an employ-
ee whose year-to-date earnings will
cross over the FICA limit on the next
paycheck, for example, the person per-
forming the test must either find such an
employee in the subset, modify one, or
add one. If the test is automated, it too
must be modified for the new employee
number and related information.
Searching for an employee that meets
all the conditions you are interested in is
like searching for a needle in a
haystack.
Thus, the time savings are illusory
because there is limited repeatability:
all effort to establish the proper test
conditions is lost every time the extract
is refreshed.
And finally, this approach obviously
cannot be employed for new systems
under development, since no production
data is available.
Starting from Scratch
The other extreme is to start from
2 Journal of Software Testing Professionals
scratch, in effect reconstructing the test
data each time. This approach has the
benefit of complete control -- the con-
tent is always known and can be
enhanced or extended over time, pre-
serving prior efforts. Internal cohesion
is assured because the software itself
creates and maintains the interrelation-
ships, and changes to file structures or
record layouts are automatically incor-
porated.
But reconstructing test data is not free
from hazards. The most obvious is that,
without automation, it's highly imprac-
tical for large-scale applications. But
less obvious is the fact that some files
cannot be created through online inter-
action: they are system generated only
through interfaces or processing cycles.
Thus, it may not be possible to start
from a truly clean slate.
A compelling argument also might be
made that data created in a vacuum, so
to speak, lacks the expanse of produc-
tion: unique or unusual situations that
often arise in the real world may not be
contemplated by test designers.
Granted, this technique allows for
steady and constant expansion of the
test data as necessary circumstances are
discovered, but it lacks the randomness
and reality that makes production so
appealing.
Seeding Data
Seeding test data is a combination of
using production files and creating new
data with specific conditions. This
approach provides a dose of reality tem-
pered by a measure of control.
This is the strategy adopted by a major
mutual fund to enable test automation.
Without predictable, repeatable data
there was no practical means of reusing
automated tests across releases.
Although much of the data could be cre-
ated through the online interface,
including funds, customers and
accounts, other data had to be extracted
from production. Testing statements
and tax reports, for example, required
historical transactions that could not be
generated except by multiple execution
http://www.testinginstitute.com
cycles. So, the alternative of acquiring
the data from production and perform-
ing the necessary maintenance on the
tests proved to be less time-consuming.
Once the data was assembled, it was
archived for reuse.
It's still not easy. You must still sur-
mount the cohesion challenge, assuring
that the subset you acquire makes sense,
and you must still have an efficient
means of creating the additional data
needed for test conditions. Furthermore,
you must treat the resulting data as the
valuable asset that it is, instituting pro-
cedures for archiving it safely so that it
can be restored and reused.
Although a popular and sensible con-
cept, reuse brings its own issues. For
time-sensitive applications, which
many if not most are, reusing the same
data over and over is not viable unless
you can roll the data dates forward or
the system date back. For example, an
employee who is 64 one month may
turn 65 the next, resulting in different
tax consequences for pension payouts.
Luckily, in many cases, such test capa-
bilities were left behind by a Y2K proj-
ect.
Furthermore, modifications to file
structures and record layouts demand
data conversions, but this may be seen
as an advantage since--hopefully--the
conversions are tested against the test
bed before they are performed against
production.
Generating Data
Automatically generated test data can
be used to create databases containing
defined cases and conditions as well as
enough information to approximate
real-world conditions for testing capac-
ity and performance. If you need to
assure that your database design can
support millions of customers or bil-
lions of transactions and still deliver
acceptable response times, generation
may be the only practical means of cre-
ating these volumes.
Fortunately, test data generators are
becoming more powerful and available.
June 2003
These products offer powerful features
that can expedite the tiresome job of
populating files and databases with
enough data to support complex test
scenarios. A few examples include
Datatect from Banner Software
(www.datatect.com), Datamacs from
Computer Associates International Inc.
(www.cai.com), D-Generate from
Synthetic Minds (www.synthetic-
minds.com) and TurboData from
Canam Software (www.turbodata.com).
But what do these tools do? How well
do they work? And what do they test?
As you might imagine, test data genera-
tors begin with the description of the
file or database that is to be created. In
most cases, the tools can read the data-
base tables directly to determine the
fields and their type, length, and format.
The user can then add the rules, rela-
tionships, and constraints that govern
the generation of valid data.
Standard "profiles" are also offered,
which can automatically produce bil-
lions of names, addresses, cities, states,
zip codes, Social Security numbers, test
dates, and other common data values
such as random values, ranges, and type
mixes. User customizable data types are
also available in most products, which
can be used for generating unique SIC
(standard industrialization classifica-
tion) business codes, e-mail addresses,
and other data types.
A more critical feature, and more diffi-
cult to implement, is support for par-
ent/child and other relationships in
complex databases. For example, a par-
ent record, such as a customer account
master, must be linked with multiple
child records, such as different accounts
and transactions. This type of function-
ality is essential for relational database
environments where referential integri-
ty is key.
June 2003 http://www.testinginstitute.com
Some users have found it easier to use a
test data generator to create data that is
then read by an automated test tool and
entered into the application. This is an
interesting combination of data generat-
ing and seeding. The synergy between
test data generation and test automation
tools is a natural, and in some cases, the
test data generation capability is being
embedded in test execution products.
The goal of using a test data generation
product, of course, is data--tons of it.
Thousands or even millions, some
claim billions, of records containing
variations of the described data can be
generated and outputted to the database
or file format of choice. Most major
databases--Oracle, Sybase, Informix,
SQL Server, and Microsoft Access are
supported, as well as formats such as
XML or flat files.
But, like most things, it's not always
quite that easy. These days, databases
can contain more than just data, such as
stored procedures or derived foreign
keys that link other tables or databases.
In these cases, it is not feasible to gen-
erate data directly into the tables. Too
often maintaining database integrity is a
project unto itself.
And, of course, in the end volume is its
own challenge. More is not necessarily
better. Too much data will take too long
to generate, will require too much stor-
age, and may create even more issues
than not enough.
Functional testing is a different animal.
If you are testing business rules - such
as whether or not an account whose bal-
ance owed is more than 90 days past
due will permit additional credit pur-
chases to be posted - then you must
know precisely which account number
contains the condition and which trans-
actions will be entered against it. While
Journal of Software Testing Professionals
it may be easy to generate huge vol-
umes of accounts whose balances are
all over the map in terms of their
amount and due dates, it is not as sim-
ple to know exactly which accounts sat-
isfy which business rules.
In this context, the same issues exist as
they do for simply sampling production
data. Even if you are comfortable that
your data sample is representative of
the types of conditions that must be test-
ed, it's another matter altogether to
know which accounts meet which
requirements. Testing complex business
rules may require knowing the exact
state of several variables that are spread
over multiple databases, tables, and/or
files, and finding that precise combina-
tion may be like looking for a needle in
a haystack.
Can you benefit from a test data gener-
ation tool? Probably. And, if you can,
how do you select the best one for your
needs? The most important step is to
carefully research the types of data and
files that your application uses, espe-
cially the way the database is struc-
tured. Not only will this help you eval-
uate the various products, it will add to
your education about how your soft-
ware and its data are designed. And that
information is useful whether you gen-
erate data or not!
Summary
As you can see, acquiring and manag-
ing test data is not easily done no mat-
ter which route you choose, and the
amount of time and effort will be sig-
nificant in any event. It is important to
realize that your test data strategy is
integral to the integrity of your overall
test process, and without a thoughtful
and consistent approach, it will be an
expense instead of an investment.
3