Testing Co-ordination

- new activity
- currently moving in 3 directions
- directions have commonalities ....
- ... but also each has its individual strengths

Concerns:

- provide user guidance and infrastructure

- keep an eye on the clock (Rome) (short term)
- work towards a coherent flexible system (longer term)

Testing Co-ordination
P. Sherwood 1
Software week 10 Dec. 2004
 How to find out more
Wiki (web pages):
https://uimon.cern.ch/twiki/bin/viewAtlasOfflineTestingInfrastructure

- condensed overview
- links to framework documentation
- testing co-ordination interactive discussions
- available via Atlas computing web pages

Meetings
- mostly attended by testing developers
- agendas and minutes available

Testing Co-ordination
P. Sherwood 2
Software week 10 Dec. 2004
 Testing Frameworks

Split the task of testing code into two part:

specific to code ‘test’

not specific to the code (reusable) ‘testing framework’

The division is not unique.

Testing Co-ordination
P. Sherwood 3
Software week 10 Dec. 2004
 Existing Frameworks
ATNight (ATN)
User supplies short tests - run as part of the nightly build

Kit Validation (KV)

Shell based test framework initially developed for testing distribution kits.

RunTimeTester (RTT)
Python based testing framework. Currently running ~200 jobs/night at
UCL.

Testing Co-ordination
P. Sherwood 4
Software week 10 Dec. 2004
Automatic vs. Local running
Automatic running
Each framework can be run automatically:

ATN - as part of the nightly build

RTT - is run nightly as a cron job
KV - can be run as a cron job

Local Running
Running framework under user control:

KV - intrinsic
ATN - included in design. Not much experience
due to low demand.
RTT - supported

Testing Co-ordination
P. Sherwood 5
Software week 10 Dec. 2004
 User’s viewpoint
Each test frameworks run jobs, and process results.
Which to use?

depends on:
- Resources required to run tests
- Results reporting mechanism
- Test libraries
- Local or automatic (nightly) testing
- Language (shell or python)
- Facilities (e.g. regression test support)
- Ease of adding tests
puzzling for the user:
- boundaries are not so sharp
- Some diffs are due to resources, not framework
- Each framework has its own native usage
Testing Co-ordination
P. Sherwood 6
Software week 10 Dec. 2004
 How to move forward (1)
Target date 31/01/05
0. Continue current approaches
changes to be made in small increments
1. Make standard tests callable from anywhere.
e.g. tests to be run under ATN could use RTT libraries.
2. Standardise language. KV → python.
Encourages code reuse
3. Make test specification a developer responsibility.
move test specification to packages
- developers in control
- specification remains framework dependent
- use CMT technology
4. Improve user interaction:
- KV: move ATHENA transform parameters to config file
- RTT: allow directory search to pick up users tests
- RTT: allow users to specify library tests and parameters
through configuration files

Testing Co-ordination
P. Sherwood 7
Software week 10 Dec. 2004
 How to move forward (2)
4. Identify information needed by the frameworks
In config files, but also hard coded, or runtime environment
5. Create a common configuration standard
Commonalities passed in the same manner
Differences passed as framework specific (fat interface)
Test configuration becomes finer grain. Meaningful queries
become possible
6. Thin the interface
push parameter interpretation into frameworks
7. Move to a single Tester?
both commonalities and specifics will have become clear
single framework would have at least all current functionality
diminish maintenance overhead
8. Handling test metadata
what tests where run when, what conditions, what results
how to display
needs thinking about
Testing Co-ordination
P. Sherwood 8
Software week 10 Dec. 2004