How Can Testing Be Done In An Agile Context?

Jens Bostr m o Felix Hall Pontus Johansson Olof Svensson

Faculty of Engineering Faculty of Engineering Faculty of Engineering Faculty of Engineering Lund University Lund University Lund University Lund University Email: ic07jb7@student.lth.se Email: dt05fh0@student.lth.se Email: ic07pj3@student.lth.se Email: dt06os4@student.lth.se

AbstractThis article aims at making an inventory of best practices when it comes to integrating testing in agile methods. It focuses on some of the problems with testing in traditional development methods, and a few difculties in introducing testing in general, related to prioritization of work. Then it presents a few previous ndings, mainly related to Scrum and XP, of how to solve these problems when they arise. Among the conclusions of this article is the fact that communication is the single most important feature of agile in general and testing in particular.

I. I NTRODUCTION Traditionally, processes similar to those of other engineering branches have been applied to software development. While being well suited for business proposals and contract writing, they can be seen as one of the reasons for the over-budget and over-deadline problems plaguing the software industry. Recently, there has been a large shift towards processes more focused on quality- and customer related values. These processes are called agile, as they can be seen as lightweight and easy handled compared to the heavier and slowermoving waterfall model and similar methods. Scrum is a common development method used in industry today. It is often implemented in conjunction with eXtreme Programming [1]. The focus of this article lies on a combination of these two, and the problems and solutions found are often related to these. Being mainly developer-oriented, there is a need of dening the testing process related to the other activities within the framework of the agile methodology. The testing has to be done continuously, since feedback is needed for each and every functionality that enters the system as a result of one of the agile principles of always having functioning code. Meanwhile, it seems as if both testers and test methods recognized by the industry are traditional in nature, leading to problems when integrated in more agile projects. This article aims towards making an inventory of documented methods related to agile software development and analyzing these, providing a starting point for the introduction of agile testing in an organization. II. AGILE T ESTING VS . T RADITIONAL A. Scrum and eXtreme Programming In the combination of Scrum and XP, Scrum can be seen as the management model, meaning the interface between the team and the company as a whole, while XP dictates the practices within the team for facilitating communication and

collaboration. Scrum has an iterative approach to software development, in which the sprint, which can be one to four weeks long, has the central role. The team has its goals, recorded on a whiteboard or similar, called a task board. A list of tasks is called a backlog. There is one list called the product backlog, which contains all known requirements on the product. There is also a sprint backlog, which contains all tasks that are to be done before the sprint is over. During a sprint, the team works focused on their tasks, while a person, called the Scrum master, is responsible for eliminating obstacles along the way. Obstacles can be problems related to other teams, the company as a whole, line managers, and similar. The team communicates with the customer through a person inside the company, easily available. This person is called the product owner [2]. B. Traditional Testing There are several techniques and methodologies dened for doing the testing in traditional software projects. The waterfall model is the most general and classic development model still used in industry. It is a sequential design process, in which development progress can be represented as a ow through the following phases, not allowing the next phase to start before the previous is fullled: 1) 2) 3) 4) 5) Requirements specication Design Implementation or coding Verication Maintenance

One problem when using this model is when adjustment is needed after testing is done. Adjustment that effect an early stage in the development process will cause problems, since once a phase is nished, it cannot be restarted. This lack of exibility can lead to large delays, since the project is supposed to be close to its end, with customers waiting for the product [3]. The waterfall model is a formal way of describing a development process and it is hard to apply in reality. There are a lot of different models today that are modications of the waterfall model, making them easier to apply in a real project. In general the feedback is needed much earlier in an agile project, earlier than these modied waterfall models can supply it.

C. Test Automation There are lots of benets in implementing automated testing in software development, both in more traditional development models and agile. Manual testing takes large quantities of time, requires a large amount of manpower, and a large part of it is extremely repetitive. Implementing automated testing can also require a lot of time and resources, but once it is in place, it starts saving time and manpower. This is especially true for regression test suites, where a set of tests covering the area in which there has been recent activity is chosen from tests that have already been run before. To perform these tests, once automated, can be done in a few moments or overnight. In an agile context, the automatic testing integrated in the version control system can allow the developer to focus on his task, while a suite of unit- and integration tests can be run automatically on a server. If something goes wrong, the developer can go back to correct the error fast, before enough time passes for him to having switched context mentally. D. So, what is Agile Testing? In short, agile testing should be testing methods which simplify and shorten the feedback loop for developers and stakeholders within an agile organization. For the developer, this means that he can be certain that the code he just wrote is doing what its supposed to, allowing him to let it go mentally. For the stakeholders, it means that they at all times can be certain of the quality of the software, not having to wait a week for a test organization located at some other place having time to perform the verication. To be able to provide this fast feedback the testing has to be well integrated into the process. If the testing is performed by the developers themselves, this has to be taken into consideration when planning and calculating effort to implement a use case. On the other hand, if testing is done by separate testers, how should the parties involved be synced, when less formal documents are provided than in more traditional models? III. A NALYSIS A. Organization One way in which traditional testing differs from agile methods is that there is often no test team and test manager. Theres also no time specically allocated for writing test plans and making extensive retesting [3]. In an agile project everyone could write tests, including; developers, business analysts and even customers. This has proved to provide several benets. First, the testing isnt in the hands of only a few assigned testers, who might leave the team for some reason. Second, the developers test- and code awareness improves as they write test themselves. Third, it is a good idea to mix developers and testers while developing a new product, since they both have different qualities that are needed during a project [4]. Finally, the code tends to be more designed for testability as the authors know that they later have to test it themselves [3]. Let the developers develop less and test more to ease bottlenecks [5].

On the other hand, when a developer performs testing on code he himself has developed it can be hard to remain objective towards it. Test results might be biased towards the better since the developer knows exactly how the product is supposed to perform and how the code is written. By using a independent test team some new views on the piece of code can be obtained, removing the bias and nding errors that the tests written by the developer would never nd [4]. One way of solving this is by having specialized testers review unit tests written by developers before accepting them as valid for their purpose. Testers have more experience with how to outline a test than developers, the two should however work as a team to come up with the best solution. This will increase the overall efciency of the tests. Confronting the development teams with the results of their work can also add to the weight of the results. If everyone can see how they perform they will be more aware of what they produce [4]. These specialized testers would need to have knowledge of how to use the test tools available and how to design scripts and tools. They must also be able to estimate how much testing needs to be done. It is also important to know in what order to perform the tests and how the results can effect each other [6]. Depending on the depth of the test to review different qualities of the tester is required. Since there are many ways and levels of testing, seasoned testers are often specialized in different areas. Examples of these are: Test Tools and Automation Test Equipment Test Environment Architecture Architecture Verication Operations Readiness Testing Depending on what tests are in question different expertise will be needed. The project leader should make sure to recruit testers with the abilities needed [6]. B. Test Techniques and Methods One of the central techniques in XP is test-rst programming. The technique has two principles; rst, write the test case before the code and second, make them executable for regressions testing [7]. Muller and Hagner [7] made a case study to investigate what could be the benets of using test rst compared to not using it. Their results were that test-rst doesnt necessarily make a developer come up with a solution faster. It could pay off slightly in terms of reliability and the study showed that test-rst makes the developed code more understandable. C. Priority of Testing Organizations can sometimes give low priority to testing, which can cause negative effect on quality. Testing takes time and at the end of production there are no more results from testing than that the product works as it is supposed to do. This means that the marketing department in a company will have a hard time to nd gures that shows the value or contribution of testing. Also, if the developers write the tests themselves, they are not writing code during this time. This might lead

to the developers taking shortcuts, trying to get back into what they feel is productive work. One approach to make the organization promote testing is to dene untested work as no work - under all circumstances. At the end of each iteration, the team reviews and considers tasks without full test coverage as incomplete tasks [5]. D. Requirements and Formalities In traditional methods there is often a team mainly working with requirements and related documents. Because of that and because of the structure of the traditional model, there is more time dedicated to writing well documented requirements. There can also be a false sense of non-urgency when a certain amount of time has been allocated for e.g writing a requirement specication, leading to time being unevenly distributed to the phases which would better require them. As there is less time dedicated to writing documents in agile methods, the Quality Assurance team (QA) and the Product Manager (PM) could team up and compile the software requirements specication (SRS) and the test plan (TP) into a single document, since they are rather similar. This avoids so called Test Plan Alchemy, which basically is copying The software should do from the SRS and paste it into the TP and change it to Make sure that the software does. Dont forget to review test plans and make sure they are up to date. By once again merging the QA and PM the bottleneck with reviewing can be reduced [3]. Deshpande et al. thinks that the term use case is old fashioned and suggests that storytelling should be used instead. In theory it is the same thing, but use case makes one think in dry and uninspiring terms. Storytelling on the other hand encourages creativity. Furthermore they propose to integrate feature testing and development of features. The testing of a feature should be taken into account when timeestimating the development of a feature [3]. It is also important to limit how much time is going to be spent on testing and what to test during this time. One study claims that risk analysis is a good way to estimate what to test and how extensive the tests should be [4]. Another solution is using similar methods of setting priorities of test cases as those dened in Scrum itself. Guided Exploratory Testing (GET), described by Geras [8], is a method using agile principles to guide the testers to come up with tests that is of most value. Users are asked to prioritize what to be tested based on their knowledge of the system. Geras suggests three different levels of testing: 1) Transaction Testing 2) Business Function Testing 3) Final Acceptance Testing The two rst levels are supposed to be carried out by testing sub-teams, while the third and last level should be done by so called super users, meaning experts on using the specic system, in Scrum relating to the product owner. At the beginning of every test level, each testing sub-team creates a testing backlog containing test targets- which is a list of items to be tested. Geras claims that using test targets instead of

TABLE I R ELATION BETWEEN TEST LEVELS AND TEST BACKLOG ITEMS IN GET. Test Level Transaction Testing Business Function Testing Acceptance Testing Test Backlog Item Individual Transaction String of Transactions Scenarios

traditional test cases improves communication within the team because feedback can be given faster. Using a test backlog also provides a way to quickly come up with an approximate size of the test effort, which helps validating the project schedule. Table 1 shows the relation between the test levels and different backlog items. For test level one and two, the testers tasks in each test sub-team are: Related to each backlog item, run the transaction or string until its operation has been investigated and evaluated sufciently. Decide with professional judgment if it passed or failed. If it failed, document it in some way. Change the test status column on the backlog to either passed or failed. As the testers in the acceptance test are users and not very familiar with test techniques, scripts are used to carry out the tests. These scripts consists of approximately 20 questions, where a question could be; Does the screen layout correspond to the layout of the previous version?. To track the test effort burn-down charts are used. The idea is to have a burn-down chart for each and every test backlog, enabling each test subteam to view their test progress. Having an ideal curve added to the burn-down chart as well, gives a sense of what is required. Burn-down charts also enable the possibility, in a management point of view, to get an indication of how the progress is going. It prevents questions in terms of how long time the testing will take [8], which can be difcult to answer and might lead to communication problems that has a negative effect on the process [9]. E. Defect Management Defect managment in traditional projects is often a large mechanism in which many different roles are involved. This leads to large delays from the nding of an error until the time it is xed. Since agile methods strive towards being lightweight and exible, there is a need of dening other ways of providing the customer and other stakeholders with fast feedback. When it comes to defect management, Deshpande et al. suggests that a team-centered approach should be used. By adapting Scrum and XP principles the team is located in the same room and has daily meetings which make it easier to know who might be the best person to correct a defect and also delegate the task to this person. Defects should be xed as soon and efcient as possible. If the team members assign bug-xing responsibilities themselves, the average time to x

a bug would be short since the overhead of planning and prioritizing defects to be xed is avoided. This also makes new development faster as the code is clean and stable, leading to more satised customers [3]. F. Flexibility Using agile methods, a team can be more exible and less forced into using a certain method or technique. If a method used during one sprint proves to be unsatisfactory, another method can be used during the next sprint, letting the team adjust and adopt the most suited methods [4]. IV. C ONCLUSIONS Agile methods have reached maturity and are widely adapted in industry, mainly thanks to its exibility and ability to adapt to changing conditions. One of the most prominent properties of agile is the way that it promotes communication. When such methods are promoted it is the communication between developers and between the developers and the customers that is mentioned rst. Indeed, these are important properties, but perhaps one of the reasons for the success of agile it the least mentioned - the communication between the developers and the testers! If testers are integrated in the developer team, or if the developers themselves write test cases, they can be in direct contact with a product owner or even a customer. In traditional models, testers have to, in parallel with developers, make assumptions based on a requirement specication which someone tried writing from what a customer said in a contract half a year ago. The roles of the staff within an agile organization or team is more dynamic comparing to roles in traditional organizations. Because there is less time spent on writing documentation there is a greater need for communication between co-workers. With principles from both Scrum and XP, the staff is allowed to work more integrated with the development team, which greatly improves their ability to communicate directly. A common way would be to integrate developers and testers, making them keep better track on both development and testing. The effectiveness of defect management could then also increase by making the delegation of defect correction faster and at the same time provide faster feedback. V. D ISCUSSION Worth to mention is that some of the methods discussed in this paper can be implemented using other techniques than Scrum and Xp. Some do not even have to be used in agile processes. It is hard to tell what is a fact and what is a subjective opinion when i comes to agile processes. Most development teams have their own take on how to practice agile software engineering and none of them need to be wrong. This makes it hard to nd actual facts to base a scientic report on.

R EFERENCES
[1] C. Vriens, Certifying for CMM Level 2 and ISO9001 with XP@Scrum. Agile Development Conference, 2003, p. 120-124, IEEE. [2] K. Schwaber and J. Sutherland The Scrum Papers: Nuts, Bolts, and Origins of an Agile Process PatientKeeper, Inc. 2008 [3] S. A. Deshpande et al, Improving Software Quality with Agile Testing. International Journal of Computer Applications, 2010, p. 68-73, Foundation of Computer Science. [4] R. van Roosmalen, Ten Tips for Agile Testing. 2008, http://softwaredevelopmentisfun.blogspot.com/2008/04/ten-tips-foragile-testing.html, 22/02/2011. [5] D. Talby et al, Agile Software Testing in a Large-Scale Project. IEEE Software, 2006, p. 30-37, IEEE. [6] E.J. Weyuker et al, Clearing a Career Path for Software Testers. IEEE Software, 2000, p. 76-82, IEEE. [7] M.M Muller and O. Hagner, Experiment About Test-First Programming . Software, IEEE Proceedings, 2002, Pages: 131-136, IEEE. [8] A. Geras, Leading Manual Test Efforts with Agile Methods. Agile 2008 Conference, 2008, p. 245-251, IEEE. [9] M. Puleio, How Not to do Agile Testing. Agile Conference 2006, 2006, p. 305-314, IEEE.