Analysis and evaluation of negative impacts of Agile methods on software testing (A case study of an IT organization in Pakistan)

Abstract
Several software development organizations are embracing Agile methods. Their intent is to adapt this model to rapidly deliver the software to customers through iterative development cycles. This project will focus analyzing the negative impacts of Agile methodology on software testing processes in an SDLC. This analysis will conclude how effective the agile manifesto has been in improving software testing processes to improve the quality of product.

Introduction:
Software Testing:
Software Testing is the process of executing a program or system with the intent of finding errors. Or, it involves any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results [1]. The main advantage of software testing is that the software which is being developed can be executed in its real environment. In well-run projects, the mission of the test team is not merely to perform testing, but to help minimize the risk of product failure. Testers look for manifest problems in the product, potential problems, and the absence of problems. They explore, assess, track, and report product quality, so that others in the project can make informed decisions about product development. It's important to recognize that testers are not out to "break the code." We are not out to embarrass or complain, just to inform. We are human meters of product quality. [2]

Conventional Method (Waterfall)

Traditional software model approaches expect that we can foresee how the project will unfold with reasonable precision, with so much competition in the software market and new innovations popping up each passing day, anticipating the future and planning accordingly cannot be 100% accurate. Requirements may tend to change each passing

moment. Conventional methods already in place were not able to cope up continuous agility, and for the same reason agile manifesto were introduced in 2001 [3]

Figure 1: Waterfall Model [4]

Agile Method (Scrum):

Agile software development is an assembly of software development methods which are based on incremental and iterative development. Solutions and requirements are evolve through collaboration between self-organizing, cross-functional teams. It promotes adaptive planning, evolutionary development and delivery, a time-boxed iterative approach, and encourages rapid and flexible response to change. It is a conceptual framework that promotes foreseen interactions throughout the development cycle [5].

Figure 2: Agile Workflow

Agile Manifesto
The Agile Manifesto reads, in its entirety, as follows: We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more. [6]

Research Methodology:
An online survey form was developed and circulated to around 10 software test managers and test leads in a software development organization which embraced Agile methods few years back. Targeted audience had experience working in software projects based on Conventional Model (Waterfall) as well as Agile model. The questions in the survey form were carefully discussed and finalized keeping in view the important testing areas which directly influence the product and project quality. Testing professionals answered those

questions using best of their knowledge and hands-on experience with testing activities in both waterfall and agile modes.

Software Testing Parameters

The following parameters in software testing processes were analyzed based on the survey conducted and the interviews done with the testing managers/ leads in software testing field. 1. Incomplete Test Planning: Test Planning is very critical phase for a project. Test management would like to have complete or significant part of project and software scope defined before it can commence work on the following testing activities: Test Strategy Test Plans Test Environments

With Agile in practice, Requirements and Scope continue to change, adding to the original testing scope planned for the project. During the project, added scope affects testing as much as the dropped scope. For example during the course of a project, the testing effort would have already spent on certain OS, Database or office version. So dropping the support of one of those tested versions late in the project doesnt help testing. Survey results: 2. Ineffective Test Case Designing With inadequate Requirement Analysis and Design, testing team doesnt get necessary details in consolidated form (like a doc or share-point) from where it can get the parameters for designing the test cases. Result: Quick iteration cycles dont allow proper time for test case designing. An early delivery of working software to testing means, there is no benchmark to compare the results with rather the software itself. Poorly designed test cases which dont cover all test scenarios Insufficient details in test cases which yield different testing results when executed by different testers Important user scenarios and requirement is not covered in the test cases, resulting in major defects by users after software is delivered.

Survey results:

3. Correction rather than Prevention It is commonly believed that the earlier a defect is found the cheaper it is to fix it. The following table shows the cost of fixing the defect depending on the stage it was found. For example, if a problem introduced during the Requirements is found during Construction, it would cost 5~10 times more.
Cost to fix a defect Requirements Phase Where Architecture Introduced Construction Phase Where Detected Requirements Architecture Construction System test Post-release 1 3 1 510 10 1 10 15 10 10100 25100 1025

Since Agile prefers to have a working software early in the SDLC, most of the requirement defects are found late by testing during alpha testing (post implementation) phase which otherwise could have been prevented during comprehensive requirement review phase. 4. Testing Re-Work Agile methods promote iterative development through frequent build cycles which results in testing re-work and make it less efficient: Testing cycles, specially integration with other products, cant be properly tested in shorter cycles. Smoke Testing captures most of the testing effort since it is repeated for each build. As a result less time is available for regression, functional and integration testing. Defect failures increase due to quick builds. Programmers get less time for comprehensively resolve the defects. Results is either failure of original defects or introduction of new defects around resolved defects. With constant change in the project and product requirement, testing teams has to add new scope close to project end. A smaller enhancement for development could mean a very comprehensive regression and functional testing effort. For example support to new OS, Database or newer version of third party software. Breaking software into smaller chunk of code and merging into mainstream may look like a simple development activity. However for testing this has resulted duplication of testing effort both at branch and main code due to regression problems.

Survey results:

5. Less Testing Coverage Due to frequent and quicker builds in Agile, some of testing types are given more coverage by testing team at the expense of others. Result is that not all the test plans planned for the project are executed. Secondly some of the important test plans which are deemed critical from user perspective and planned in the project, are given less coverage. Good testing coverage during Agile: Smoke Regression Less testing coverage during Agile: Functional Integration Performance Load Usability

Survey results:

6. Poor Software Quality upon release Software testing department analyzes the quality of the released software in the light of following parameters: Effective testing coverage according to plan Testing results and resolution of important defects Behavior of the software according to the expectations

However Agile based projects allowed less time for the complete test coverage. This resulted into the products which had some workarounds for the major functionalities or were followed by quick patches or updates. Survey results: 7. Ineffective Test Legacy Data For the software testing team this is important to create test documentation and re-use it in the subsequent projects. Those projects could be the main projects where the subprojects got merged OR the upgraded release of the existing project. In either case, testing team working on the new project heavily depends on the test plans, test reports and the open defects from previous projects. With Agile in practices, testing team deals mostly with the working software compromising on the creation of legacy test documentation system.

While the team on the project may get necessary design and testing details through informal communication with the project team (like through meetings, chat, daily calls, emails etc.) the subsequent testing teams would not get the same details. Lack of such legacy test information impacts the efficiency of the testing team working on the subsequent projects. Survey results: 8. Lack of Training Opportunities In Agile (specially in scrum model), the project is broken down into monthly sprints. Testing work increases due to frequent daily or bi-weekly builds. Also most of the information required for the proper test design and execution is communicated informally through email/chat but not through formal documentation. In case of a resource turn-over and allocating new hire on the project, it becomes difficult to get the new person on board since all the information is available in the scattered form. Testing teams found it hard to recruit and allocate a new hire during the course of an agile project. On the other hand, this process was much streamlined and smooth in case of conventional model. Survey results: 9. Increased Requirements for Testing Resources Agile adds to the testing overhead through: Re-execution of failed test plans Re-testing of the failed defects. Frequent builds Test environment preparation for new requirements Loss of time spent on items dropped as result of updated requirements. Repeated failures of components after integration Need of more smoke/regression testing Daily Meetings, Daily/Weekly reports. Frequent meetings with programmers , product management in the absence of formal and consolidated documentation system for Requirement and Design. Incomplete test plans.

Survey results:

Conclusion:
1. Quality is about doing it right the first time, but Agile Manifesto promotes the concept of making a program quicker even in the presence of partial requirements. This results in the bad software and increases the project timeline. 2. Software development Software is not code only. It creates experience.

3. Development teams are not merely coders. They are experience creators. They should be design and domain experts. 4. Process is bankrupt without design. You get what you design, so you better get the design right. 5. Software is a creative endeavor, not an industrial process like building automobiles. The methodology is structured to support the creative talent. 6. Any success in Agile is based on short-lived increase in productivity which is more evident on smaller projects which deal with components of the larger applications. However Agiles contribution to sustaining the same level of quality and productivity in large scale projects is not evidenced yet.

Bibliography
[1] "IEEE Defination," [Online]. [2] "http://bazman.tripod.com/what_testing.html#services," [Online]. [3] "http://en.wikipedia.org/wiki/Agile_software_development#cite_noteAgile_Manifesto-0," [Online]. [4] http://www.expertprogrammanagement.com/wp-content/uploads/2010/12/waterfallmodel-graphic.jpg. [Art]. [5] http://en.wikipedia.org/wiki/Agile_software_development. [Online]. [6] "http://agilemanifesto.org/," [Online]. [7] Mike Gualtieris Agile Software Is A Cop-Out; Heres Whats Next http://blogs.forrester.com/mike_gualtieri/11-10-12agile_software_is_a_cop_out_heres_whats_next [8] Hawthorne effect http://en.wikipedia.org/wiki/Hawthorne_effect