NZTester

the inaugural issue!

Interview with Angela Nash of Qual IT KWST Workshop #2 STANZ 2012
Possum Testing Testing With Personas Kiwi Test Teams Great Defects We Have Known & Loved

The Quarterly Magazine for the New Zealand Software Testing Community and Supporters
ISSUE 1 OCT-DEC 2012 FREE

NZTester Magazine
Editor: Geoff Horne ed@nztester.co.nz geoffh@isqa.com ph. 021 634 900 P O Box 48-018 Blockhouse Bay Auckland 0600 New Zealand www.nztester.co.nz (coming soon!)

Disclaimer:
Articles and advertisements contained in NZTester are published in good faith and although provided by people who are experts in their fields, NZTester make no guarantees or representations of any kind concerning the accuracy or suitability of the information contained within or the suitability of products and services advertised for any and all specific applications and uses. All such information is provided as is and with specific disclaimer of any warranties of merchantability, fitness for purpose, title and/or non-infringement. The opinions and writings of all authors and contributors to NZTester are merely an expression of the authors own thoughts, knowledge or information that they have gathered for publication. NZTester does not endorse such authors, necessarily agree with opinions and views expressed nor represents that writings are accurate or suitable for any purpose whatsoever. As a reader of this magazine you disclaim and hold NZTester, its employees and agents and Geoff Horne, its owner, editor and publisher, harmless of all content contained in this magazine as to its warranty of merchantability, fitness for purpose, title and/or non-infringement. No part of this magazine may be reproduced in whole or in part without the express written permission of the publisher . Copyright 2012 - NZ Tester Magazine, all rights reserved.

IN THIS ISSUE.

A Journal For New Zealand Test Professionals

Having seen a myriad of magazines and journals around testing spring up from various countries around the world, I decided it was high time that New Zealand has one as well. However rather than the usual technical tips approach I thought why not have a magazine that balances this with a focus on testers and their experiences. The idea has been kicking around in my mind for some time now and like many of my crazy ideas, I wondered whether it would ever see light of day but lo and behold, here it is! In this inaugural issue we have an interview with Qual IT Solutions Auckland General Manager, Angela Nash . I first met Angela at the StarEast Conference in 2007 where for the first time, I realised that I wasnt the only kiwi to traipse three-quarters of the way around the globe to attend a testing conference. However StarEast is held in Orlando, Florida so there are more than a few additional drawcards! We also have reviews of two New Zealand conferences; Kiwi Workshop on Software Testing #2 (KWST2) in Wellington and STANZ 2012 and Auckland and Christchurch.. We also have what I hope will be regular spots where the NZ testing community is invited to share experiences, send in test team photos, submit articles of interest (see Great Bugs pg23) and generally able to contribute in any way that is deemed of interest. So if any budding writers, pundits, observationists etc. out there would like to see name in print, heres your opportunity. Email me at ed@nztester.co.nz if you have something to say. I cant promise that every submission will be accepted but hey, if its good stuff well do our best to squeeze it in somehow. Finally, thanks to all of our contributors for this issue. Without your input, it would have been full my ranting's only and Im not sure that would be in everyones best interests. Keep well.

3 6

Interview With Angela Nash, Auckland General Manager, Qual IT Solutions Ltd

The Second Kiwi Workshop for Software Testing - David Greenlees, Australia (honorary Kiwi).

12 15 17 20

STANZ 2012 - Kim Partridge, Marketing Manager, Software Education Ltd

John Lockhart outlines the lean, mean & hungry approach to testing

Championing the End-User Using Personas - Michael Talks, New Zealand

Possum Testing: Defining Fearful Testing - Brian Osman, New Zealand

REGULAR FEATURES. 12 23 26 Kiwi Test Teams Great Bugs We Have Known and Loved And Now Its Your Turn.. Feedback Sign Up to Receive NZTester Advertising

Our inaugural NZTester interviewee is:

Angela Nash
General Manager, Qual IT Solutions, Auckland

Our first NZTester interview is with Angela Nash, Auckland General Manager for Qual IT Solutions Ltd She has previously been with Telecom, Datacom and Vero Insurance Ed. NZTester: Can you please describe Qual IT? Qual IT was founded in 2004 and now has offices in Auckland, Hamilton, Wellington and Christchurch that deliver test and assurance related services throughout New Zealand and offshore. We maintain a large team of permanent staff supplemented by a core group of long-term contractors which means we manage IP retention and systems knowledge for our clients. We offer the usual core testing services and assurance as well as non-functional testing, process reviews and specialist testing such as infrastructure. We also partner with some specialist vendors in areas that really require that extra level of capability whilst maintaining our independence from any one tool vendor or integration partner. We really focus on our ability to deliver and our pragmatism around the balance between the theory of doing what we should do versus the reality of doing the best with what you can do. NZTester: What services does Qual IT offer? The whole gambit from start to finish. If we dont have the absolute expertise in a particular area e.g. Security/Penetration testing we work with world class partners or vendors to secure the best option for the client. Given the broad spectrum of our client base we also do some ad hoc/off spec work that would not necessarily be considered true testing activities but we love being able to provide that little bit extra and our team members like the chance to think outside of strict testing every now and then as well.

NZTester: What do you believe makes Qual IT different? The level of pragmatism that runs from the directors right through to the team on the floor and the genuine approach we have with our clients and partners. We really do work to the long term with a view to delivering the best needs of the client which means being flexible, managing the risk and having sensible conversations with people. Ability to deliver is our core driver supplemented by good test practice and great people.

NZTester: What do you think makes a Test Manager or Analyst come to work for Qual IT? We offer a genuine career path in testing with options from specialist through to management and we offer a huge variety of work and lots of opportunity. We have multiple service offerings so we can flex to suit the individual and we work across four major NZ cities so there is always the option of moving around if you want to. Also, it is a company owned and run by people who have been testers so they genuinely know their stuff.

NZTester: Where do you believe the challenges for NZ testing companies lay? There are many! I think that there is still a lot of education that is needed on what testers actually do. The difference between quality assurance and testing seems obvious to us but for many it is still lumped together as the same thing. Going forward I believe the challenges also lay in understanding the evolving needs of our customers and partners and ensuring that our approach and methodologies suit that need.

This is an industry where we are constantly moving forward and breaking new ground but the danger with that is we also flood ourselves with new buzz words and approaches that can confuse people. At the end of the day it doesnt matter what you call your approach or methodology, what tools you love or hate or even what terminology you choose to use, ability to deliver is king and so being collaborative in environment that is tension-filled is the biggest challenge.

Yes absolutely. Ten years ago we were doing things so differently and just having testers involved in things was a huge step. Now our baseline of what is considered the bare minimum has moved dramatically which is great and shows the tangible value that testing is having. You know being a career tester is a real thing now and that in itself shows the maturing of our industry and along with that comes an uplift in standards and experience.

NZTester: Where do you believe NZs approach to testing is going well? That is a difficult question. I think that there is clearly some areas where an approach may be considered good because it suits a need, delivers to that need and fits within budgets, quality levels and other business drivers. Other areas are not so good because they are ill suited to the task at hand. No one industry or area really stands out to me, there are pockets of excellence everywhere. It is so dependant on where, what, who and when.

NZTester: Where do you believe the next initiatives in testing lay? Whats coming next? In NZ? Internationally? I think we are on the brink of some fundamental changes to how we do things. There are step changes happening all over the place with tools to support testing and new approaches and everyday I speak to clients who have new ideas on what they want from us. Off the back of the recent financial issues so many companies have had, there is a real driver to save money and, as always, necessity is the mother of invention and people are looking at things in different ways. I think it is great to be part of what feels like an energetic time.

NZTester: Where do you believe NZs approach could improve? In a very general sense I think testers tend to be more combative than they need to be. We operate in a small industry, comparatively speaking, and I think the value of collaborating with each other a bit more and being accepting of each others views and approaches would go a long way.

NZTester: Do you have a testing horror story to share? Horror stories. Yes. Many and numerous but absolutely none I want to share!

Many thanks Angela, we appreciate your time and insights. Having been around for the ten year period NZTester: Do you believe that overall the standard Angela refers to (and if Im honest, its plenty more than of testing in NZ is improving? that!) it is great to see how far testing has come both here in NZ and internationally - Ed.

Gilbs Law: Anything can be made measurable in a way which is superior to not measuring at all - Tom Gilb
5

KWST2: What a Ride!

by David Greenlees

Kia ora! Well my first official self-funded testing event; Kiwi Workshop on Software Testing 2! So, why does an Aussie head to a Kiwi Workshop on Software Testing? I had maybe hoped that wouldnt be asked however that is the great thing about the testing community: everyone is welcome (at least to begin with). I was the token Aussie as a colleague unfortunately could not attend at the last minute. There are some great people putting this together. It all started 12 months ago via KWST1 and has continues to grow in strength with KWST3 already in planning. Maybe a challenge for the coordinators is to run it every six months rather than twelve. that this is a good thing as I think that we can all learn a lot from these situations - certain truths tend to bubble to the surface. Please dont take this the wrong way though. The aim is not to bring attendees down rather to learn. How can we ever get to improve at something without asking questions? I dont think we can. Even if we practice in isolation, we need to be questioning and testing our own abilities if were to improve.

As mentioned, the topic of Ethics in Software Testing was a difficult one, especially for those who also had The format is based on LAWST. Readers might like to come to terms with the format. Many attendees to look up http://lawst.com/?page_id=15 to gain mentioned that it was great to see and hear that they an understanding if not familiar with it. werent the only ones dealing with ethical dilemmas in the workplace. I think I can safely say that we all The topic this time round was Ethics in Software shared many experiences or at least have done so in Testing; a VERY challenging one, driven by very the past. Examples ranged from being asked to personal beliefs and moral fibre. There was a very perform tasks that we consider inappropriate eg. diverse group of attendees: James Bach (Content Owner), Brian Osman (Facilitator), Richard Robinson look, just it sign off OK to cultural ethics and how they play a part in our common offshoring world. I (Facilitator), Aaron Hodder, Andrew Black, Andrew Robins, Chris Stapleton, Donna Chin, Farid Vaswani, personally learned a lot from these experience reports. Ive come back armed with various ways to Geoff Horne, Jeff Bedwell, John Lockhart, Katrina handle different situations, all of which I believe will Edgar, Katrina McNicholl, Liz Kitching, Mike Talks, be helpful during my career. Mike Ward, Oliver Erlewein, Sheryl Toenders and Sophia Hobman. I also had some great conversations with peers over drinks and dinner and on the first evening I got to For many attendees, it was a case of first-time with the Peer Conference format, myself included. Having know Andrew Robins better. Andrew is a test read up on LAWST and KWST1, I had as much of an manager from a mission critical environment who has worked extremely hard over the past ten years to understanding as I could although Im not sure that build his credibility which now allows him to forge any of us had quite the same understanding. There ahead in the context-driven world! I love the way he were some who were surprised at the approach relates his many years of Aikido training to his however handled it reasonably well. For those who outlook on life and testing. This is also a passion of dont know, these can be quite daunting. Getting up in front of people and providing a topical experience mine and relates very closely to a book project Im working on. I plan on speaking with Andrew a report is scary enough, let alone the open season lot more. that follows. Questions are asked, advice is given and critique is provided, whether one likes it or not. This was followed by more great conversation with Depending on the flow of the discussion, proceedings John (Lockhart) and Geoff (Horne), along with a can become somewhat heated. I personally think lovely piece of steak and a pale ale brewed in NZ!

The second evening was a highly educational experience over dinner with James (Bach), Oliver (Erelwein) and Geoff (and the All Blacks playing Ireland Ed.). During day one, the issue of counting test cases arose which ended up being one of those heated discussions I referred to earlier. There was no real resolution prior to heading into a tea break where discussion continued. It was decided that, over dinner on the second evening it would be covered again. It was great to listen to James, Oliver and Geoff talk about the subject - I learned a lot from it. I added my two cents worth here and there however being the introvert that I am, I mainly listened and reflected.

So now Im organising OZWST (Australian Workshop on Software Testing), unless of course that acronym is taken, and Ill be inviting my Kiwi colleagues along. If you ever have an opportunity to attend a Peer Conference, I do urge going. Last but not least, a very special thank you to Brian (Osman) for the personal invite that I am so grateful for. We need to get together at some stage and have a chat about tisting versus chicking! E haere ra, kia waimarie. David Greenlees is a Test Lead at the Bendigo and Adelaide Bank, Adelaide, South Australia. He can be contacted on xtremedmg@gmail.com.

Left to right: Katrina McNicholl, David Greenlees, Aaron Hodder, Jeff Bedwell, Katrina Edgar, Brian Osman (facilitator), Geoff Horne (obscured), Oliver Erlewein, Richard Robinson (facilitator), John Lockhart, James Bach (content owner), Mike Ward, Liz Kitching, Chris Stapleton, Andrew Robins, Mike Talks, Sheryl Toenders, Donna Chin, Andrew Black, Farid Vaswani. Absent: Sophia Hobman.

KSWT2 was held in June 2012 following the success of KWST1 in 2011. As for KWST1, KWST2 was chaired by James Bach of Satisfice, USA and hosted in Wellington by Software Education Ltd. Attendance is by invite only and is restricted to 20 places otherwise the proceedings could become difficult for the facilitators to appropriately manage. Should you wish to be considered for KWST3 in 2013, please email Brian Osman at brian.osman@gmail.com.

KWST2 In Pictures

The troops await while Richard tends to the techie bits.

James in full swing

Andrew making his point..which seemly includes guiding someone toward the exit door! Geoff & James after a healthy debate over a good steak

Most of the people we come into contact with aren't very good at accepting the way things are. That's what makes them effective. When you deal with some of those irritating people at work, the first thing you should try to figure out is are they working for good? Annoyance can simply be annoyance, or it can be a positive influence on everyone. That pebble in the shoe that reminds you of something that needs to be attended to. Cherish the constructive irritant in your company. They'll make everyone think, and that can't hurt.

Courtesy of The Media Guy & @gapingvoid

Geoff Horne, 021 634 900, geoffh@isqa.com

O O O O O O O O O Testing directorship, assurance and advisory Test estimation & forecasting tools & services Testing tool and service provider evaluation & selection Test management systems (HPQC, Spiratest, Zephyr etc.) establishment and administration Programme/project quality assurance Programme/project risk management Quality Review Board establishment and management Conference & in-house presentations and tutorials NZTester Magazine editor

Kiwi Test Teams

If you would like to see your team splashed across the pages of NZTester in glorious colour, then send your snaps in to me at ed@nztester.co.nz. Photos need to be taken with a digital camera of 5 megapixels or more and team member names should be provided left to right. Please ensure that you have your employers OK.
Left to right Hus Chimma, Rod Finlayson, Alan Smith. Jill Hayman, Rochelle Bradbury, Sameer Youssef, Shentelle Levi, Tina Tumupu, Amika Prasad, Deepa Kirtikar, Ratnesh Gautam, Julie Graddon, Katya Klimkina, Geoff Horne, Nimi Rose. Absent: Karen Hunter, Tina Grey, Zankar Patel, Shaun Falconer, Hayley Malhotra, Jeannie Pillay.

So given this is the first issue of NZTester, Ill start the ball rolling with the test team from Watercare in Auckland. We recently wrapped up a successful programme of work and have since gone our separate ways however it was lots of fun (with lots of challenges) while it lasted - Ed.

10

In March 1999, a man living in a rural area received a bill for his as yet unused gas line stating that he owed $0.00. He ignored it and threw it away. In April he received another bill and threw that one away too. The following month the gas company sent him a very nasty note stating they were going to cancel his gas line if he didn't send them $0.00 by return mail. He called them, talked to them, they said it was a computer error and they would take care of it. The following month he decided that it was about time that he tried out the troublesome gas line figuring that if there was usage on the account it would put an end to this ridiculous predicament. However, when he went to use the gas, it had been cut off. He called the gas company who apologised for the computer error once again and said that they would take care of it. The next day he got a bill for $0.00 stating that payment was now overdue. Assuming that having spoken to them the previous day the latest bill was yet another mistake and he ignored it, trusting that the company would be as good as their word and sort the problem out. The next month he got a bill for $0.00 stating that he had 10 days to pay his account or the company would have to take steps to recover the debt. Finally, giving in, he thought he would beat the company at their own game and mailed them a cheque for $0.00. The computer duly processed his account and returned a statement to the effect that he now owed the gas company nothing at all. A week later, the manager of the local bank called our hapless friend and asked him what he was doing writing cheque for $0.00. After a lengthy explanation the bank manager replied that the $0.00 cheque had caused their cheque processing software to fail. The bank could therefore not process ANY cheques they had received from ANY of their customers that day because the cheque for $0.00 had caused the computer to crash. The following month the man received a letter from the gas company claiming that his cheque has bounced and that he now owed them $0.00 and unless he sent a cheque by return mail they would be taking steps to recover the debt. The man then tried to file a debt harassment claim against the gas company. It took him nearly 2 hours to convince the clerks that he was not joking but convince them he

An Oldie But A Goodie

did. They subsequently provided statements which were considered substantive evidence of the aggravation and difficulties the man had been forced to endure during this debacle. The matter was heard in the local Magistrate's Court and the outcome was this: The gas company was ordered to:

Immediately rectify their computerised accounts system or show cause, within 10 days, why the matter should not be referred to a higher court for consideration under Company Law. Pay the bank dishonour fees incurred by the man. Pay the bank dishonour fees incurred by all the banks clients whose cheques had been bounced on the day our friend's had been. Pay the claimant's court costs; and Pay the claimant a total of $1500 per month for the 5 month period March to July inclusive as compensation for the aggravation they had caused their client to suffer.

NZTester comment: All for the sake of a test case to trap $0.00 balances! While we can laugh at the absurdity of this comedy of errors, when we consider how much time and money it cost, not to mention the frustration for all concerned, it shows just how much havoc even simple software defects can cause (as if we didnt know already). Article is of unknown origin. Ed.

11

By Kim Partridge

Every year Software Education gathers together some Julie then provided every STANZ attendee with a of the best minds in software testing from around the personal testers survival kit which included, among world at our STANZ conference. other things, a chocolate bug! For the first time ever, we went to Christchurch and Auckland to host two one-day conferences with international speakers Elisabeth Hendrickson, Alan Page and Julie Gardiner. Day 1 started at STANZ Christchurch. Julie Gardiner was first up with her keynote presentation on the 2012 Survival Guide: Lessons for Every Tester. Julie can be seen at bottom right of the photo below gearing up to take the stage. Alan Pages keynote was entitled: Testing Upside Down; Customer Focused Test Design. He noted that in many organisations, performance, security and reliability testing is performed only towards the end of a project. Basing his points on research undertaken at Microsoft, Alan outlined how these can be quite customer-impacting and that there is a strong case for starting these non-functional types of testing much earlier in the development lifecycle and turn the traditional method of testing upside down. Alan also discussed scenario testing, shifts and sparks, and live testing, which may seem like a scary prospect but is certainly possible providing there is a robust (and quick - Ed.) release management process in place to roll back if necessary. The last of the morning keynotes was presented by Elisabeth Hendrickson. In Testing Reframed, Elisabeth first looked at historical views of testers which she described either as gatekeepers of quality, product assurers or independent assessors of product quality. Elisabeth outlined why she believe these views are outdated and mostly misguided anyway. She observes that times are changing, the context in which testing occurs is changing, technology and platforms are changing and development methods are moving from a Waterfall to an Agile approach. As a result, Test Driven Development (TDD) and Acceptance Test Driven Development (ATDD) have become increasingly popular. With the help of continuous integration tools, feedback around the quality of a product can now be obtained in seconds rather than days. Another takeaway from Elisabeths session was her view that the definition of testing can be split into either or both of exploring and checking. The afternoons were spread across three streams: Sharon Robson presenting Leading the Way

In her presentation, Julie outlined five principles she believes every tester needs to follow in order to not simply survive as a tester but thrive: 1. Stand out by estimating quality 2. Stand out by having passion - enjoy testing and have fun! 3. Stand out by taking ownership of your career 4. Stand out by demonstrating and reporting the value of testing 5. Stand out by retaining integrity

12

From the Back of the Room, Geoff Horne with A Method of Assessing Testing Risk Mitigations and Alan Kan showing How to Make Your Testing More Agile. This was followed by three workshops in Christchurch and two in Auckland where Julie Gardiner presented Usability Testing in a Nutshell (Christchurch only), Alan Page presented The Big Picture of Test Automation and Elisabeth Hendrickson presented Beginning with the End in Mind: Acceptance Test Driven Development (ATDD) in Practice.

of the year! It was quite a different experience for our presenters as well, some of whom were also speaking at our Fusion conference in Sydney later that week. Those from North America noted quite a difference between internal flights in NZ compared to the US. One thing I did miss about the two day events was the usual evening dinner on the first night. Last year we got to informally sit, chat, eat and drink away from conference proceedings for a few hours, which is always very interesting and a lot of fun. This year we hosted drinks and canapes at the end of each day however there are always trains and planes to catch at the end of a conference. We had a lovely time talking to those who did stay back and received much praise for Kiwi testers from our international experts, not to mention many compliments around how beautiful Aotearoa is. So, for the next STANZ, wherever it is and for however long it may be, we hope you can join us. Kim Partridge is the Marketing Manager at Software Education, Wellington. She can be contacted at kimp@softed.com

STANZ Auckland

It was a successful STANZ from Software Educations perspective as our customers provided us with great feedback. On a scale of 1 to 5, our international testers all scored above 4 plus we also saw 4+ for the range of topics, quality of speakers and overall value of the conference. However even we have to admit we are not always perfect and we did get some interesting feedback on things we could do for the next STANZ Conference eg. bring it back to Wellington! Some attendees preferred the one day format whereas others liked being able to interact with speakers more over a two-day period. We are really pleased that our gamble to take STANZ on the road to Christchurch and Auckland paid off! It provided the opportunity to talk to testing communities in both locations and indulge in a little travel ourselves. There were only a few logistical challenges eg. transporting all materials with us on the plane, not to mention flying out of Wellington during one of the windiest weekends

Software Education hosts on the bench in Auckland

13

14

WebTest believes Kiwi companies should apply their own strengths to help improve testing efficiency
Auckland-based company WebTest is a growing force in web application testing and CEO John Lockhart is not afraid to voice strong and perhaps controversial views on the testing process. WebTest works with companies that use a variety of development models and not just in web application testing, despite the name. John believes that combining a Kiwi approach with the principles of lean development can greatly improve testing. John comes from a development management background and hates to see client staff performing pointless work, identifying three primary inefficiencies affecting many testing efforts:

status comment for each one (not applicable, complete, blocked etc).

Test cases or scripts can also be documented however he recommends simple tabular forms or using the 'Given, When, Then' syntax from Behaviour Driven Development.

John believes these approaches have key advantages:

They enforce a discipline of keeping tests specific, independent and at the business level rather than going into a level of detail that makes them likely to become out of date and wastes effort. They support automation directly without change and form readable documentation of the business rules replacing much of the need for detailed functional specifications and other documents.

Rather than actual testing, analysts spend time writing documents that may never be read. Testing is being performed without a clear understanding of why, just because the tests are in scripts someone wrote years ago. Much of the testing effort may be spent on work that is unlikely to find significant defects rather than mitigating serious risks. Companies have employed people who are not competent testers and lack domain and system knowledge. Using five people who lack capability and soak up management time is less efficient than using one or two experienced people who can complete focused testing with minimal overheads.

It might be thought that these approaches can only work for functional requirements however nonfunctional requirements such as load/ performance/stability testing also work well when performed in this manner. Says John: "WebTest encourages clients to view automation as an integral part of the development process and ensure the entire team is involved. "Use tooling that is flexible, well architected, compatible with any developer tools used by the team and supports the executable specification approach. "Automation and tooling in general should not be considered optional as any professional should have, and use, the tools of their trade appropriately without having to jump through management hoops or deal with micromanagement", says John. "This assumes you are using test professionals of course, key people should have years of experience in testing."

John proposes a few solutions.

Ensure that documentation serves a purpose and is only detailed enough to achieve that purpose. WebTest works with a client to develop a project test strategy of less than one page complemented by a constantly changing test plan in a wiki of perhaps a few pages. Instead of massive test analysis, test case and test script documentation John advocates the use of bullet-pointed checklists with a simple

15

If there are no internal specialists available, John advises companies to bring in an expert provider. "It's important to be sceptical about sales campaigns or demonstrations from testing vendors and have someone available who knows enough about testing to keep them honest." Go for quality rather than quantity in testing, John advises. Hire the very best people - at least one if developers are closely involved; more if not. Then hire the brightest juniors; "they can be highachieving university IT graduates or people from other backgrounds. The crucial factor is they have the intelligence and attitude required to flourish in testing and on graduate pay levels, companies can afford to train them and keep them on between projects. They'll find plenty to do if they are bright and motivated." A key objective should be to smooth the process which is one of the under-rated keys to agile practices. This principle comes from lean manufacturing and can be applied to traditional development approaches. "Keep all parts of the process running in parallel, in small increments, and get fast feedback from your test team", says John. "Ideally defects are found and fixed within a day or two of being discovered with minimal or no documentation." This reduces three common inefficiencies:

"The only documentation needed is a 'lessons learned' list or modifications to tests if they did not catch the issues well." By following these principles, WebTest has found that companies are able to achieve massive improvements in efficiency. They are able to speed delivery and become more responsive to market demands. "By smoothing the process, the levels of stress, chaos and panic drop drastically. This is great for everyone except managers whose only skill is crisis management - they have to find another a company that still does things the traditional way!" says John. In his view, Kiwis common sense and famed 'number eight wire' mentality ties in well with lean principles; "I'm mystified why so many Kiwi companies follow the so-called 'best practices' of large overseas companies, which in our testing experience often turn out to be worst practise. With expert help those same Kiwi companies can bring their inherent strengths into the testing process and see vast improvements." - by NZTester Staff Writer John Lockhart is CEO of WebTest - a growing testing services companies which helps some of New Zealand's high profile companies achieve great testing outcomes. He can be contacted at 021 757 546 or email at john@webtest.co.nz.

the defect management and review process time for developers to track down defects which is much harder if time has passed the cost of working around existing defects in testing, documentation and support

16

Championing the End-User Using Personas

By Michael Talks

The other week I attended what was to be a fascinating presentation by Optimal Usability (http: //www.optimalusability.com/) regarding the use of personas within organisations entitled Putting Personas to Work. A persona is a named fictitious character whose personal outlook and details sketched out. They represent an important demographic as an end user or customer however learning through the details provided around their habits and behaviour, we can ensure our business endeavours to champion and represent these people. Richard Douglass of Optimal Usability talked in great detail about the process of how such personas can be drawn up. However I'm going to attempt to outline how I've seen them used initially and how useful they can be for understanding the mind set of an end user. Back in 1996 I worked for Thompson Marconi Sonar in Weymouth, England. The companys main target sector was the UK Defence industry however since the end of the Cold War that market had declined significantly. Like many companies of the time it was attempting to diversify and beat swords into ploughshares to gain new commercial markets. During transition, Thompson Marconi found its sonar technology had applications within the salmon farming industry. A salmon farm is typically a netted area within a river or lake where the fish are hatched and allowed to grow in containment. However the lure of so many in an enclosed environment was just too much for the local seal population which would often attempt to break through the nets. For the farmer this is devastating and usually leads to whole broods of salmon devoured or escaping through damaged nets. Their

fate is no better than that of the seals which become entangled in the nets, many suffering terrible injury and often death. The solution was to use one of company's aquaphones to emit a signal which acted as an aquatic scarecrow to the seals while not providing any harmful effects on the fish. However that was the easy part..... The hard part was putting the technology into a device suitable for its operators, in effect the product end-users and this is where the team at Thomson Marconi hit its first obstacle. In its defence sector, the company understood its end -user very well. A typical sonar operator: Had come through secondary school Received a year's training on his equipment Was supported in the sonar suite by technical staff and a well-equipped storeroom Had access to a wide range of documentation to assist Quite comfortable with technology

Thomson Marconi put together an early prototype SealScrammer which was programmable much like a simple video recorder in that it could be activated at a specific time of the day, operating on a custom battery. There was sizeable early interest in the device however much of this did not translate into sales, leaving the Thompson Marconi team scratching its head. One of the biggest markets for the SealScrammer was Chile, where it was trialled, with the company sending technical staff to work with potential clients

17

there. The Technical Lead returned with many experiences of client end-users (and a Chilean wife, I don't quite know how that happened to this day) and it soon became clear why sales had failed to materialise. It was discovered that the average Chilean SealScrammer end-user: Had very poor literacy skills, some were almost completely illiterate mainly due to living and working in remote areas from a young age. Had little access to technology with the SealScrammer likely to be the most advanced device owned to date. Was however very clever mechanically and learned very early in life how to maintain cars etc. as being so remote there were no other options available (especially given the nearest town for spares could be several hours drive away. So quite a different proposition to the typical naval operator within the UK Defence. For the SealScrammer to work in Chile, it had to have value to this persona. The Thompson Marconi team set about defining the persona, documenting it and displaying prominently on the SealScrammer team wall. This led to changes to the design, making the device much more intuitive to use. Frequently serviced parts were made much easier to recognise thus reducing reliance on the documentation, which incidentally was written in English - some users could barely read their own language let alone someone elses. The device was also made to work using a standard car battery so it could be easily replaced or serviced it when it ran down. Finally, it was no longer programmable. There was no point selling a device as easy to program as a video recorder to users who didn't own a video recorder, nor would probably not know what one was. The device was triggered instead by a radio control similar to most garage door controls with just a single button. The end-users were told to press this every couple of hours and became quite tickled at being in control and knowing when it was activated. The end result was a device with a lot less functionality than the original but a lot more value to the end-user. It then sold well and when in 1999

Thomson Marconi Sonar was purchased by Thales, one of the SealScrammer team bought the business back from them. John Ace Hopkins, who was quite a character, built Ace Aquatic (http:// www.aceaquatec.com/) into a successful business with end-users in mind. Sadly, whilst researching this piece I learned of his death last year. Today many companies and government organisations are researching into personas to gain a better understand of their potential end users. Usually, unlike Thomson Marconi which had two personas only, there are many more they're there to ensure that no customer gets left behind. Hence when working on a website, it's important that it is usable not only by Chels, a recent University media graduate who never goes far without her MacBook, but also by Dot, who is retired and uses computers only as a last resort. She is still on Windows NT (she doesn't like change) and has a low- resolution CRT screen. Oh and the only internet she has is dial-up. As testers we need to ensure that we champion these and other end-users in our testing and not favour the high-specd machines over the older, slower ones. That means testers, who may prefer other browsers like Chrome and Opera, still need to test on Internet Explorer because it is still the most popular browser in use today. Personas can also be useful when advocating the impact of the bugs raised. By relating to business owners how a bug will affect one of the core business personas is a powerful tool in the testers arsenal for getting the message across. Personas, like characters in a much- loved TV show are characters we can all relate to and can bring different areas of a business together. This was seen in the Optimal Usability presentation where I networked with many from my own company whom I'd never met in my time here. They weren't even testers but were definitely interested in how testing can champion these personas and I think any event which allows us to share the values of testing with a wider audience is a good thing!

Michael Talks is a Senior Tester (and all round decent bloke - Ed.) at Kiwibank. He can be contacted on mike_talks@hotmail.com Also watch out for his article series Creating a Testing Culture starting in the next issue of NZTester.

18

Check This One Out

A particular young mother is not quite in a position to buy an iPhone 5 so she decides to take advantage of the drop in iPhone 4 prices (in anticipation of the iPhone 5 release). After all, her trusty iPhone 3 has served her well however is now on its last legs, having been dropped numerous times, sat on, thrashed and argued over by a 6 year old and a 3 year old wanting to play games and generally doing what iPhones do best, being iPhones! Being the computer-savvy consumer she is, onto her Mac she goes and logs onto the website of her local iPhone retailer. To her delight, sure enough there they are; iPhone 4s at $599.00 and 4Ss at $799.00. However before clicking on the buy button she decides to have a look around to see what else might be available and surfs through a number of pages before returning to the iPhone 4. To her much greater delight she finds that the price has now dropped to $199.00 and all while she was looking around! Now, being of a questioning persuasion she is never one to merely accept things as they are and will always probe to ensure there is nothing there that could burst her this is too good to be true bubble. But, as she fossicks around on the site, the $199.00 price is holding firm. Still not convinced however our heroine decides to buy over the phone, at least that way she can verify immediately whether there is small print that she might have missed. After wading through the layers of customerservice voice menus she finally gets to speak with a support representative who assures her that the price is indeed $599.00 and insists that the website does not show $199.00 as claimed. So young mother offers to capture a screenshot and send it to the retailer, which she duly does and finally gains acknowledgement that indeed it is showing as $199.00. Not only that but the support representative has now managed to reproduce the $199.00 price herself and after a little more exploration, finds that navigating to the iPhone 4 page in a particular sequence is what generates the error condition, which is exactly the sequence that our young mother had followed.

Now, most of us testers might jump up and down here; see what happens when testing is not done properly and there is a point there. However in reality, finding something like this would probably happen only by chance as opposed to specific a test case being designed to cover that very sequence of page jumps. After all, there could have been so many different permutations of jumps to test, it would not have been viable to cover them all. Perhaps sampling a subset of permutations or using an all pairs approach (testing that any page can successfully jump to any other page) may have spotted it however this too would have been unlikely. Perhaps adding this scenario to the regression test suite might not go amiss though. So what was the outcome? The retailer to its credit honoured the $199.00 price without question then on finding that iPhone 4s were unable to be supplied right away, called the intrepid young mum back and asked whether she would mind taking an iPhone 4S instead - still for $199.00! Young mum was left counting her Christmaseswouldnt you? - by NZTester Staff Writer

NZTester comment: We have no idea whether anyone else encountered the same problem nor how many $199.00 prices were honoured however I think it is unlikely to have only been a single occurrence. And while the impact of honouring the price across the breadth of the retailers business is probably very small, there is never any guarantee that this will always be the case. Incidentally, young mum is the Editors daughter and he can vouch for exactly what 6 year-olds and 3 yearolds thrash and argue over Ed.

19

Possum Testing: Defining Fearful Testing

by Brian Osman

A possum is a marsupial introduced into New Zealand around 1837 to help establish a fur trade. The problem with possums in New Zealand is that they are considered to be pests with up to 30 million possums destroying native vegetation and fauna. So why am I talking about possums? Because a possum is synonymous with something that produces more harm than perceived good. Therefore the question is:What is possum testing? Possum testing was defined as "Testing that you don't value motivated by fear on some level". What do I mean by this? Early in my testing career, I was taught that good testing was dependent on test scripts and the documentation produced. Even then, I did not feel comfortable with what I was doing but continued forward because I did not know of an alternative and I was motivated by fear (losing credibility, not seen as doing good work, not doing *real* testing and possibly losing my job). I quickly realized that I did not value what I knew as testing but this realisation came second to the motivation of fear. So let's explore this further. According to Michael Bolton, testing is a process of exploration, discovery, investigation, and learning1 . This definition clearly highlights testing as a creative, sapient activity that helps a tester become more productive through exploring, discovering and learning the product under test. A good tester is someone that understands these elements and by applying these elements in a practical way, allows useful information to be fed back to management. This is one particular value that testers can add. The crux of possum testing is the concept of value. Value is subjective because it is dependent on what

we hold to be worthy or important. When looking back at my early days of testing, I was writing test scripts based only on requirements documents. The value I placed on these scripts was high due to the fact that I saw testing as artefacts based and not on the information learned or issues found. The value that we place on our testing is contextual dependent on the situation and project we are in. On some projects, the testing was either not valued by the project team or I did not value the work that was done. For example, on one project, we wrote approximately 400 test scripts for each tester (16 testers on the team) and re-wrote our test scripts each time we received a new version of the functional specification (which was at least seven new versions). The project eventually failed and the artefacts produced were near worthless. The worth of these test artefacts were high in terms of documentation produced but low in terms of information gained and shared with management. For this particular project, fear became a motivating factor thereby affecting the value of the testing undertaken. Fear is something that we are afraid of and the fear that ran through the project was the fear of failure, fear that contracts would be terminated and fear that testers reputations may be tarnished even though the product was not up to par. Fear is something that we face as we are often on the end of project emotions from management, developers and business as we present information that may be contrary to what the project sees as success. On another project, testers were told to stop calling bugs bugs or defects and call them issues. While seemingly innocuous on the outside, the motivating fear by the test manager was in upsetting the relationship with the vendors.

20

This fear was a motivator which devalued the effort and In short, possum testing can take many forms and can be present at different levels. Understanding what you value and what motivates you can help you understand the worth of the work you do. If it is not of value then something needs to change to make you a better tester. What do you value, what motivates you and what is the fear that surrounds you? Remember, if your work is something motivated by fear then you may be doing possum testing. 1 http://www.developsense.com/ blog/2009/08/testing-vschecking/ Brian Osman is the Principal Consultant at OsmanIT. He can be contacted at brian.osman@gmail.com.

Testing Events Coming Up...

21

Reproduced with the kind permission of Andy Glover

22

Great Bugs We Have Known and Loved

By NZTester Staff Writer

We are hoping that this subject can become a regular feature in NZTester so we hope youll be keen send in your favourites. Our Staff Writer will kick the first one off a dreaded memory leak Ed. In the dark distance past ie. circa 20 years ago, the best performance testing tool available was the Friday night pizza party. This was usually conducted out of hours with project management feeding those who had volunteered to stay back with stacks of pizzas in return for everyone clicking <enter> buttons at the same time. The technicians behind the scenes would then monitor systems activity across the various scenarios of lets all click at once to see how the system would perform. At this time test automation was in its infancy however there were a few products on the market that could effectively capture and play back keystrokes plus enter test cases at specific points in the generated code. We had yet to discover the maintenance overheads of automated testing however that was not too far around the next bend. However one particular test automation developer came up with an idea: why not run automated tests on multiple desktops at once? That would effectively create the same system activity that many users would without the need for fingers to click <enter> buttons. All that was needed was some trigger code on the desktops that could be fired off centrally and Bobs your uncle. And so it became.

that could be distributed across 400 workstations; he doing the doing, I doing the managing. It took us a couple of weeks but we were finally ready for the big night. We had installed the agent software on the desktops, made sure we had all 400 running and engaged our respective fit and healthy teenage sons to run up and down 6 floors of office space to make sure that none of the desktops fell over or off the network. Our central controller was set up and we had run a few sample executions, building the number of desktops in use up as we went. Once we got to 50 we decided take the plunge and kick off the first official 400-desktop run. BANG! She was away, we could see the automated scripts fire up on the desktops, sons verified that they were doing the same on other floors and after 20 minutes we had around 120 in action. Our central controller was capturing the performance data that was being sent back by the desktop agents and dumping it to a database. It was all very exciting and we used another technology still in its infancy at the time to maintain contact with our floor runners the cellphone. But then. CRASH! The test server ground to a halt and error messages were flashing up everywhere. Cellphone calls from sons went ballistic and we started to see our central controller become very confused before it too ground to a halt. What on earth had happened?

The backroom boys rebooted the server however At around the same time, a large NZ company had we decided to call it a night and try again the next implemented a client/server Customer Relationship night, which we did. However the same thing Management (CRM) system purchased from a happened. After around 20 minutes of running and software company in the USA. A project manager had at around 120 desktops in flight, the server would seen the opportunity for test automation, grabbed it inexplicably freeze. So we tried again the next night and engaged tools and services from the company I and the next. After around six attempts all trying was contracting to at the time. So a colleague and I something different to correct the error, we had to started work on building a set of automated tests throw in the towel.

23

Meanwhile the infrastructure chaps were also at a loss. They had started investigating from the very first night and came up empty. The US-based software developer had been notified on the third night and immediately pointed finger at the test automation tool then at my colleague and I. Class! However a throw-away comment from one of the junior database analysts alerted me to something. He had mentioned in passing that he thought our error message might be the same as one that was being encountered around once every 2-3 months or so. The server was simply rebooted when it happened and business carried on for another 2-3 months. Because of the frequency and the simple fix, it had received a low priority on the support register.

who had recently discovered that they were running up and down stairways for love and not money. Fast forward 20-odd years. Performance test tools are now so intelligent they not only detect issues but also provide analysis and recommendations of possible actions to take. We dont need fleets of desktops any more, tools can now simulate user loads virtually, right down to the browser (now thats a word we didnt know 20 years ago) to be used and probably many more options that I wouldnt even know about.

One thing hasnt changed though. The need for Performance Testing. An under-performing system or application has the same impact on business now as it did back then, even more so now that technology and our ways of conducting business Back at the ranch, my colleague had run through have changed so radically, via the Internet, mobiles, all of his automated scripts countless times over broadband, tablets, wireless, faster, bigger, wider, so I suggested that we execute each one manually fatter, less expensive etc. etc. Im sure you see while also running a bunch via the central controller the point. to generate a background load. We d been going Oh and by the way, sons were paid in the end, just in for a few days when all of a sudden bingo, up comes hamburgers rather than cash! the error and down goes the server. So we take out the background load and run the particular script manually over and over on multiple desktops and boof, again! So we had reproduced the problem manuallythat ruled out the tools and the testers! After much investigation, my colleague found the trigger. It happened when a particular tab on a particular screen was clicked, activity took place on it and the tab exited. Doing it time and time over caused the error. The software developer was advised and came back almost immediately workspace on the server was being grabbed when the tab on the client was clicked on and was not being released upon exiting so over many executions the allocated workspace continued to increase in size until the server had had enough poor thing. Simple fix, simple release into Production and the client was away again. No more 2-3 month halts and it was reported that overall system performance had increased as well. We discovered later that the software developer released the remedy as an emergency fix to all of its 70-odd customers around the world at the time and that a few had reported the same error that we had encountered more than once before! So, everybody was happy - all except sons,

In the next issue of NZTester Magazine (Jan 2013):

Interview with another NZ testing personality First of Creating a Test Culture series by Michael Talks First of Testing Practices vs. Commercial Realities series by Geoff Horne StarEast 2012 Review by Brian Osman Kiwi Test Teams Great Bugs We have Known and Loved

24

And now its your turn

If you would like to be involved with and/or contribute to future NZTester issues, youre formally invited to submit your proposals to me at ed@nztester.co.nz Articles should be a minimum of 1/2 A4 page at Cambria 11pt font and a maximum of 2 A4 pages for the real enthusiasts. If you wish to use names of people and/or organisations outside of your own, you will need to ensure that you have permission to do so. Articles may be product reviews, success stories, testing how-tos, conference papers or merely some thought-provoking ideas that you might wish to put out there. You dont have to be a great writer as we have our own staff writer who is always available to assist. Please remember to provide your email address which will be published with your article along with any photos you might like to include (a headshot photo of yourself should be provided with each article selected for publishing). As NZTester is a free magazine, there will be no financial compensation for any submission and the editor reserves the sole right to select what is published and what is not. Please also be aware that your article will be proofread and amendments possibly made for readability. And while we all believe in free speech Im sure, it goes without saying that any defamatory or inflammatory comments directed towards an organisation or individual are not acceptable and will either be deleted from the article or the whole submission rejected for publication.

Feedback
NZTester is open to suggestions of any type, indeed feedback is encouraged. If you feel so inclined to tell us how much you enjoyed (or otherwise) this first issue, we will publish both praise and criticism, as long as the latter is constructive. Email me on ed@nztester.co.nz and please advise in your email if you specifically do not want your comments published in the next issue otherwise we will assume that youre OK with this.

Qual IT Solutions Assurity Consulting QAVerify LoadVerify Software Education

4 9 21 22 back cover

Sign Up to Receive NZTester

Finally, if you would like to receive your own copy of NZTester completely free, even though were real low tech. right now theres still three easy ways: 1) connect with me (Geoff Horne) on LinkedIn, 2) go to www.isqa.com/nztestermagazine.html, or 3) email me at geoffh@isqa.com with Subscribe the subject line. -Ed. 25

If you are interested in advertising in NZTester Magazine please email ed@nztester.co.nz. We can assist with graphic design and photography if required (fee-based).

26