ARTICLE IN PRESS

International Journal of Industrial Ergonomics 36 (2006) 345352 www.elsevier.com/locate/ergon

Usability in a medical technology context assessment of methods for usability evaluation of medical equipment
Erik Liljegren
Department of Product and Production Development, Division of Human Factors Engineering, Chalmers University of Technology, TEBORG, Sweden SE-412 96 GO Received 28 September 2004; received in revised form 30 September 2005; accepted 5 October 2005 Available online 17 February 2006

Abstract The increased complexity of medical technology makes usability an important selection criterion when new equipment is purchased. However, this requires an understanding of what usability is in a medical technology context and what usability evaluation methods are suitable. A questionnaire was used to investigate what users of medical technology regard as the largest component of usability. The component difcult to make errors was regarded as being 30% of overall usability. The components easy to learn, efcient to use, easy to remember made up 20% each of overall usability. Satisfaction only made up 10% of overall usability. Four common methods, hierarchical task analysis, cognitive walkthrough, heuristic evaluation and usability tests were evaluated according to thoroughness, validity, reliability, cost effectiveness and clarity. Usability tests are recommended to be the primary method in usability evaluations at hospitals, as they full the criteria and address the difcult to make errors aspect of overall usability. Hierarchical task analysis and cognitive walkthrough full some criteria. Cognitive walkthrough also addresses the difcult to make errors aspect. Relevance to industry There is an increasing awareness of the need for higher usability of medical technology. This requires an understanding of what usability is and what usability evaluation methods are suitable, both in the design process and when medical technology is purchased at hospitals. r 2006 Elsevier B.V. All rights reserved.
Keywords: Usability; Usability tests; Hierarchical task analysis; Cognitive walkthrough; Heuristic evaluation; Medical technology

1. Introduction 1.1. Background Medical technology is becoming more important in modern health care. This increases the treatment and diagnostic capabilities, but also increases the complexity of health care. The increased complexity has consequences for patient safety, as it can increase the risks for human error (Reason, 1990). The Harvard Medical Practice Study showed that human error is the cause of up to 69% of injuries to patients (Leape, 1994). A survey by Weinger (1999) showed that between 69 and 82% of mishaps in
Tel.: +46 31 772 3611; fax: +46 31 772 3660.

E-mail address: erik.liljegren@me.chalmers.se. 0169-8141/$ - see front matter r 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.ergon.2005.10.004

anaesthesia were due to human error. Deciencies in the design of medical technology can increase the risk of human error (Hyman, 1994). Studies have shown, that medical technology in use today can exhibit common manmachine interaction aws, or latent errors (Reason, 1990), such as poor feedback about operating mode, hidden controls, arbitrary and complex sequences of operation, and ambiguous alarms (Cook et al., 1991; Obradovich and Woods, 1996; Liljegren et al., 2000). Some systems are simply too complex, such as a patient monitoring system used during cardiac anaesthesia, where the users had difcult interacting with the system as it had so many functions that the control of these functions became a problem (Cook and Woods, 1996). In some cases the cause of an incident can be directly traced back to design features (Cook et al., 1992).

ARTICLE IN PRESS
346 E. Liljegren / International Journal of Industrial Ergonomics 36 (2006) 345352

However, poor design of medical technology also affects the work environment of the staff. Studies have shown that medical staff often have a feeling of helplessness and frustration when handling technology (Hellstro m, 1995; Persson et al., 1992). This can generate an increased sense of worry, stress and uncertainty among the staff. Effectiveness and safety of medical technology depends on factors such as user competence, safe use and appropriate application (Persson et al., 1992). All of these factors can be inuenced by the design of medical technology. Designers in other elds, e.g. software design and consumer products, have been aware that their products need to be designed so that users can use the products to a satisfying degree. The designers have aimed for products with high usability. Usability is not a single property, but a combination of many different properties and attributes. The ISO denes in the standard ISO 9421-11 that usability is the effectiveness, efciency, and satisfaction with which specied users achieve specied goals in particular environments (ISO, 1998). Effectiveness: is the accuracy and completeness with which specied users can achieve specied goals in particular environments. Efciency is the resources expended in relation to the accuracy and completeness of goals achieved. Satisfaction is the comfort and acceptability of the work system to its users and other people affected by its use. There is an increasing awareness from both the health care community and medical technology industry that new medical equipment needs to have higher usability. Two approaches can be used to increase the usability of medical technology (Cook et al., 1992; Goodman and Ahn, 1999): 1. The equipment designers and manufacturers need to take human capabilities and limitations into account during design. The results from human factors research has been highly benecial in e.g. aviation and control room design. There is an increasing understanding that human factors knowledge has a lot to offer in medical device design as well (Klatzky and Ayoub, 1995). 2. The purchasers and users of medical technology need to evaluate the usability of new equipment when it is selected and purchased. It has long been regarded as important to evaluate the usability of any system (Chapanis, 1991), but today there is hardly any formal usability evaluation performed at hospitals. When medical devices are purchased, decision-makers usually have information about e.g. cost and functionality, but very little information about the usability of the systems under consideration for purchase. However, today very little, if any, research is done on how usability evaluations can be integrated into purchasing situations and what methods are suitable. In Sweden there is an increasing interest for usability evaluation of medical

devices, and some hospitals have started to include usability evaluations when they purchase new equipment. However, much of the evaluation is on an ad hoc basis. This generates a need for usability evaluation methods that are suitable for application in health care and can be used to provide decision-makers with information about usability qualities of systems under consideration for purchase. However, rst two questions have to be answered: 1. What is usability in a medical technology context? The ISO denition includes two major parts upon which usability depends: three measurable components (effectiveness, efciency and satisfaction) and three specic prerequisites (specied users, specied goals and particular environments). This makes usability a context-dependent property, as the same device used in two different environments can be seen as having different usability depending on the users, and what they are using it for. Also, usability depends to an extent on what the users perceive as usability. The easiest way to answer this question is to simply ask the specied users what they consider usability to be. The rationale behind this is that if what the users perception of usability is known, then methods for evaluation can be selected accordingly. However, then the ISO denition is difcult to use as it is not easily translated into questions that are easily answered by users. Nielsens (1993) denition of usability is more helpful as it is focussed on equipment properties and attributes and easily translates into questions. The denition states that usability is associated with ve components:

  

 

Learnability: The system should be easy to learn so that the user can rapidly start getting some work done with the system. Efciency: The system should be efcient to use, so that once the user has learned the system, a high level of productivity is possible. Memorability: The system should be easy to remember, so that the casual user is able to return to the system after some period of not having used it, without having to learn everything all over again. Errors: The system should have a low error rate, so that users make few errors during the use of the system and error recovery is easy. Satisfaction: The system should be pleasant to use, so users are subjectively satised when using it.

These ve components can all be translated into components of the ISO denition. The ISO component effectiveness relates to the Nielsen component few errors. The three Nielsen components learnability, efciency and memorability all relate to the ISO component efciency, i.e. the resources expended in relation to the accuracy and completeness of goals achieved. The nal Nielsen component, satisfaction, corresponds to the ISO component satisfaction. Using Nielsens (1993) denition, the overall concept of usability consists of ve components that constitute different percentages of overall usability.

ARTICLE IN PRESS
E. Liljegren / International Journal of Industrial Ergonomics 36 (2006) 345352 347

2. What methods are suitable to evaluate the usability of medical technology? Existing usability evaluation methods evaluate different aspects of usability, and provide different information. It is important to choose methods that evaluate the desired aspect(-s) of usability, as e.g. evaluating only the satisfaction aspect of a safety-critical system, or the efciency aspect of a computer game would be inadequate. 1.2. Purpose and aims The purpose of this study was to investigate what users of medical technology regard as the greatest component or components of usability and assess four usability evaluation methods according to the users interpretation of usability and methodological criteria. The aim was to suggest suitable usability evaluation methods that can be used when hospitals purchase medical devices. 2. Users perception of usability 2.1. Method A questionnaire was distributed to six wards with a high usage of technology (Table 1). The head nurse at each ward distributed the questionnaires. A total of 105 questionnaires were distributed. Four Intensive Care Units and two Surgical Units at two Swedish University hospitals participated in the study. These two types of wards were selected because they use medical technology to a large extent. The devices often have graphical user interfaces and typical devices that are commonly used are patient monitoring systems, ventilators and infusion pumps. The questionnaire was in Swedish and divided into three parts. Part 1 collected background information such as personnel category, age, gender, type of ward, number of years at present ward, how often the respondent used medical technology with menu systems or user interfaces. Part 2 asked the respondent to grade usability based on ve components. The concept of overall usability was taken as 100% and the respondents were asked to grade from 0 to 100% how much each of the ve components made up of overall usability. The components were taken

from Nielsens (1993) denition of usability and formulated as follows: 1. The equipment should be easy to learn, so you can start to use it quickly. 2. The equipment should be easy to remember, so you can start using it quickly after a period of absence. 3. The equipment should be easy to use, so you do not have to direct all attention at handling the equipment. 4. It should be difcult to make errors. 5. The equipment should be pleasing and comfortable to use. The third and nal part of the questionnaire provided for general comments. 2.2. Results from the questionnaire A total of 80 questionnaires were returned. One questionnaire was not lled in and was excluded. Eightytwo percent of the respondents were women and eighteen percent men. The average age was 40 years (SD 9 years, range 2158 years). Forty-one percent of the respondents were assistant nurses, 54% were nurses and ve percent medical doctors. Ninety-six percent of respondents used medical devices daily and four percent two to four times a week. The average time the respondents had worked at the present ward was 9 years (SD 7 years, range 0, 530 years). The results for all professional categories are shown in Fig. 1. The component difcult to make errors comprised the greatest part of usability, with a median grading of 30%. The three components easy to learn, efcient to use, easy to remember all received median gradings of 20%. However, these three components can be grouped under the ISO component efciency. The nal component, satisfaction, received a median grading of only 10%. 2.3. Comments on methodology The criteria used for selecting those who lled in the questionnaire were unknown to the author, as the head nurse in each ward distributed the questionnaires. However, the respondents were realistically distributed over professional categories. Many respondents also used technology daily. This indicates that the selection of respondents realistically reects the intended user group. The component difcult to commit errors was regarded as being the largest part of usability. However, errors in healthcare differ greatly both in character and consequence, from diagnostic errors to fatal treatment errors. This difference was not addressed in the questionnaire. However, in modern healthcare it is important to reduce all errors so this difference in error types is. The ISO denition includes three prerequisites, specied users, specied goals and particular environments. In this study, the specied users were assistant nurses, nurses and

Table 1 Distribution of questionnaires to the two hospitals Type of ward Number of questionnaires distributed 16 16 25 16 16 16

General intensive care unit, hospital 1 Cardiac intensive care unit, hospital 1 Surgical unit, hospital 1 General intensive care unit, hospital 2 Thorax intensive care unit, hospital 2 Surgical unit, hospital 2

ARTICLE IN PRESS
348 E. Liljegren / International Journal of Industrial Ergonomics 36 (2006) 345352

60 50 Frequency 40 30 20 10 0 30
2.4 Difficult to make errors 2.5 Pleasant to use

medical doctors. The particular environments were ICUs and Surgical wards, two types of environments that are characterised by high usage of advanced technology. Both these prerequisites were addressed in the questionnaire, although indirectly. The third prerequisite, specied goals, was not directly addressed. The specied goals when using different kinds of medical devices are different. However, as the component difcult to commit errors was regarded as the greatest component, one conclusion is that the users have a specied goal of operating a medical device without error. However, this prerequisite can be studied in further detail. 3. Usability evaluation in technology acquisition 3.1. Introduction

Frequency

20

10

0 50 40 Frequency 30 20 10 0 50 40 Frequency 30 20 10 0 50 40 Frequency 30 20 10 0 0-10 20-30 40-50 60-70 10-20 30-40 50-60 70-80
Fig. 1. Results from the questionnaire, showing the distribution of gradings for the ve components of usability.
2.1 Easy to learn 2.2 Easy to remember 2.3 Efficient to use

The purpose of a usability evaluation is to nd usability problems, which are problems that users encounter in real use, in the actual work context, that will affect the efciency, effectiveness and/or satisfaction, with which users use a device. A usability evaluation of medical technology in a selection process should consist of three parts: 1. Finding usability problems, which generates a need for suitable usability evaluation methods that can be used in a clinical setting. 2. Estimating the severity of these usability problems, so that decision-makers can decide if the usability problems are severe enough to inuence the decision of which system to purchase. 3. Determining the possible clinical outcomes if these usability problems trigger a human error. This part has not been addressed in this paper.

3.2. Usability evaluation methods Usability evaluation methods (UEMs) can be divided into two groups: analytical and empirical. Analytical UEMs rely on the judgement of one or more evaluators and do not involve actual users. Empirical UEMs rely on data from actual users, either user performance or user opinions. All UEMs are able to nd usability problems, but there are four criteria UEMs should full to be suitable (Hartson et al., 2001): 1. Thoroughness: the methods should nd as many usability problems as possible when the user performs tasks with the evaluated system. 2. Validity: the methods should nd usability problems that are real problems in use. Common complaints about medical equipment are generally expressed as subjective opinions that are difcult to transform into information that is useful in a selection process. The

ARTICLE IN PRESS
E. Liljegren / International Journal of Industrial Ergonomics 36 (2006) 345352 349

complaints can be grouped under three headlines: Complexity of tasks, ambiguous and unclear markings and complex and illogical menu systems. Therefore, the validity of a method can be assessed according to how well the results address these informal complaints and the difcult to make errors component of usability. 3. Reliability: the results from the methods should be consistent and independent of the individual performing the evaluation. 4. Cost effectiveness: if evaluations are performed within budgetary or schedule restraints, the methods should also be as cost or resource efcient as possible when used by trained evaluators. In a purchasing situation, a fth criterion can be added: 5. Clarity: the results should be understandable and usable in a decision-making process, where the decision-makers are not trained ergonomists. This criterion has two aspects (a) how the results are presented and (b) what information the results provide. Four UEMs that are common and current are assessed according to these ve criteria: three analytical UEMs (hierarchical task analysis, cognitive walkthroughs and heuristic evaluation) and one empirical UEM (usability tests). Hierarchical task analysis (HTA) is a task description method based on an overall goal being re-described into the sub-goals and operations needed to attain these goals, and the conditions under which the operations have to be performed (Kirwan and Ainsworth, 1992). The process is known as progressive redescription and continues until a stopping criterion is met. However, in practice the redescription often continues as long as it is meaningful to do so. The result is a hierarchy of sub-goals and operations needed to perform a task successfully and a list of conditions called plans that relate to e.g. timing, choices and sequencing of operations. HTA is suitable for analysis of existing systems and can be used as a separate UEM to describe tasks for comparison of operations and task content. This way, the efciency component of the ISO denition of usability can be evaluated by comparing the number of operations needed to perform a task. Usability problems found with HTA are e.g. overly complex tasks and illogical task sequences (Fig. 2). Cognitive walkthrough (CW) is a method that evaluates the ease with which a specied or typical user can successfully perform a task using a given interface design (Polson et al., 1992; Lewis and Wharton, 1997). The purpose is to suggest to the designer where his or her design will create problems for the user and why these problems will occur. Usually, cognitive walkthroughs are performed as part of a design process but cognitive walkthroughs have properties that make them suitable for usability evaluation of nished systems as well. A CW analysis is performed in three steps: preparation, analysis and follow-up (Lewis and Wharton, 1997). In the preparation step the intended user background is dened, tasks are selected for evaluation, the

intended way to perform these tasks is described, preferably with HTA, and the user interface states for each step in the task are dened. When doing a CW analysis on an already existing system, the system is available for analysis, hence the user interface states for each step are known. In the analysis step, one or more analysts work through the correct or intended way to perform these tasks. For each action, the analyst asks four questions:

1. Will the user try to achieve the right effect? 2. Will the user notice that the correct action is available? 3. Will the user associate the correct action with the desired effect? 4. If the correct action is performed, will the user see that progress is being made?

Each question is answered with a success or a failure story that describes why or why not the user will succeed or fail. In the nal follow-up step, the results from the analysis are compiled into a list of possible usability problems. Heuristic evaluation (HE) is a usability inspection method that investigates whether a user interface complies with recognised usability principles commonly called heuristics (Nielsen and Mack, 1994). Table 2 presents a typical list of heuristics.

0. Change alarm limits via the menu system 1. Open correct submenu 1.1. Press MORE MENUES 1.2. Press ALARM CONTROL 1.3. Press ALL LIMITS 2. Perform desired 3. Finish desired task task 3.1. Press MAIN 2.1. Go to desired MENU 3.2. Press X in top parameter with left corner of sub-menu 2.2. Go to desired limit with 2.3. Change limit with 2.4. Press RETURN to exit sub-menu

Plan 3: do either 3.1 or 3.2

Fig. 2. Hierarchical task analysis used to describe the task Change alarm limits via the menu system, a task commonly done with patient monitoring systems.

Table 2 A typical list of heuristics, originally presented by Nielsen and Mack (1994) Simple and natural dialogue Speak the users language Minimize the users memory load Consistency Feedback Clearly marked exits Shortcuts Precise and constructive error messages Prevent errors Help and documentation

ARTICLE IN PRESS
350 E. Liljegren / International Journal of Industrial Ergonomics 36 (2006) 345352

A HE is performed in two steps, analysis and aggregation. In the analysis step, each evaluator inspects the user interface alone to avoid biases in the analysis. Usually 3 to 5 evaluators are necessary, with typically 1 to 2 h per evaluator (Nielsen and Mack, 1994). The evaluators are free to inspect any part of the interface, but should be guided by the usability heuristics when inspecting the interface. It is recommended that the evaluator goes through the interface twice. Each problem is documented. In the aggregation step the evaluators ndings are aggregated. The result is a list of usability problems in the user interface and how these usability problems violate the heuristics. The most commonly used empirical UEM is usability tests (UTs), described by e.g. Nielsen (1993) and Jordan (1998). A usability test, UT is performed in three stages: preparation, test and follow-up (Nielsen, 1993). The preparation stage consists of:

 

Selecting test users, where the test user group should have the same characteristics as the intended user group. In a clinical setting, the test users are assistant nurses, nurses and/or medical doctors. Choosing tasks to be evaluated, usually the selected tasks are relevant, important and representative of the way the system is used. Data to be analysed, the data can be objective e.g., time on task, errors, corrected errors, or subjective e.g. individual preferences.

The actual test is then performed, and the test users perform the tasks included in the evaluation. During the test, the test users are encouraged to think aloud. The test is usually recorded for further analysis. After the test, the test users are debriefed, either they ll in questionnaires about subjective ratings or interviewed. In the follow-up stage the collected data is analysed and compiled into a report. A suitable set of data for analysis has been suggested by Liljegren (2004) and includes task completion, errors and subjective data. 3.3. Assessment of usability evaluation methods according to criteria The thoroughness criterion was expressed as nding as many usability problems as possible when the user performs tasks with the evaluated system. HTA only partly fulls this criterion to begin with, as only usability problems related to the ISO component efciency can be found. Also, before any usability problems can be found, the HTA must describe how tasks are actually performed. It is usually recommended that data for an HTA be gathered from many sources. An initial HTA using the manual is often made, and this manual-based HTA is then compared to how the task is really done by users. Therefore, the thoroughness criterion is only fullled if the HTA is based on how tasks are actually performed. It

can be argued that HTA is then transformed into an empirical UEM but empirical UEMs depend on user performance data. Observation of users is in this case a part of the HTA analysis. HE does not focus on the users tasks but more on the user interface as a whole, so HE only partly fulls this criterion. HE can evaluate tasks, but not in the same structured way as CW or HTA. CW on the other hand is entirely focussed on the users tasks, so fulls the thoroughness criterion. However, most studies agree that HE usually nds more usability problems than CW but CW usually nds more real usability problems. Also, cognitive walkthroughs usually also nd the most serious usability problems (Sears, 1997). However, Jeffries et al. (1991) found that HE found more usability problems than CW. The validity criterion is also only partly fullled by HTA, as the method only nds efciency-related usability problems in the rst place. CW has an advantage over HE as CW is focussed on users task and aims at nding possible usability problems during task performance. UEMs that focus on user tasks are probably more effective when user interfaces are evaluated (Beer et al., 1997). Jeffries et al. (1991) also found that the problems found with HE were more severe than those found with CW. There is a trade-off between reliability and cost effectiveness, as the reliability of analytical UEMs increases with the number of analysts performing the analysis. However, in health care it might not be possible to have more than one or two analysts doing the analysis. This means that the structure of the methods is important for the reliability. In HTA, the combination of describing tasks using the manual, and observing users doing them, makes HTA full the reliability criterion. In CW, the analyst is guided by the questions and answers them with either a success story or a failure story. In HE, the analyst is deciding whether a user interface complies with recognised usability principles. There is more room for individual variation in HE than in HTA or CW. Therefore, HE fulls the reliability criterion to a lesser extent than HTA or CW. All three analytical methods are similar from a cost effectiveness perspective in that they require training and expertise to be used successfully. HE is usually regarded as easier to learn than CW and HTA. However, the better cost effectiveness of HE is set against the lesser reliability imposed by the structure of the method. The clarity criterion was expressed as the results should be understandable and usable in a decision-making process, where the decision-makers are not trained ergonomists. All methods can full the aspect of how the results are presented, as the results can be presented so that decision-makers can understand them. However, the results from HTA, CW and HE provide different information. In the case of HTA, the results are a description of how a task is performed and the number of operations required by the user to complete the task. CW presents a list is of potential problems the user might encounter when he or she performs a task. The results from

ARTICLE IN PRESS
E. Liljegren / International Journal of Industrial Ergonomics 36 (2006) 345352 Table 3 Fullment of usability evaluation criteria by the described usability evaluation methods (F fulls criterion, p fulls criterion in part) Method Criterion Thoroughness Hierarchical task analysis Cognitive walkthrough Heuristic evaluation Usability tests p F p F Validity p F p F Reliability p p p F Cost effectiveness F F F p Clarity F F p F 351

HE are a list of where the user interface violates heuristics or usability guidelines. When the list of staff complaints (complexity of tasks, ambiguous and unclear markings and complex and illogical menu systems) is compared to the results generated from these three methods, the results from HTA and CW are more in line with the complaints. Therefore, HTA and CW fulls both aspects of the clarity criterion whereas HE only fulls the rst aspects. UTs are the most comprehensive way to evaluate the usability of any system. The thoroughness is usually high, especially if more test users are included. A minimum of ve to six test users is recommended and then around 7580% of usability problems are discovered (Virzi, 1992; Nielsen, 1993). It is also acknowledged that the most serious usability problems are discovered in the two to three rst test users. The validity criterion depends on what tasks and what test users are included in the UT but usability problems found in a UT are real problems encountered by actual users, albeit in a test situation. Provided the selection of test users and test tasks is suited to the actual use situation, the validity criterion is fullled. The results from UTs are user performance so the reliability mainly depends on user experience. When equipment is purchased and subjected to clinical trials, test groups are usually formed. The test users should then be from this test group. The cost effectiveness of UTs is a trade-off with the thoroughness and validity criteria. Even though UTs require test users and post-test analysis of the tests, the thoroughness and validity of UTs mean that UTs are cost effective provided the tests have been properly prepared. A post-test analysis without previously chosen data to be analysed is not cost effective. The clarity of UT is high, as both aspects are fullled. Table 3 presents the fullment of criteria. 3.3.1. Assessment of usability evaluation methods according to questionnaire results The results from the questionnaire show that UEMs should primarily focus on evaluating the component difcult to make errors, followed by easy to learn, efcient to use and easy to remember. UTs can be used to evaluate a system according to all these components, as these components can be built into the evaluated tasks. Cognitive walkthroughs primarily evaluate difcult to make errors, as the analysis generates a list of potential

problems, i.e. opportunities to make errors. Cognitive walkthroughs can also be used to evaluate the component easy to learn, as the original purpose was to evaluate user interfaces from a learning perspective. As originally formulated, is based on the theory of learning by exploration. The components efcient to use and easy to remember are not evaluated in a cognitive walkthrough analysis. HTA primarily focuses on the component efcient to use, as the outcome are descriptions of how tasks are performed, i.e. the number of operations required to successfully completing a task. Heuristic evaluation on the other hand does not focus on any of the ve usability components, but on whether the user interface complies with usability principles. 4. Estimating the severity of usability problems The severity of a usability problem is a combination of three factors (Nielsen and Mack, 1994): 1. Frequency, i.e. how often the usability problem occurs. 2. Impact, i.e. how easy it will be for the user to overcome the usability problem. 3. Persistence, i.e. will users overcome it after the rst time or will they repeatedly be affected by the usability problem. The frequency of usability problems identied with HTA can be found by counting the number of times this problem appears in the HTA description. The other two factors are not addressed by HTA. As HTA descriptions are also used in the CW analysis, CW also provides information about frequency of usability problems. CW can also be used to estimate the impact and persistence of a usability problem as the CW analysis includes a question about feedback. Overcoming a usability problem is a learning procedure and the original purpose of CW was to evaluate how easy a user interface is to learn. The impact and persistence of a usability problem found in a CW analysis can be estimated from e.g. error messages or feedback and how well this helps the user identify to overcome a problem. However, only the correct or intended way to do tasks is analysed in a CW analysis. Therefore, the impact and persistence are usually not assessed, unless the analysis is also extended to cover these parts of a task. Question 4 in the CW analysis

ARTICLE IN PRESS
352 E. Liljegren / International Journal of Industrial Ergonomics 36 (2006) 345352 Cook, R.I., Woods, D.D., 1996. Adapting to new technology in the operating room. Human Factors 38 (4). Cook, R.I., Potter, S.S., Woods, D.D., McDonald, J.S., 1991. Evaluating the human engineering of micro-processor-controlled operating room devices. Journal of Clinical Monitoring 7, 217226. Cook, R.I., Woods, D.D., Howie, M.B., Horrow, J.C., Gaba, D.M., 1992. Unintentional delivery of vasoactive drugs with an electromechanical infusion device. Journal of Cardiothoracic and Vascular Anesthesia 6 (2), 238244. Goodman, C.S., Ahn, R., 1999. Methodological approaches of health technology assessment. International Journal of Medical Informatics 56, 97105. Hartson, H.R., Andre, T.S., Williges, R.C., 2001. Criteria for evaluating usability evaluation methods. International Journal of HumanComputer Interaction 13 (4), 373410. rden (Use of Hellstro m, M., 1995. Teknikanva ndning och tillbud i sjukva technology and incidents in health care), paper at the Medical Technology 95 conference, AIC Conferences January 1718 (in Swedish). Hyman, W.A., 1994. Errors in the use of medical equipment. In: Bogner, M.S. (Ed.), Human Error in Medicine. Lawrence Erlbaum Associates, Inc. ISO 9241-11, 1998. Ergonomic requirements for ofce work with visual display terminals (VDTs), Part 11: Guidance on usability. Jeffries, R., Miller, J.R., Wharton, C., Uyeda, K.M., 1991. User interface evaluation in the real world: a comparison of four techniques. In: CHI Conference on Human Factors in Computing Systems. ACM, New York, pp. 119124. Kirwan, B., Ainsworth, L.K., 1992. A Guide to Task Analysis. Taylor & Francis, London. Klatzky, R.L., Ayoub, M.M., 1995. Health care. In: Nickerson, R.S. (Ed.), Emerging Needs and Opportunities for Human Factors Research. National Academy Press, Washington, DC. Leape, L.L., 1994. Error in medicine. JAMA 272 (23). Lewis, C., Wharton, C., 1997. Cognitive walkthrough. In: Helander, M., Lander, T.K., Prabhu, P. (Eds.), Handbook of HumanComputer Interaction. Elsevier Science BV, London. Liljegren, E., 2004. Increasing the usability of medical technology, methodological considerations for evaluation. Ph.D. Thesis, Department of Product and Production Development, Chalmers University of Technology, Gothenburg, Sweden. Liljegren, E., Osvalder, A.-L., Dahlman, S., 2000. Setting the requirements for a user-friendly infusion pump. Proceedings of the Human Factors and Ergonomics Society, 44th Annual Meeting, San Diego, USA, July 30August 5, 2000. Nielsen, J., 1993. Usability Engineering. Academic Press, San Diego. Obradovich, J.H., Woods, D.D., 1996. Users as designers: how people cope with poor HCI design in computer-based medical devices. Human Factors 38 (4), 574592. n, M., 1992. Work organisation, work Persson, J., Ekberg, K., Linde environment and the use of medical equipment: a survey study of the impact on quality and safety. Medical and Biological Engineering and Computing 31, HTA20HTA24. Polson, P., Lewis, C., Rieman, J., Wharton, C., 1992. Cognitive walkthrough: a method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies 36, 741773. Reason, J., 1990. Human Error. Cambridge University Press. Virzi, R.A., 1992. Rening the test phase of usability evaluationhow many subjects is enough? Human Factors 34 (4), 457468. Weinger, M.B., 1999. Anesthesia equipment and human error. Journal of Clinical Monitoring and Computing 15, 319323.

(If the correct action is performed, will the user see that progress is being made?) can be re-formulated to also include erroneous action, e.g. If the erroneous action is performed, will the user see that the action is wrong? This reformulated question can then be included in an analysis. UTs address all three factors. The frequency and impact of a usability problem are two of the most common objective data gathered from a UT. The frequency is estimated from the number of occasions a particular usability problem occurs. The impact of a usability problem is estimated from e.g. the test users reaction. UTs can also be used to determine the persistence of a usability problem if the test tasks include the problemrelated task more than once. This shows whether the test user overcomes the problem the second time it occurs. Heuristic evaluation does not address the three factors explicitly. The frequency of a usability problem can be estimated from how many times a user interface does not comply with the usability principles. The impact and persistence of a usability problem are not addressed at all. 5. Conclusions about suitable methods Technology plays an increasingly important role in modern health care. This increases the importance of usability when new medical technology is purchased. Suitable UEMs should full the ve criteria of thoroughness, validity, reliability, cost effectiveness and clarity, and focus primarily on the usability component difcult to make errors. Also, they should address the severity, impact and persistence of usability problems. Usability tests are the primary usability evaluation method to be used in selection processes at hospitals. UTs full all criteria and usability problems related to all components of usability can be found, and their frequency, impact and persistence assessed. Cognitive walkthrough and hierarchical task analysis are suitable complements to UTs. Cognitive walkthrough can be used to nd usability problems related to the components difcult to make errors and easy to learn, and their frequency can be assessed. The impact and persistence of usability problems can also be estimated. HTA can be used to nd usability problems related to the usability component efcient to use and their frequency can be assessed. The impact and persistence of usability problems can however not be estimated. References
Chapanis, A., 1991. Evaluating usability. In: Shackel, B., Richardson, S.J. (Eds.), Human Factors for Informatics Usability. Cambridge University Press, Cambridge, pp. 359395.