Unit 3.

Think-aloud Usability Testing

 3.1 Basic Issues In Think-Aloud Testing
 3.2 How To Conduct A Think-Aloud Test

 3.3 Think-Aloud Testing Vs Heuristic

Evaluation
 Resource
 In the ftp server

 Audio and video clips

CDForThinkAloud
 Transcripts:
SSD4_TA_ALL_transcripts_v1.4.2.doc
 3.1 Basic Issues in Think-Aloud Testing

 3.1.1 What Is Think-Aloud Usability

Testing?
 3.1.2 Ethics for Empirical Studies
 3.1.1 What Is Think-Aloud Usability Testing?
 an empirical technique for assessing the
usability of a prototype of an interface
 "may be the single most valuable usability
engineering method"-- Nielsen, 1993
 How to do?
 ask a user to think-aloud while performing a task
on your system
 you watch silently and learn
 how the user thinks about the task
 where the user has problems using your system
 3.1.1 What Is Think-Aloud Usability Testing?
 root in psychology research — two
observational techniques
 Think-Aloud Protocol Analysis
 Understanding Verbalization

 Understanding Working Memory

 Critical Incident Analysis

 Think-Aloud Protocol Analysis

 Cognitive psychology
 interested in understanding how people solve problems
 discover and understand the details of

 what information people pay attention to

 how they represent that information

 how they bring prior knowledge to bear

 what transformations they make to information in the

course of solving some puzzle or performing some
task
 Think-Aloud Protocol Analysis

 think-aloud protocol analysis: been in

use for 40 years
a method of understanding these details of
thought
 In cognitive psychology research, this method
has two parts
 collect think-aloud data (protocols)

 analyze the data by building a model of it

(usually a computer simulation).
 1st parts to this method
 Collecting think-aloud data
 proven to be extremely useful in understanding
the usability of computer systems
 cognitive psychologists have studied it quite
thoroughly, and UI design can learn a lot from
their studies
 two parts to this method-2
 making a formal model( 形式模型 ) of the
data and processes
 has not been as useful in most UI design
 although there have been some dramatic
successes
 The analysis step of think-aloud data in UI
design uses the critical incident technique
 Understanding Verbalization

 three types of verbalizations of thoughts

 Talk-Aloud (Type 1)
 Think-Aloud (Type 2)

 Mediated processes (Type 3)

 these three types can be understood by

thinking about what is in Working Memory
(WM).
Understanding Working Memory
 Understanding Working Memory

 WM stores several types of things

 It stores all the
results of perception once those
things have been understood by the person,
 E.g., The picture of the Lion in last slice

 WM also stores all the information that is

brought in from long term memory (LTM) to
solve a problem
 WM also holds all the intermediate states in a
problem solution, information that is figured out
along the way to the solution
 Understanding Working Memory

 WM stores several types of things

 WM holds a lot of clues as to what a person was
thinking about as they solved the problem or
performed a task
 On the other hand, WM does NOT hold the
processes that are used on the information
 WM holds to generate those intermediate
pieces of information.
 Those processes are used by the cognitive
processor but are not explicitly represented in
WM.
 Theory behind Think-aloud protocols
 People can verbalize( 描述 ) the linguistic
contents （可用词 语表达的内容 ） of
their WM
 A lot of information in WM is already in
linguistic form (expressed in words)
 Type 1 protocol:
 to"talk aloud," — get these pieces of
information to come "out of their mouth" right
after they enter WM
Type 1 protocol
a simple addition problem: 2+4 =?
 Cognitive psychologists have shown that:
 Cognitive psychologists have shown that
 for the
most part, asking a person to "talk aloud"
as they work on a task does not
 change their thinking strategies or

 slow them down in their thinking.

However, much of the information is not linguistic in nature

 with modern GUI computers systems

 may include information about space, color, time,
or other things that are not naturally spoken
about with words
 This is not to say that they can’t be
expressed with words, people have to learn
a vocabulary
 how to translate the perceptual information they
are getting into that vocabulary
 Learn how to express those new thing
 An example
 wine tasting
 if you are not familiar with the skill, you will not
be able to translate the sensations on your tongue
into words
 you will have sensations on your tongue when
tasting the wine (non-linguistic information),
 you could learn the translation into words given
enough time and exposure to the "language."
Type 2 protocol
 Another example
 do a jigsaw puzzle picturing a beach
 Thinking aloud as you do the jigsaw puzzle
would be as shown in the figure of next slice
Another example
 What Cognitive psychologists tell us?
 asking people to translate everything they are
thinking into words ("think aloud") does not
change the way people think about problems
 it does slow them down （ ？ ）
 In fact , some time it will speed them
up(according to the book of Usability
Engineering )
 This is the type of protocol most often used in
usability research
 can get a lot of information about the quality of
the UI
 The Type 3 protocol
 The Type 3 protocol is when you ask
someone to verbalize, but make some sort
of demand on them to add more processing
to the information.
 This type of protocol is not recommended
for UI design because psychologists have
shown that
 this does change the way people think as they
solve problems,
 in addition to slowing them down.
the Type 3 protocol
asking a person doing the jigsaw puzzle to explain how he
found each piece.
 Conclusion of type 3
 In type 3:
 the information state reported was one that the
person never passed through without the
instruction to "explain,"
 the explanation of the process was not what she
actually did.
 Conclusion of type 3
 For these reasons
 Type 3 instructions should be avoided
 if you notice that a person seems to
 be explaining rather than just reporting what they
think
 you need to put them back on the right track.
 Critical Incident Analysis

 Collecting the data.

 by video tape
 by action-and-voice software

 Analyst role in designing the computer design.

 analyze this data to decide how to improve the computer
system design
 Critical Incident Analysis

 History
 developed during World War II
 develop procedures for the selection and classification of
aircrew
 This method also has parts, involving several different
people in different roles
 collecting the data and

 analyzing it

 UI design draws from both parts of this technique for its

use.
 original critical incident technique
 observers report critical incidents that they
witness in the course of performing a task
 E.g., combat veterans reported actions of officers
they observed during combat missions.
 These observations are usually gathered
through … after the task is performed
 interviews or

 questionnaires.
 original critical incident technique
 Concurrent recording of these observations
is recommended in the original papers, but
they were acknowledged to be impractical
in many of the situations the critical
incident technique was used
 e.g., during combat missions.
 The observations gathered are then
categorized and interpreted by analysts and
put into a final report that summarizes the
findings.
 the general procedure for the critical incident
technique
 Someone performs a task in the real world
 e.g., combat missions.
 An observer (who may or may not be the
"someone" performing the task) reports critical
incidents after the fact in interviews or
questionnaires administered by an analyst.
 The analyst categorizes and interprets the
observations.
 The analyst writes a summarizing report of the
data and interpretations.
 modified procedure of Usability studies in UI design have
 A user thinks aloud as he or she performs a task using a
prototype of the computer system being evaluated,
usually in a laboratory setting, usually videotaped or
using screen-and-voice capture software.
 An analyst (who is not the user performing the task)
looks at the recording of the think-aloud protocol
session and reports critical incidents using the UAR
format.
 The analyst categorizes and interprets the observations.
 The analyst writes a summarizing report of the data
and interpretations.
 Differences between the two procedure
 users take the role of the "someone
performing the task,"
 the usability analyst takes both the roles of
the observer and the analyst in the original
critical incident technique.
 Watching the videotape and writing up
UARs takes the place of after-the-fact
interviews or questionnaires.
 Definition of the term “critical incident” （关键事
件）
 The definitions used in the 1954
Psychological Bulletin paper are as follows:
 “By incident is meant any observable
human activity that is sufficiently complete
in itself （本身） to permit inferences
and predictions to be made about the
person performing the act."
 Definition of the term “critical incident”
 “To be critical, an incident must occur in a
situation where the purpose or intent ( 目
的 ) of the act seems fairly clear to the
observer and where its consequences are
sufficiently definite （明确） to leave
little doubt concerning its effects."
 "Such incidents are defined as extreme
behavior, either outstandingly effective or
ineffective with respect to attaining the
general aims of the activity."
 The three important concepts mapped into UAR format

 The three important concepts in these

definitions can be mapped into our format for
UARs
 Map table
 What a usability study includes?
 the procedure for doing a usability study combines
 the best of think-aloud protocols

 the critical incident technique.

 It gets at users’ thoughts

 what they pay attention to,

 what information they miss,

 what prior knowledge they bring to the task, and

 what is puzzling or clear to them.

 it also provides a tractable way to

 record important data in critical incident UARs


 3.1.2 Ethics For Empirical Studies
 Testing the Interface,not the participant
 Voluntary Participation

 Maintain Anonymity

 Informed Consent

 Laws
 You are testing the Interface, Not the Participant

 Your attitude
 you are testing the interface, not the participant
 including

 everything you do with the participants in your

study
 everything you do with the data they generate

 come up in every aspect of

 the study ethics

 procedures

 data analysis
 You are testing the Interface, Not the Participant

 Mantra : The user is not like me.

 You are the system designer
 You can never think like a typical user
 You know too much about the system
 You cannot prevent that knowledge from helping
you use the system
get information about how real users will deal with the system

 To get information about how real users

will deal with your system,
 must do empirical testing with people who know
as much or as little about computers and your
specific system as you expect your real users will
know
 This leads to the inescapable conclusions
that
 participants in your study are extremely
important
 the Participant

 They are always giving "good data" whatever

they say or do, Why?
 Since whatever they say or do
 comes from a basis of knowledge like that of
your eventual actual users
 gives you a valid indication of what your
actual users will think about your system.
 You are testing the Interface, Not the Participant
 Kinds of people to use the Software
 Participant will not pay attention always

 What we do?
 No matter what the participants do,
 always ask yourself what about your system
indicate them in that direction,
 rather than blaming the participant for not
knowing enough or not paying attention or not
reading.
 You are testing the interface, not the participant.
 Voluntary Participation

 Participation must be voluntary

 Do not put any pressure on people to continue in a study
once they have started
 Or It may get useless data

 It is unlikely when sth. objectionable found about

participating, if so , let them choose quit or not
 it is important that you create an atmosphere in the
testing situation
 Tell the participants that they are allowed to stop at
any time.
 Pay bonus to the people
 Watch carefully and stop at proper time
 Be sensitive to stop
 to be sensitive to the many ways people express a desire
to stop a session
 Some people would never say something as direct as "I
want to stop now.“
 But they may express a lot of negative emotions

 "I'm so stupid I'll never be able to finish this,“

 "This seems like it will go on forever, stupid

&%@#$* computer!!!“
 Watch carefully and stop at proper time
 What you must do
 gently ask, "Would you like to stop now?"
 Do not wait for tears or for the user to throw the
equipment across the room!
 When they are in highly emotional state, you will
not get any additional usable data from them
anyway
 it is better for both of you if you diplomatically
terminate the session
 remember to ask yourself later what in your system
brought this user to this emotional state
 Maintain Anonymity

 It is your responsibility to maintain the anonymity of

your participants, do as follows,
 Store data under a code number
 Map the number and name
 Do not record participants face
 Do not show videotapes without participants consent
Mapping between consent participant names and numbers
Name Number
Jane Doe U1
Fran Diaz U2
Chris Smith U3
... ...
 Informed Consent
 is fundamental to every kind of experiment using
human participants.
 Ethically Obligated to tell the particiant
 what the experiment is about
 what procedures will be used
 what compensation they will receive
 what they can do if they object to something in the study
 because their participation is voluntary, they are free to
stop and leave at any time.
 Informed Consent
 Written Consent: be sure that you have
informed each participant of these things,
 give them a written consent form to read and sign
 build enough time into your testing schedule to allow
them to read it at a leisurely pace
 ask questions if they have any problem
• When parents or guardians must also be
informed and sign a consent form
– Children
– others who cannot legally give consent (e.g., mental
patients),
– people under severe stress (e.g., severely ill patients,
incarcerated prisoners),
 Example of Informed Consent


 Laws
 laws that govern the use of humans in empirical
studies.
 consist of federal regulations
 tell you
 when they apply and
 what you have to do to comply with them
 what types of observations are exempt from the
regulations
 Institutional Review Board( 评审 委员会 )
 Be dictated when an organization must form
"Institutional Review Board"
 to oversee all observations of people
 all universities that receive federal funding must have
IRBs
 Laws
 Most recent version is available in the website
 the regulations for the "Protection of Human Research
Subjects,“
 http://www.access.gpo.gov/nara/cfr/cfr-
retrieve.html#page1
 In USA,it is your and your organization's
responsibility to understand how these laws
apply to your work
 In otherconutry, it is your responsibility to
determine what the relevant laws are and apply
them
 All the steps like maintaining anonymity and
informed consent has to be followed even if your
are exempted from government regulations
 3.2 How to Conduct a Think-Aloud Usability Test

 3.2.1 Defining the study’s Framework

 3.2.2 Choosing what to Observe
 3.2.3 Preparing for Observation
 3.2.4 Introducing the Participants to the
procedure
 3.2.5 Conducting the observation
 3.2.6 Analyzing the observation
 3.2.7 Finding possible redesigns
 3.2.8 Writing a summarizing report
 3.2.1 Defining the Study’s Framework
 What should our development team do?
 come to a consensus as to
 the purpose of the system
 the usability observation
 WHY? It will influence choices in the more actual
steps.
 list questions (in next slice)
 asked and answered them for several times earlier
in the development
 right before usability tests are conducted is a good
time to ask them again
 summarize the answers
 Problems asked yourself
 W1: What problem is the system trying to solve?
what work is it trying to support ？
 help in choosing the test suite of tasks.

 W2: What level of support will the user have?

 training to use this system

 documentation

 online help

 other resources

 help in developing materials, choosing tasks, and

in setting up a realistic situation.
 Problems asked yourself
 W3: What type of usage to evaluate?
 Walk-up-and-use/first-time use?

 Users skilled in the task domain using this particular

system/for the first time?
 Exploratory use with no time pressure / goal-directed
use under time pressure?
 Any other aspects?

 The answer to this question will influence choice of

 tasks

 participants

 definition of what a problem is in this system.

 Problems asked yourself
 W4: What usability goals for this system?
 That 90% of users can accomplish a simple task
within 3 minutes with no training?
 That half of the people trained in using this system
will perform with less than one recoverable error per
task?
 help in choosing tasks and defining the criteria for
identifying problems ／ good features of the system.
 About frameworks
 not the only framework that would be
reasonable
 one of several good approaches.
 example: the test of Date/Time control panel
 The Date/Time control panel : study
framework
 supports setting a computer's date, time, and time
zone.
 It is particularly useful to people traveling with
laptops.
 The control panel should require no training or
online tutorial: all owners of computers should be
able to use it intuitively (a walk-up-and-use
situation).
 example: the test of Date/Time control panel
 The Date/Time control panel : study
framework
 Every user should be able to complete the tasks
of setting the date, time, and time zone.
 It is not critical that there be no errors committed
in performing the task, but no complete task
should take longer than 3 minutes.
 3.2.2 Choosing What To Observe
 The first thing
 choose the tasks you want to give participants
 Note: The choice of tasks is influenced by several
things
 the content of the tasks
 the need for training to do the tasks,
 the duration of the tasks,
 integration of the tasks

 E.g.,
 Choosing task to test the date/time control panel
 The Content of the Tasks
 pick tasks that reflect actual or expected
use of the system
 If asimilar system is already in place (even a
paper-based system), actual tasks can be
determined by observing what people already do
with the existing system.
 The Content of the Tasks
 pick several tasks from the many you
identify and include them in the test suite.
 When You identify many real tasks through
observation, interview, or sometimes data
collected for other purposes
 include the most frequent tasks in the test
 getdata on how your system supports the tasks
users will do on the most regular basis
 The Content of the Tasks
 include tasks that cover the range of
functionality of the system
 exercise every part of it.
 If you do not do this, then the entire system is not
tested, only the subset represented in the tasks
you chose.
 Make sure you include not only tasks that create
things from scratch but also delete or modify
existing things.
 The Content of the Tasks
 include error-recovery tasks
 e.g., put the
person in a situation where they
would be had they made an error and ask them to
recover from it.
 include in the test suite very important
tasks, though possibly infrequent
 E.g., emergency procedures are used
infrequently, but should be tested because they
are safety-critical.
 The Need for Training to Do the Tasks
 the training must be counted as part of the
test suite of tasks
 If require training to accomplish those tasks.
 The task set becomes
Should be Rather than
1."Learn to do 1."Task A."
Task A“
2."Task A,"
 The Need for Training to Do the Tasks
 This allows you to test training materials as
well as the system
 test training materials :

 paper-based reference materials

 Web pages

 lectures

 online tutorials.
 The Duration of the Tasks
 users cannot do tasks for a relatively long
time, E.g.,
 more than an hour at a time, or
 more than two hours on a single day.

 More than this is usually too tiring for

participants.
 Design your test suite to allow for
 hourly breaks and

a break for the day after two hours.

 The Duration of the Tasks
 This duration includes the training to do
the task.
 If thetraining is on one day and the task
performance is on the next, you will have to
review the training on the second day to refresh
the user's memory.
 The Integration of Small Tasks
 include tasks required the integration of
several system features.
 E.g., do not just have small tasks that
 create a new object,

 but

 start the user out with a system full of objects

already created
 make them navigate through those objects,

 modify them

 create new ones

 The Integration of Small Tasks
 include tasks that test the integration of
your system with other systems they might
use. E.g.,
 how might the users include the results of
something your system did into a report
written in a word processor?
 How might they include those results in email?
On a Web page?
 How might someone send them information
through email that needs to be used in your
system?
 Example: Choosing Tasks to Test the Date/Time Control
Panel
 The Date/Time control panel can only support a
very limited set of tasks.
 setting the time,
 setting date
 setting time zone of a person's computer.

 we see that
 the usability target in this case will not require user
training and
 that the tasks will last no more than a few minutes .
 These facts make it reasonable to require participants to
carry out all tasks in one single test session.
 the Date/Time control panel no tasks would integrate
with other applications.
 Example: Choosing Tasks to Test the Date/Time Control
Panel
 The suite of tasks to use in testing the usability of
the Date/Time control panel would include the
following:
 Set the time
 Set the date
 Set the time zone
 3.2.3 Preparing For Observation

 Note :
 Before running any actual think-aloud sessions, there are
several preparations need to make.

 Contents
 Setting Up a Realistic Situation for Data Collection
 Writing Up the Task Scenarios

 Practicing the Session

 Recruiting Users
 Setting Up a Realistic Situation for Data Collection
 This situation will differ depending on the
system you are testing.
 If system is a desktop system for a PC, an office-like set-
up is fine.
 traditionally set-up for usability labs in large
corporations, Microsoft and Sun.
 Setting Up a Realistic Situation for Data Collection
 For many more types of computer systems,
realistic set-ups take on new characteristics.
 Testing the interface to a microwave oven is best done in
a kitchen-like setting.
 Testing a PDA can be done almost anywhere, but you
need to pay attention to lighting conditions that will be
found in offices, in cars, outdoors, and on public
transportation.
 Navigation devices to be placed in cars need to be tested
while driving (or in driving simulators).
 Setting Up a Realistic Situation for Data Collection
 The variety is endless today with the ubiquity of
computer systems.
 In general, think carefully about how your
system will eventually be used and set up the
situations to match those uses as closely as
possible.
 Setting Up a Realistic Situation for Data Collection
 Difference situations , difference the data-collection
process.
 traditionally capture the actions and voice of the users :
 fixed video cameras
 microphones.
 In the more mobile situations, more dynamic data-
capturing techniques must be employed.
 wireless microphones
 operate a camcorder and follow the user around as
they work.
 Some new software capture and playback the user's
actions and voice with a portable computer.
 Setting Up a Realistic Situation for Data Collection
 each situation has its own requirements, the solutions
will vary. capture at least :
 what users do with your system (including observations about
what is being displayed to your users)
 what users say as they think aloud, synchronizing what they
say with what they do.
 When record the user's actions, you will need record the
time during the session.
 some video recorders can put a timestamp on the signal.
 screen-capture software have a system clock within the area of the
screen when the software is capturing.
 Writing Up the Task Scenarios
 Write a statements of tasks for the
participants to perform. How to do?
 each task is written on a separate sheet of paper
 given to the user one task at a time.

 Depending on the task, these descriptions are

either purely in words, or they include pictures or
diagrams to help describe the task.
 In addition to being described individually, a
series of tasks usually is accompanied by a cover
story that links the tasks in the series into a
meaningful narrative.
 Practicing the Session
 Do not underestimate the difficulty of
making a session run smoothly.
 A saying among experimental psychologists
applies to usability testing: "Subjects are
like pancakes; you always have to throw
the first few away."
 Practicing the Session
 take time to get everything to work in concert.
 The hardware and software must perform
normally.
 The written instructions must be clear, or you will
meet problem of comprehending the instructions
instead of problems with the system you are
testing.
 Giving of verbal instructions must be smooth;
otherwise, you will confuse the participant.
 The data-capture equipment must work.
 The solution to all these problems is practice,
Practice, PRACTICE.
 Practicing the Session
 Practice first with yourself
 which can at least debug the procedures of
 what software must be running when,
 what windows must be open, etc.
 Walk through the entire session,
 reading every piece of information,
 doing every task.
 Write everything down in a script so you can
reproduce the session the next time you run it.
 Practicing the Session
 Practice first with yourself
 Make sure your script includes reminders for
things you should do
 e.g., to make sure there is a tape in the video
camera or that the sound level is adequate.
 If you find problems with your procedure when
you walk through it yourself, fix those problems
before bringing in anyone else to practice on.
 Practicing the Session
 Practice next with a friend:
 different eyes will see things you did not catch yourself
in your instructions, tasks, or equipment set-up.
 If your friend has enough time to spare, again, walk
through everything in the entire session.
 fix any problems your friend finds before the
next practice session.
 Practicing the Session
 Finally practice with someone who has a similar
background to the users you will be using in
your study.
 By this third session, you might expect all the software
and hardware to work,
 but don't be surprised, if yet another set of hands and
eyes finds something else wrong!
 This is an opportunity for you

 to practice your delivery of the instructions

 to determine whether the instructions themselves are

comprehensible to someone who is much like your
users.
 Practicing the Session
 Purpose in conducting these tests is to find
problems with your computer system
 If you uncover any problems while running these
tests of your procedures, write them up in UARs.
 It is not "cheating" to discover system problems
during this preparatory phase of the testing.
 Don't throw away valuable information just
because you weren't expecting to get any during
the preparation phase.
 Recruiting Users
 different systems require different types of users
to provide data.
 As different systems require different situations in order
to provide a realistic test
 The most important consideration is
 that the participants in your usability tests have the same
background knowledge as your eventual users will have.
 Recruiting Users
 E.g.,
 if your system is for airline pilots to program the
destinations of their aircraft, you need to use airline
pilots as participants in the usability tests
 if current airline pilots are not available, retired airline
pilots would be the next best thing.
 Educational software should be tested by students in the
grade where the software fits into the curriculum.
 Medical systems should be tested by doctors or nurses .
 the Disney Company tests their new virtual reality
games with visitors to their existing theme parks.
 some users are more difficult to obtain than
others, but it is crucially important to get at least
a handful of participants who represent the
actual user group
 Example: Preparing for a Think-Aloud Usability Test of the
Date/Time Control Panel
 chose a screen-capture program to record both
the user's interactions with the system and their
voice.
 Make sure that the system clock appeared on the
bottom menu bar of the screen, so we would
always have a record of the time of the
interaction.
 wrote up two of the tasks. one on each page.
 Imagine the following scenario
 You are a new reporter for the Pittsburgh Post
Gazette. There has been some recent unrest in
the Philippines, and you have volunteered to go
to Manila to cover the story. You are waiting to
board your flight at Pittsburgh International
Airport.
 You have a few minutes to spare before your
flight. Using the Date-Time control panel on
your laptop computer, adjust the time zone to
the correct one for your destination.
 When you arrive, you discover that the battery
in your laptop is dead. Re-set the time on your
computer to 2:35 PM, which is the current local
time.
 What to do in this example
 We composed a script for the test, which you will
see and hear in the next section.
 Since we were using these recordings for course
materials, as well as using them to find usability
problems in the Date/Time control panel, our
script had to be a little different from what you
would normally use: we stated specifically that
the recordings would be used in a class.
 We also composed a consent form specific to this
testing session , which also mentioned the
classroom use of the recordings.
 Finally, the analyst rehearsed the script six times
before collecting any actual data.
 What to do in this example
 Any potential owner of a computer could
be a participant in our study.
 ask participants about their computer
background.
 Questions about computer use were written into
the script.
 for run the observations at the airport, with
travelers who had quite some time to wait before
their next flights.
 Sample consent form
 Sample consent form
 3.2.4 Introducing the Participant to the Procedure

 Describe the purpose of the study in

general terms
 Train the user to “Think-Aloud”
 Explain the rules of the observer
 Example: Introducing Participants to a
Think Aloud-usability Test of the
Date/Time control panel
 Recording of participant briefings
 When firstly meet your test participant

 give the participant enough

information for him or her to feel
comfortable doing the think-aloud
 describe the purpose of the study
 explain how to think aloud
 provide time to practice the technique
 explain the rules of the observation
 Describe the Purpose Of Study
 Give a brief introduction to the entire study with the
following elements.
 Introduce yourself (name and title).
 Introduction of Organisation and Purpose
 Tell the user
 the goal of the particular study
 testing the Computer, not testing the user
 Participation is purely voluntary
 it is OK if they want to stop at any time(check the computer
system at this time)
 Signature in consent form
 Show the Equipment and explain how to use
 Show the participant how you are going to record their
voice
 When done, ask participant if OK to start recording
 if so, start the recording device.
 If "no" ask them
 what else they need to know
 answer any questions.

 If they persist in not wanting to be recorded, tell

them their actions and voice must be recorded,it
is a requirement in the study.
 Excuse them if they continue to refuse.
 Train the User to “Think Aloud”
 Training the user is a bit awkward at first
 Should give them practice doing on tasks
 Ericsson and Simon’s Protocol Analysis
 successfully to elicit good think-aloud behavior
with users.
 adapt these instructions to your task situation,
script it, and read it almost verbatim (word by
word) to your participants.
 Train the User to “Think Aloud”
 How to do the observation
 Ask the user to perform a sample Think-Aloud
 Ask the user to verbalize things he is searching
for and see
 Train the User to “Think Aloud”
 If the user stop , request him to talk again
 During the practice session, the participant stops
talking for 5 to 10 seconds, say "Please keep
talking."
 Don't say "What are you thinking" or "Please
explain what you are doing"
 because these phrases tend to encourage
people to switch to Type 3 verbalizations
(explaining or filtering what they are thinking),
 Explain the Rules of the Observation
 before moving on to the observation phase of the
usability test, explain the "rules" of the
observation.
 Cannot answer questions during the observation

 Can ask questions but you'll record them and

explain them after observation
 Prompt the user to continue if user keeps quite

 After explaining
 ask the participant if any questions exist about
the think-aloud procedure or anything else so far.
 Example: Introducing Participants to a Think-Aloud
Usability Test of the Date/Time Control Panel
 For preparation the test of the Date/Time control panel,
incorporated all the information into a script.
 The analyst follow the script when testing the software.
 Following the script are several recordings that capture
our delivery of this introductory material to a typical
user.
 As with all informal observations, the analyst may not
have adhered perfectly to the script.
 Deviations or omissions from the script in the recordings can be
found
 Typically experienced analysts are able to follow a script closely
without appearing to be unnatural or to lack spontaneity.
 Recordings of Participant Briefings
1. The first recording is of the analyst describing the purpose of the
study in general terms. Refer to that subsection of this module,
the script above, and the consent from, as you view it. Think
about what the analyst is doing according to the procedures
presented in this module and what she is doing that deviates from
those procedures.
 Click Introduction_&_Consent to view the recording.
 After you have viewed the recording, click Intro_Critique for our critique of
the recording.
2. The second recording is of the analyst introducing the think-
aloud technique. Refer to the relevant previous subsection, and to
the script above, as you view it. Think about how the analyst is
following the procedures presented in this section and what she is
doing that deviates from those procedures.
1. Click How-to-TA to view the recording.
2. After you have viewed the recording, click TA_Critique for our critique of the
recording.
 Recordings of Participant Briefings
1. The third recording is of the analyst demonstrating the think-
aloud technique and giving the participant a chance to practice.
Refer to that subsection of this section and the script above as
you view it. Think about where the analyst follows and deviates
from the procedures presented in this section.
1. Click Practice to view the recording.
2. After you have viewed the recording, click Practice_Critique for our critique
of the recording.
2. The last recording is of the analyst explaining the rules of the
observation and giving the participant the task. Refer to that
subsection of this section and the script above, as you view the
recording. Think about where the analyst follows and deviates
from the procedures presented in this section.
1. Click Rules_& Tasks to view the recording.
2. After you have viewed the recording, click Rules-&-Tasks_Critique for our
critique of the recording.
 3.2.5 Conducting the Observation

 ready to conduct the observation

 After introduced the participant to the general
purpose and procedures of the study
 After trained him/her how to think aloud

 Contents
 Introduce the observation phase
 Begin the observation

 Conclude the observation

 Example : Conducting an observation of Date/Time

control panel
 Introduce the Observation Phase
 Describe the system
 Tell the participant as much about the
system as you expect real users will
know
if they will only have seen it through
trade-magazine advertising or TV
commercials, give them that level of
overview.
If they will have gone through an
hour's training, this is the time to do
that training.
 Introduce the Observation Phase
 Tasks you wish him / her to do
 If there is more than one task, it is best to do
one at a time rather than describe all of them
and then set the user free to do them all on
their own.
 Have the tasks written on a sheet of paper,
give to the participant, and tell them about it
verbally so they get the information twice.
 If diagrams or pictures might make the task
easier to understand, take the time to prepare
these ahead of time and walk through them
with the user.
 Introduce the Observation Phase
 Ask the participant if she has any questions
 any questions about the goals of the study,
the procedures, the product, or the task.
 Answer any questions that clarify what the
user has to do, but not any that solve some
of the problems they might encounter in
doing the task. E.g.,
don't answer questions like "How do I do
that task?"
if asked reply "That's exactly the sort of
thing I need to observe. I would like to see
how the system helps you figure that out."
 Begin the Observation
 Check with the recording devices.
 Keep away from the user
 monitor their progress from another room
 If running the test in the field, move out of the user's line
of sight and sit quietly
 Note down their questions
 tell the user you will not answer questions about the
product while they were working
 should speak their questions aloud

 jot down any questions and answer them later

 Begin the Observation
 Prompt if they keep quite
– Just say "Please keep talking”
– Don’t say "What are you thinking?" or "Please explain
what you are doing"
 Be sensitive to their willingness to quit
 Answer Their Questions
 Note their opinions and suggestions
 Conclude the Observation
 S1: When finished, answer any questions you
jotted down on your notes
 S2: Ask them if has any more questions about the
system, the study, or the organization.
 S3: Answer these questions
 If you can right then answer,
 If you can not, put the user in touch with someone who
can answer the questions.
 Although the test has got reliable data, it is a
good idea to ask the user these questions
 (1) it gives them a chance to express their opinion, which
people often want to do
 (2) users some times have great ideas that you can only
collect by asking.
 Conclude the Observation
 S4: Ask for any opinions about the product they
just tested.
 S5: Ask for any suggestions to improve the
product.
 s6: Thank for their participating.
 Reiterate that their participation will help you identify
problems with the system
 Give the participant whatever compensation you had
promised them when they were recruited, or arrange for
the compensation to be sent to them (e.g., fill out any
necessary paperwork to pay them or get their address to
send them a copy of the final report).
 Thank the participant again when they leave.
 Example :Conducting An Observation With Date/Time
 Click at the link and view the recording

 Introduce yourself

 Introduce the task

 Get the user’s consent

 Explain the task

 Ask the user to begin and prompt if he stops

 Example :Conducting An Observation With Date/Time
 In the observation, we decided not to give
any training or introduction other than the
name of the panel.
 This was to simulate how real users would be
introduced to this control panel.
 it would just come installed on their machine,
ostensibly without help or documentation.
 Give the descriptions of the tasks to the
participant, each task on a separate sheet of
paper. as you view the recording.
 Think about how the analyst follows and
deviates from the procedures presented in
the course material.
 Example :Conducting An Observation With Date/Time
 At one point, the participant says, "I give up"
but then he continues to look for an answer, so
the analyst correctly lets him continue for some
time.
 He tries some ideas, summarizes the problem,
tries some more, and, finally pauses for a length
of time and gives the analyst an imploring look
(which is not visible on this recording).
 The analyst is sensitive to the participant's
frustration, and decides to suggest that the
participant move on to the next task.
 He gratefully accepts and continues to "think-
aloud" very well.
 Another recording of a different participant
 The script for the analyst to conduct and conclude observation.
 The Conclusion recording.
 Refer to the previous material in this section as well as to this
script as you view the recording. Think about where the analyst is
following and diverging from the procedures presented in the
course material.
 When done, read the critique.
 The recordings represent quite good Type-2 verbalizations.

 For contrast, we have included an example of a think-aloud

session where the participant explain her thoughts rather than
merely report them.
 The analyst should have interrupted the participant to ask her
not to explain (as the script earlier requires), but failed to do so.
When you view this recording, notice how much slower and less
natural the process seems.
 Click the following link to view the DateTime2 recording.
 3.2.6 Analyzing the Observation

 contents
 Establish criteria for Critical Incidents

 View the recorded behavior and write

UARs
 Analyze the observations of the date/time
control panel
 3.2.6 Analyzing the Observation

 Having collected think-aloud usability data from

several participants
 you now must analyze the data to find

 good features that you want to preserve in

future versions of your system,
 problems with the system you want to fix,

 possible solutions to those problems.

 Establish Criteria For Critical Incidents
 What is a problem?
 It isimportant to think hard about what behaviors
should be considered critical incidents
 what observable behaviors indicate that a feature
is so well designed that it should be preserved in
future redesigns
 Establish Criteria For Critical Incidents
 What is a critical incident?
 extreme behavior, either outstandingly effective
or ineffective with respect to attaining the general
aims of the activity
 not everything a user does will be "critical"; not
everything is worth thinking whether it should be
fixed or preserved in the next version of the
software.
 Re-Defining Critical Incident
 Different design situations will generate different
criteria for criticality
 Establish Criteria For Critical Incidents
 List criteria for … before view the recorded data
 problems
 good features

 List them in a similar table, using it as a

reference as you review the data.
 no more than about 10 criteria
 keeping more than that in mind as you review the
recordings can get unmanageable.
 Some useful criteria for systems and are easily
prototyped in Visual Basic.
 View The Recorded Behavior And Write UAR’s
 View the record
 Identify a critical incident
 Give it a unique name and mark as a good
feature or a bad feature
 Find relationships with other UAR’s
 Evidence For The Aspect
 It should state the actual behavior, includes
 what they said
 what they did (typed words, clicked buttons).

 what was on the screen at the time of the incident

 something on the screen does NOT in itself mean that the

user actually noticed it
 Evidence is just “facts”, NOT an interpretation
 may be more than one interpretation of the same facts
 Interpretation will appear in the "explanation" slot

 not in the "evidence" slot

 Evidence For The Aspect
 Include pointer for replay it anytime later
 tape-counterfor a videotape
 time stamp on the videotape
 time shown on a clock displayed on a computer
screen
 Purpose and consequence should be fairly
clear, include enough context to permit
those inferences
 what they knew
 what they paid attention to
 how they approached a problem
 Evidence For The Aspect
 Evidence should start with statement of goal
 It is much easier to understand how the interface supported or
failed to support a user
 User will tell what he is trying to accomplish by saying something
like
 "I wanna find…"
 "Now, let's try to…"
 "Gotta go do…"
 when not explicitly voice the goal, other actions is evidence. E.g.,
 the user's goal is to do the part of the task she just read from the
task description .
 the act of reading the task description would be evidence for that
goal.
 Sometimes the system sets the user's goal. E.g.,
 system presents a modeled dialog box. In VB, msgbox
 Therefore, the appearance of a dialog box and the fact that it is
modeled should be recorded as evidence for the user's goal.
 Evidence For The Aspect
 Include the effects of the user's actions in the evidence
slot, These are usually:
 screen shots
 descriptions of what happened on the screen and with the system.
 Some side effects have no visible effect on the screen.
 include a factual description of those side effects in evidence.
 When the consequence of a user's actions not be seen
until much later in the recording.
 include the evidence for this effect, even if it is not contiguous
with the incident itself.
 The evidence you include in the report must be complete enough
to describe both the goal and the effects, no matter how separated
they are by time.
 On-line UARs vs. Paper UARs
 In on-line UARs
 easyto include actual clips of the recording
 Can play the multimedia contents

 Paper UARs
 continue to be needed for many reasons
 easier to carry and hand out copies at a meeting

 even include dynamic(On-line play)

evidence, consider using it as a back-up to
static(Paper based) evidence.
 Explanation Of The Aspect

 Explanation is your hypothesis when they

were performing the acts , about
 what the user was seeing
 Interpreting

 Understanding

 guessing

 In the UARs, Clearly say

 your understanding of theuser’s background
 your understanding of how the system actually
works
 Explanation Of The Aspect
 Sometimes there will be more than one
plausible explanation for the evidence
 Record all the possible explanations
 Different explanations point to different
solutions
 look for more evidence confirms or disconfirms
the other explanations
 Sometimes have to try more users on a
shorter, more focused task
 gather evidence to distinguish between different
explanations.
 Severity of the Problem or Benefit of the Good
Feature
 severity is related to the criteria used to
identify the incident as critical.
 giving up on a task is more severe than the
user expressing distressed surprise
 Severity of the Problem or Benefit of the Good
Feature
 there are no standard criteria for
identifying critical incidents
 what counts as critical varies from one design
situation to the next
 what counts as severe also varies with the design
situation
 establish criteria for measuring severity as
each design situation dictates （设计规
定） .
 Possible Solution, Including Trade-Offs
 It is so important to find evidence for the
user’s goals
 A solution to a usability problem comes from supporting
the user's goals directly.
 Questions can lead to inspiration for a solution( 设计灵
感 ).
 Ask
 if the goal the user care is supported by the system
 if not, why not?
 Is it a reasonable goal, if so, how can you support it?
 If it isn't a reasonable goal, ask what it was about the
system that guided the user to form an unreasonable
goal.
 Possible Solution, Including Trade-Offs
 Record solutions generated by users themselves
 Sometimes users will generate solutions themselves
 Record these suggestions and give them consideration
much as those from the development team.
 Do not give them more importance just because

 they come from a user—users not knowing what

features would actually serve them in the long run.
 Any suggestion arising in a usability test is likely to be
a reaction to a local difficulty and may not take into
account enough understanding of the whole system to
be truly insightful.
 Example : Analyze The Observation Of Date/Time

 Analyze the observation made in 3.2.5

 Use the criteria for identifying problems
and good features presented in the table.
 Identified 3 critical incidents and write 3
UARs, 2 problems and 1 good feature.
 UARs :
 TA UAR 1

 TA UAR 2

 TA UAR 3
 Example : Analyze The Observation Of Date/Time

 Notice: TA UARs seems bigger than HE UARs.

 the critical incident technique requires incidents to be
"complete,"
 they include the setting of a goal, the problem-solving
required to accomplish that goal, and its final resolution
(whether it is success, abandonment, or a change of
goals).
 HE UARs can identify problems in the small steps
towards a goal (e.g., understanding a label, or searching
through a cluttered screen)
 TA UARs require a complete chain of events towards a
goal.
 there is often evidence supporting the HE UARs within
the TA UARs, but they will not be identical in scope.
 3.2.7 Finding Possible Re-Designs

 After writing up UARs from several

usability tests
 Step back and generate ideas about how to
redesign the system to fix any problems you have
found.
 Contents
 Relating different Usability Aspects
 Determine Possible Solutions

 Example: Looking for Possible Re designs of the

Date/Time control panel
 Relate Different Usability Aspects
 Relate repeated actions on different objects
 UARs of same goal arose by different users or
under different circumstances
 slightly different goals arose with the same user

 "Similar" goals usually share an action or an

object.
 Look for repeating the same action on different
objects
 E.g., deleting a file and deleting a folder are
both expressed as an action (delete) on an
object (file or folder).
 Relate Different Usability Aspects

 A string of different actions on the same

object may relate several UARs
 E.g., copying file1 to another folder, then
deleting file1 (which has the effect of moving the
file).
 Such strings of related goals often show a larger
goal that is not being supported well in the
system.
 Relate Different Usability Aspects
 Many problem UARs is about system feature
relate them
 E.g., problems with the spelling checker, perhaps
the operation of that entire feature should be
rethought
 Examine UARs that relate to other parts of the
computer system outside the system you are
designing
 This integration with other applications is often a
source of usability problems.
 Relate Different Usability Aspects
 Ways relate UARs:
 heavily depend on the particular system you are
designing.
 Find commonalities
 Always think about how to relate the UARs, these
patterns have the potential to inspire better designs.
 don't get bogged down in trying to impose patterns
where they do not jump to mind (e.g., don't spend more
than a day thinking about this).
 Sometimes there really are no big patterns, and the
right thing to do is just to go about fixing the little
things.
 Determine Possible Solutions
 Write a solution that supports the users goals
directly
 If UARs suggests a larger user goal seems to be
problematic, Ask yourself :
 Is this larger goal intended to be supported by the
system, and, if not, why not?
 Is it a reasonable goal, and, if so, how can you support
it?
 If it isn't a reasonable goal, why the system guided the
user to form an unreasonable goal?
 Determine Possible Solutions
 Write a solution that supports the users goals
directly
 If a set of UARs about a specific system feature,
check ：
 what user goal the feature was designed to support.
 Is this a reasonable goal?
 Was there any direct evidence that users would form
that goal
 did anyone articulate that goal? If not, what other
goals did the users express in place of that goal?
 if they didn't voice the goal the feature was designed
to support, what goals were they working on when
they did use that feature?
 Determine Possible Solutions

 Write a solution that supports the users goals

directly
 If UARs concerning with other applications on the
computer,
 make sure you list the goals.
 These goals are only a subset of what people will
actually want to do with your system
 use this list as a starting point, and generate other
possible integration requirements.
 Determine Possible Solutions
 Make sure your solution does not violates any
heuristics
 Besides the general idea that the system redesign
should support users‘ goals more directly, there are
no hard and fast rules （硬性规定） about how to
generate new designs.
 Use the UARs and their relationships as inspiration
for a redesign
 check your ideas with a quick Heuristic Evaluation
to make sure you are not violating heuristics.
 prototype the new design and iteratively user test it
again.
 Example: Possible re-design of Date/Time
 Check with the complete set of Date/Time UARs

 Follow the material given under this topic

 3.2.8 Writing a Summarizing Report

 Summarizing Reports
 Example : A summarizing Report for the
Date/Time
 Summarizing Report
 Communicate the Results of the usability
analysis
 If the problem is severe it should not exceed three
pages
 If the problem is small then ranked them
 In the report usability aspect must be fixed along
with usability problem
 If needed produce “highlight” videotape to
support your report that summarizes all the
problems mentioned in the UAR’s
 Example: A Summarizing Report for the Date/Time Control
Panel
 If the think-aloud data were all you had to
report on, a summarizing report would not
be necessary, because
 thethree UARs themselves would be sufficiently
short for everyone to read.
 For finding possible redesigns include the
28 UARs of Heuristic Evaluation

 SampleOfSummarizingReport.doc
 Take assessment
 Exercise 7
 3.3Think-Aloud Testing vs. Heuristic Evaluation
 3.3.1 Comparing Think-Aloud Usability
Testing With Heuristic Evaluation
3.3.1 Comparing Think-Aloud Usability Testing With Heuristic
Evaluation
 In the section, we will discuss
 Many usability aspects identified in
HE are
confirmed in Think-Aloud Usability Tests
 When HE predictions are not confirmed by
Think-Aloud.
 “False Alarms” Vs True Problems

 Think-Aloud Usability tests can show Things HE

can’t show
 TA & HE
 Two techniques for evaluating the interface
designs.
 heuristic evaluation: HE is an analytic technique
that you can use quickly at a very early stage in
design
 (e.g., with rough, paper sketches of ideas
without the exact procedures worked out).
 think-aloud usability testing: requires a more
detailed design and is easily conducted on
prototypes you can build in VB
 TA & HE
 Why teach both techniques?
 many usability aspects predicted by HE are
confirmed in usability tests
 these two techniques will sometimes give
conflicting information.
 HE can uncover usability aspects that think-aloud
tests cannot.
 think-aloud usability tests can address usability
aspects that HE cannot.
 they do not overlap completely and are best used
together.
 Many Usability Aspects Identified in HE are Confirmed in
Think-Aloud Usability Tests
 Heuristics are general principles distilled
from many years of design experience.
 Heuristics can predict the usability
problems that are revealed in think-aloud
usability tests.
 E.g., DateTime3 recording of the Date/Time
control panel.
 problems uncovered by heuristic evaluation
confirmed by the think-aloud usability tests.
 though not completely.
 Example
 UAR HE2 predicted the problem
 the map being so large in that it would have interaction
but not being supportted
 In think-aloud recording,
 the participant brings up the Time Zone tab, he moves
the mouse pointer over the map and says (time
16:28:02):
If my knowledge of geography were better, I would
instantly be able to locate it [i.e., the Philippines] on the
map…[moving the mouse pointer around the Pacific
Ocean] but I can't.
 Example again
 the interpretation of this behavior is that
 he thought he could do something with it, if he could
have found the Philippines on the map, he would have
wanted to click on it.
 Later he gives up searching through the time
zone list, if you listen carefully, you can hear him
click on several places in the map (time
16:30:02)
 Example of the list of time zones
 Several HE-based UARs note the list of time zones
 HE18:
 Provides too much irrelevant information .

 HE21:
 Is inefficient, since a visual search through such a long
list takes too long .
 TA:
 ...of course, it isn't in any kind of order, so we just have
to sit here and scan through lots and lots and lots of
cities (in an exasperated voice)…and then when we
realize we went in the wrong direction, we have to go
alllll the way back up (the user sighs).
 evidence of the user's perception of inefficiency
 And …
 HE27:
 thevery incompleteness of the list of cities would
cause users confusion.
 TA:
 he fails to find either Manila or the Philippines
and finally gives up on the task (time 16:30:00)
 All this evidence supports the problem
identified in HE28:
 theListBox is a bad control for setting the time
zone.
 Example for supporting good feature
 HE5 and HE16
 TA
 Theuser has no difficulty using the OK button in
several instances (times 16:30:10 and 16:30:50),
 HE22
 seeing the time zone setting on the Date/Time tab
 TA
 "of course, we are still in the wrong time zone"
 When HE Predictions are not Confirmed by Think-Aloud Usability Tests

 Happen in two ways:

 Think-aloud testing contradicts an HE prediction.
 HE predicts a good feature

 participants in TA tests have problems

 Think-aloud testing yields no evidence to support

an HE problem prediction.
 HE predicts a usability problem,

 TA give no evidence to show users actually

having that problem.
 When HE Predictions are not Confirmed by Think-Aloud Usability Tests

 When HE predicts a good feature, but users

have problems in TA tests ： Believe the think-
aloud data!
 Unless some aspect of the testing situation that
makes it undeniably anomalous, E.g.,.
 the participant became ill and could not continue with
the test
 had noticeable difficulty concentrating during the
test's last 15 minutes
 "False Alarms" vs. True Problems
 There is a debate in the human-computer interaction
community
 whether HE-predicted problems not be shown up in TA
tests are
 “false alarms“ or

 true problems undetected in the tests.

 fixing "false alarms," will waste time—or worse,

decreases the usability of the system!
 true problems undetected by the think-aloud testing,
fixing them is a good use of effort.
 "False Alarms" vs. True Problems
 To the heuristics, some situations unlikely to be covered
in TA tests. E.g.,
 most TA tests cover new users' experience of the
system, not skilled users' experience.
 TA tests use people who have never seen the system
before, give these people a minimum of training, and
record their initial interactions with the system.
 This approach yields valuable information about how
new users think as they use the system—but not about
how experienced users might think.
 "False Alarms" vs. True Problems
 several heuristics give advice on how to
handle errors and about help and
documentation
 errors that participants in think-aloud tests may
or may not make
 test participants may or may not choose to use
help and documentation
 We suggest two rules of thumb to apply to
usability problems identified in HE but not
confirmed in TA tests
 two rules of thumb - 1
 If an HE problem does not show up in TA,
review TA data to see if the situation the HE
refers to did indeed arise.
 If arise, then it is possible that either
 1) the HE UAR reflects a false alarm
 2) other users in other circumstances will indeed have
the trouble the HE predicted.
 After all , only tested a few users on a few tasks,
 we cannot possibly test all potential users.
 the next user may have the problem
 if more than one participant encounter this situation
and none of those had problems
 trust the data and assign the problem a low priority
on the list of things to fix.
 two rules of thumb - 1
 If an HE problem does not show up in TA,
 If the situation did not arise or if you had only one user
who encountered
 you have no reliable data to contradict the HE-
predicted problem.
 then judge the problem on the basis of its
 Severity
 relationship to other UARs
 the difficulty involved in fixing it
 two rules of thumb - 2
 A system being used often (e.g., word 、 Excel),
problems predicted by the Flexibility and
Efficiency of Use heuristic , even if not show up
in the TA tests.
 That heuristic predicts usability for skilled users, not
new users,
 TA tests typically will not provide data for that type of
user.
 Example for Applying these rules of thumb to Date/Time
control panel
 Date/Time control panel is not used very
often, so it falls first rule.
 Since we had only one participant in our
think-aloud test, all our HE predictions fall
under the first rule's second clause —where
we are advised to trust HE predictions.
 Example for Applying these rules of thumb to
Date/Time control panel
 UARs HE6, HE8, HE12, and HE14 all predict
problems with the standard OK, Apply, and
Cancel buttons, but the user seemed to have no
problems with these.
 Upon a closer examination:
 we discover that the user only used the OK
button when he was finished using the control
panel.
 He had no opportunity to encounter problems
with these buttons because he never had the goals
to apply or cancel, only to accept the changes he
had made and close the window.
 Therefore, we have no evidence for or against
those HE predictions.
 Example for Applying these rules of thumb to
Date/Time control panel
 There is no evidence in the recording for or
against the two UARs concerning help (HE23
and HE24) because the participant never used
the help facility.

 In such circumstances
 probably decide to fix these problems predicted
by heuristic evaluation,
 Based on their relative severity and cost of
applying the fixes.
 Think-Aloud Usability Tests Can Show Things HEs Can't
Show
 some problems heuristic evaluation cannot show in
principle
 Some ohter does not show in practice.

 When an HE is done using a paper prototype it cannot

identify problems with the dynamics of the system.
 E.g., it cannot indicate where system speed will be a
problem—but where users may become annoyed or
impatient because the system seems slow.
 HE will not detect situations where the feedback is so
delayed , but the user becomes confused as to what is
happening with the system.
 Think-Aloud Usability Tests Can Show Things HEs Can't
Show
 interacting with a running prototype
during usability testing , these problems
happened.
 only interaction with a real system will
uncover problems in the code that will
cause the system to crash.
 HE assumes the system will work as
sketched or prototyped.
 Think-Aloud Usability Tests Can Show Things HEs Can't
Show
 if the code in the deployed system is
different from the code in the prototype.
 TA tests with a prototype will not uncover this
type of problem
 So , extensive testing with actual users must
be done before any major release of
software.
 Practice tell us ……
 HEs are done without framing the analysis in
terms of a real-world task,
 TA tests typically try to simulate real task
situations
 This task-oriented framework often reveals
usability aspects of the interaction
 between disparate features in the system
 between it and other software in the users' environment.
 E.g.,
 when copying something from the system into an e-mail message
or a slide presentation, difficulties often arise.
 in practice, these usability aspects tend to appear
more often during TA than during HE.
 Date/time panel sample
 the prototype responds almost instantaneously to
the user's actions,
 no occasion to complain the speed of the interface.
 The prototype is also sufficiently robust
 that it does not crash during the tests.
 it is isolated enough
 that it is unlikely to be used in integrative tasks that span
Both of these are dynamic artifacts of the system
 Remember!
 it is an endless source of challenge to designers of
user interfaces.
 remember: continued observation of users
 whether with think-aloud usability tests
 Or informally use product in the field

 It is an integral part of iterative design and the

product life cycle
 Take assessment
 Multiple-choice quiz 8
 Practical Quiz 3