OMS CS:6750 Assignment M4

Background

The problem I am attempting to design an interface for is: how do we help users ​navigate
indoors.​ The 3 ideas/prototypes were:

● Console at entrance with ‘video tour’ from current location to destination (textual)
● Mobile app with location awareness and mapping+navigation (verbal)
● Robot which greets user and escorts them to destination (wizard of oz)

Qualitative Evaluation: Plan

Prototype: Mobile app with location awareness and mapping+navigation (verbal)

Method: In-person interviews
Participants: 4 or more friends. No identifying information captured.
Location: Convenient locations (home, coffee shop, office). Quiet place where it is easy to have
a conversation. Ensure that there is sufficient time (about 20 minutes), ie that the participant is
not rushed.
Recording method: Note taking

Qualitative Evaluation: Method

The evaluation will be fairly sequential: greeting/introduction, read out interface description
script (previous assignment), then ask questions about it.

Questions to ask:
● (General) Would you find utility in such an interface? Why or why not?
● (Learnability) How would you use the app?
● (Efficiency) Would this app get you to your destination sooner?
● (Efficiency) Is there anything described that you don’t find useful?

Then just thank the participant for their input.

Qualitative Evaluation: requirements assessment

The goal of the qualitative assessment is to:

● (in)validate the overall goal of the interface, ie understand if that is even a problem
● (in)validate our initial set of user requirements
 ● inform interface design nuances not captured in the requirements yet and answer some
of the open questions

Specifically, the verbal prototype evaluation should (in)validate the following requirements:
● Is a mobile app the right medium -- would it be too distracting
● Need to show the current location
● Provide turn by turn directions
● Work through elevators etc.
● Is privacy a concern

Empirical Evaluation

Prototype: ​Console at entrance with ‘video tour’ from current location to destination

Conditions

This evaluation should compare the efficiency/accuracy between:

● Interface A (base): a static map and directory (typically seen in malls)
● Interface B (proposed): an interactive console with video tour to the destination

What are we testing:

● Will the proposed interface make it ​faster​ for a typical user to get to their destination?
● Will the proposed interface let users get to their destination with ​less effort?

Hypothesis

Null hypothesis: the proposed interface does not make the task significantly faster or easier
Alternate hypothesis: proposed interface makes the task significantly faster or easier or both

Experimental Method

Experimental method: written survey

Study design: this should be a within-subjects study. There will be 2 sets of interview questions
and subjects will be required to answer both. The order in which surveys are presented will be
randomized.

Survey questions:

<Describe the interface and the use case. Use same questions for both>. Please answer the
following questions about this interface:

Q: How often would you use the interface:

 ● Never
● A few times
● Quite frequently
● Always

Q: If you replied never, please provide a reason.

Q: Would you find this interface easy to use:

● Very easy
● Somewhat easy
● Difficult
● Very difficult

Q: Would this interface help you save time:

● Yes
● No

Q: Do you have any comments/questions about the interface?

Lurking variables

There are some systematic differences: interface A exists today, and users may already have
used it. Interface B will require some imagination on the part of the user, which could introduce
variability in the results. This also tied into the fact that we are testing for ​broad ​differences here,
ie 2 radically different interfaces where the number of differing variables is huge (like, will the
search work effectively, will the videos be slow/fast enough etc)

Data analysis method

● Design: within subjects

● Data: discrete/categorical -> may be converted to continuous
● Methodology: ANOVA

The result we want to get out of this is eg is one ​easier​ than the other, ie is there significant
difference in responses for those 2 questions.

Predictive Evaluation

Prototype: Robot which greets user and escorts them to destination (wizard of oz)

Cognitive Walkthrough
 Sub Task Notes

Walk into the mall and * User walks into the mall and sees the robot docked in front with a
discover the interface signboard describing its purpose (personal guide?)
(robot) * User walks up to it. User is greeted by the talking robot.
* The surroundings may be noisy so user may not be able to hear.
* User replies to robot’s greeting. User understands that robot can
be used to escort them to where they need to go.

Converse with the robot * Robot needs to perceive commands and speak back in a
and communicate personal way. Volume, body language etc should be appropriate.
desired destination * Because the technology is new, user is either delighted or highly
frustrated.
* As destination is agreed upon, the user should receive feedback
so that both parties know exactly where they will land.

Follow the robot to * There needs to be a calibration of robot speed and human speed
destination * If user stops for some reason, robot should recognize that
* If user changes their mind (different destination, don’t want robot
around), robot should recognize that
* There should be a way for robot to recognize when it is being
addressed (vs regular chatter)

Next assignment

Evaluations selected:
● Qualitative evaluation through in-person interviews
● Empirical evaluation through surveys

Why:
● The requirements/prototypes are in early stages, so broad, qualitative feedback will be
very helpful
● I would have preferred to administer the wizard of oz prototype instead of the empirical
survey, but don’t want to recruit so many in-person participants