Week 1 Quiz: Should we build it?

Did it
matter?
Quiz, 5 questions
3/5 points (60%)

Try again once you are ready.

Required to pass: 80% or higher

You can retake this quiz up to 3 times every 8 hours.
Question 1
Correct
1/1
point

1. Question 1
You're collaborating with a group of developers to build an app that addresses a
common problem facing your city: finding parking spaces. Together, you've
finished customer discovery and validated your core propositions. You've built
an initial product based on user stories that are tied to those propositions. After
building a minimum viable product based on your hypotheses, your analytics
show that users are accessing the app regularly but aren't successfully finding
parking spaces each time they use it. Those analytics and user follow-ups suggest
this is because they are having trouble with the search function. What problem
might your team be facing?

Your team developed a minimum viable product that isn't tied to user stories.

Your team did not adequately explore why they were building the app.

Your team needs to build and test a completely new prototype based on the product feature
assumptions.
Your team needs to review 0 day success criteria on usability, particularly around search.

Correct

Particularly with new products, it's often more motivation than usability that's the issue, but
every product is different. This case suggests a current issue with usability around search.

Question 2
Incorrect
0/1
point

2. Question 2
Your team built an online payment system for school lunches. Overall, the
product is working well and the segments it services (students, parents, and the
cafeteria team) find it valuable. However, parents have asked for some new
features. The schools have collected the inputs and they're a little patchy--ideas
for an app, an email, and a web page. Your initial discovery interviews reveal
that parents want to know what their child is purchasing, both to monitor their
spending and their nutrition. What is the best next step?

Conduct a feature experiment using a concierge MVP that gives parents the ability to see what
food their child purchases each day.

Create a funnel experiment to find the best way to get parents to sign up for the online payment
system if it has these new features.

The team should create a cohort experiment to compare customers over time to see who is most
likely to use the new feature.

Use an A/B test to analyze quantitative data and outcomes of two different landing pages
describing the potential new features.
This should not be selected

A/B tests have many uses, including incrementally improving something that's already
fundamentally sound or comparing alternative updates to the current baseline. In this situation,
the team needs a different type of test to experiment with how valuable users find various takes
on the potential new feature. Review the introduction to experiment types in the "Five Core
Experiments" video and then follow up on specific experiment types in the remaining course
videos.

Question 3
Correct
1/1
point

3. Question 3
Your team has just conducted an A/B test on a new cart feature and has some
puzzling results. What should the team do next?

Run a cohort test to compare different ways to navigate through the cart feature.

The team should review the fundamental propositions through qualitative research with
customers to better understand what's driving behavior around the shopping cart.

Correct

When A/B testing (or quantitative research in general) reveals puzzling results, it may be time
for the team to shift from testing what to reexamining why in order to determine if and how a
proposition is in fact valuable.

Scrap the feature until a future product iteration is released.

A/B tests are easy to set up, so the team should run quite a few to get more quantitative data.

Question 4
Incorrect
0/1
point

4. Question 4
Why might a team decide to scrap a feature after testing it?

After 30 days, users are taking advantage of the feature in less than [threshold percentage] of
interactions.

After a month, more users than the failure threshold are using the feature.

After several months, the feature is continually used and users share anecdotal evidence that they
like what the feature is doing for them.

In usability testing, the majority of users can't complete the core tasks with the interface

This should not be selected

This scenario is typical of a 0 day 'failure', but a given feature could reasonably go through
several iterations of tuning the user interface--that's normal. Test early and test often. This would
not be a good reason for the team to eliminate the feature. Review how to use metrics in the
"Testing Features: Running the Experiment" video.

Question 5
Correct
1/1
point

5. Question 5
A music video website enables users to record and upload videos set to popular
songs. There's a feature that allows viewers on the site to rate posted videos, and
the highest rated videos are listed in the trending favorites area. In terms of the
Hook Framework, why might this feature be included in the product?
The trending favorites area will deepen the user's investment in the product and make it less
likely for the user to switch to another similar product.

The trending favorites area increases the anticipation of an uncertain reward.

Correct

A reward is the unpredictable but tangible gratification from an action. Users enjoy seeing if
their video is rated highly enough to be featured in the trending favorites area, which increases
the likelihood that they'll use the website enough to become a habit.

The trending favorites area is acting as an external trigger and encourages the user to use the
website.

The trending favorites area increases the action required of the user.

Week 1 Quiz: Should we build it? Did it

matter?
Quiz, 5 questions
4/5 points (80%)

Congratulations! You passed!

Question 1
Correct
1/1
point

1. Question 1
You're collaborating with a group of developers to build an app that addresses a
common problem facing your city: finding parking spaces. Together, you've
finished customer discovery and validated your core propositions. You've built
an initial product based on user stories that are tied to those propositions. After
building a minimum viable product based on your hypotheses, your analytics
show that users are accessing the app regularly but aren't successfully finding
parking spaces each time they use it. Those analytics and user follow-ups suggest
this is because they are having trouble with the search function. What problem
might your team be facing?

Your team did not adequately explore why they were building the app.

Your team developed a minimum viable product that isn't tied to user stories.

Your team needs to build and test a completely new prototype based on the product feature
assumptions.

Your team needs to review 0 day success criteria on usability, particularly around search.

Correct

Particularly with new products, it's often more motivation than usability that's the issue, but
every product is different. This case suggests a current issue with usability around search.

Question 2
Incorrect
0/1
point

2. Question 2
Your team built an online payment system for school lunches. Overall, the
product is working well and the segments it services (students, parents, and the
cafeteria team) find it valuable. However, parents have asked for some new
features. The schools have collected the inputs and they're a little patchy--ideas
for an app, an email, and a web page. Your initial discovery interviews reveal
that parents want to know what their child is purchasing, both to monitor their
spending and their nutrition. What is the best next step?

Use an A/B test to analyze quantitative data and outcomes of two different landing pages
describing the potential new features.

Create a funnel experiment to find the best way to get parents to sign up for the online payment
system if it has these new features.

This should not be selected

The team is not focused on finding the best way for customers to sign up for the online payment
system. Instead they need to test the value and usability of new features. Review the introduction
to experiment types in the "Five Core Experiments" video and then follow up on specific
experiment types in the remaining course videos.

Conduct a feature experiment using a concierge MVP that gives parents the ability to see what
food their child purchases each day.

The team should create a cohort experiment to compare customers over time to see who is most
likely to use the new feature.

Question 3
Correct
1/1
point

3. Question 3
Your team has just conducted an A/B test on a new cart feature and has some
puzzling results. What should the team do next?
Run a cohort test to compare different ways to navigate through the cart feature.

Scrap the feature until a future product iteration is released.

A/B tests are easy to set up, so the team should run quite a few to get more quantitative data.

The team should review the fundamental propositions through qualitative research with
customers to better understand what's driving behavior around the shopping cart.

Correct

When A/B testing (or quantitative research in general) reveals puzzling results, it may be time
for the team to shift from testing what to reexamining why in order to determine if and how a
proposition is in fact valuable.

Question 4
Correct
1/1
point

4. Question 4
After observing a new feature introduced to a set of users for 30 days, your team
finds that usage of the feature is below the failure threshold. The team eliminates
the feature and several users complain. When company leadership hears of this,
an emergency meeting is called. How could you explain the decision?

The person responsible for the failure will be removed from the team.

There's a threshold for relevance we apply to all new features--adding marginal features crowds
out future innovation.

Correct
No one likes to hear complaints--and complaints are certainly a bad sign. But in practice you
can't please every single user all the time. You might want to explain in more detail: The team's
working guideline is that if new features aren't used [a certain amount by a certain number of
users; insert your metrics here], they're not economically viable for the company. Supporting
features costs money and clutters the product for all the other users, all of which crowds out
future innovation that really will matter. Because we are working iteratively, we know that the
need will come up again if it is truly important.'

The team will use a success metric for the next experiment so the results are more definitive.

Because we are relying on quantitative rather than qualitative observations, we feel confident
that the results support elimination of the feature.

Question 5
Correct
1/1
point

5. Question 5
A music video website enables users to record and upload videos set to popular
songs. There's a feature that allows viewers on the site to rate posted videos, and
the highest rated videos are listed in the trending favorites area. In terms of the
Hook Framework, why might this feature be included in the product?

The trending favorites area will deepen the user's investment in the product and make it less
likely for the user to switch to another similar product.

The trending favorites area increases the anticipation of an uncertain reward.

Correct

A reward is the unpredictable but tangible gratification from an action. Users enjoy seeing if
their video is rated highly enough to be featured in the trending favorites area, which increases
the likelihood that they'll use the website enough to become a habit.
The trending favorites area is acting as an external trigger and encourages the user to use the
website.

The trending favorites area increases the action required of the user.

Week 1 Quiz: Should we build it? Did it

matter?
Quiz, 5 questions
5/5 points (100%)

Congratulations! You passed!

Question 1
Correct
1/1
point

1. Question 1
Your team is working on a plant identification app for a client. The client feels
they have a clear vision for the project and provided you a list of features. The
development team built a 1.0 based on the client's specifications and have
released it to a test group. Initial feedback was encouraging and usability tested
well. However, you've found that usage is dropping off significantly after the 30-
day mark. What might be the problem?

Your team did not adequately explore what constituted a valuable outcome for the user

Correct

Your team may have relied on the client's assumed problem scenarios and value propositions
without adequately exploring how to test whether users were achieving outcomes valuable
enough to keep them using the app. If so, explore what constitutes a valuable outcome, how to
instrument observation and testing for it, and then update the product (or non-product MVP
vehicle) and re-test.

Your team needs to focus on what features the product should have.

Your team needs to build and test a new prototype based on the product feature assumptions.

Your team developed a minimum viable product that isn't testable in the field.

Question 2
Correct
1/1
point

2. Question 2
Your client, a book seller, wants to figure out the best way to get more customers
to sign up for their monthly email with personalized suggestions for new books.
Right now sign-ups are OK but not great. Initial interviews with customers
suggest that a basic desire to be connected with a great next book exists. Those
that have the service enjoy it and continue it. How should your team experiment
to make this happen for your client?

Generate some specific hypotheses about how to better deliver on the proposition of personalized
reading suggestions, rank them, execute them, and run A/B tests on the customer base to test
them.

Correct

While it's not the only place they're useful, A/B tests are particularly well suited to incrementally
improving something that's already fundamentally sound. Assuming your hypotheses have to do
with executions like changing/adding follow-up emails, tweaking the checkout page, etc., A/B
tests are a good way to compare alternative updates to the current baseline.
The team should create a cohort experiment to compare customers over time to see who is most
likely to subscribe to the monthly book club.

Create a concierge MVP experience and run it

Create a product experiment featuring the monthly email.

Question 3
Correct
1/1
point

3. Question 3
A team fairly new to agile practices is working on a brand new product. What
questions should they be asking and what types of experiments should they be
running as they get started?

The team should create funnel tests to see if they can convert more users to revenue.

The team should focus on testing user motivation.

Correct

When first building a new product, experiments should focus on user motivation and why
questions rather than usability. You can deliver outstanding usability but if the user just isn't very
motivated it will be waste (and this happens a lot).

The team should focus on testing usability.

The team should design a cohort test to analyze how changes in the user journey will affect
outcomes.

Question 4
Correct
1/1
point

4. Question 4
Why might a team decide to scrap a feature after testing it?

After 30 days, users are taking advantage of the feature in less than [threshold percentage] of
interactions.

Correct

If, despite initial 0 day success on usability, users are no longer using the feature after a month, it
probably means it's not valuable enough in its current form and should be scrapped or
substantially rethought.

After a month, more users than the failure threshold are using the feature.

In usability testing, the majority of users can't complete the core tasks with the interface

After several months, the feature is continually used and users share anecdotal evidence that they
like what the feature is doing for them.

Question 5
Correct
1/1
point

5. Question 5
You're part of a team developing a healthy living app. The app is designed to
help users monitor blood pressure and heart rate and also encourages users to
make healthy decisions such as getting more sleep, exercising, and drinking
water. The team's tests find that some users have made the app a habit, but the
team would like for more of them to use it more regularly. In terms of the Hook
Framework, why might the team investigate the impact of sending text message
reminders for users to drink a glass of water at periodic intervals throughout the
day?

The text is acting as an external trigger for the user to take action with the app.

Correct

A trigger is an internal or external stimulus that leads to action. Successful triggers help products
become habits.

The text increases the anticipation of an uncertain reward.

The text will deepen the user's investment in the app and make it less likely for the user to switch
to another similar product.

The text increases the action required of the user.

QUIZ-2

Week 2 Quiz: Is it usable?

Quiz, 5 questions
4/5 points (80%)
Congratulations! You passed!

Question 1
Correct
1/1
point

1. Question 1
Your team has a set of user stories they're preparing for implementation.
They've reviewed patterns and comparables, chosen an approach they want to
explore, created an interactive protototype, used the template to prepare
research details for the first usability test, and screened subjects. What else do
they need to do in order to make sure they are ready to test with subjects?

Test the test plan.

Correct

Getting subjects is expensive, so the team should run through the test to make sure the
infrastructure is set and the moderator has practiced.

Decide what kind of test to use.

Make sure the subjects selected will be appropriate for the test.

Prepare questions about signifiers, including color and style.

Question 2
Correct
1/1
point

2. Question 2
Your team's first usability test bombs. The test subjects seem perplexed by the
test items and indicate that they aren't familiar with the narratives/actions being
tested and wouldn't need to execute these tasks in their daily work. What could
your team have done to prevent this from happening?

Use a different kind of test so users will be less confused

Make sure the subjects selected are appropriate for the test

Correct

Screening test subjects is an important step and will ensure that the test provides valuable
feedback.

Practice and make sure the infrastructure of the test site is functioning smoothly

Prepare questions about signifiers, including color and style

Question 3
Incorrect
0/1
point

3. Question 3
During usability tests, you observe that subjects are particularly confused at a
couple of key points where the website uses different approaches/styles for the
same kind of action. How could the team remedy this problem?

Write a positioning statement briefly describing the company and the product.
Ask someone on the team to articulate their favorite design styles and adopt that across the
application.

This should not be selected

While consistency would certainly help alleviate some of the usability issues raised by test
subjects, the design choices should be meaningful and relevant to the users and not to an
individual on the team. Review the important ways design can improve usability in the "Creating
Consistency" video.

Create a style guide so the design will be consistent across the product.

Embrace current design styles, especially spare designs featuring limited text and other features.

Question 4
Correct
1/1
point

4. Question 4
You are creating a website for a network of bed and breakfast establishments.
Which of these would you avoid as you are developing the product?

Hire a professional designer, if the budget allows.

Invent novel signifiers and ways of categorizing and filtering the properties.

Correct

Users have learned behaviors and do best with commonly encountered patterns. It would be best
to avoid novel signifiers and affordances unless there's an extremely compelling reason.
Look at how popular vacation rental websites display features and information about each
property.

Conduct usability tests, even for seemingly simple items like finding a location.

Question 5
Correct
1/1
point

5. Question 5
A colleague just handed you a draft of an introduction to a usability test and
asked for feedback. How would you suggest he revise the introduction below?

Introduction DRAFT Thanks so much for taking time out of your busy schedules to help us out
today. I know we are on a tight schedule, so I'll get us started. To begin, let's look at this
screenshot of the registration page an undergraduate would see after selecting classes that are
required for her major. Tell me what you see here. If you were this student, how would you
proceed from here?

Provide a specific motivation/goal to the subject so they have a clear understanding of their
objectives.

Correct

This draft introduction combines motivation and usability, which will jumble the results. Since
you're testing usability, not motivation, it is vital to supply specific motivation to the subject
(such as, "Go ahead and register for Econ 101") and see if they're able to accomplish the task. As
you begin asking questions, you'll move test subjects from the user story to the test items.

Ask the test subjects about their signifier preferences.

Add reassurances that you'll be there to guide them every step of the way.
Advise your colleague to hold off on the usability test until a working version of the website can
be tested, rather than an interactive prototype.

Week 3 Quiz: Did it break?

Quiz, 5 questions
3/5 points (60%)

Try again once you are ready.

Required to pass: 80% or higher

You can retake this quiz up to 3 times every 8 hours.
Question 1
Correct
1/1
point

1. Question 1
What's one reason a tester might prefer to work on a DevOps-oriented team
practicing continuous development, rather than in a more traditional, legacy-
style environment?

Automated testing frees up time to focus on building smarter process that avoid tedium

Correct

Rather than focusing on manual testing, testers in a DevOps environment are more likely to be
able to thoughtfully consider post mortems, discover why are things breaking, invest in
automation, and determine how to improve processes.

In DevOps, coders are more likely to be blamed for errors than testers.

There's far less testing in a DevOps environment.

The tester prefers the clear handoffs typical of DevOps teams.

Question 2
Correct
1/1
point

2. Question 2
You are managing a complete, interdisciplinary agile team that is working
together for the first time. What habits and routines should you try to instill in
the team?

Let's not spend time establishing a shared vocabulary or formalizing polices.

As we approach the end of an iteration, acceptance tests become a more important than user
stories.

As long as we catch the bugs, it doesn't matter when we find them.

As we progress through the continuous delivery pipeline, we can allow more time for the later
stage test suites to run.

Correct

The amount of time automated tests take to run increases as the build goes from small to large
tests. At later points, teams are more confident that the code is working and are ready to broaden
the scope and duration of the test suites.

Question 3
Correct
1/1
point
3. Question 3
Your team is thinking about how to prioritize and focus the functional (medium,
large) testing. What would you suggest they do?

Make sure the testers on the team have their time freed up to focus on the testing, and have
developers ready to tackle the list of issues as soon as it is complete.

Make sure to budget enough time at the end of the iteration for testing

Systematically identify all possible tests and motivate the team to finish them as efficiently as
possible.

Pull in the narrative and observational results you've used in development

Correct

You can't test everything. Looking at the current iteration as a uniform lump of software that you
should test as much as you can will result in a wasteful allocation of resources. Instead, test the
'happy path(s)' all the way through the software using realistic input the team has drawn from
their user research. Leave testers time to explore and consider how best to test the features as
they're implemented. Then move down to lower priority narratives. This prioritization should
roughly mirror the work you did with prioritization through story mapping (horizontal slices,
etc.).

Question 4
Incorrect
0/1
point

4. Question 4
After release, a problem is discovered in the train ticketing application your
team has built. How should the team proceed?
Someone should quickly fix the problem by any means necessary.

Be sure to document any configurations or updates required for a fix through version control.

The team should start with a fresh install by hand to be sure everything is correct.

Fix the problem as efficiently as possible and don't worry about the documentation needed for
changes in other environments.

This should not be selected

It can be tempting to just start manually tweaking things in production to get it fixed--and when
it comes down to it it's important that individuals do have discretion to do what they think makes
sense. But even if this results in faster resolution (a proposition which itself is debated), you now
have a production asset with undocumented or semi-documented changes. Will you remember
all these particular details in production next time you need to diagnose an issue? Will someone
else? How will this affect your ability to confidently test new code and updates upstream in your
continuous delivery pipeline? See the "Release Stage" video to review.

Question 5
Incorrect
0/1
point

5. Question 5
A team is gradually adopting continuous development practices. They get
together to discuss how things are going after every sprint, but they are having
trouble finding the best way to look at their progress over time. How would you
recommend that the team improve its retrospectives?

The team should take notes on their retrospectives so they can keep track of decisions and
evaluate their improvement.
The retrospective facilitator should prepare for each meeting by creating an agenda.

The facilitator should ask team members why they think things are happening.

The team should hold a 1-2 hour retrospective after every two-week sprint.

This should not be selected

This team is holding regular retrospectives, but there are ways they can better assess their
improvement over time. Review how agile teams can best conduct and learn from retrospectives
in the "Test-Driven General Management" video.