Playtesting for The Automatician Block D, 2016

Goal
Our initial goal of the playtest session was to get a feel for the difficulty of the
puzzle. Another thing we wanted to learn was what makes or what does not make
the puzzle fun to play, and get a general response to the way puzzle pieces
connect to each other. We also recorded the time taken for each puzzle.
Before the playtest started we thought that the number of puzzle pieces highly
influences the difficulty and that the fun the player experiences is closely related
to the difficulty.

Set-up
We knew we wanted to ask the player some questions during and after the
playtest but were not sure what the best way to do this was. At first we started
with a survey that the players could fill in after the playtest but the problem could
be that players would forget the first few puzzles and the later puzzles would
influence their grading of the first puzzles.
Instead of the survey at the end of all the puzzles we decided to ask players to
grade the difficulty and fun, how much they enjoyed the puzzle, after every puzzle
they played. We asked for a grade between 0-10, 0 being the lowest and 10 being
highest, we did this so we could easily compare the results between players and
playtests. It was also useful for the quantification of the puzzles, which was being
worked on by another team member at the time.
For the physical set-up we used two computers and a second screen. We would sit
players on one side of the table with one of the computers and have the other one
on the opposite end with the second screen. One person would explain the game
and the mechanics to the player and ask and answer questions. The other person
would note down the answers given, to the player this was the only thing the
second person was doing. But on the second screen the person noting down the
answers could also see the players screen and would look at the players behavior
and add additional notes based on the players behavior.
This set-up was used so that the player, who was unfamiliar with the game and
the environment they were in did not feel pressured and would act the same as
they would in a more familiar environment. When the player think no one is
looking at their screen they are not as afraid to make mistakes and try out new
things, otherwise players might get embarrassed and feel bad when they cannot
find the solution quickly.
On the next page images can be found of the set-up used during most of the
playtests.

Execution
The first time we set-up for the playtest we got a couple of people from the team
who enthusiastically wanted to try out the new puzzles we created. We used this
opportunity to fine tune and test our process. It quickly became clear that 1 puzzle
(Music room 3) was clearly too hard for player in the current state, some players
would take over 30 minutes to complete a single puzzle. This is not what we and
the player wants, and we decided that if the player has not found the solution after
5 minutes there is a slim chance they will find it after that. Therefore we decided
that after 5 minutes we would place the first puzzle piece for the player and every
3 minutes after that the next piece. This was done to speed up the process and not
the bore the players too much.
A spreadsheet was created to quickly gather all the data. Here we would put in
the raw data and after each playtest we would the data to a different
spreadsheet.

Figure 1 - Spreadsheet for raw data

The image above is a screengrab from the spreadsheet were the raw data was put
in. We asked players how experienced they were with puzzle games, recorded the
time, noted their different approaches for each level, and asked their enjoyment
and difficulty.
We were also invited to an elementary school to show our game and explain
what we do (at NHTV). I used this opportunity to let children experience and
playtest the game.

While the children respond

positively to the game and
puzzles, their playtest data
was not used as it was too
biased. We did not get to use
our normal set-up and the
children pressured each other
into solving the puzzles as
quick as possible. The
questions we asked were also
aimed at adults and not
children, the children had a
hard time grading the
difficulty and fun from 0-10.

Analyzing
After each playtest we would analyze the data gathered and make changes based
on the data and player behavior. While at times we had many thing we wanted to
change, the key was to focus on few changes to ensure that the results are relevant
to what changed.

Figure 2 Analysing playtest results on a whiteboard for quick notes

Sometimes we did make multiple changes, to ensure the changes did not
influence each other we had control puzzles where we would only change 1 thing
and could compare that puzzle to the others. This made sure that the 2 changes
did not influence each other as expected.

When we were happy with the change(s) made we would set-up a new playtest
session, we had a total of 4 playtest sessions with around 8-15 people each.

Conclusion
Overall I think we had four successful playtest session and learned a lot about our
puzzles during that time. While there are still some point to improve on, like a
broader audience playtesting and explaining the game to each player in the same
way, the results were still valid and came to good use iterating on the puzzle we
had created. After having completed four playtest sessions we narrowed down
where our problems are, what players want to see more and were able to sort our
puzzles by difficulty.

Results
Playtest 1 vs. Playtest 4

Figure 1 - Time taken per puzzle, playtest 1 vs playtest 4

The average total has not changed, playtest 1 had an average time of 29:52 and
playtest 4 a time of 29:56. What we do see is the spikes are a lot less and the time
taken scales well over time. But most importantly not a single puzzle stands out
from the others.

Figure 2 - Difficulty per puzzle, playtest 1 vs playtest 4

Similarly to the time taken per puzzle, most of the spikes have been removed.
Puzzle do not reach too high numbers anymore as in playtest 1 (music room 3 had
an average difficulty rating of 9.6), the highest difficulty rating from playtest 4 was
7.7.

Figure 3 - Fun rating per puzzle, playtest 1 vs playtest 4

The overall fun has increased, mostly in the earlier puzzles, while also reaching its
highest point in playtest 4 in the last puzzle (an average fun rating of 8.1).
Increasing the enjoyment players have during the puzzles was the hardest as
different players have different definitions of fun, however from this graph we
conclude that the third and last puzzle have been positively changed.