27 May 2021
Preparing a Hot/Cold Meter Usability Study
UX research is a muscle every product designer should be able to flex. Side projects are a great way to refine and understand the research process. Following a process, eliminating bias, practicing empathy, and gaining insights from real users are all skills that grow with practice. We can also share all the details of our research including our methods and process with you. This is the first of 3 posts centered around our usability test conducted for our Geo Racer app.
Geo Racer and the Hot & Cold Meter
Geo Racer is a multiplayer racing app our Gaslight team built for a development competition. Our goal was to build something that our families and friends could play while also exploring the possibilities of Elixir and Phoenix LiveView. LiveView is a promising technology that allows us to update the browser with server-side code.
At the core of our app’s gameplay is the hot and cold meter. It is designed to help players find waypoints in a designated racecourse before their opponents do. As the name suggests, the hot and cold meter will change color from blue to red to indicate as each user gets closer to the next point.
Usability Testing
Usability testing is one of many UX research methods for building useful and beloved products. In its simplest form, a usability test should explore users in the application in order to identify frustrations and problems with a feature or task.
Defining Goals and Scope
With every type of UX test or interview, there should be a goal/hypothesis and a defined scope. I always recommended limiting the scope as much as possible. Examining one feature at a time helps focus the feedback and ensures a more accurate outcome.
There are many features we would like to test in the Geo Racer app, but the hot and cold meter is the most logical starting point. The meter is the main tool used to play and win the game. It’s also a unique element and not based on existing UI patterns, making it a clear candidate to be tested.
The game is designed to be played with other racers. There are some fun elements of the game when playing other opponents, like sending hazards that obstruct each other. Because our focus is studying their ability to find waypoints with the meter, we will have each racer complete the race alone.
We also want to keep our testing goal and hypothesis focused within our scope. Our goal needs to define our scope and focus while the hypothesis provides us a measurable usability goal to prove or disprove.
Goal: To understand how effective the hot and cold meter is at relating a waypoint’s direction to the player.
Hypothesis: The more a player uses the hot & cold meter in Geo Racer, the quicker they will be able to find waypoints.
In order to measure the success of a user’s ability to read the meter, the test will require us to collect quantitative data. We will also need qualitative data to provide deeper insights into what a user does and does not understand. A combination of rating questions, long-form questions, and recorded times will help us measure the experience from multiple perspectives.
Qualifying Survey
Qualifying surveys are very helpful when targeting test users that will potentially use an application in real life. In this usability test, we are hoping to get a range of perspectives on how useful our hot and cold meter is. Our only real qualifying question is confirming the user has a device they can play the game on. We will use the survey to balance out our test player’s age range, mobile gaming experience, and outdoor game enthusiasm. That is why we are keeping this survey short and simple.
Questions
- Please Provide Your Age
- Do you have a mobile device with cell service?
- How many people live in your household?
- How often do you play games on your mobile device?
- Do you play outside mobile games like Pokemon Go? Can you please name them?
- Do you play games outdoors with friends and family?
- Are you interested in finding new and fun games to play on your mobile device?
Testing Script
A simple and clear script can be one of the best tools for a consistent unbiased user test. The script can help us avoid providing different instructions and contexts for all the test users.
Script
Hello! I am [ test guide name ] with Gaslight. We have built an app called Geo Racer. You would normally play with friends using the app on your own mobile devices. The goal is to reach the waypoints before your opponents and there are tools in the game to make it difficult for your friends to finish. Today, we would like to observe you racing to several waypoints we have placed around this park. I will start the game for you now and you will have to use the game screen to find each of the four points. While you are racing, we encourage you to talk aloud about what you are seeing and experiencing. Please yell out when you reach each waypoint.
[ the player races while our team records observations ]
Thank you for playing. I am now going to ask several questions about your experience. Please provide as much detail to your answers as possible.
Thank you again for your time. You can find Geo Racer at georacer.io and save the app to your home screen.
Test Questionnaire and Recording Data Points
Our Questionnaire will provide the majority of the data for our research and will be completed by the user immediately following their race. While this test calls for observing a complete race before asking questions, for many studies it can be helpful to ask questions while a user completes a task.
Questions
In our questions below, we have mixed in multiple ones that require long-form answers and some that require a rating. This provides the ability to measure responses and have the detailed thoughts of real users. We also end with a solicitation to download the app today, to see if the gameplay was enjoyable enough for them to commit to putting the game on their device. The questions follow a logical progression through the gameplay.
- First Name and Last Initial
- Were you able to complete the Geo Race? If so, how easy or difficult was it to complete?
- Could you tell what the meter was doing when you started the race?
- What indicated to you how the meter worked?
- Describe what it was like finding each waypoint with the meter?
- On a scale from 1 to 10 (1 being least and ten being most), what was the effort needed to find waypoints in the app?
- Did the effort to find waypoints change as the game continued? If so, explain how.
- Were there any problems finding the waypoint? If so, explain what happened.
- On a scale from 1 to 10 (1 being least and ten being most), how likely would you be to play again?
- Does the game remind you of another game or app you have played before? What was that game like?
- Was the game fun to play? What did you like about it?
- On a scale from 1 to 10 (1 being least and ten being most), how enjoyable was the game to play?
- Would you download the app today to play again? Why or why not?
Data Points to Record
While the user is completing the race our team we are observing. We will be making notes on what the tester says and recording data points using a stopwatch that will help us measure against our hypothesis.
- The time between the start and moving towards the first waypoint
- Time to find the first waypoint
- Time to find the second waypoint
- Time to find the third waypoint
- Time to find the fourth waypoint
- Time to complete the game
Preparing Docs And Tools
There are many great tools that can help us record and keep track of our data. Because we will be outside conducting this study I would like to keep it simple and have something I can use on my tablet.
We will use a Google form for our qualifying survey and a separate Google form for recording the question responses and race data. Both forms will be converted to a Google sheet after the field test. We will also use a tool like Hotjar or Fullstory to record the screen sessions of each tester. This will help us verify the waypoint times and possibly gain other insights into their race.
Now that we have our survey, script, and testing questions it’s time to play some Geo Racer with our friends. We will be back soon to share how we conduct our usability test.
