Thinking Aloud Test Plan

1 Introduction

The website is an online retailer that specializes in selling a variety of sports equipment. These include dumbbells in all variations, machines for muscle training or cardio. You can also find other usefull gadgets like massagers, table tennis equipment, trampolins and little plastic bikes for kids. They also ship in every european country.

The goal of a thinking aloud test is to really get a picture of what a possible user might be thinking while browsing through the website. To achieve this goal the participants are asked to verbalise their thoughts while performing certain tasks. They should provide commentary on what they are seeing, thinking and doing. There will be one facilitator who should encourage the participant by giving feedback or asking questions to keep up a flowing commentary.

2 Test Methodology

A TA Test (or thinking aloud test) is a test where a certain number of participants are browsing through the website we want to test and giving live feedback while performing tasks. In other words, their task is to provide running commentary on what they are seeing, thinking and doing. Thinking aloud tests usually have a relatively small number of test users. We plan on interviewing 5 people where the first one will be more of a test run to make some last second adjustment to our setup or other parts of the test and therefore will be approximately one day before the others.

For each test, there will be one facilitator [Pid2008] who should give feedback to the participant or ask them questions to keep the commentary flow. The facilitator should for example ask the participant what they are trying to do, what they are reading, what they find confusing or what decicions they are making throughout the test. The facilitator should interact with the user in small actions like head nodding or simple feedback prompts like "uh huh" or "yeah". He will not answer to actual questions until after the thinking aloud test is completed. In order to prepare our test users we will show them video clips of our test run aswell as making a practice run with another website so they get a feeling for their task.

The thinking alout test will be recorded via screen recording like in the heuristic evaluation. Additionally we will record the user via a camera behind them to see the screen and some of their body language. We will put a mirror on the desk to also capture their facial expressions.

The reason why we choose to do a thinking aloud test is that we want to find as many usability problems as possible and why they occur. A lot of these reasons can be found on the Nielsen Norman Group Website [NNG2012]. And since it doens't require a large amount of participants, the thinking aloud test is perfect to achieve our goal. The only real drawback of this test is that the user might change his problem solving behaviour since they might think before acting.

A general overview of the thinking aloud test can be found in Keith Andrews' course notes [And2023]. The thinking aloud method is described in detail by Barnum [Bar2020].

3 User Profiles

The website tries to attract everyone who wants to engage in sports. From the casual athlete to the determined fitness-freak. Since the website has a really broad collection of sports equipment it really offers something for everyone. We came up with the following groups:

• Dedicated/high performance group: People who follow a certain sport on a professional level, like boxers, weightlifters, etc.
• Casual workout group: People who set themselves a goal, like loosing weight, building up strength, etc. These people probably visit gyms regularly and want some of the sports equipment in their own home.
• Entertainment group: People who search for entertainment more than having a set goal in mind. This group is more interested in trampolines, table tennis, slacklines, etc.
• Wellness, Yoga, Massage group: People who enjoy a more calm sports like yoga or gymnastics or just want to relax a bit.
• Sportswear, additional features group: People who don't look for sport devices but rather sportswear, or small fitness helpers like pulse monitors or cycling watches. We also added users who are interested in sport nutrition to this group.
• Committed givers: People who want to buy a present for children, like small scooters, gokarts or stall bars.

For our thinking aloud test we decided to focus on the user group "Casual workout group". Everyone of our participants visited a gym at least once in the last 30 days but eveyone for their own individual reasons.

4 Test Users

Table 1 gives an overview of the test users planned to participate in the study.

5 Test Environment

The planned environment for the tests is shown in Table 2.

In order to get a somewhat similar system for everyone we decided that our test users should not have access to an ad blocker. The management of cookies altough will be left up to the users preference since some of our findings include the management of cookies. But in order to have the same chances for all users they will be reset after every test.

6 Recording

To record the thinking aloud test we will use the external equipment which is provided to us via our lecturer Keith Andrews. The usability kit contains:

• Digital Camcorder Sanyo Xacti HD1010 (SD-Card)
• Transcend SD HC Card 16 GB
• Stereo Table Microphone Philips SBC ME570
• Rucksack LowePro
• Tripod Hama Profil 74
• Headphones

For screen capturing we use OBS Studio. After that we will put the video into Adobe Premiere Pro and start cutting the clips to illustrate our findings. This program also offers the function to blurr out faces which we will use to protect the privacy of our test users.

7 Training

Everyone of our test users is confident with using a laptop or a technical device in general so there is no need to train how to use those. But if any questions from our participants arise we planned a short time window before the test to answer them. For all five participants we will tell them the general process of a thinking aloud test. Once our pilot test is done we plan on showing the other four participants clips from the first test. The facilitator will then demonstrate how the thinking aloud tehcnique works by giving a short example. We also plan on making a small practice run with each test user on a completely unrelated website to give them a feeling on what to expect and train how to keep the commentary flow going.

8 Test Tasks

Table 3 shows the tasks which we suggest might be used.

9 Interview Questions

The following questions will be asked of each user immediately after the final task.

• Opening Question
  • "Wie war's?"
    ["How was it?"]
• Standard Questions
  1. "Gab es etwas, dass besonders gut oder positiv aufgefallen ist?"
     ["Did anything strike you as particularly good?"]
  2. "Gab es etwas, dass besonders schlecht war?"
     ["Did anything strike you as particularly bad?"]
  3. "Hatten Sie Probleme mit den Aufgaben die wir Ihnen gegeben haben?"
     ["Did you have problems with the tasks we gave you?"]
  4. "Fühlten Sie sich überfordert, als Sie dass erste mal die Website besuchten?"
     ["Did you feel overwhelmed when navigation through the website for the first time?"]
• Individual Questions
  At this point we will ask questions depending on each individual test user. For example if the user struggled with a certain task of the test we will now ask what the problem was. If some findings match the ones from the log we will also evaluate on them. Also, if the user had some questions during the test we will answer them in this section.

10 Data Collection and Collation

In order to keep our participants identity private, we are using first name aliases and assign everyone a testperson ID (TPid).

Since our tasks have a time limit we have to define a clear start and end point. Every task will be read to the test user by one of us and once the task is clear we will start the timer. We will stop the timer as soon as the test user has successfully completed one of the tasks. If the test user is not sure whether he has completed the task or not, we will give feedback. A new task will only start if the test user tells us that he is ready.

The result spreadsheet will have five main topics:

• Background Q: This section will focus on the test user and their background. We will insert age, gender, occupation, their education, experience with computers and technology in general, sight impairment and if this is their first Usability Test. This data is useful so we can check the differences between our participants. It is important that our test users are not too similar to get better results.
• Task Success: In this section we will keep track of the ammount of tests each individual tester completed successfully.
• Negative findings (Problems): Just like in the Heuristic Evaluation in this section we will enter the problems our test users encountered during the Usability test and evaluate them. We make clips from the original recording to back up the found problems.
• Positive findings: This section is identical to the one above but this time we keep track of the positive findings instead of the negative ones.
• Feedback Q: In this section we will insert the results of our feedback-sheet. The test user has to rate the website on different aspects like appearance, consistency, quality, etc. We will then continue to calculate the mean all results to make the feedback as accurate as possible.

The data and results from the tests will be collated in the spreadsheet shown in Appendix A.7.

Every item which contains personal data of the test users (like name, signature or the video recordings) will be stored as a proof for one year and afterwards they will be deleted.

Once again to keep the identity of our test users private, we will blurr out their faces in the video recordings.

References

[And2023]

Keith Andrews; Human-Computer Interaction: Course Notes; 13 Mar 2023. http://courses.isds.tugraz.at/hci/hci.pdf

[NNG2012]

Nielsen Norman Group; Thinking Aloud: The #1 Usability Tool; 15 Jan 2012. https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/

[Pid2008]

Pidoco Powerful Prototyping; Moderator (Description Page); https://pidoco.com/en/help/ux/moderator

[Bar2020]

Carol M. Barnum; Usability Testing Essentials; 2^nd Edition, Morgan Kaufmann, 2020. ISBN 0128169427.

A Test Team Materials