Thinking Aloud Test Plan

1 Introduction

The website that will be tested will be the website, of the in Graz based, Clothing shop Kastner & Öehler. User can browse the website for the products they are offering and also order them online. With this thinking aloud test we want to get insights of the thought process of different chosen users by letting them perform some prepared tasks.

2 Test Methodology

The definition of a thinking aloud test is, that you ask several people from different user groups to use a specific system, for example a website on a specific device and while doing this they should speak out their thoughts loud while they are navigating through the interface. It depends on which interface or system you are willing to test but basically this test will be done as follows. First you recruit different users that fit your needs or skill level for the system to be tested, then you hand them out representative tasks to perform. And while they are working with the given tasks you just listen to their thoughts while doing so and write them down, or take a video or audio of the session. So this test doesn’t take a lot of time planning and it is really cost efficient, but you also get a lot of different opinions since you have many test users, maybe from different user groups. [Nie2012]

With the use of a thinking aloud test a users’ behaviour and interaction with a website can be observed and evaluated. In such a way the software programs or websites can be designed that the customers’ expectations are more satisfied. There are several advantages of such tests. One example would be the low cost since no experts and no special equipment is needed. Another one would be that spontaneous reactions are gathered from the test users. Even if the methodology is not followed correctly, valuable results can be gathered still. The maybe biggest advantage is that this method is easy to teach, so no specialists are required. One disadvantage would be that such tests are mostly created under laboratory and not authentic conditions. Since the test user has to think out loud, this can have an influence on the test users’ actions and increase the cognitive load. [Med2022]

3 User Profiles

Since Kastner-oehler.at is mainly an online store for fashion and clothing, it’s target group are people who are interested in buying new clothes and have a certain degree of fashion understanding. The product catalog of this online store includes clothing for children, men, women and additionally extra sections for certain groups such as the “Infected”-section, which mainly targets teenagers. Furthermore, the online store has sections for other products such as beauty products or things in the field of home and living to offer. Therefore, the site is trying to attract a very broad field of potential customers, which in general are very hard to group. One way of grouping these users, is after their own fashion understanding respectively their fashion sense and their reasoning for the site visit. People which are into fashion and beauty are more interested in finding new trendy clothes and products and are more selective in their choosing and therefore need more options of selection and filtering to get to the things which they have in mind. In contrast users, which are not really into fashion, want to get multiple recommendations of standard things so that they can search and compare which of the recommendations suits them best. But for all of them the main task is the searching and buying of products.

For testing, the main target user group should be a group of young adults who are experienced with online shopping and generally are comfortable navigating though websites on various platforms. Because of the past experience with different online shopping platforms they may experience things while performing tasks which is, in their eyes, better or worse than at other comparable websites. But since the website targets a wide range of user groups, also less experienced users should be considered to ensure a easy usage of the offered functions and the navigation through the website.

4 Test Users

Table 1 gives an overview of the test users planned to participate in the study.

5 Test Environment

All users will use the same test environment for the thinking aloud test. The planned environment for the tests is shown in Table 2.

For this test, all adblockers will be disabled. The acceptance of cookies will be left up to each test user, because users should act as they would when browsing this website privately, but it can also be a reason of the design if users will just quickly click through the prompt for the cookies or if they deactive some of them.

6 Recording

For the external video capture we will use a Nikon D5300 with an 200mm lens on a tripod. As a microphone we will use a HyperX Cloud 2 Headset.

The screen, during the testing process, will be captured by the software "OBS Studio" which is an free and open source software that can be used, among other things, for screen, webcam, a integraded camera from realtek and microphone recordings.

The recorded clips will be further processed with the program Sony Vegas Pro 15, which is a video editing software. For the blurring of the faces of the test users, we will use the boxblur function of ffmpeg. With this function a box can be created in the video which will be blurred.

7 Training

For every test user the same notebook with the same mouse and keyboard will be provided by the test team. Since the test users are familiar with the use of computers and interacting with the internet/website, there is no special training needed.

In the beginning the facilitator will explain to the test user how the thinking aloud test will be done. Also, a simple task is performed by the facilitator for better understanding. If something is still unclear, the user is always allowed to ask questions. Furthermore, a video of a previous thinking aloud test is available and can be shown to the user if needed.

8 Test Tasks

Table 3 shows the tasks which we suggest might be used.

9 Interview Questions

The following questions will be asked of each user immediately after the final task.

• Opening Question
  • "How was it?" ["Wie war's?"]
• Standard Questions
  1. "Did anything strike you as particularly good?"
  2. "Did anything strike you as particularly bad?"
  3. "Which task was the hardest and why?"
  4. "Do you think the website is easy to use? With reasoning please.”
• Individual Questions

  If during the test with each test user any questions come up, the user will be asked that questions as well.

10 Data Collection and Collation

Before the task starts the user gets a simple introduction and has to sign a declaration of consent. Then the facilitator asks the test user the questions from the background questionnaire and fills in the given answers. All collected data will be assigned to a test user id and first name aliases.

The test user will get a sheet of paper with all the tasks on it. The user has to read the tasks and if everything is clear the test will get started. Otherwise, the user can ask questions. A task is either over if the test user reaches the goal or the facilitator decides that the task takes too long. After all tasks are done the test user gets a feedback form, which he has to fill out.

Immediately after the thinking aloud test is done the handwritten questionnaires (background, feedback) as well as success measures and observed problems are transferred to a spreadsheet shown in Appendix A.7.

Any faces occurring in the recordings will be blurred or pixelated. All collected personal data, like name, signature and the full video recordings will be deleted after one year.

References

[Nie2012]

Nielsen Norman Group; Thinking Aloud: The #1 Usability Tool.; retrieved on 28 Apr 2023 https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/

[Med2022]

Torresburriel Estudio; Thinking Aloud Test:pros and cons; retrieved on 29 Apr 2023 https://uxtbe.medium.com/think-aloud-test-pros-and-cons-4cce1452f693

A Test Team Materials