Thinking Aloud Test Report

3.1 Test Methodology

A TA Test (or thinking aloud test) is a test where a certain number of participants are browsing through the website we want to test and giving live feedback while performing tasks. In other words, their task is to provide running commentary on what they are seeing, thinking and doing. Thinking aloud tests usually have a relatively small number of test users. We plan on interviewing 5 people where the first one will be more of a test run to make some last second adjustment to our setup or other parts of the test and therefore will be approximately one day before the others.

For each test, there will be one facilitator [Pid2008] who should give feedback to the participant or ask them questions to keep the commentary flow. The facilitator should for example ask the participant what they are trying to do, what they are reading, what they find confusing or what decicions they are making throughout the test. The facilitator should interact with the user in small actions like head nodding or simple feedback prompts like "uh huh" or "yeah". He will not answer to actual questions until after the thinking aloud test is completed. In order to prepare our test users we will show them video clips of our test run aswell as making a practice run with another website so they get a feeling for their task.

The thinking alout test will be recorded via screen recording like in the heuristic evaluation. Additionally we will record the user via a camera behind them to see the screen and some of their body language. We will put a mirror on the desk to also capture their facial expressions.

The reason why we choose to do a thinking aloud test is that we want to find as many usability problems as possible and why they occur. A lot of these reasons can be found on the Nielsen Norman Group Website [NNG2012]. And since it doens't require a large amount of participants, the thinking aloud test is perfect to achieve our goal. The only real drawback of this test is that the user might change his problem solving behaviour since they might think before acting.

A general overview of the thinking aloud test can be found in Keith Andrews' course notes [And2023]. The thinking aloud method is described in detail by Barnum [Bar2020].

3.2 User Profiles

The website tries to attract everyone who wants to engage in sports. From the casual athlete to the determined fitness-freak. Since the website has a really broad collection of sports equipment it really offers something for everyone. We came up with the following groups:

• Dedicated/high performance group: People who follow a certain sport on a professional level, like boxers, weightlifters, etc.
• Casual workout group: People who set themselves a goal, like loosing weight, building up strength, etc. These people probably visit gyms regularly and want some of the sports equipment in their own home.
• Entertainment group: People who search for entertainment more than having a set goal in mind. This group is more interested in trampolines, table tennis, slacklines, etc.
• Wellness, Yoga, Massage group: People who enjoy a more calm sports like yoga or gymnastics or just want to relax a bit.
• Sportswear, additional features group: People who don't look for sport devices but rather sportswear, or small fitness helpers like pulse monitors or cycling watches. We also added users who are interested in sport nutrition to this group.
• Committed givers: People who want to buy a present for children, like small scooters, gokarts or stall bars.

For our thinking aloud test we were notified, that we should investigate more on the user group "people who are interested in home fitness". So we searched for people at our university who fit into the test group and were willing to participate in the thinking aloud test. Since the description covers a broad variety of people, because "interested in home fitness" can mean that they are also very experienced on that topic or have no idea of home fitness but want to get started we tried to get a good mix of opinions.

3.3 Test Users

To select our test users we went to our university, the TU Graz and asked people who passed by us if they wanted to participate in a thinking aloud test. If people seemed interested we started asking them more specific questions about their fitness procedures to find out whether or not they fit our choosen test group. We asked questions like "Do you have any kind of sport equipment at home?" or "Do you visit the gym regularly? If yes do you also want to workout at home?". After gathering five participants for our thinking aloud test we were done.

Due to a member of our group not being able to record on the 16th of May we shifted our whole sheduling for one day. Therefore the pilot recording was on 17th of May instead of the 16th of May and the other test users were recorded on the 18th of May instead of the 17th of May.

Table 1 gives an overview of the test users who participated in the study. The mapping of real names to aliases is included in Appendix C.1. Scans of the original completed questionnaires are included in Appendix B.1.

3.4 Test Environment

The test room is shown in Figure 1.

The environment used for the thinking aloud tests is shown in Table 2.

The exact web browser version used for the tests is shown in Figure 2.

In order to get a somewhat similar system for everyone we decided that our test users should not have access to an ad blocker. That is why we decided to disable them on our system. The management of cookies altough will be left up to the users preference since some of our findings include the management of cookies. But in order to have the same chances for all users they will be reset after every test.

3.5 Recording

To record the thinking aloud test we used the external equipment which was provided to us via our lecturer Keith Andrews. The usability kit contains:

• Digital Camcorder Sanyo Xacti HD1010 (SD-Card)
• Transcend SD HC Card 16 GB
• Stereo Table Microphone Philips SBC ME570
• Rucksack LowePro
• Tripod Hama Profil 74
• Headphones

For screen capturing we used OBS Studio. After that we put the video into Adobe Premiere Pro and started cutting the clips to illustrate our findings. Since this program also offers the function to blurr out faces we will use that function.

3.6 Training

Everyone of our test users is confident with using a laptop or a technical device in general so there was no need to train how to use those. But if any questions from our participants arose we planned a short time window before the test to answer them. We told all five of our participants the general process of a thinking aloud test. We also asked every participant whether or not they wanted to see a short clip of a thinking aloud test to give them a feeling of how the test works and how they should behave. We not just showed them, but asked them in order to not put them under any pressure while doing the recording. Thoughts like "Oh I have to act like in the video, I need to talk more" could potentially lead to mistakes or being way too careful and we won't get a realisitic result from the TP.

3.7 Tasks

The internal task list used by the test team is shown in Table 3.

3.8 Interview Questions

The following questions were asked of each user immediately after the final task. Transcripts of the interviews can be found in Section 4.8.

• Opening Question
  • "Wie war's?"
    ["How was it?"]
• Standard Questions
  1. "Gab es etwas, dass besonders gut oder positiv aufgefallen ist?"
     ["Did anything strike you as particularly good?"]
  2. "Gab es etwas, dass besonders schlecht war?"
     ["Did anything strike you as particularly bad?"]
  3. "Hatten Sie Probleme mit den Aufgaben die wir Ihnen gegeben haben?"
     ["Did you have problems with the tasks we gave you?"]
  4. "Fühlten Sie sich überfordert, als Sie dass erste mal die Website besuchten?"
     ["Did you feel overwhelmed when navigation through the website for the first time?"]
• Individual Questions

  After the standard questions above, any individual questions which arose during the test with each particular test user were asked. These are detailed in the transcripts in Section 4.8.

3.9 Feedback Questionnaire

After the interview, the user was asked to fill out the feedback questionnaire given in Appendix A.7. The feedback questionnaire was given on paper. An overview of the results can be seen in Section 4.9.

3.10 Data Collection and Collation

In order to keep our participants identity private, we used first name aliases and assigned everyone a testperson ID (TPid).

Since our tasks have a time limit we had to define a clear start and end point. Every task was read to the test user by one of us and once the task was clear we started the timer. We stopped the timer as soon as the test user has successfully completed one of the tasks. If the test user was not sure whether he has completed the task or not, we gave them according feedback.

Data from the background questionnaires, task success measures, observed problems, and ratings from the feedback questionnaires were collated into the data spreadsheet (see Appendix A.8).

Every item which contains personal data of the test users (like name, signature or the video recordings) is stored as a proof for one year and afterwards they will be deleted.

Once again to keep the identity of our test users private, we will blurr out their faces in the video recordings.

4.1 Discussion and Analysis

The following will discuss and analyze the main lessons we learned from the thinking aloud test. Four out of the five tasks were all completed by all test users, which is a good indication for the design of the website. Only one task was universally difficult or not completed, which is the task to find the shipping cost for a 30kg package. Most test users had no idea where to look, which has to do with them not knowing that much about shopping websites. Typically the shipping costs are situated at the very bottom. Some test users knew where to look, but still couldn’t find it. This has to do with shipping being situated in an atypical position. The lesson learned here is that conventions should have been followed more closely and maybe the makers of the website could have found a way to make it easier for the average user to find.

The next lesson, though not nearly as significant, is about the user login page. All users could find the page but as soon as they were on it you could see overwhelmed faces. On the left there's the user login, on the right you can create a new account. There is no clear indication which one is which at a first glance, which is a very important distinction to make life easier for the user. Some users even started typing in the wrong one before realizing it. So as a designer you should make sure the user can easily differentiate between the two.

Another lesson is about task 5. Users were asked to add a full set of equipment to the cart. Everyone was able to complete the task but you could see similar struggles with most users. The categories are not really made for this task, some articles were in the wrong category and users only found them by chance. This may be by design to make customers buy a finished set. However in general, more categories would have been a great addition.

4.2 Task Completion

A summary of how many users completed each task and whether any assistance was given is shown in Table 5.

4.3 Impressions

Tables 6a, 6b, and 6c show a summary of the responses of the test users to the questions asked after the first task.

4.4 Top Three Positive Findings

The three most positive findings according to their average (mean) positivity ratings are described in more detail below. The positivity rating scheme used to rank positive findings is shown in Table 7.

4.5 List of All Positive Findings

Table 8 shows a list of all the positive findings which emerged from the test, sorted in descreasing order of average (mean) positivity, i.e. the most positive are at the top of the table. Positivity ratings were assigned by members of the test team, their name codes are shown in Table 9.

4.6 Top Five Problems

The five most serious problems according to their average (mean) severity ratings are described in more detail below. Problem number 1 is the problem (negative finding) with the highest mean severity. The severity rating scheme used to rank problems is shown in Table 10.

4.7 List of All Problems Found

Table 11 shows a list of all the problems observed in the test, sorted in descreasing order of average (mean) severity, i.e. the most severe are at the top of the table. Severity ratings were assigned by members of the test team, their name codes are shown in Table 12.

4.8 Interviews

TP1 "Stuart" really liked the design of the website, even though he stated, that he was overwhelmed by the amount of information on the front page at first. Somehow he wasn't asked for a cookie selection box and he didn't like that. Since he couldn’t find the information about the delivery fee he made a few suggestions about how to solve the problem. But overall his impression of the website was very positive.

TP2 "Silvia" stated, that she was very overwhelmed of the websites design. She told us about it in almost every answer of the interview. The only positive thing mentioned about the website was, the fact, that she had seen worse websites of that kind. She also did not like the amount of products in a single category, it was not enough for her likings, and found that a bit disturbing, but it did not seem to bother her that much, in comparison to the overwhelming design of the website.

TP3 "Sally" on the other hand really liked the design. Also she didn't have any complaints, she just gave a suggestion on how to solve the delivery fee problem, but overall it was a positive experience for her.

TP4 "Martin" was pretty neutral about the website. He said he didn't like the design that much, but it is not a big problem for him as well. He just didn’t like the way the delivery fee information was handled by the websites designer and he thinks the frontpage is way too clutterd with all that information that is thrown at you.

TP5 "John" said he was not disturbed by the front page that much. He said it was a lot, but he kind of expected it to be, since amazon was looking the same to him. But he also told us, that the website wasn't special in any way to him, just a shopping site that gets the job done. But he was very impressed by the categories and the search method, because it shows you exactly what you think you are looking for.

4.9 Feedback Questionnaires

Table 13 shows a summary of the ratings given by users in the feedback questionnaire at the end of the test. The neutral scale in the original feedback questionnaire has been mapped to a weighted scale between 6 (best) and 0 (worst). The numbers in bold indicate the (rounded) mean rating.

The original blank questionnaire can be found in Section A.7. Scans of the questionnaires as completed by each test user can be found in Section B.2.

A.1 Checklist

The following checklist was used for the tests: checklist.html.

A.2 Orientation Script

The following orientation script was used for the tests: orient-de.html.

A.3 Background Questionnaire

The following blank background questionnaire was used in the tests: background-de.html.

A.4 Consent Form

The following blank consent form was used in the tests: consent-ta-de.pdf.

A.5 Task Slips

The following task slips were presented to the test users: external-tasks-ta.pdf.

A.6 Data Collection Form

We did not use a specific data collection form.

A.7 Feedback Questionnaire

The following blank feedback questionnaire was used in the tests: feedback-de.html.

A.8 Data Spreadsheet

The data from the questionnaires and the other results were collated in the spreadsheet: data-ta.xlsx.

A.9 Presentation Slides

The slides used for the presentation are: ta-slides.pdf.

B.1 Completed Background Questionnaires

For each test user, the completed background questionnaire was scanned as PDF:

• tp1-bq.pdf
• tp2-bq.pdf
• tp3-bq.pdf
• tp4-bq.pdf
• tp5-bq.pdf

B.2 Completed Feedback Questionnaires

For each test user, the completed feedback questionnaire was scanned as PDF:

• tp1-fq.pdf
• tp2-fq.pdf
• tp3-fq.pdf
• tp4-fq.pdf
• tp5-fq.pdf

C.1 User Aliases

The mapping of test users' aliases to their real names is documented in the file users.html.

C.2 Signed Consent Forms

For each test user, the signed consent form was scanned as PDF:

• tp1-consent.pdf
• tp2-consent.pdf
• tp3-consent.pdf
• tp4-consent.pdf
• tp5-consent.pdf

C.3 External Videos

For each test user, the entire test was captured with a tripod-mounted external video camera in the following files:

• TP1 “Stuart”: tp1-ext.mp4.
• TP2 “Silvia”: tp2-ext.mp4.
• TP3 "Sally": tp3-ext.mp4.
• TP3 "Martin": tp4-ext.mp4.
• TP5: “John”: tp5-ext.mp4.

C.4 Session Capture Videos

For each test user, the entire test session on the computer was captured in the following files:

• TP1 “Stuart”: tp1.mp4.
• TP2 “Silvia”: tp2.mp4.
• TP3 "Sally": tp3.mp4.
• TP3 "Martin": tp4.mp4.
• TP5: “John”: tp5.mp4.

C.5 Session Capture Video Transcripts

The coding scheme for session capture video transcripts is shown in Table C.1. A chronological protocol of events during the test sessions is given in Tables C.2 through C.6. Times are given in hours, minutes, and seconds of elapsed time on the session capture video for that user.