Thinking Aloud Test Report

3.1 Test Methodology

The definition of a thinking aloud test is, that you ask several people from different user groups to use a specific system, for example a website on a specific device and while doing this they should speak out their thoughts loud while they are navigating through the interface. It depends on which interface or system you are willing to test but basically this test will be done as follows. First you recruit different users that fit your needs or skill level for the system to be tested, then you hand them out representative tasks to perform. And while they are working with the given tasks you just listen to their thoughts while doing so and write them down, or take a video or audio of the session. So this test doesn’t take a lot of time planning and it is really cost efficient, but you also get a lot of different opinions since you have many test users, maybe from different user groups. [Nie2012]

With the use of a thinking aloud test a users’ behaviour and interaction with a website can be observed and evaluated. In such a way the software programs or websites can be designed that the customers’ expectations are more satisfied. There are several advantages of such tests. One example would be the low cost since no experts and no special equipment is needed. Another one would be that spontaneous reactions are gathered from the test users. Even if the methodology is not followed correctly, valuable results can be gathered still. The maybe biggest advantage is that this method is easy to teach, so no specialists are required. One disadvantage would be that such tests are mostly created under laboratory and not authentic conditions. Since the test user has to think out loud, this can have an influence on the test users’ actions and increase the cognitive load. [Med2022]

3.2 User Profiles

Since kastner-oehler.at is mainly an online store for fashion and clothing, it’s target group are people who are interested in buying new clothes and have a certain degree of fashion understanding. The product catalog of this online store includes clothing for children, men, women and additionally extra sections for certain groups such as the “Infected”-section, which mainly targets teenagers. Furthermore, the online store has sections for other products such as beauty products or things in the field of home and living to offer. Therefore, the site is trying to attract a very broad field of potential customers, which in general are very hard to group. One way of grouping these users, is after their own fashion understanding respectively their fashion sense and their reasoning for the site visit. People which are into fashion and beauty are more interested in finding new trendy clothes and products and are more selective in their choosing and therefore need more options of selection and filtering to get to the things which they have in mind. In contrast users, which are not really into fashion, want to get multiple recommendations of standard things so that they can search and compare which of the recommendations suits them best. But for all of them the main task is the searching and buying of products.

For testing, the main target user group should be a group of young adults who are experienced with online shopping and generally are comfortable navigating though websites on various platforms. Because of the past experience with different online shopping platforms they may experience things while performing tasks which is, in their eyes, better or worse than at other comparable websites. But since the website targets a wide range of user groups, also less experienced users should be considered to ensure a easy usage of the offered functions and the navigation through the website.

3.3 Test Users

For the selection process of the test users we mainly choose friends in the same range of age as the Falicators which are also currently attending a University. We had an even amount of men and women which all of them, some more, some less, the same experience with various online shops. And they also have a solid knowledge base on using Computers/Tables and Smartphones and the WWW in general.

Table 1 gives an overview of the test users who participated in the study. The mapping of real names to aliases is included in Appendix C.1. Scans of the original completed questionnaires are included in Appendix B.1.

3.4 Test Environment

The test room is shown in Figures 1a and 1b .

The tests were held on the 24th of May between 14:00 and 17:30 and on the 26th of May between 17:30 and 19:00.

The environment used for the thinking aloud tests is shown in Table 2.

The exact web browser version used for the tests is shown in Figure 2.

For this test, all adblockers were disabled. The acceptance of cookies were left up to each test user, because users should act as they would when browsing this website privately, but it can also be a reason of the design if users will just quickly click through the prompt for the cookies or if they deactive some of them.

3.5 Recording

For the external video capture we chose a Nikon D5300 with an 200mm lens on a tripod. As a microphone we used a HyperX Cloud 2 Headset.

The screen, during the testing process, was captured by the software "OBS Studio" which is an free and open source software that can be used, among other things, for screen, webcam, a integraded camera from realtek and microphone recordings.

The recorded clips were further processed with the program Sony Vegas Pro 15, which is a video editing software. For the blurring of the faces of the test users, we used the boxblur function of ffmpeg. With this function a box can be created in the video which will be blurred.

Due to technical difficulties with the camera we had to cut the session capture (timestamp 00:12:04) and external session video (timestamp: 00:19:21) of TP2. Because of issues with the recording software, not the complete interview of TP1 is on the session capture recording, but the external session video contains the information. Additionally, we had to cut TP4's external recording at timestamp 00:02:36, 00:02:40, 00:02:44 and 00:02:48 due to technical errors with the camera. In the screen capture video of TP5 we had to cut at timestamp 00:10:25 and 00:11:10, because of technical errors of the laptop.

3.6 Training

For every test user the same notebook with the same mouse and keyboard was provided by the test team. Since the test users are familiar with the use of computers and interacting with the internet/website, there is no special training needed.

In the beginning the facilitator explain to the test user how the thinking aloud test will be done. Also, a simple task was performed by the facilitator for better understanding. Furthermore, a video of a previous thinking aloud test was shown to the test users.

The test users did not receive any further training or practicing for the TA test, but the facilitator demonstrated a small example, so each user knew how they should do the test. The website used was "uncut.at" where the facilitator had to find the cinema program for a specific date, on a specific cinema.

3.7 Tasks

The internal task list used by the test team is shown in Table 3.

3.8 Interview Questions

The following questions were asked of each user immediately after the final task. Transcripts of the interviews can be found in Section 4.8.

The following questions will be asked of each user immediately after the final task.

• Opening Question
  • "How was it?" ["Wie war's?"]
• Standard Questions
  1. "Did anything strike you as particularly good?"
  2. "Did anything strike you as particularly bad?"
  3. "Which task was the hardest and why?"
  4. "Do you think the website is easy to use? With reasoning please.”
• Individual Questions

  After the standard questions above, any individual questions which arose during the test with each particular test user were asked. These are detailed in the transcripts in Section 4.8.

3.9 Feedback Questionnaire

After the interview, the user was asked to fill out the feedback questionnaire given in Appendix A.7. The feedback questionnaire was given on paper. An overview of the results can be seen in Section 4.9.

3.10 Data Collection and Collation

Before the task started the user got a simple introduction and had to sign a declaration of consent. Then the facilitator asked the test user the questions from the background questionnaire and filled in the given answers. All collected data will be assigned to a test user id and first name aliases.

The test user got a sheet of paper with all the tasks on it. The user had to read the tasks and if everything is clear the test would get started. A task was either over if the test user reached the goal or the facilitator decided that the task takes too long. After all tasks were done the test user got a feedback form, which he had to fill out.

Immediately after the thinking aloud test is done the handwritten questionnaires (background, feedback) as well as success measures and observed problems are transferred to a spreadsheet shown in Appendix A.7.

Any faces occurring in the recordings were blurred or pixelated. All collected personal data, like name, signature and the full video recordings will be deleted after one year.

4.1 Discussion and Analysis

After all test participants were done with their Thinking Aloud Test, we noticed that the users found far more negative things then positive ones. Things like pop ups at the checkout, non-fitting suggestions or the overall bad response time from the website were some of the negative findings. While trying to order multiple items the users noticed the most negative aspects. It is not possible to put more items than one into the shopping cart, instead the number has to be adjusted in the shopping cart. Furthermore, it is not possible to order more then 9 pieces of the same item. The third problem that arose was the bad feedback, when there are not enough pieces of one item anymore. One of the participants also pointed out, that the design/layout of the website could be improved, by minimizing the white spaces on the left and right side and also increasing the items shown per row to make it more comfortable when looking around the website. The most positive about the website, which was noticed by the users, was the back to the top button. Things like the offering of multiple pictures for one item, or the already prepared outfits were also mentioned. The smaller positive things found were the suggestions of certain items as well as the option to subscribe to a newsletter. We noticed that no one of the test participants used the search bar function, instead they all chose to use the filter options to look for the items. Since the tasks provided were solved by everyone and some of them had some troubles with task 5, we think that the chosen tasks were pretty fitting.

4.2 Task Completion

A summary of how many users completed each task and whether any assistance was given is shown in Table 5.

4.3 Impressions

Tables 6a, 6b, and 6c show a summary of the responses of the test users to the questions asked after the first task. "-" means, that the user was not asked this question in the interview.

4.4 Top Three Positive Findings

The three most positive findings according to their average (mean) positivity ratings are described in more detail below. The positivity rating scheme used to rank positive findings is shown in Table 7.

4.5 List of All Positive Findings

Table 8 shows a list of all the positive findings which emerged from the test, sorted in descreasing order of average (mean) positivity, i.e. the most positive are at the top of the table. Positivity ratings were assigned by members of the test team, their name codes are shown in Table 9.

4.6 Top Five Problems

The five most serious problems according to their average (mean) severity ratings are described in more detail below. Problem number 1 is the problem (negative finding) with the highest mean severity. The severity rating scheme used to rank problems is shown in Table 10.

4.7 List of All Problems Found

Table 11 shows a list of all the problems observed in the test, sorted in descreasing order of average (mean) severity, i.e. the most severe are at the top of the table. Severity ratings were assigned by members of the test team, their name codes are shown in Table 12.

4.8 Interviews

TP1 "Sarah" overall liked the site despite some small inconvienences because she could find a lot of clothing and options there.

TP2 "Josh" stated that he didnt like the pop up in the shopping card which seemed to appear every few seconds and that the size of the banner with the countdown changed with the width of the characters, also that the pop up when adding clothes to the shopping cart didnt close by itself. He couldnt find anything where he would say thats "super nice" but overall still good.

TP3 "Bruce" said that the website was very respoonsive and loading fast, but he didnt like the unexpected pop Ups and also that he cant put more than 9 of the same shirt at once in the shopping cart.

TP4 "Sylvia" stated that the website looks like any other website, so pretty basic. TP4 didn't like the pop ups and the bugs with the shopping card, for example only adding 9 pieces of the same items.

TP5 "Phil" said, that it is a good normal website and if you already used some it won't be hard to use this one. TP5 didn't like the thick white spaces on the left and right of the items, as well as the much scrolling which was involved. Also the problem with the maximum of 9 pieces per item were annoying for TP5.

4.9 Feedback Questionnaires

Table 13 shows a summary of the ratings given by users in the feedback questionnaire at the end of the test. The neutral scale in the original feedback questionnaire has been mapped to a weighted scale between 6 (best) and 0 (worst). The numbers in bold indicate the (rounded) mean rating.

The original blank questionnaire can be found in Section A.7. Scans of the questionnaires as completed by each test user can be found in Section B.2.

A.1 Checklist

The following checklist was used for the tests: checklist.html.

A.2 Orientation Script

The following orientation script was used for the tests: orient.html.

A.3 Background Questionnaire

The following blank background questionnaire was used in the tests: background.html.

A.4 Consent Form

The following blank consent form was used in the tests: consent-ta.pdf.

A.5 Task Slips

The following task slips were presented to the test users: external-tasks-ta.pdf.

A.6 Data Collection Form

We did not use a data collection form.

A.7 Feedback Questionnaire

The following blank feedback questionnaire was used in the tests: feedback.html.

A.8 Data Spreadsheet

The data from the questionnaires and the other results were collated in the spreadsheet: data-ta.xlsx.

A.9 Presentation Slides

The slides used for the presentation are: ta-slides.pdf.

B.1 Completed Background Questionnaires

For each test user, the completed background questionnaire was scanned as PDF:

• tp1-bq.pdf
• tp2-bq.pdf
• tp3-bq.pdf
• tp4-bq.pdf
• tp5-bq.pdf

B.2 Completed Feedback Questionnaires

For each test user, the completed feedback questionnaire was scanned as PDF:

• tp1-fq.pdf
• tp2-fq.pdf
• tp3-fq.pdf
• tp4-fq.pdf
• tp5-fq.pdf

C.1 User Aliases

The mapping of test users' aliases to their real names is documented in the file users.html.

C.2 Signed Consent Forms

For each test user, the signed consent form was scanned as PDF:

• tp1-consent.pdf
• tp2-consent.pdf
• tp3-consent.pdf
• tp4-consent.pdf
• tp5-consent.pdf

C.3 External Videos

For each test user, the entire test was captured with a tripod-mounted external video camera in the following files:

• TP1 "Sarah": tp1-ext.mp4.
• TP2 "Josh": tp2-ext.mp4.
• TP3 "Bruce": tp3-ext.mp4.
• TP4 "Sylvia": tp4-ext.mp4.
• TP5: "Phil": tp5-ext.mp4.

C.4 Session Capture Videos

For each test user, the entire test session on the computer was captured in the following files:

• TP1 "Sarah": tp1.mp4.
• TP2 "Josh": tp2.mp4.
• TP3 "Bruce": tp3.mp4.
• TP4 "Sylvia": tp4.mp4.
• TP5: "Phil": tp5.mp4.

C.5 Session Capture Video Transcripts

The coding scheme for session capture video transcripts is shown in Table C.1. A chronological protocol of events during the test sessions is given in Tables C.2 through C.6. Times are given in hours, minutes, and seconds of elapsed time on the session capture video for that user.