Thinking Aloud Test Plan

1 Introduction

The web site we are going to evaluate provides information about the Technical University Graz. The visitor gets informed about the range of studies, facilities and institutes.

We are going to evaluate this web site, because we participate in the course Human Computer Interaction. We also want to get a good grade.

2 Test Methodology

The purpose of a thinking aloud test is to be able to read a users thought and thus indicating why exactly he is doing things. If you ask a user after he made a mistake why exactly he did something, it could be possible that he simply gives a wrong but still plausible answer. To give his actions correct reasonings he has to speak aloud (thus the name thinking aloud test) what he is doing and what he is planning to do. Now, if he makes a mistake, you can identify the source and cause of his mistake and furthermore improve the design to prevent such issues. "The fact that the same method has remained #1 for 19 years is a good indication of the longevity of usability methods." [Nie2012]

This method has several advantages, for example it can be started very early during development and the test users don’t have to be very sophisticated as everyone who is capable of using a computer can try using the software product. Also by making developers watch the testing process where a normal user keeps making mistakes with their products may soften up their stubbornness and thus the developers increase their focus on usability. One big disadvantage of TA tests is that not only does it take more time during testing, also could the thinking aloud process alter the test users behaviour.

It is very important that you create a comfortable environment for the test user and make it clear that he is not the one being tested. His performance during the test process is not of importance, only the system should be eyed critically. During the testing process the test user has to know this fact. Otherwise he could fear that he may not sound smart enough and think through his actions before doing anything which defeats the purpose of usability testing. Additionally it also matters that the user has a constant stream of thoughts and never stops talking. The facilitators can encourage this behaviour by asking neutral questions, for example “What’s on your mind currently?”. But the facilitators have to refrain from using direct questions as it would again influence his testing activity. "It is interesting to note that Ericsson and Simon actually suggest that instead of saying Keep talking after a subject falls silent, a researcher might choose to play a “reminder tone”". [Bor&Ram2000]

3 User Profiles

The typical user that generates the most traffic would be a prospective student. Since the lifeblood of a university are the people studying there, the university also wants to attract as many of them as possible. In addition to that, the site also wants to provide information about the quality of the university to employers of graduates. Another possible user group would be Jobseekers, who want to look up available job offers and most likely the pay. Students and graduates could also visit the site from time to time.

The prospective student is used to computers, knows his way around but also wants to see modern design. Since the user is used to responsive web and doesn't know the internet before it was as beautiful as it is now. The employer is most likely a business man using computers constantly since he looked up the university of his prospective employee. He doesn't want to spend a lot of time on the site and wants information as fast as possible. Jobseekers probably want to invest a lot of time in the site to gather as much information as possible about available job offers and the required qualifications for these. A student is already familiar with the site and uses it from time to time. He is very familiar with computers and uses them regularly. The graduate wants to connect with his fellow graduates. He doesn't use computers too much. Unless the interface is complex to handle, he/she is able to use it.

Our typical user, the prospective student, wants to get information about available studies and how to enroll into courses. The Curriculum of the study he is interested in is also very important for him. The employer wants to see the quality of the university in comparison to other national or international universities. The jobseekers want to look up available job offers and the amount of salary. Students want information about deadlines and the Curriculum. Graduates want to use the Alumni network to connect with fellow graduates.

The user group we intend to test will be the prospective students. Not only because they are the typical users of the website, also because they are most easily to find as test users.

4 Test Users

4 Test Users

Out test users with their assigned aliases and additional relevant information about them is shown in Table 1.

5 Test Environment

The test environment (room, computer, browser, internet connection) which will be used for the test is shown in Table 2. The ad blocker that is installed on the test machine will be disabled during the test.

6 Session Capture

For the session capturing we are going to use the software Monrae as in the lecture suggested. The webcam we use will be the 'Creative Live! Cam Chat HD'. It is a USB webcam and captures in 720p HD. The webcam also has a good microphone build in, we will not use an external one.

Morae provides a function to make high resolution screenshots from the captured sessions. We are going to use this function for our screenshots in our testings.

The tripod, we are going to use, will be the 'Rollei C4i Titan'. The camera we use is the 'Canon EOS 100D'. It can caputre videos up to 1080p full HD. For a better sound recording we use an external microphone, the 'TAKSTAR SGC-598'.

7 Training

8 Test Tasks

Table 3 shows the tasks which we suggest might be used.

9 Interview Questions

The following questions will be asked of each user immediately after the final task.

• Opening Question
  • "How was it?"
• Standard Questions
  1. "Did anything strike you as particularly good?"
  2. "Did anything strike you as particularly bad?"
  3. "What was the first thing you recognized?"
  4. "Which task was hard to perform and explain what was difficult?"
  5. "How difficult is it to find specific information on the website?"

10 Data Preparation

At first we set the browser website to 'www.google.com'. Then we give the test user the task printed out on paper. We tell the test user to carefully read the task and we ask him, if there are any questions to the task. After 1 minute we ask if everything is clear and if the test user has no back questions we tell him to start the task. At the same moment we start to measure the time. Every task is over when the test user reaches the goal, or the time is over. We tell the test user to stop and we also stop the time measurement.

The collected data from the hand written questionnaires will be typed into the PC right after the test user is done. The test user should be available until the questionnaire is typed in. If there are back questions from the person who types the questions into the PC, or any answer is unclear, the test user can be asked immediately. Also task success measures and observed problems are going to be evaluated right after every individual test user is done.

The highlight videos will be edited with Adobe Premiere Pro CS6. We use Premiere because we already have a licence and a little bit experience in video editing with this software.

References

[Nie2012]

Jakob Nielsen; Thinking Aloud: The #1 Usability Tool; Visited 2017-04-22. https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool

[Bor2000]

Ted Boren & Judith Ramey; Thinking aloud: Reconciling theory and practice; Visited 2017-04-22. https://www.researchgate.net/publication/3230127_Thinking_aloud_Reconciling_theory_and_practice

A Test Team Materials