Heuristic Evaluation Plan

1 Introduction

The website to be evaluated is "www.oebb.at", the official website of the austrian railway network. The website is used to gather information about the timetable, ticket costs and services of the oebb and to acquire tickets. We want to evaluate this web site, because it is very important for many people depending on public transportation to have a reliable platform.

2 Evaluation Methodology

Heuristic evaluation is defined by Jakob Nielsen and Rolf Molich as an informal method of analyzing an interface design in 1990. [Nie1990]

Heuristic evaluation is a practical method to help developers evaluate a user interface design with a small set of test users. It is a fast, cheap and easy approach to detect usability problems. This analysis is intended to improve the user-friendliness of interfaces, for example a website. To achieve a significant evaluation result, a group of 5-8 people should examine the design for usability problems. The evaluators have to understand the purpose of the system or product and what the target users goals are. To understand this needs, user personalities have to be defined before the evaluation. It is important that the test users evaluate the user interface individually. This ensures a higher percentage of the heuristic problems found. The individual test users capture the positives and negatives they discovered ranked according to their severity. The notes must be precisely specified and identified. [Phil2017]

For this evaluation, the "Andrews General Usability Heuristics", shown in Appendix A.1, will be used. These are based on and slightly adapted from Nielsen's revised set of ten usability heuristics. This illustration provides guidance for the evaluators, pointing out what they should pay attention to and what the system or product should be able to do. For each heuristic problem a scale from 0-5 is used. [Nie1994].

It is good practice to evaluate the product at least twice. A roughly walk through the interface followed by a more detailed analysis. If the system is used by several kinds of devices, it also makes sense to perform the evaluation on a few different ones. This way it can be ensured that the interface is likely to work for the end user as well. Once the results of the individual evaluators have been fully documented, they can exchange views and determine which heuristic problems are serious and which are minor and list them. This way, the expert knows what to focus on during the revision. [Nie1995]

3 User Profiles

The typical users of the website are people who use the public transport system. The variety of the user is therefore huge. Users that come to your mind are seniors, teenagers, environmentally conscious, tourists, disabled people, students, workers, school classes and families. These users can be grouped in four categories: people with handicaps (seniors, disabled), groups (tourists, school classes, families), people with technical knowhow (student, worker, teenager), people without technical knowhow (seniors, workers).

The general user will use the website to book a ticket and find all available trains on the timetable.

People with handicaps can use the website to gather information about the handicapped accessible of the train station and trains. This group will also often want to make seat reservations.

Groups will use the website to book a large number of tickets and try to find some information about a group discount or the capacity of the train. If you are in a big group it is important to have a reliable information source to plan in advanced.

People, who travel alone and have no experience with online booking, need a pretty simple and manageable website. Additionally the information about changes of arrival and departure time should be very easy to find.

For people who have experience with the website will want a fast way to book a ticket, in the best case, the system remembers the usual journeys and makes a suggestion. Frequent users will properly use the mobile version more than the other groups to book and find the right train.

4 Extent of the Evaluation

We are evaluating the entire english version of the website https://www.oebb.at/, excluding external links (adds, related links, ..).

5 Evaluators and Evaluation Environments

All evaluators (Windows and Linux) are planning to use OBS Studio for screen recording. Planned video settings: Container: MP4 Output Video Resolution: 1920×1080 Frame Rate: 20 Codec: H.264 Rate Control: VBR Bit Rate: 5000 Kbps Planned audio settings: Channels: Stereo Codec: AAC Sample Rate: 44100 Hz Bit Rate: 160 Kbps Bits per Sample: 32

On all operating systems the evaluators will use LosslessCut to extract smaller video clips from their screen recording sessions.

References

[Nie1990]

Jakob Nielsen and Rolf Molich; Heuristic Evaluation of User Interfaces; Proc. Conference on Human Factors in Computing Systems (CHI’90). ACM. Seattle, Washington, USA, Apr 1990, pages 249–256. doi:10.1145/97243.97281

[Nie1994]

Jakob Nielsen; Enhancing the Exploratory Power of Usability Heuristics; Proc. Conference on Human Factors in Computing Systems (CHI’94). ACM. Boston, Massachusetts, USA, Apr 1994, pages 152–158. doi:10.1145/191666.191729

[Nie1995]

Jakob Nielsen; How to Conduct a Heuristic Evaluation; 01 Jan 1995. https://nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

[Phil2017]

Miklos Philips; Heuristic Analysis for UX – How to Run a Usability Evaluation; Visited 2020.09.09. https://www.toptal.com/designers/usability/usability-analysis-how-to-run-a-heuristic-evaluation

A Materials

The following materials will be used by the evaluation team.