Heuristic Evaluation Plan

1 Introduction

The usability of the official tourism portal of the city of Graz will be inspected using an Heuristic Evaluation. The website provides information on various sight-seeing highlights, events and local restaurants. It proposes shopping adventures as well as trips in and around Graz. In order to convince people visiting Graz and to leave a positive impression, the website should appeal to as many user groups as possible across different societies. To achieve the aims of the portal, usability is of the utmost importance. With the Heuristic Evaluation of this website, we want to filter out the positive and negative aspects to ensure a high quality website with a great usability.

2 Evaluation Methodology

The heuristic evaluation is a specific method of analysing usability for various types of user interfaces, first described by Jakob Nielsen and Rolf Molich in 1990. A team of so called 'evaluators' systemically checks and analyses the interface design for potential usability-problems taking a small number of different principles into account. For this evaluation, the "Andrews General Usability Heuristics" (based on Nielsen's revised set of ten usability heuristics [Nie1994]), shown in Appendix A.1, will be used.
According to a study conducted by Nielsen and Molich in 1990, it is important to have a whole team of evaluators in order to be able to find as many flaws in the design as possible and to deliver good results. Individuals, in contrast, only found between 20% and 51% of the known problems according to this study. [Nie1990]

An Heuristic Evaluation step by step [And2021, Won2020]:

1. The design (e.g. the website) is determined and the ten heuristics are presented to the evaluators for analysis. It should be noted that - in order to find as many flaws as possible without a having a huge number of participants - these evaluators are not the end users, already have domain expertise in the field of industry of the product and are experts in usability (so called 'double-experts'). [Nie1992]
2. At first, evaluators work on their own. Each evaluator will search for potential problems and positive findings. All findings are gathered in a table. In addition, they identify specific parts they want to assess. Afterwards, the noted potential problems will be examined in detail.
3. Then, they should record problems as accurately as possible.
4. Finally, the findings of all experts will be aggregated and again individually rated with respect to the problem's severity such that afterwards a full list, order by severity, can be provided. In the end, the result is presented to the customer and possible solutions to these problems are discussed.

In relation to other methods, heuristic evaluation is rather cheap and can already be used at the start of the development process of a product. The application is also quite self-explanatory and usually both major and minor problems are identified and dealt with later. Although it can happen that domain-specific problems could possibly be bypassed, the advantages outweigh the disadvantages. [And2021]

3 User Profiles

The site is trying to acquire as much users as possible who are interested in visiting Graz and / or in getting to know which attractions, sight-seeing spots, restaurants, current events and trip possibilities Graz offers.
Therefore, we can distinguish between the following kinds of users:

• People of other countries who have chosen to visit Graz and want to acquire more information on what to do in Graz.
• People of other countries who want to visit Austria and stumbled across Graz and want to inform themselves about Graz.
• Austrian citizens who want to visit Graz and plan their trip as well as acquire information on what to do in Graz.
• People currently visiting Graz and searching for specific information, e.g. fine dining restaurants.
• People living in Graz who want to make a trip with friends or find a new restaurant they have not yet been to.
• People who are looking for one specific event, which is announced on the website.
• People who are preparing a presentation about Graz (e.g. students of Upper Austria for school purposes)

These kinds of users can, furthermore, be grouped according to distinct shared attributes:

1. People not living in Austria, who are likely not speaking german and who are trying to plan a trip to Graz.
2. Austrian people planning a trip to Graz.
3. People who are just gathering some information about Graz, with no intention of visiting.
4. People searching for a specific information on the website.

The goals and tasks of these groups are respectively:

1. Obtain information on what are the most important sight-seeing spots in Graz, get to know the culture of Graz, probably search for some already planned trip (as detailed knowledge of what can be visited when and how is not present), find an accomodation and organize transportation. Maybe also assess whether a trip to Graz is worth the circumstances.
2. Obtaining information on what sight-seeing attractions the users have not seen yet. Maybe they do not want to take part in an organized trip but instead explore the city on their own. Furthermore, obtaining information on shopping and restaurant possibilities may be more important to this group than to the first. Apart from that, Austrian visitors tend to have a more concrete expectation and reason from and for visiting Graz than the first group.
3. People just gathering information about Graz will likely do not want to get a ton of offers for accommodation and currently existing promos in shopping halls but a discrete and full list of e.g. the sight-seeing attractions.
4. This group of users wants to get the specific information they are looking for as fast as possible without any further distraction and long ways of click through the website.

4 Extent of the Evaluation

We will be evaluating all parts of the website.
In addition to Graz Tourismus, we will also have a look at Graz Tourismus Gruppenangebote, Graz Tourismus Kongress und Veranstaltungsservice and Graz Tourismus Advent.

5 Evaluators and Evaluation Environments

Each evaluator will evaluate the web site with both a mobile web browser and a PC web browser. Mobile devices will be operated in portrait mode and PC web browsers in landscape mode. The evaluators will use the mobile evaluation environments shown in Table 1 and the PC evaluation environments shown in Table 2.

Anna (using Ubunt) and Christof (using macOS) will use OBS Studio to make screen recordings with audio commentary. Emil and Fabian on macOS will use the built-in macOS screen recorder for creating the recordings.
As far as possible (local restrictions may applay, e.g. Anna's browser resolution does not support Full HD), we will record with the following settings:
Planned video settings: MP4 Output Video; Resolution: 1920×1080; Frame Rate: 20; Codec: H.264; Rate Control: VBR; Bit Rate: 5000 Kbps.
Planned audio settings: Stereo Codec: AAC; Sample Rate: 44 kHz; Bit Rate: 160 Kbps; Bits per Sample: 32.

If possible, all positive and negative findings will be recorded seperately and commented accordingly after the first session. In case some findings cannot be reproduced, we will extract the corresponding video clips out of the fully recorded initial session. For this, Anna will use LosslessCut 3.32.0 on Ubuntu, whereas, Christof, Emil and Fabian on macOS will use QuickTime Player to extract video clips from their screen recordings.

References

[And2021]

Keith Andrews; Human-Computer Interaction Course Notes; http://courses.iicm.tugraz.at/hci/hci.pdf Visited 2021-03-13

[Nie1990]

Jakob Nielsen and Rolf Molich; Heuristic Evaluation of User Interfaces; Proc. Conference on Human Factors in Computing Systems (CHI’90). ACM. Seattle, Washington, USA, Apr 1990, pages 249–256. doi:10.1145/97243.97281

[Nie1992]

Jakob Nielsen; Finding Usability Problems Through Heuristic Evaluation; Proc. Conference on Human Factors in Computing Systems (CHI’92). ACM. Monterey, California, USA, May 1992, pages 373–380. doi:10.1145/142750.142834

[Nie1994]

Jakob Nielsen; Enhancing the Exploratory Power of Usability Heuristics; Proc. Conference on Human Factors in Computing Systems (CHI’94). ACM. Boston, Massachusetts, USA, Apr 1994, pages 152–158. doi:10.1145/191666.191729

[Won2020]

Euphemia Wong; Heuristic Evaluation: How to Conduct a Heuristic Evaluation; July 2020. https://www.interaction-design.org/literature/article/heuristic-evaluation-how-to-conduct-a-heuristic-evaluation

A Materials

The following materials will be used by the evaluation team.