Heuristic Evaluation Report

706.021 Human-Computer Interaction 3VU SS 2006

Group 5-06

Joachim Lechner
Marcel Medwed
Markus Tatzgern
Michael Hofmann

Heuristic Evaluation of the Web Site

http://www.one.at/

Report of 1st May 2006

1 Executive Summary

The evaluators had the assignment to investigate the homepage of the mobile phone telecommunications vendor ONE, http://www.one.at/. Therefor user groups of the homepage and their expectations in it had to be identified. Roughly, the following user groups were found: private, business and prepaid card customers. The last ones are a special case, because they do not want to be associated to the company by an agreement. The goal of the user groups can differ. Some may be new customers looking for a cheap plan, others are perhaps already customers and want to take a look into their bill (cost control), others probably just want to download the latest games or background images for their mobile phone. Because the extended services (cost control, downloads, ...) of the homepage were limited to registered customers and therefore not accessible to the evaluators, the main focus lay on the publicly available parts of the page. Mainly the behaviour of new customers was investigated, but also other parts were looked at.

In the next step each evaluator independently looked at the homepage. Problems and positive findings were categorised by applying so called heuristics. These can be found in the appendices. Afterwards the problems were collected and the evaluators independently assigned a severity rating to the problems. The complete list of problems can be found in section 6 starting with the most severe problems.

The most positive findings were counted at the webshop for new customers. It is very well structured. The customer has the choice to go through the shopping procedure by first selecting the plan and in the second step the mobile phone, or vice versa. Also very good is the option to sort out the mobile phones by narrowing down the features the customer needs. The progress bar always indicates how many steps are left to complete the ordering procedure and helps the customer to keep the overview.

The main menu of the http://www.one.at/ homepage helps the different users to keep track of their steps by coloring the menu according to the type of customer he is (private, business, prepaid card). Other positive things noted by the evaluators are the "Handyberater" and the "Handyvergleich", which can be used to look for the proper mobile phone with the wanted features or to compare phones to each other.

But the evaluators also found some problems. The most severe ones deal with the navigability of the homepage. For example, it is not possible to bookmark certain parts of the homepage, because the link always points to the starting page. Another related problem is, that the reload button does not reload the actual page, but resets the browser to the starting page. This is one of two not so obvious possibilities to return to the home of www.one.at. Another way is to press the ONE logo at the top left of the screen. The evaluators stumbled upon this option merely by coincident, because it is not marked as home button in any way.

Other problems deal with cosmetic and irritating things of the homepage. In one part the text produces a so called underline hover effect. The user may conclude that the text is a clickable link, which it actually is not. One part of the page was found which actually was not consistent with the rest, which can also be frustrating for the unexperienced user.

All in all the page offers many things for users. People can look for plans, mobiles, downloads, games, and so forth. For this amount of information it is not easy to produce a consistent and easy navigable homepage. In many cases the designers succeeded, but in some cases there is room for improvement.

2 Introduction

The Heuristic Evaluation is a technique to analyse the usability of interface designs (software). For this purpose a number of evaluators in the range of unexperienced to experienced investigates the design (typically 3-5 test users per user group result in the best cost-benefit ratio (see How to Conduct a Heuristic Evaluation )). Generally it is the case that some experienced testers will find the same number of issues as a greater amount of unexperienced ones.

For the evaluation of the interfaces so called usability heuristics are used (see Ten Usability Heuristics and this document on heuristics). Each test user tests the web page against these generic recognised usability principals. The main goal is to find as many of the existing usability problems as possible, but also positive aspects should be noted. Additionally screenshots and notes may be used to describe the results. To achieve a high fault detection coverage it is necessary to combine the individual evaluation results. This originates from the fact that the single test results only cover a subset of the total issues. Furthermore it is required that the evaluations are independent from each other. Influences may lead to a lower overall percentage of detected faults, after the results are combined. The individual results are aggregated together by the evaluation manager (remove duplicates, merge same results, categorise, etc.). Afterwards each evaluator independently assigns a severity rating for each problem. Finally the average value of the weights is calculated to find the issues with the highest rating.

This method doesn't provide information on how to correct the found usability issues. It is a very cheap and relatively fast method for finding design problems, which not only finds major problems but also minor ones. Another advantage of the heuristic evaluation is that it can be applied very early in the development process (e.g. pen-and-paper). This results in an early detection of design problems.

2.1 User Profiles

Mainly the side tries to attract the following three types of users. This is already obvious through the main page of the website.

• Private customers
• Business customers (firms)
• Prepaid card customers (TAKE ONE)

Additionally these users can be grouped into the following two categories:

• Persons, who are already ONE customers. They want to use the additional features provided by ONE, which can be accessed via the web page (view invoices, download games/news/pictures/..., online SMS, change settings for the connection (change customer data, cost control, etc.), etc.).
• Persons, who want to become customers of ONE. These persons want to collect information about the contract possibilities. Also roaming-customers are covered by this category.

Furthermore we found the following special user groups, which also get some attention:

• Acoustically impaired people (SMS FOR FREE Gehörlos)
• Disabled people (All in ONE Fair)
• Students (ONE home4students)
• Persons, who want to use the Internet, but don't have a possibility to connect at the current location (I-ONE)
• Press
• Persons, who want to get a job at ONE
• Business competitors

Overall the web page is designed for all ages. But of course a personal computer with an Internet connection and basic web surfing knowledge is required. Furthermore money is required for using the available offerings. Because of this the web page attracts persons older than about 14 years (our esteem).

2.2 Extent of the Evaluation

The evaluators could only impersonate new customers, so they were locked out of some parts of the homepage, which required the registration via a ONE mobile phone number. Nevertheless the most important parts could be investigated. The main focus lay on the behaviour of new customers. They would first of all look for information on the different plans provided and perhaps visit the webshop for new customers. After this stage other parts of the homepage were visited, like the entertainment section where games can be downloaded.

The main goal was to find problems for new customers. A second run dealt with finding cosmetic errors and minor problems of the homepage.

2.3 Structure of the Report

In section 3 the evaluation environment of each evaluator is described. The description includes amongst others the used browser, the operating system and the speed of the user's Internet connection. In section 4 some positive impressions of the homepage are mentioned, while in section 5 the identified major problems are discussed. For the whole list of found problems see section 6.

In the appendices the individual log-files of the evaluators can be found as well as the list of the applied heuristics.

3 Evaluation Environment

A description of the hardware and software used by each evaluator.

4 Positive Impressions

The webshop for new customers of www.one.at is done very well. First of all the user has the choice of selecting a mobile phone or a plan as starting point of his ordering (See figure 1).

Figure 1: Home of the webshop for new customers

In the following steps he always has the progress of the shopping procedure in view by the progress bar to the left of the screen shown in figure 2.

Figure 2: Progress bar to the left of the screen

When choosing a mobile phone the user can narrow down his selection by defining the technical parameters the mobile should fulfil (See figure 3).

Figure 3: Selection of mobile phones

Another positive thing was found at the main page of www.one.at. There the different colors for the customers, i.e. private customers with a yellow color, business customers using a red color and prepaid card customers having an orange color, facilitate the navigation for the user (See figure 4).

Figure 4: Different colors for different customers

At the main page of www.one.at two helpful tools were found, namely the "Handyberater" and the "Handyvergleich". These help the customer when choosing a mobile phone fitting to his needs and comparing similar ones. The "Handyberater" is shown in figure 6, while screenshots of the "Handyvergleich" are found in figure 7a and 7b.

Figure 6: "Handyberater"

Figure 7a: "Handyvergleich" selection

Figure 7b: "Handyvergleich" comparison

5 Analysis of the Main Problems

The most sever problem identified was, that the different parts of the www.one.at homepage could not be bookmarked. This means, if the user wants to store for example the page with the plan relevant for him, he cannot do this. Only the main page can be stored. This clearly violates the heuristic of Flexibility and Efficiency of Use. There exist no screenshot of this problem, because it can be found all over the page.

When pressing the reload button of the browser at any point of the www.one.at homepage the current page is not reloaded, but the user gets back to the starting page (See figure 8a and 8b). The heuristic applied here was the one of Reversible Actions. This behavior was not observed in the webshop for new customers.

Figure 8a: Before browser reload

Figure 8b: After browser reload

In the game download section for private customers text was marked clickable when pointing at it with the mouse. The reason for this is the underline hover effect. Nothing happened when clicking on what seems to be a link for the user. Because this thing was not found at other points of the homepage this is a problem of Consistency. (See figure 9)

Figure 9: Underline hover effect

In the game download zone (Ladezone) of the private customer section there was found a link leading to a completely differently designed part of the www.one.at homepage, i.e. it features another link behaviour. The new page is not loaded beneath the main menu as usual. This clearly violates the heuristic of Consistency and can be seen in figure 10.

Figure 10: Different link behaviour

The button for returning to the starting page of the www.one.at homepage is hidden in the ONE logo at the top left of the screen. There exist no other possibility to return to the homepage. Only in the webshop for new customers an additional link was found at the bottom left of the screen. (See figure 11)

Figure 11: Logo at top left is return

6 List of Problems Found

Aggregated list of all problems found, in descending order of average severity.

Appendix A: Heuristics Used by the Evaluators

The evaluators used the set of heuristics found here heuristics.pdf

In the hardcopy a printout of this list of heuristics is included.

Appendix B: Individual Evaluation Logs

The raw, individual evaluation logs of each evaluator. In the hardcopy include printouts of each of the evaluator logs.

XHTML 1.1   CSS2