Heuristic Evaluation Report
INB.06021UF Human-Computer Interaction 3VU SS 2021
Group G6-01
Anna Glaser
Christof Feischl
Emil Winterleitner
Fabian Burgmann
Evaluation of the Web Site
Graz Tourismus
HE Report of 05th May 2021
1 Executive Summary
The report contains the heuristic evaluation of www.graztourismus.at.
First, every evaluator has executed the heuristic evaluation (a specific usability analyzation method, see chapter 3.1) individually using different browsers and on distinct devices and setups.
The findings have been documented in log files and supported with short descriptive videos. Finally, the results have been aggregated and ranked by severity.
One crucial negative finding was the misleading and inconsistent design which often lead to confusion. For example: some icons do not flip on sections where it should actually be the case, hyperlinks are not highlighted, some images/text paragraphs are missing, some pictures/elements are cut off, errors are not immediately recognizable, the documentation provided is not applicable and insufficient clarity on how to open or close something (navigation elements, menu options, ...).
Another finding to be improved is the long loading time. Especially with the video on the start page or when downloading brochures, you have to wait a while for the actual task to begin. That is not justifiable with todays' technological possibilities. Regarding the brochures, furthermore, the download size should be stated to avoid unwanted mobile data usage.
There is a lack of orientation, too. A guide to make the first use easier would be nice. For example, a misplaced click takes one to another page. But if one wants to go back, the site is newly loaded and one has to search from the top again. Furthermore, descriptions for abbreviations that a user might not be familiar with should be provided.
Another aspect to be improved was that a number of elements were not displayed completely legibly. The background images were missing behind some of the subtitles and a map was cut off once. Some filter and search options did not perform the expected, intuitive behavior.
Various forms throughout the website lacked correct input validation and it was possible to send the forms with syntactically invalid input.
Last but not least, we listed the missing translation of some parts of the website. If one had set the website's language to English, one would not expect to encounter any german paragraph.
However, the website also offers many outstanding positive features.
As the best feature of the website the section "Our tips" had been selected by the evaluators. This was very helpful for the user and gives a good overview about the most important facts of a selected destination / chosen element.
The second best aspect was the searching function in general. We could go back to the history of our queries and then edit them.
The design of the map and the options (zoom, destinations displayed with background information) were very informative and appealing.
Finally, we selected the watchlist. It can be very helpful if you are planning a trip and want to save the destinations as a PDF and send them to friends by email, for example.
2 Introduction
The usability of the official tourism portal of the city of Graz was inspected using an Heuristic Evaluation. The website provides information on various sight-seeing highlights, events and local restaurants. It proposes shopping adventures as well as trips in and around Graz. In order to convince people visiting Graz and to leave a positive impression, the website should appeal to as many user groups as possible across different societies. To achieve the aims of the portal, usability is of the utmost importance. With the Heuristic Evaluation of this website, we wanted to filter out the positive and negative aspects to ensure a high quality website with a great usability.
3 Evaluation Procedure
3.1 Evaluation Methodology
The heuristic evaluation is a specific method of analysing usability for various types of user
interfaces,
first described by Jakob Nielsen and Rolf Molich in 1990.
A team of so called 'evaluators' systemically checks and analyses the interface design for potential
usability-problems
taking a small number of different principles into account. For this evaluation, the "Andrews
General Usability Heuristics" (based on Nielsen's revised set of ten
usability heuristics [Nie1994]), shown in Appendix A.1, will be used.
According to a study conducted by Nielsen and Molich in 1990, it is important to have a whole team of
evaluators in order to be able to find as many flaws in the design as possible and to deliver good results.
Individuals, in contrast, only found between 20% and 51% of the known problems according to this
study.
[Nie1990]
An Heuristic Evaluation step by step [And2021, Won2020]:
- The design (e.g. the website) is determined and the ten heuristics are presented to the evaluators for analysis. It should be noted that - in order to find as many flaws as possible without a having a huge number of participants - these evaluators are not the end users, already have domain expertise in the field of industry of the product and are experts in usability (so called 'double-experts'). [Nie1992]
- At first, evaluators work on their own. Each evaluator searched for potential problems and positive findings. All findings were gathered in a table. In addition, they identify specific parts they want to assess. Afterwards, the noted potential problems were examined in detail.
- Then, they should record problems as accurately as possible.
- Finally, the findings of all experts were aggregated and again individually rated with respect to the problem's severity such that afterwards a full list, order by severity, can be provided. In the end, the result should be presented to the customer and possible solutions to these problems are discussed.
In relation to other methods, heuristic evaluation is rather cheap and can already be used at the start of the development process of a product. The application is also quite self-explanatory and usually both major and minor problems are identified and dealt with later. Although it can happen that domain-specific problems could possibly be bypassed, the advantages outweigh the disadvantages. [And2021]
3.2 User Profiles
The site is trying to acquire as much users as possible who are interested in visiting Graz and / or
in getting to know
which attractions, sight-seeing spots, restaurants, current events and trip possibilities Graz
offers.
Therefore, we can distinguish between the following groups of users:
- People not living in Austria, who are likely not speaking german and who are trying to plan a trip to Graz.
- Austrian people planning a trip to Graz.
- People who are just gathering some information about Graz, with no intention of visiting.
- People searching for a specific information on the website.
The goals and tasks of these groups are respectively:
- Obtain information on what are the most important sight-seeing spots in Graz, get to know the culture of Graz, probably search for some already planned trip (as detailed knowledge of what can be visited when and how is not present), find an accomodation and organize transportation. Maybe also assess whether a trip to Graz is worth the circumstances.
- Obtaining information on what sight-seeing attractions the users have not seen yet. Maybe they do not want to take part in an organized trip but instead explore the city on their own. Furthermore, obtaining information on shopping and restaurant possibilities may be more important to this group than to the first. Apart from that, Austrian visitors tend to have a more concrete expectation and reason from and for visiting Graz than the first group.
- People just gathering information about Graz will likely do not want to get a ton of offers for accommodation and currently existing promos in shopping halls but a discrete and full list of e.g. the sight-seeing attractions.
- This group of users wants to get the specific information they are looking for as fast as possible without any further distraction and long ways of click through the website.
3.3 Extent of the Evaluation
We evaluated all parts of the website.
In addition to Graz Tourismus, we also had a look
at
Graz Tourismus Gruppenangebote,
Graz Tourismus Kongress und Veranstaltungsservice and
Graz Tourismus Advent.
3.4 Evaluators and Evaluation Environments
Each evaluator evaluated the web site with both a mobile web browser and a PC web browser. Mobile devices were operated in portrait mode and PC web browsers in landscape mode. The evaluators used the mobile evaluation environments shown in Table 1 and the PC evaluation environments shown in Table 2.
| Evaluator | Anna Glaser (AG) | Christof Feischl (CF) | Emil Winterleitner (EW) | Fabian Burgmann (FB) |
|---|---|---|---|---|
| Age | 21 | 21 | 21 | 24 |
| Gender | female | male | male | male |
| Device | Sony Xperia XZ3 | iPhone Xs | iPhone 7 | iPhone Xs |
| OS and Version | Android-Version 10 | iOS 14.4 | iOS 14.4 | iOS 14.4.1 |
| Screen Size | 6" | 5.8" | 4.7" | 5.85" |
| Screen Resolution | 1440×2880 | 2436×1125 | 750×1334 | 2436×1125 |
| Web Browser | Firefox Browser 86.1.1 | Safari 14 | Google Chrome 87.0.4280.77 | Brave 1.23.1 |
| Ad Blocker | none | none | none | Brave built-in Ad Blocker |
| Internet Connection | WiFi | WiFi | WiFi | WiFi |
| Download Speed | 38.2 Mbps | 72.9 Mbps | 106.58 Mbps | 26.34 Mbps |
| Browser Resolution | 1440×2880 | 886x1920 | 750×1334 | 886x1920 |
| Screen Recording Software | AZ Screen Recorder | iOS Screen Recording | iOS Screen Recording | iOS Screen Recording |
| Planned Date of Evaluation | 2021-03-30 | 2021-03-30 | 2021-03-30 | 2021-03-30 |
| Evaluator | Anna Glaser (AG) | Christof Feischl (CF) | Emil Winterleitner (EW) | Fabian Burgmann (FB) |
|---|---|---|---|---|
| Age | 21 | 21 | 21 | 24 |
| Gender | female | male | male | male |
| Device | Medion Erazer X7819 | MacBook Pro 2019 | MacBook Pro 2019 | MacBook Pro 2018 |
| OS and Version | Ubuntu 20.04.2 LTS | macOS BigSur 11.2.2 | macOS Catalina 10.15.7 | macOS BigSur 11.2.3 |
| Screen Size | 17.3" | 16" | 16" | 15.4" |
| Screen Resolution | 1920×1080 | 3072×1920 | 3072×1920 | 2880×1800 |
| Web Browser | Mozilla Firefox for Ubuntu canonical - 1.0 | Safari 14.0.3 | Google Chrome 89.0.4389.90 | Microsoft Edge 89.0.774.57 |
| Ad Blocker | uBlock Origin | AdGuard für Safari | Adblock Plus 3.10.2 | none |
| Internet Connection | WiFi | WiFi | WiFi | WiFi |
| Download Speed | 38.2 Mbps | 72.9 Mbps | 106.04 Mbps | 29.59 Mbps |
| Browser Resolution | 1834×882 | 1920×1080 | 1920×1080 | 1920×1080 |
| Screen Recording Software | OBS Studio 26.1.2 | OBS Studio 26.1.2 | macOS Screen Recorder | macOS Screen Recorder |
| Planned Date of Evaluation | 2021-03-30 | 2021-03-30 | 2021-03-30 | 2021-03-30 |
Anna (using Ubuntu) and Christof (using macOS) used OBS Studio to make screen recordings with audio commentary.
Emil and Fabian on macOS used the built-in macOS screen recorder for creating the recordings.
As far as possible (local restrictions may apply, e.g. Anna's browser resolution does not support Full HD), we recorded with the following settings:
Planned video settings: MP4 Output Video; Resolution: 1920×1080; Frame Rate: 20; Codec: H.264; Rate Control: VBR; Bit Rate: 5000 Kbps.
Planned audio settings: Stereo Codec: AAC; Sample Rate: 44 kHz; Bit Rate: 160 Kbps; Bits per Sample: 32.
All positive and negative findings were recorded seperately and commented accordingly after the first session.
In case some findings cannot be reproduced, we extracted the corresponding video clips out of the fully recorded initial session.
For this, Anna used LosslessCut 3.32.0 on Ubuntu, whereas, Christof, Emil and Fabian on macOS used QuickTime Player to extract video clips from their screen
recordings.
4 Results of the Evaluation
This section describes the results of the heuristic evaluation.
4.1 Top Three Positive Findings
The top three positive findings according to their average (mean) positivity ratings are described in more detail below. The positivity rating scheme used to rank the positive findings is shown in Table 3.
| Positivity | Meaning |
|---|---|
| 4 | Extremely Positive |
| 3 | Major Positive |
| 2 | Minor Positive |
| 1 | Cosmetic Positive |
| 0 | Not a Positive |
P01. Our Tipps
The finding is summarized in table 4.
| Description | Our Tipps is really helpful and easily reachable also it is shown in the right context. |
|---|---|
| Video Clip(s) | p01-fb-pc-pos02-our-tipps.mp4 |
| Location (How Reproducible?) | Go to menu - all submenues have different 'Our Tipps' |
| Mean | 3.50 |
The placement and content of 'Our Tipps' was received really well by all evaluators. Each Menu consistently has a different 'Our Tipps' content and is placed at the same position. As can be seen in Figure 1.
P02. Search History Feature
The finding is summarized in table 5.| Description | The search history feature is really nice and old searches can even be deleted. |
|---|---|
| Video Clip(s) | p02-fb-pc-pos04-search-history-feature.mp4 |
| Location (How Reproducible?) | Go to start page - search something - search again - a popup with old searches should appear - click on 'X' to remove search entry |
| Mean | 3.50 |
The search history works really well and it's a nice bonus that old searches can be deleted, Figure 2 shows it in Action.
P03. Map functionality
The finding is summarized in table 6.
| Description | All elements in Graz Tourism Webportal which are associated with a specific location can be viewed in a map directly on the according subpage. |
|---|---|
| Video Clip(s) | p03-cf-pc-pos02-map.mp4 |
| Location (How Reproducible?) | Go to a restaurant - Open map |
| Mean | 3.50 |
The map that can be found on all of the subpages of the different places is a really nice way to visualize the location in a comfortable way as can be seen in Figure 3.
4.2 List Of All Positive Findings
Table 7 lists all positive findings that were found by our evaluators in descending order.
| No. | Title | Description | Video Clip(s) | Location (How Reproducible?) | Found by | Severity rating | Mean | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CF | AG | FB | EW | CF | AG | FB | EW | ||||||
| 1 | Our Tipps | Our Tipps is really helpful and easily reachable also it is shown in the right context |
p01-fb-pc-pos02-our-tipps.mp4 | Go to menu - all submenues have different 'Our Tipps' | n | n | y | n | 3 | 4 | 4 | 3 | 3.50 |
| 2 | Search History Feature | The search history feature is really nice and old searches can even be deleted |
p02-fb-pc-pos04-search-history-feature.mp4 | Go to start page - search something - search again - a popup with old searches should appear - click on 'X' to remove search entry | n | n | y | n | 3 | 3 | 4 | 4 | 3.50 |
| 3 | Map functionality | All elements in Graz Tourism Webportal which are associated with a specific location can be viewed in a map directly on the according subpage. | p03-cf-pc-pos02-map.mp4 | Go to a restaurant - Open map | y | n | n | n | 4 | 3 | 4 | 3 | 3.50 |
| 4 | Watchlist with save as PDF and send via Mail functionality | Throughout the Graz Tourism web portal you can add elements to the watchlist. Afterwards, you can print the remembered elements or send it to your mail for late use and easily finding the elements again. | p04-cf-pc-pos01-watchlist.mp4 p04-ew-pc-pos03-watchlist-valid.mp4 |
Go to start page - somewhere and add various elements (like restaurants) to the watchlist - watchlist - save as PDF | y | n | n | y | 3 | 4 | 3 | 3 | 3.25 |
| 5 | Good error handling in form | When you try to order a brochure and you have forgotten to fill in e.g. your first name, the site jumps up to the field, highlights it and remarks that you have to fill in the first name. | p05-cf-mob-pos02-form-error-handling.mp4 | Go to Service - Good to know - Order a brochure | y | n | n | n | 3 | 3 | 4 | 3 | 3.25 |
| 6 | Searching for venues | When searching for a bar or other location, one can easily filter out unwanted results. Searching for specific places is made easy that way. |
p06-ew-pc-pos01-search.mp4 | Go to Eat & Drink - Restaurant Guide Search | n | n | n | y | 3 | 3 | 3 | 3 | 3.00 |
| 7 | Hotels and Apartments - Easy going back | When choosing a hotel and clicking on it in order to have a closer look, you can then just press the back button of your browser or click "Back to search results" and you get back to exactly the location where you have been at before. | p07-cf-mob-pos01-easy-going-back.mp4 | Go to Hotels and Apartments - Hotels - Choose a page - Click on a hotel - Go back | y | n | n | n | 3 | 2 | 3 | 3 | 2.75 |
| 8 | Same design helps the user | The design is pretty much the same on all subpages. Everywhere there are subsections that you can open and close immediately or that you can click directly on via the category. So everything is relatively clear for the user. |
p08-ag-pc-pos03-samedesignsubpages.mp4 | Go to start page | n | y | n | n | 2 | 4 | 3 | 1 | 2.50 |
| 9 | Navigation across subpages | The directory like structure allows to navigate the website in a pleasant way |
p09-ag-mob-pos02-goodbackoption.mp4 | Go to a subpage - see navigation line in the upper part of the website (above subsection) | n | y | y | n | 3 | 3 | 2 | 2 | 2.50 |
| 10 | Messages - Easy go back | The Messages box opens and closes without changing the websites current state (reloading/changing website) |
p10-fb-mob-pos01-messageseasy-go-back.mp4 | Go to start page - messages (left-bottom corner) - use it - close | n | n | y | y | 2 | 2 | 3 | 3 | 2.50 |
| 11 | Connection to map | If you click on the name of the destination, you can see the location on the map to the right. You not only have the description of the destination, but also the location. |
p11-ag-pc-pos01-connectionmapanddestination.mp4 | Go to start page | n | y | n | y | 2 | 2 | 1 | 2 | 1.75 |
| 12 | Adjusting the width | When you rotate your mobile phone no space gets wasted by the website. Then, the searching bar gets wider and some sections adapt to the new width. |
p12-ag-mob-pos03-adjustingwidth.mp4 | Go to start page | n | y | n | n | 1 | 2 | 2 | 1 | 1.50 |
| 13 | Getting feedback when loading the video | If you want to watch the video and click on it, you can watch the progress of the loading of the video via busy cursor. This makes you feel more informed because you get feedback from the website. |
p13-ag-pc-pos02-watchprogress.mp4 | Go to start page | n | y | n | n | 2 | 2 | 1 | 1 | 1.50 |
| 14 | Home button redirect | Home button redirects to same website | p14-ew-mob-pos01-home-button.mp4 | Go to start page - somewhere - click on the 'home'-button | n | n | n | y | 1 | 1 | 0 | 1 | 0.75 |
| 15 | Icon size | The size for the + and - (expand / collapse) icons is the same size. | p15-ew-mob-pos02-icons.mp4 | Go to start page - menu entry to expand / collapse it | n | n | n | y | 0 | 1 | 0 | 1 | 0.50 |
| 16 | Quick navigation at destinations | If you want to get more information about the destination at "Sightseeing - Top 10", you will be linked to the corresponding See & Do page. You can watch a video there. A route planner with google.maps is also linked and offers fast navigation. |
p16-ag-mob-pos01-quicknavigation.mp4 | Go to start page - see and do - sightseeing - sights - Landhaus Courtyard | n | y | n | n | 0 | 1 | 0 | 0 | 0.25 |
4.3 Top Five Negative Findings
The top five problems according to their average (mean) severity ratings are described in more detail below. Problem number 1 is the problem (negative finding) with the highest mean severity. The severity rating scheme used to rank the problems is shown in Table 8.
| Severity | Meaning |
|---|---|
| 4 | Catastophic problem |
| 3 | Serious problem |
| 2 | Minor problem |
| 1 | Cosmetic problem |
| 0 | Not a problem |
N01. Watchlist deleted after sending
The finding is summarized in table 9.
| Description | When sending the watchlist to an email address, the watchlist is then deleted automatically (reset). If you misspelled the email address or the email address does not reach out to you, then the watchlist has been lost completely and can not be easily resent. |
|---|---|
| Video Clip(s) | n01-cf-mob-neg12-watchlist-deleted-after-sending.mp4 |
| Heuristic | A05 Error Prevention |
| Location (How Reproducible?) | Go to start page - somewhere and add items to the watchlist - Send it to a (syntactically correct) email address |
| Mean | 4.00 |
The watchlist sending via email is a good idea, but the watchlist is deleted after sending. If the user later wants to edit it again or if he mispelled the email, he has to start from scratch. The Problem can be seen in Figure 4.
N02. Sending the form with wrong information is possible
The finding is summarized in table 10.| Description | The form for ordering brochures gets sent with obviously wrong information like with entered text instead of numbers for the zip code. No further verification of the entered data is done and mistakes by the user cannot be reviewed. |
|---|---|
| Video Clip(s) | n02-cf-pc-neg14-sending-invalid-form.mp4 |
| Heuristic | A05 Error Prevention |
| Location (How Reproducible?) | Go to Good to know - Service - Order Brochures - Enter text instead of numbers for the zip code |
| Mean | 3.75 |
The data entered in the ordering brochures form should be checked for validity where it can be. Maybe with a red border around wrongly entered data, and if the user clicks 'send' it could show an alert that some data is wrong. See Figure 5.
N03. Suggested Drag and Drop does not work
The finding is summarized in table 11.
| Description | The help / documentation provided on the site is not applicable, i.e. it does not work as described. |
|---|---|
| Video Clip(s) | n03-cf-pc-neg21-drag-and-drop.mp4 n03-ew-pc-neg06-watchlist-sort.mp4 |
| Heuristic | A10 Help and Documentation |
| Location (How Reproducible?) | Go to start page - somewhere and add items to the watchlist - go to the watchlist |
| Mean | 3.75 |
The Drag&Drop reordering as suggested in the documentation does not work, the user can drag items, but on release nothing happens. See Figure 6.
N04. English version misses correct translations
The finding is summarized in table 12.
| Description | When switching to the English version of Graz Tourism, some parts have not been translated accordingly. |
|---|---|
| Video Clip(s) | n04-cf-pc-neg30-missing-translations.mp4 n04-ag-mob-neg11-germanparagraph.mp4 n04-ew-pc-neg12-livechat-language.mp4 n04-fb-mob-neg03-watchlist-translation-error.mp4 |
| Heuristic | A02 Speak the Users' Language, A04 Consistency |
| Location (How Reproducible?) | "Go to Graz Tourism - English - Imprint Go to Graz Tourism - English - Data Protection Go to Graz Tourism - English - Chat Go to Graz Tourism - B2B - English Go to Congress - planning organization - conference subsidies Go to watchlist - Navigation bar Go to start page - live chat symbol Go to start page - eat and drink - Restaurant guide - Restaurant guide search - For the last tickbox the translation is missing" |
| Mean | 3.75 |
Some parts of the english website are not translated. In some cases it redirects the user to the german website and he has to change the language back to english. See Figures 7 and 8.
N05. Error message not well visible
The finding is summarized in table 13.
| Description | When entering a wrong verification code into the form for ordering brochures, the error message appears out of the user's field of view. The user has to scroll up to see the error message. |
|---|---|
| Video Clip(s) | n05-cf-pc-neg13-message-out-of-view.mp4 |
| Heuristic | A09 Good Error Messages |
| Location (How Reproducible?) | Go to Good to know - Service - Order Brochures |
| Mean | 3.25 |
When entering a wrong verification code and submitting the ordering form, the error message appeares at the top of the form. The user has to scroll up to see it, to solve it the error message could appear as an alert or next to the 'submit' button. See Figure 9.
4.4 List Of All Negative Findings
Table 14 lists all negative findings that were found by our evaluators in descending order.
| No. | Title | Description | Video Clip(s) | Heuristic | Only on Browser / OS | Location (How Reproducible?) | Found by | Severity rating | Mean | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CF | AG | FB | EW | CF | AG | FB | EW | ||||||||
| 1 | Watchlist deleted after sending | When sending the watchlist to an email address, the watchlist is then deleted automatically (reset). If you misspelled the email address or the email address does not reach out to you, then the watchlist has been lost completely and can not be easily resent. | n01-cf-mob-neg12-watchlist-deleted-after-sending.mp4 | A05 Error Prevention | Go to start page - somewhere and add items to the watchlist - Send it to a (syntactically correct) email address | y | n | n | n | 4 | 4 | 4 | 4 | 4.00 | |
| 2 | Sending the form with wrong information is possible | The form for ordering brochures gets sent with obviously wrong information like with entered text instead of numbers for the zip code. No further verification of the entered data is done and mistakes by the user cannot be reviewed. | n02-cf-pc-neg14-sending-invalid-form.mp4 | A05 Error Prevention | Go to Good to know - Service - Order Brochures - Enter text instead of numbers for the zip code | y | n | n | n | 4 | 4 | 3 | 4 | 3.75 | |
| 3 | Suggested Drag and Drop does not work | The help / documentation provided on the site is not applicable, i.e. it does not work as described. | n03-cf-pc-neg21-drag-and-drop.mp4 n03-ew-pc-neg06-watchlist-sort.mp4 |
A10 Help and Documentation | Go to start page - somewhere and add items to the watchlist - go to the watchlist | y | n | n | y | 4 | 3 | 4 | 4 | 3.75 | |
| 4 | English version misses correct translations | When switching to the English version of Graz Tourism, some parts have not been translated accordingly. | n04-cf-pc-neg30-missing-translations.mp4 n04-ag-mob-neg11-germanparagraph.mp4 n04-ew-pc-neg12-livechat-language.mp4 n04-fb-mob-neg03-watchlist-translation-error.mp4 |
A02 Speak the Users' Language, A04 Consistency | Go to Graz Tourism - English - Imprint Go to Graz Tourism - English - Data Protection Go to Graz Tourism - English - Chat Go to Graz Tourism - B2B - English Go to Congress - planning organization - conference subsidies Go to watchlist - Navigation bar Go to start page - live chat symbol Go to start page - eat and drink - Restaurant guide - Restaurant guide search - For the last tickbox the translation is missing |
y | y | y | y | 4 | 4 | 4 | 3 | 3.75 | |
| 5 | Error message not well visible | When entering a wrong verification code into the form for ordering brochures, the error message appears out of the user's field of view. The user has to scroll up to see the error message. | n05-cf-pc-neg13-message-out-of-view.mp4 | A09 Good Error Messages | Go to Good to know - Service - Order Brochures | y | n | n | n | 3 | 3 | 4 | 3 | 3.25 | |
| 6 | Sending unvalidated form is possible | The form for ordering group trip brochures ("manuals") can be submitted without choosing the brochure to be ordered and without having entered syntactically correct data. This may lead to unintentional errors. | n06-cf-pc-neg27-sending-unvalidated-form.mp4 | A05 Error Prevention | Go to Graz Touristik - Good to Know - Group manual/group trip brochures | y | n | n | n | 4 | 4 | 2 | 3 | 3.25 | |
| 7 | Random ordering in the Shopping Guide | If you open an entry in the Shopping Guide and want to go back via the browser's back button then you come back to the Shopping Guide. However, the entries are now ordered in a different way and you have to look through all again. | n07-cf-mob-neg13-shopping-guide-order.mp4 | A03 Reversible Actions | Go to Shopping & Lifestyle - Shopping Guide - Choose one - Go back via the browser's back button | y | n | n | n | 4 | 3 | 3 | 3 | 3.25 | |
| 8 | Contradictory information provided | On the bottom of the page it says "Mo - So 10.00 - 17.00". However, in the Service section it is specified that we can reach out to the office by phone Mo - Fr 09:00 - 17:00 and Sa - So 10:00 - 17:00. | n08-cf-pc-neg09-inconsistency-office-hours.mp4 | A04 Consistency | Go to Good to know - Service - Information Bureau and then scroll down to the bottom of the page | y | n | n | n | 3 | 4 | 2 | 3 | 3.00 | |
| 9 | Navigation cannot be opened | In the Social Media Channel, the navigation menu item is visible as usual. However, you cannot open the navigation anymore. | n09-cf-mob-neg26-navigation-opening-issue.mp4 | A04 Consistency | Go to Good to know - Social Media & Co - Social Media Channel | y | n | n | n | 3 | 3 | 3 | 3 | 3.00 | |
| 10 | Image cut off | The image of the Austrian motorways is embedded in such a way that Graz is not visible. The user has to download the image to be able to see the motorways around Graz. | n10-cf-mob-neg27-image-cut-off.mp4 | A08 Aesthetic and Minimalist Design | Go to Graz Touristik - Good to know - Directions to Graz | y | n | n | n | 3 | 3 | 3 | 3 | 3.00 | |
| 11 | Map not optimized for mobile | When using the map on a mobile device one notices that the map is not optimitzed for mobile usage. Having a look at a restaurant and opening the map, you can see where the restaurant is located. If you then close the map and open it again later on (without refreshing the page), then the map has zoomed in drastically such that you have to zoom out a lot before being able to see something sensible again. When opening a popup in the map, it opens in a way that the content is not visible/partially visible and it is not possible to arrange it in a readable way |
n11-cf-mob-neg28-map-zoom-issue.mp4 n11-fb-mob-neg08-map-unfortunate-zoom.mp4 |
A04 Consistency A07 - Flexibility and Efficiency of Use |
Go to Graz Touristik - Good to know - Restaurants suitable for groups - Choose one Go to watchlist/place - open map - click on place in map |
y | n | y | y | 3 | 2 | 4 | 3 | 3.00 | |
| 12 | Missing information in map | Some places in the map have no information associated with them |
n12-fb-pc-neg26-missing-information-in-map.mp4 | A02 - Speak the User's Language | Go to a venues site (e.g. Aiola im Schloss) - open the map - Information should be undefined | n | n | y | n | 3 | 3 | 3 | 3 | 3.00 | |
| 13 | On site search only searches current website | If one wants to search Christmas related topics on the main page he will not find the desired results. | n13-ew-mob-neg07-search.mp4 | A10 Help and Documentation | Go to the main page - search for a christmas topic | n | n | n | y | 3 | 3 | 3 | 3 | 3.00 | |
| 14 | Filtering with keywords | Although at least five brochures have the keyword "Graz" already on their cover, when filtering for "Graz", we only obtain three brochures. As the keyword filtering is not explained, I cannot make up how it works as my first intuition seems to not be applicable. | n14-cf-pc-neg12-unexpected-filter-behavior.mp4 | A02 Speak the Users' Language | Go to Good to know - Service - Brochures | y | n | n | n | 2 | 3 | 4 | 2 | 2.75 | |
| 15 | No lazy loading with images provided | The page tries to load all images (thumbnails of the images) at once. Therefore, long waiting times occur. | n15-cf-pc-neg17-no-image-lazy-loading.mp4 | A01 Feedback | Go to Good to know - Social Media & Co - Image database - Fast Access | y | n | n | n | 3 | 3 | 2 | 3 | 2.75 | |
| 16 | Misleading wording | In the watchlist you can click "Show further filters" and then the possibilities to save the watchlist as PDF and send it via mail get revealed. | n16-cf-mob-neg03-misleading-wording.mp4 n16-ew-mob-neg02-watchlist-send-mail.mp4 |
A02 Speak the Users' Language | Go to watchlist - Show further filters | y | n | y | y | 3 | 3 | 4 | 3 | 2.75 | |
| 17 | Going back implies a reload | When having a closer look at a specific event and then going back again, the previous page is loaded again and jumps up to the top of the page. | n17-cf-mob-neg06-going-back-jumps.mp4 | A03 Reversible Actions | Go to See & Do - Events - Choose one - Go back again | y | n | n | n | 3 | 2 | 4 | 2 | 2.75 | |
| 18 | Viewing Watchlist leads to context switch | When viewing the watchlist from e.g. Graz Touristik sub section of the Graz Tourism portal, you switch back to the main Graz Tourism portal and are no longer in the previous sub section (e.g. Graz Touristik) without any information, so the user might wonder why the navigation now looks different. | n18-cf-mob-neg29-watchlist-context-switch.mp4 | A02 Speak the Users' Language | Go to Graz Touristik - watchlist | y | n | n | y | 2 | 3 | 3 | 3 | 2.75 | |
| 19 | Missing part of the subsection | At the category "Good to know" under the section "social-media-co/social-media-channel", there should be four possibilites (facebook, instagram, webcam and YouTube). At portrait mode, you can only see the sections "facebook", "instagram" and "YouTube" with pictures and plugins. The whole "Webcam" section is missing. On the other hand, if you put the mobile phone at landscape mode, you can see all four sections. |
n19-ag-mob-neg08-missingpart.mp4 | A04 Consistency | Firefox Android [on other Browser-OS combinations the "Webcam"-section is not visible at all, neither in protrait, nor in landscape mode] | Go to Good to know - social media and co - social media channel | n | y | n | n | 3 | 4 | 2 | 2 | 2.75 |
| 20 | Favorites - No Feedback | It doesn't give any feedback when adding an item to favorites, besides changing the icon. Thus it is not clear what difference it makes. |
n20-fb-pc-neg01-favoritesno-feedback.mp4 | A05 - Error Preventio | Go to watchlist - add to favorites | n | n | y | n | 3 | 2 | 4 | 2 | 2.75 | |
| 21 | Booking.com not mentioned | It is nowhere mentioned that the booking box is from booking.com |
n21-fb-mob-neg17-booking.com-not-mentioned.mp4 | A02 - Speak the User's Language | Go to the main page - scroll down until booking box | n | n | y | n | 2 | 2 | 4 | 3 | 2.75 | |
| 22 | Loading a video takes very long | Although a busy cursor and some loading symbol is provided, loading a short video takes longer than a user would accept it to do. | n22-cf-pc-neg02-video-loading-time.mp4 | A01 Feedback | Go to start page - "Graz Video" | y | n | n | n | 3 | 3 | 2 | 2 | 2.50 | |
| 23 | 404 Error Message when wrong email entered | In case you enter a wrong email to which your watchlist should be sent to, you get an 404 error. This error is actually meant for another kind of error and does not provide sensible error information. | n23-cf-pc-neg22-404-error-message.mp4 | A09 Good error messages | Go to start page - somewhere and add items to the watchlist - go to the watchlist - enter "a" for the mail address - click "Send email" | y | n | n | y | 3 | 4 | 1 | 2 | 2.50 | |
| 24 | Going back leads to a jump | On the homepage, when clicking on an event and going back again, the site jumps back to a different point on the homepage and you have to scroll down again. | n24-cf-pc-neg03-jumping-when-going-back.mp4 | A03 Reversible Actions | Only on Safari | Go to TOP EVENTS - select one - go back again | y | n | n | n | 2 | 2 | 3 | 2 | 2.25 |
| 25 | Last nav element needs special treatment | Usually, all side navigation elements can be opened by clicking on their name. However, the last element ("our tips") can only be expanded when clicking on the plus symbol. | n25-cf-pc-neg08-inconsistency-last-nav-element.mp4 | A04 Consistency | Go to See & Do - last element of the side navigation | y | y | n | y | 2 | 3 | 2 | 2 | 2.25 | |
| 26 | Abbreviation not explained | In the Service section it says "Mo - So/Ft: 10.00 - 17.00 Uhr" where "Ft" is an abbreviation nowhere explained and which I do not know. | n26-cf-pc-neg10-unexplained-abbreviation.mp4 | A02 Speak the Users' Language | Go to Good to know - Service - Information Bureau | y | n | n | n | 2 | 3 | 2 | 2 | 2.25 | |
| 27 | Webcam counterintuitive | Regarding the second webcam on the webcam sub-page, you cannot go to full screen as suggested by the icon. Furthermore, it is counterintuitive how to play the video linked to the sight-seeing spot. | n27-cf-pc-neg15-webcam-counterintuitive.mp4 | A02 Speak the Users' Language | Go to Good to know - Service - Webcam Graz - Second webcam | y | n | n | n | 3 | 2 | 2 | 2 | 2.25 | |
| 28 | Ordering of brochures | The brochures of the city magazines are not ordered in an obvious way. The user has to go through all when searching for a specific brochure. | n28-cf-pc-neg25-city-magainze-brochure-ordering.mp4 | A02 Speak the Users' Language | Go to Shopping & Lifestyle - Good to Know - City Magazines | y | n | n | n | 3 | 1 | 3 | 2 | 2.25 | |
| 29 | Content cut off | Some links are cut off and the user cannot read where the link redirection will lead to in the Data Protection page. Furthermore, the statistics of the capacities of the hotels in Graz is cut off too in Graz Congress section. | n29-cf-mob-neg08-links-cut-off.mp4 | A08 Aesthetic and Minimalist Design | Go to start page - section 'Data Protection' / 'Datenschutzbestimmungen' - Scroll down to the first clickable links Go to Graz Congress (Convention Bureau) - Planning and Organisation - Graz Convention Bureau |
y | n | n | n | 3 | 2 | 2 | 2 | 2.25 | |
| 30 | Cookie consent message | If you want to order gift cards for the city of Graz, you go to the corresponding page and have to type in how many gift cards you would like to order. If you type in a number for the first time in this session, then you are redirected to another page which says that cookies may now be saved by Safari although not explicitly accepted for this specific page and the message is shown only a short amount of time such that you cannot read it. | n30-cf-mob-neg17-cookie-consent-message.mp4 | A01 Feedback | Go to Shopping & Lifestyle - Graz City Gift cards - Order Graz City Gift Cards - Enter a number in the field of how many gift cards you want to order (In the meanwhile has the function been deactivated) |
y | n | n | n | 3 | 2 | 2 | 2 | 2.25 | |
| 31 | Autofill in the brochure form does not work | When trying to order a brochure, the autofill is not fully working. E.g. the first name and last name fields cannot be autofilled automatically (whereas the zip code works as expected). | n31-cf-mob-neg23-form-autofill.mp4 | A07 Flexibility and Efficiency of Use | iPhone using Safari | Go to Good to know - Service - Order a brochure | y | n | n | n | 2 | 1 | 4 | 2 | 2.25 |
| 32 | Missing prices | At "Incoming" under the section "programme-suggestions", there is no price information, although there is the € symbol. | n32-ag-mob-neg10-missingprices.mp4 | A06 Recognition Rather Than Recall | Go to Groups - programme suggestions | n | y | n | y | 2 | 3 | 1 | 3 | 2.25 | |
| 33 | Advents Map | Newly opened popups/places open below the previous popup and overlay parts of it, this causes them to also show outside of the map region |
n33-fb-pc-neg28-advents-map.mp4 | A08 - Aesthetic and Minimalist Design | Go to Graz Advent subpage - scroll down - open the map - open a second map | n | n | y | n | 3 | 2 | 2 | 2 | 2.25 | |
| 34 | Printing will cut off content | Although the option to print is presented explicitly on the "Our Team"-page, it will not work nicely as on the second page, one row of people will be cut off. | n34-cf-pc-neg06-printing-cut-off.mp4 | A08 Aesthetic and Minimalist Design | Only on Safari | Go to Good to know - Our Team - "Print this page" | y | n | n | n | 2 | 3 | 1 | 2 | 2.00 |
| 35 | Download starts without stating the size | When downloading a brochure, you do not have any information on how large the download will be. When having download restrictions, it may be important to know that the download will have e.g. about 10 MB in advance. | n35-cf-pc-neg24-download-size-not-stated.mp4 | A05 Error Prevention | Go to Good To Know - Service - Brochures - Download | y | n | n | n | 3 | 2 | 1 | 2 | 2.00 | |
| 36 | Add to watchlist - reversible | If a place is added to the watchlist, it is not possible to undo it without doing so in the watchlist |
n36-fb-pc-neg04-add-to-watchlistreversible.mp4 | A03 - Reversible Acions | Go to place - scroll down till save to watchlist button - click - scroll down again | n | n | y | n | 2 | 1 | 3 | 2 | 2.00 | |
| 37 | More Events - Long loading | Opening the more events page takes a long time compared to other pages on graztourismus |
n37-fb-pc-neg19-more-eventslong-loading.mp4 n37-ew-pc-neg19-loading-time.mp4 |
A01 - Feedback | Go to start page - Events - Click 'more events' | n | n | y | y | 3 | 2 | 1 | 2 | 2.00 | |
| 38 | URL not shortened when printing | When storing the watchlist as a PDF, users might print the PDF and if they want to get further information later on again, they have to enter the full URL of this sightseeing-spot. URL shortening would have led to more efficiency. | n38-cf-pc-neg23-url-shortening-for-printing.mp4 | A07 Flexibility and Efficiency of Use | Go to start page - somewhere and add items to the watchlist - go to the watchlist - give it a name and store it as a PDF | y | n | n | n | 2 | 2 | 1 | 2 | 1.75 | |
| 39 | Navigation inconsistency | Usually, in the side navigation we have subcategories for the main category we have chosen in the upper head navigation. But this convention is broken within the Congress and Exhibition Calendar category. | n39-cf-pc-neg28-navigation-inconsistency.mp4 | A04 Consistency | Go to Graz Congress - Congress and Exhibition Calendar | y | n | n | n | 2 | 2 | 1 | 2 | 1.75 | |
| 40 | Dropdown menu not easy to close | The specified dropdown menu can only be closed by clicking somewhere else on the page but not by clicking on the dropdown button itself again. | n40-cf-mob-neg01-dropdown-closing.mp4 | A02 Speak the Users' Language | Go to Graz Tourism Homepage - 'our websites' / 'weitere Portale' dropdown menu | y | n | n | y | 2 | 2 | 1 | 2 | 1.75 | |
| 41 | Clickable symbol covered | On the bottom of the page, there is a symbol which is clickable but very hard to reach as the red chat symbol almost all of the time covers the other symbol. | n41-cf-mob-neg04-symbol-covered.mp4 | A08 Aesthetic and Minimalist Design | Go to Do & See - Choose a sight - scroll down to the bottom | y | n | n | n | 1 | 2 | 2 | 2 | 1.75 | |
| 42 | Watchlist with zero elements | When trying to save an empty watchlist as a PDF, the site opens a new tab as if it were working. However, the site neither generates an error message nor an empty PDF. | n42-cf-mob-neg10-watchlist-zero-elements.mp4 | A09 Good Error Messages | Go to the Watchlist without having added elements - Save the watchlist as a PDF | y | n | n | n | 2 | 2 | 1 | 2 | 1.75 | |
| 43 | Click outside | The menu popup doesn't allow clicking outside, but the search popup does |
n43-fb-mob-neg07-click-outside.mp4 | A04 - Consistency | Go to the menu - click outside - close menu - click search - click outside | n | n | y | n | 2 | 2 | 1 | 2 | 1.75 | |
| 44 | Place icon buttons | The icon buttons of places do not look different if they are clickable or not, for example the accessible button is not clickable, but the family button is |
n44-fb-mob-neg12-place-icon-buttons.mp4 | A04 - Consistency | Only on the english site | Go to a place - scroll down - click accessible - nothing happens - click on family - family website opens | n | n | y | n | 2 | 1 | 3 | 1 | 1.75 |
| 45 | Navigation issue with 'home'-symbol | The 'home'-symbol in the navigation is clickable in a position where it seems like I would click on another navigation point. This issue only occurs for the left-most navigation entry in the main navigation on top of the site. | n45-cf-pc-neg01-navigation-home-symbol.mp4 | A02 Speak the Users' Language | Go to 'home'-symbol (in the vertical middle) until right until the navigation menu is opened | y | n | n | n | 1 | 2 | 2 | 1 | 1.50 | |
| 46 | Inconsistency with opening images in full screen | As learned throughout the website, the yellow rectangle with the black arrow in it in the right-bottom corner of the image suggests that you can click the image and open it in full screen. However, this does not work in the presented representation. | n46-cf-pc-neg16-full-screen-images-consistency.mp4 n46-fb-pc-neg25-image-gallery-menu,-unclear-button.mp4 |
A04 Consistency | Go to Good to know - Service - BackCup Go to Eat and drink - Events - Styrian Menues - scroll down - click on yellow arrow button |
y | n | y | n | 2 | 1 | 2 | 1 | 1.50 | |
| 47 | Sliding window inconsistency | On the homepage, in the second window you can slide through the content. However, in the first window which looks exactly the same you cannot slide, you have to click the left / right buttons. | n47-cf-mob-neg02-sliding-window-inconsistency.mp4 | A04 Consistency | Go to Graz Tourism - First two sliding windows | y | n | y | n | 1 | 2 | 1 | 2 | 1.50 | |
| 48 | Search function is not visible everywhere | If you are looking for someting in the search bar, the URL path to "good-to-know" is opened and then searched for and displayed everywhere. On the other hand, if you go in via the menu tab on the main website via "good-to-know", you won't see any search entry where to look. |
n48-ag-pc-neg04-searching.mp4 | A04 Consistency | Go to Good to know | n | y | n | y | 2 | 2 | 1 | 1 | 1.50 | |
| 49 | Missing picture or preview at facebook | At the category "Good to know" under the section "social-media-co/social-media-channel", there are four possibilites (facebook, instagram, webcam and YouTube). A picture or preview at Facebook is missing. |
n49-ag-pc-neg17-missingatfacebook.mp4 | A04 Consistency | Firefox Ubuntu | Go to Good to know - social media and co - social media channel | n | y | n | n | 2 | 2 | 1 | 1 | 1.50 |
| 50 | Not interactable area | In Graz advent, two image boxes that redirect to a subpage when clicked on do not respond to click on the upper half of the control. |
n50-fb-pc-neg20-not-interactable-area.mp4 | / | Edge | Go to Christmas - scroll down to Christmas Highlights - Click on the upper area of one of the two rightmost boxes | n | n | y | n | 2 | 1 | 2 | 1 | 1.50 |
| 51 | Our website - open tab/open in same tab | The websites in 'Our websites' open the selected website in a new tab, except if you select 'Graz Tourism' |
n51-fb-pc-neg23-our-websiteopen-tab-open-in-same-tab.mp4 | A04 - Consistenc | Go to 'Our Websites' - notice how website is opened | n | n | y | n | 1 | 2 | 2 | 1 | 1.50 | |
| 52 | Cookies - unclear hyperlinks | The hyperlinks in the cookies notification are unclear, they look like they belong together (one hyperlink) but they are two different links |
n52-fb-mob-neg14-cookiesunclear-hyperlinks.mp4 | A08 - Aesthetic and Minimalist Design | Go to the website with deleted browser data - check the notification at the botton | n | n | y | n | 2 | 2 | 1 | 1 | 1.50 | |
| 53 | Links not highlighted nor clickable | Usually, links are highlighted in a specific way and are made clickable throughout the website. However, in the data protection information text the convention was broken. | n53-cf-pc-neg05-links-not-highlighted.mp4 | A04 Consistency | Go to start page - section 'Data Protection' / 'Datenschutzbestimmungen'. | y | n | n | n | 1 | 2 | 1 | 1 | 1.25 | |
| 54 | Background image missing | Usually, all subpages of that specific kind have background images. However, these subpages do not. | n54-cf-pc-neg07-background-image-missing.mp4 n54-ag-pc-neg14-missingpicturecasinograz.mp4 |
A04 Consistency | Go to Good to know - About us - Tourism Coach Go to Shopping & lifestyle - our tips - Casino Graz Go to travel and transport - getting here - by caravan |
y | y | n | n | 1 | 1 | 2 | 1 | 1.25 | |
| 55 | Clickable button does nothing | In the calendar view of the events, there is a clickable button with an arrow down symbol. However, the button does nothing. | n55-cf-pc-neg20-clickable-button-doing-nothing.mp4 | A04 Consistency | Go to See & Do - Events - Event calendar - Calendar View - Choose a date | y | n | n | n | 1 | 2 | 1 | 1 | 1.25 | |
| 56 | Description might be missing | At the top on the right side there is the button "watchlist". But there is no description and no further instructions. | n56-ag-pc-neg02-watchlistdescription.mp4 | A10 Help and Documentation | Go to watchlist | n | y | n | n | 1 | 2 | 1 | 1 | 1.25 | |
| 57 | Different tips on the subpage | In the middle at the section "Our tips", there are four suggestions ("Graz tourist information", "webcam", "Just arrived?", "Corona information"). If you click on "more information", you come to the subpage "see-and-do" and get the tips from there, but they are not the same as the suggestions on the main page. |
n57-ag-pc-neg05-differenttips.mp4 | A04 Consistency | Go to See & Do | n | y | n | n | 1 | 2 | 1 | 1 | 1.25 | |
| 58 | Unnecessary subsections of events | At the category "See & Do" under the section "events", almost all of the subsections lead to the "Event calendar". Only the category to be selected has adapted to the subsection. These subsections can be left out. |
n58-ag-pc-neg12-unnecessarysubsections.mp4 | A08 Aesthetic and Minimalist Design | Go to See & Do - events - Event calendar | n | y | n | n | 2 | 1 | 1 | 1 | 1.25 | |
| 59 | Missing additional description at subtitle | At the category "Shopping & Lifestyle" under the section "markets/flea-markets", there is a lonely subtitle "Markets off Fröhlichgasse street at the Grazer Messe parking lot". There is not any text/description, picture or link. |
n59-ag-pc-neg15-lonelysubtitle.mp4 | A10 Help and Documentation | Go to Shopping and lifestyle - markets - flea-markets (Make sure to switch to the English version; does not occur in the German version) |
n | y | n | n | 1 | 1 | 2 | 1 | 1.25 | |
| 60 | Christmas Graz Menu - Items placement | The items placement in the menu of Christmas Graz Website has no real structure, items seem randomly placed with unpleasant gaps between them |
n60-fb-pc-neg18-christmas-graz-menuitems-placement.mp4 | A08 - Aesthetic and Minimalist Design | Go to Christmas Graz - Hover over menu | n | n | y | y | 2 | 1 | 1 | 1 | 1.25 | |
| 61 | Empty menues | Even if a menu has no items it shows the expandable button |
n61-fb-pc-neg31-empty-menues.mp4 | / | Go to Hotels & Appartements - look at the sidebar - expand and collapse Graz Packages | n | n | y | n | 1 | 2 | 1 | 1 | 1.25 | |
| 62 | Sightseeing - different behaviours | The Sightseeing control on graztourismus behaves differntly if it is fully loaded or not, if it is fully loaded places are opened as popups inside the Sightseeing area. If it is not fully loaded places are opened like a link. |
n62-fb-pc-neg32-sightseeingdifferent-behaviours.mp4 | / | Go to the main page - scroll down - click on a place in sightseeing - again go to main page - scroll down - click on a place in sightseeing (make sure to be fast) | n | n | y | n | 1 | 2 | 1 | 1 | 1.25 | |
| 63 | Icons should flip by convention | On the homepage, usually all icons adhering to this type of link flip when hovering over them. With one such link this mechanism is broken. | n63-cf-pc-neg04-icon-not-flipping.mp4 | A04 Consistency | Only on Safari | Go to start page - section "Sightseeing - Top 10" - hover over the link in the right bottom corner | y | n | n | n | 1 | 1 | 1 | 1 | 1.00 |
| 64 | Broken alignment of brochures | Not all brochures are aligned in the same way, therefore, it seems broken to the user. | n64-cf-pc-neg11-brochure-alignment.mp4 | A08 Aesthetic and Minimalist Design | Only on Safari | Go to Good to know - Service - Brochures | y | n | n | n | 1 | 1 | 1 | 1 | 1.00 |
| 65 | Typo in best times for visiting | When having a look at when to do this trip in Graz, it says "best time" where March is stated twice and February no single time. | n65-cf-pc-neg18-typo-visiting-time.mp4 | A04 Consistency | Go to See & Do - Trips around Graz - Choose one - Scroll down | y | n | n | n | 1 | 1 | 1 | 1 | 1.00 | |
| 66 | Redirection to Facebook site where other behavior expected | When clicking on an item in the side navigation, we usually get to a site of the Graz Tourism platform. However, in this case we instantly get to a Facebook site. | n66-cf-pc-neg19-facebook-redirection.mp4 | A04 Consistency | Go to See & Do - Guided tours - Graz Rikscha | y | n | n | n | 1 | 1 | 1 | 1 | 1.00 | |
| 67 | Accommodation legend needs to be remembered | When going to the accommodation overview a legend is provided. When switching to e.g. the Hotel section, this legend is no longer presented. | n67-cf-pc-neg26-legend-memorization.mp4 | A06 Recognition Rather Than Recall | Go to Hotels & Appartements - Accommodation overview and then switch to Hotel | y | n | n | n | 1 | 1 | 1 | 1 | 1.00 | |
| 68 | Side Navigation Highlighting convention broken | In Advent in Graz under Christmas Markets the selected sub-sub-categories, like "Hauptplatz" are usually marked as being selected when having clicked on them. However, this convention is broken here. | n68-cf-pc-neg29-navigation-highlighting-broken.mp4 | A04 Consistency | Go to Graz Advent - Christmas markets - Choose a sub element like "Hauptplatz" | y | n | n | n | 1 | 1 | 1 | 1 | 1.00 | |
| 69 | Drop down element with only one element | After having selected a shop in the shopping guide, you are presented with a dropdown menu. In this dropdown menu you have only one possible selection. The user might ask what the dropdown menu is for. | n69-cf-mob-neg14-dropdown-one-element.mp4 | A08 Aesthetic and Minimalist Design | Go to Shopping & Lifestyle - Shopping Guide - Choose one | y | n | n | n | 1 | 1 | 1 | 1 | 1.00 | |
| 70 | Discover Graz Image not clickable | Usually, you can click on the whole image in order to open the respective page. However, on the homepage, for the first entry "Discover Graz" the image is not clickable and you have to click on the text. | n70-cf-mob-neg20-image-not-clickable.mp4 | A04 Consistency | Go to Graz Tourism - First image vs second image underneath | y | n | y | n | 1 | 1 | 1 | 1 | 1.00 | |
| 71 | Other naming might be confusing | At the top of the main page, if you click on the subpage "Christmas", a new page is loaded, but with URL ".../advent". The other naming might be confusing for people who e.g. don't know the word "Advent". |
n71-ag-pc-neg01-othernaming.mp4 | A02 Speak the Users’ Language | Go to Christmas | n | y | n | n | 1 | 1 | 1 | 1 | 1.00 | |
| 72 | Different sizes of "+" and "-" | On all categories from the menu bar, on the left side, there are the different menu registers/sections. The "+" symbol is bigger than the "-" symbol if the section is open. |
n72-ag-pc-neg11-minussize.mp4 | A04 Consistency | Go to See & Do | n | y | n | n | 1 | 1 | 1 | 1 | 1.00 | |
| 73 | Events - Text too long | The text of some events boxes doesn't fit in the box and is not ended with an ellipse(... for example) |
n73-fb-pc-neg05-eventstext-too-long.mp4 | A08 - Aesthetic and Minimalist Design | Go to start page (Congress/Tourism/...) - scroll down till violet boxes | n | n | y | n | 1 | 1 | 1 | 1 | 1.00 | |
| 74 | Search icon border | The border around the search icon is not aligned correctly, the border sticks to the icon on the left side |
n74-fb-pc-neg21-search-icon-border.mp4 | / | Edge | Go to start page - search icon in the search bar | n | n | y | n | 1 | 1 | 1 | 1 | 1.00 |
| 75 | Map - unclear icons | The map has two icons on the top, it looks like both are clickable, but they belong to the same button => same action |
n75-fb-mob-neg13-mapunclear-icons.mp4 | A08 - Aesthetic and Minimalist Design | Go to watchlist - open the map - click on icons - map closes no matter where clicked | n | n | y | n | 1 | 1 | 2 | 0 | 1.00 | |
| 76 | Expandable menu | If you are on a subpage and you open the menu, you can collapse menu entries. But if you collapse the item you are currently in, the menu item collapses but the icon does not change to a + sign |
n76-fb-mob-neg15-expandable-menu.mp4 | / | Go to a page with a menu - click on it - collapse it | n | n | y | n | 1 | 1 | 1 | 1 | 1.00 | |
| 77 | Hover of preview buttons | If you click on an arrow in the preview, the button is highlighted and stays that way until clicked elsewhere |
n77-fb-mob-neg16-hover-of-preview-buttons.mp4 | / | Go to the main page - scroll to first preview - click on the arrow | n | n | y | n | 1 | 1 | 1 | 1 | 1.00 | |
| 78 | Open map button | When clicking open map, the open map button stays and an additional close map button becomes visible, only one of the two should be visible at a time |
n78-fb-pc-neg27-open-map-button.mp4 | A08 - Aesthetic and Minimalist Design | Go to the watchlist - open the map | n | n | y | n | 1 | 1 | 1 | 0 | 0.75 | |
| 79 | Omit unnecessary designation | At the upper right corner on the main page and on all subpages, "Italian" and "Deutsch" are enough. "Languages" before these words is not necessary. In this case it complements the pattern, e.g. also for "Our websites", but it can be omitted because it is self-explanatory anyway. |
n79-ag-pc-neg03-languages.mp4 | A08 Aesthetic and Minimalist Design | Go to start page | n | y | n | n | 0 | 0 | 1 | 0 | 0.25 | |
References
- [And2021]
- Keith Andrews;
Human-Computer Interaction Course Notes;
http://courses.iicm.tugraz.at/hci/hci.pdfVisited 2021-03-13 - [Nie1990]
- Jakob Nielsen and Rolf Molich;
Heuristic Evaluation of User Interfaces;
Proc. Conference on Human Factors in Computing Systems (CHI’90).
ACM. Seattle, Washington, USA, Apr 1990, pages 249–256.
doi:10.1145/97243.97281 - [Nie1992]
- Jakob Nielsen;
Finding Usability Problems Through Heuristic Evaluation;
Proc. Conference on Human Factors in Computing Systems (CHI’92).
ACM. Monterey, California, USA, May 1992, pages 373–380.
doi:10.1145/142750.142834 - [Nie1994]
- Jakob Nielsen;
Enhancing the Exploratory Power of Usability Heuristics;
Proc. Conference on Human Factors in Computing Systems (CHI’94).
ACM. Boston, Massachusetts, USA, Apr 1994, pages 152–158.
doi:10.1145/191666.191729 - [Won2020]
- Euphemia Wong;
Heuristic Evaluation: How to Conduct a Heuristic Evaluation;
July 2020.
https://www.interaction-design.org/literature/article/heuristic-evaluation-how-to-conduct-a-heuristic-evaluation
A Materials
The following materials were used by the evaluation team.
A.1 Heuristics
The evaluators have used the Andrews General Usability Heuristics 2013
found in file:
heuristics.pdf.
A.2 Log Files
The individually collected notes can be seen in the following (plain text) log files.
- Anna Glaser:
log-ag-mob.txtandlog-ag-pc.txt. - Christof Feischl:
log-cf-mob.txtandlog-cf-pc.txt. - Emil Winterleitner:
log-ew-mob.txtandlog-ew-pc.txt. - Fabian Burgmann:
log-fb-mob.txtandlog-fb-pc.txt.
A.3 Spreadsheet
The merged individual lists can be seen in the following file:
helist.xlsx.
B List of Abbreviations
| Abbreviation | Definition |
|---|---|
| AG | Anna Glaser |
| CF | Christof Feischl |
| EW | Emil Winterleitner |
| FB | Fabian Burgmann |
| y (Regarding the table for the positive and negative findings) | Yes, this person did find this issue |
| n (Regarding the table for the positive and negative findings) | No, this person did not find this issue |