Heuristic Evaluation Report

1 Executive Summary

1.1 Methodology

1.2 Main findings

1. Valid mail address is invalid
   
   According to the RFC 3696 standard which is defining if a mail addresses is valid or not, the address "_somename@example.com" should be valid. However, it is marked as invalid in the form.
2. The search form is very small
   
   The search from is too small on mobile devices. The user is forced to zoom manually into the form.
3. No direct enrollment link
   
   If you have decided to start a study at the TU Graz and want to enroll to a study you won't immediatey find something like a enrollment menu point.
4. Only hard to remember Username/Password possible
   
   Usernames mustn't contain your own name which is ridiculous because it doesn't provide any means of security. Additionally, the password restrictions make it very hard to make a password rememberable.
5. Invalid E-mail
   
   In the feedback form is an optional field for the e-mail address and there is no check if the e-mail is valid or not.
6. Search site has different design
   
   In the search page the menu bar, footer etc. is missing in comparison to the main page. If the URL wouldn't be a subdomain of tugraz.at you would think that you have been redirected to a scammer site.
7. Link failure
   
   There are four links on this page which are not working. When one of them is clicked, the website offers no return and nothing happens.

2 Introduction

The web site we are going to evaluate provides information about the Technical University Graz. The visitor gets informed about the range of studies, facilities and institutes.

We are going to evaluate this web site, because we participate in the course Human Computer Interaction. We also want to get a good grade.

3 Evaluation Methodology

A heuristic evaluation is a fast, cheap and efficient way to check the usability of an user interface. To achieve this goal, a group of evaluators checks the UI individually and notes all the negatives and positives.

Heuristic Evaluation (originally proposed by Nielsen and Molich, 1990) is a discount method for quick, cheap, and easy evaluation of the user interface. [Danino2001]. At first the evaluators have to consider which target group is using the UI and describe their needs and tasks. The evaluators try to think and act like the typical user. This is necessary to find all the problems the users could face. (...) differences between heuristic evaluation sessions and traditional user testing are the willingness of the observer to answer questions from the evaluators during the session and the extent to which the evaluators can be provided with hints on using the interface. [Nie1995].

The evaluators also have to test on different platforms and devices the main target group may use. All the problems the evaluators find have to be documented for analyse purpose. During the documentation screenshots and screen captures are made for better visualisation and comprehensibility of the problem. For this evaluation, the "Andrews General Usability Heuristics", shown in Appendix A, will be used. These are based on and slightly adapted from Nielsen's revised set of ten usability heuristics [Nie1994]. These ten heuristics contain rules how a UI should look and response to an user. Then every evaluator puts his notes into a list and gives them grades depending on the importance of the problem. After the validation, the individual lists are merged into one big list and then get sorted by the importance of the problem. With this list of positives and negatives they can create the heuristic report. This detailed report will be given to the customer the heuristic evaluation was made for.

4 User Profiles

The typical user that generates the most traffic would be a prospective student. Since the lifeblood of a university are the people studying there, the university also wants to attract as many of them as possible. In addition to that, the site also wants to provide information about the quality of the university to employers of graduates. Another possible user group would be Jobseekers, who want to look up available job offers and most likely the pay. Students and graduates could also visit the site from time to time.

The prospective student is used to computers, knows his way around but also wants to see modern design. Since the user is used to responsive web and doesn't know the internet before it was as beautiful as it is now. The employer is most likely a business man using computers constantly since he looked up the university of his prospective employee. He doesn't want to spend a lot of time on the site and wants information as fast as possible. Jobseekers probably want to invest a lot of time in the site to gather as much information as possible about available job offers and the required qualifications for these. A student is already familiar with the site and uses it from time to time. He is very familiar with computers and uses them regularly. The graduate wants to connect with his fellow graduates. He doesn't use computers too much. Unless the interface is complex to handle, he/she is able to use it.

Our typical user, the prospective student, wants to get information about available studies and how to enroll into courses. The Curriculum of the study he is interested in is also very important for him. The employer wants to see the quality of the university in comparison to other national or international universities. The jobseekers want to look up available job offers and the amount of salary. Students want information about deadlines and the Curriculum. Graduates want to use the Alumni network to connect with fellow graduates.

5 Extent of the Evaluation

The field of the evaluation includes the whole website. The site consists of the main site and main menu which is separated in TU Graz, Studying and Teaching, Research, Faculties and Institutes and Information for. We assess every menu item in order to their usability. Each of the menu items provides specific information for the visitors of the website. As part of the evaluation we try to think like a typical and potential user and use the website like they would do. Every menu item is more or less important for specific users.

6 Evaluators and Evaluation Environments

The hardware and software used by each evaluator is shown in Table 1.

MW used the native screenshot function by Samsung for his Android phone and for screen recordings the app AZ Screen Recorder was used. FM used the same tools for screenshots and screen recordings like MW. PG used RecordMyDesktop for Video Capture and the Ubuntu Native screenshot feature for screenshots. Quicktime player was used by FL for his MacBook.

7 Positive Findings

The top three to five positive findings according to their average positivity ratings are described in more detail below. The positivity rating scheme used to rank the positive findings is shown in Table 2.

The main positives, which were found by the eveluators, refer to the design of the website. The first thing is the very important overview over all the studies.

7.1 Studies Overview

This site has a very good overview of all the Bachelor and Master studies which were offerd by the University. Upcoming students can easily find out more about the studies they are interested in. You can see the overview in Figure 1.

7.3 Clean Information Categorization

The tile design provides not only a clear information seperation (deadlines, links etc.) but also presents this information in an asthetic design. You can see it in Figure 3.

8 Analysis of the Main Problems

The top five problems according to their average severity ratings are described in more detail below. Problem number 1 is the problem (negative finding) with the highest average severity. The severity rating scheme used to rank the problems is shown in Table 3.

8.1 Correct Mail Address is Invalid

A main problem founde by the evaluators is, that an RFC 3696 valid e-mail address is marked as invalid in the University registration form as shown in Figure 4.

8.2 Very small Search Form

On mobile devices, the search form the website provides, is too small. The user has to zoom in as shown in Figure 5.

8.3 No direct Enrollment Link

If you have decided to start a study at the TU Graz and want to enroll to a study, you won't immediately find something like an University enrollment menu point. The menu is shown in Figure 6.

8.4 Too complex Username and Password

The University username can't contain your own name which is ridiculous as it doesn't provide any means of security. Also the password restrictions make it very hard to make a password rememberable. The problem is shown in Figure 7.

8.6 Different designed Search Page

Also on the PC the Search page looks different. It is missing the menu bar, footer etc. the main page uses. If the URL wouldn't be a subdomain of tugraz.at you would think that you have been redirected to a different site. The page is shown in Figure 9.

9 List of Problems Found

References

[Nie1994]

Jakob Nielsen; Enhancing the Exploratory Power of Usability Heuristics; Proc. Conference on Human Factors in Computing Systems (CHI’94). ACM. Boston, Massachusetts, USA, Apr 1994, pages 152–158. doi:10.1145/191666.191729

[Nie1995]

Jakob Nielsen; How to Conduct a Heuristic Evaluation; Visited 2017-03-23. https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

[Danino2001]

Nicky Danino; Heuristic Evaluation - a Step By Step Guide Article; Visited 2017-03-27. https://www.sitepoint.com/heuristic-evaluation-guide/

A Heuristics Used by the Evaluators

The evaluators used the Andrews General Usability Heuristics 2013 found in file: heuristics.pdf.

B Individual Evaluation Logs

The individual evaluation logs are listed below.

C Spreadsheet

The filled-out spreadsheet of all problems and positives including their ratings is helist.xls.