First high-fidelity prototype

The first high-fidelity prototype was designed to analyse the performance of parallel programs, as articulated in Chapter 6 about the High-fidelity prototype. This first interactive prototype design was developed to enable users to interact with the system by performing visual queries on it. In this exercise, the aim was to understand both how users interact with the systems and how the system interacts with the users. Director of the University of Tennessee’s Innovative Computing Laboratory requested various computer scientists to help us evaluate the new callgraph visualisation, consequently, those chosen scientists performed practical evaluation on the first interactive prototypes of the callgraphs. Seven users evaluated the usability of the first high fidelity prototype design. The same visual queries performed by an OU expert on the low- fidelity prototypes were refined and executed on the first interactive prototype system by UT users. Table 4.2 depict the visual queries performed by the users on the first high fidelity prototype and original design of the callgraph visualisation:

Indicate objects that consumed normal, moderate and excessive execution time Establish the status of a particular node

Identify details of the node:

Which routine name is represented by a node? Which procedure called an object?

What amount of time is consumed by different nodes?

What is the relationship between an object and its associates? Demonstrate how the system works:

Which nodes have children nodes? How to expand or collapse nodes?

How to filter and display a node with its associates?

How to return back to main screen after expanding or collapsing nodes? Which objects cause performance bottlenecks?

56

During the evaluation process, the same visual queries performed on the first high fidelity prototype were also executed on the original design of callgraph visualisation. Users performed these queries in a controlled laboratory (office) with no access of the public members. The visual queries (Table 4.2) helped us to identify usability problems on both first interactive prototype and original design of the callgraph visualisation. It was anticipated that the queries would provide answers to the following:

Can users perform the important visual queries quickly and effectively? Does the design provide various ways in which to obtain performance information about the program?

Does the callgraph design meet the users’ needs and expectations? Is the design informative to the users?

Does the system design offer a distractive user interface?

Does the final design expect users to carry out multiple tasks to acquire information?

Table 4.3: Answers to the visual queries performed by users.

After performing visual queries, users were interviewed and completed questionnaire about the first high fidelity prototype and original design of the callgraphs. The questionnaire focused on how easy or hard it was to use both the original and first high-fidelity prototype design of the callgraph visualisation system. In the questionnaires (survey), users indicated job titles and number of years (experience) in the computational science field.

During completion of the questionnaires, each user was interviewed to understand more about the physical and conceptual design of both the original and first interactive prototype design of the callgraph visualisation system. Users performed tasks on both the original and first high- fidelity prototype design of the callgraph system for comparisons. Afterwards, interviews were conducted with the users to record feedback about the systems. During the interview, the users were asked the following questions (Table 4.4):

57

How helpful is the visualisation designs in terms of optimising programs? Which design(s) is more helpful?

Do you like the physical layout of the screens?

What will you like to see in the new callgraph design? How easy or difficult was it to learn the system? Which visual queries were easier to do than others?

Which questions were hard or easy to answer with the design?

What are the most important queries that users perform on callgraphs?

Table 4.4: Users questionnaires about the original and first high fidelity prototype

design

We conducted the interview by asking questions to the users. For interview purpose, all the questions were given to each user before the discussion can start. The user’s answer to each question was written on the question paper and responses were also recorded for future analysis of the interview. Every user was interviewed on a one-to-one basis immediately after performing visual queries on the callgraph visualisations.

4.2.3.1 Evaluation

The method of observing users when performing tasks was precisely applied to evaluate both the original and the first interactive prototype system. The controlled settings evaluation was practised to evaluate the first high-fidelity (interactive) design and original callgraphs in a laboratory where users were tasked to perform visual queries on the systems. Users were recorded using video footage when performing tasks on both the original and first high-fidelity prototype design. Consequently, audio was recorded, which was analysed after the evaluation in order to understand the interaction between the users and callgraph visualisations. Furthermore, we took note of the users’ reactions when using the original and the first high- fidelity prototype system. As discussed in Section 4.2.3, users were interviewed to get feedback about the usability of the original and the first high-fidelity prototype. The following data - visual queries, interviews, notes, video and audio recordings were assessed using quantitative and qualitative analysis techniques to ensure that the system met users’ requirements. All the

58

interviews, notes, videos, audio recordings and visual queries performed by the users on the first and second high fidelity prototypes can be found our website 9 by clicking on the “Research” tab followed by “Visualization of parallel programs”. The results were also interpreted using the method of converting data into percentages and averages, as discussed in Chapter 6. Graphical presentation of the data collected was also analysed and interpreted accordingly, as discussed in Chapter 6. Nonetheless, analysis, interpretation and presentation of the observation/interviews data helped to determine the usability of the first high-fidelity prototype design. The process of re-designing and evaluating the prototypes was continuously repeated when the system did not meet users’ requirements and needs, until the final visualisation managed to fulfil both.