Evaluation Techniques

Several techniques were employed when preparing for this period of evaluation. These included: i) the use of a questionnaire; ii) field observation; and iii) logging user interactions.

i) Questionnaire Design

In designing the questionnaire, three separate categories were chosen to be included: 1) A Familiarisation or “Training” Section; 2) Literary/Scholarly Tasks, and 3) Interface Issues. The familiarisation section allowed users to become acquainted with the application and “trained” them to be able to complete specific tasks that were referred to again in more detail further into the questionnaire. In using the questionnaire format, data could be gathered about specific elements of the application as well as user views on the application as a whole.

ii) Field Observation

By observing users in the environment in which they would usually use an application such as this, any outcome is generally more authentic than if the subject had been analysed interacting with the application in a foreign environment. It is important to make use of field observation early in the design stage of the application, with the aim of increasing understanding about what users do naturally164 and how they interact with objects, people and situations in the field.

164

iii) Logging User Interaction

By using a simple piece of JavaScript code, it has been possible to log user interactions with the application, by having a timer and recording the number of mouse clicks by the user in a specified time frame (for example every ten minutes). This demonstrated approximately how much time users were spending on specific tasks or questions. Although this was a basic level of experimentation with logging, it has encouraged ideas about where logging user interactions could prove useful in future evaluative sessions. Preece et al165 suggests that an advantage of logging user activity is that it is unobtrusive, but that also by informing users that their interactions are being logged might in turn influence their behaviour, and therefore any outcomes of the logging.166

|££] 1 Unmodernised No. of clicks: 45 log= 6,9,9,37,40 Fig. 17 An example of data logging information on the status bar

The illustration in Fig. 17 shows where the data appears on the status bar. Firstly there is the version of the poem that the user has most recently positioned the mouse over, then there is the total number of mouse clicks the user has made, followed by the number of mouse clicks made in each timed period of ten minutes (up to a total of 50 minutes).

Fig. 17 shows that in this case the mouse was clicked 6 times during the first ten minutes, then 3 more times in the second ten minutes, then not at all in the third ten minutes, and so on. These example results indicate that the user did not interact with the application very much during the “Familiarisation” section of the questionnaire, which incorporated reading time, but interacted most during the “Literary Questions/ Tasks”, and “Application Questions/ Tasks” sections, which required the user to compose onscreen.

Fig. 18 visually presents the findings of the first group of students (Group A) in terms of their measurable levels of interactivity with the screen. Initially there is a clear increase in interactivity as users begin to familiarize themselves with what is onscreen. At approximately 15 minutes, users again increase their level of interactivity, and at 25 minutes, the majority of users reduce their number of mouse clicks again. Only one student continued beyond the time the others have ended their onscreen session.

Although the logging process primarily provides quantitative results, when used in conjunction with observed studies of users, it has been possible to interpret statistics qualitatively.

4 5 0 400 350 300 w = 250_o ° 200 6 z 150 100 50 0

Student 1a — Student 2a ——- Student 3a —— Student 4a ---Student 5a —— Student 6a —— Student 7a —— Student 8a Fig. 18 Mouse click interactions recorded with Group A

Table 1 (below) gives a numerical indication of the levels of interactivity that are possible if all variants are revealed. In order to observe how much interaction was achieved by the user, I calculated the maximum number of clickable choices (variant words) from each edition, and the maximum number of mouse clicks it would take for the user to cycle through every variant word in each stanza of the poem.

/A

/

\ /V

a

\ \

'

—\Vy \

7 J V

\

\\\ \

"

\ \ X: \\

"

Var 1 Var 2 ro N U1 £ u J3 .E 1 0 — I ro fO N U) E a) CO r - 4-1 t/3 — I r\i 03 N U) E cu iS .E C/3 —I

Table 1 Total textual variants and possible total number of mouse clicks

Table 1 shows for example that the user can click eight times in total on the first choicepoint in line 1 stanza 1 of the poem to cycle through variants from each of the editions. An additional mouse click returns the user to the initial status of the question mark. The numbers Var 1-6 across the top of the table indicate that there are between one and six choicepoints on each line of the poem, some of the cells in the table do not have entries as not all lines have as many as six choicepoints.

From this table, the total number of possible variants can be calculated. There are 80 choicepoints, and 693 clickable variants, this is the basic figure students would achieve if they were to cycle through every single variant on

the page, but does not allow for repeated clicks or for the additional click taken to return to the initial question mark state.