Objective Results Processing

The processing of the objective results was relatively straightforward for this experiment and did not follow any guidelines or theoretical models in particular, but rather would apply a descriptive statistical analysis on the data collected by the application during the experiment’s runtime. The time taken to complete the task, the total number of clicks performed by the participant, the total number of mistakes made by the participant and the Field of View (FOV) differential are the metrics which are measured and compared during the objective results processing phase.

A number of formulae are required to calculate some of these metrics. For example, to calculate the total number of mistakes made by the participant during any task, there are multiple factors which the system considers. Firstly, definitions for what counts as an error need to be established. An error is recorded on every occasion that the participant performs an action which does one of two things:

 Any action which does not progress the participant towards the completion of the experiment

 Any action which is performed incorrectly

An example of the first error would be if the participant selects the wrong tool. If, for example, the participant was asked to perform Task A, which involves colouring each square individually and they select any tool other than the ‘Fill’ tool, this would be perceived as an action which does not further their progress towards the completion of the task, so an error is recorded. Likewise, if the user repeatedly clicked to select a tool or colour which was already selected, this would be perceived as an error by the system.

The second error type is recorded when the participant attempts the correct function, but does so incorrectly. An example of this would be if they were attempting Task A and they coloured any of the squares the wrong colour (e.g.) all participants are asked to colour the top left square of the grid blue, so an error would be recorded if they filled the top square with any colour other than blue. Similarly with Task B, where participants are instructed to select the correct statement based on the pattern shown to them, if the participant selects an untrue statement an error is recorded.

The second statistic which was necessary to calculate the total number of mistakes was the number of times either the ‘Undo’ or ‘Redo’ tools were utilised by the participant.

The idea behind recording this is that by hitting either of these buttons, the participant is displaying an awareness of the error they had made. For this reason, each time these buttons were selected and an action was undone/redone, a variable which was to be deducted from the total number of errors was incremented.

The last statistic to be calculated in order to output the number of mistakes made was the total number of misclicks. A misclick was recorded on every instance the participant clicked on an area of the screen which has no functionality (i.e.) they clicked an empty space. The number of misclicks is divided by the total number of clicks (as outlined in the previous section). This number is then multiplied by the total

from the number of undo/redos deducted from the number of errors to produce the equations output.

Equation 1: The Mistakes Calculation Formula

Note: The output of this algorithm was supposed to be a floating-point number, but due to a programming error, it was actually output as an integer. Because of this, the results which were output have been rounded to the nearest whole number.

Another algorithm was required to create a measurable output for the FOV differential between the FOV viewed by the participant and the FOV utilised by the components of each interface. In order to calculate the differential, the following four statistics needed to be tracked by the application during the experiments:

 Highest X-Axis value viewed - translates to how far to their right the participant looked during the experiment.

 Lowest X-Axis value viewed - translates to how far to their left the participant looked during the experiment.

 Highest Y-Axis value viewed - translates to how far up the participant looked during the experiment.

 Lowest Y-Axis value viewed - translates to how far down the participant looked during the experiment.

These statistics would return a floating-point value, which could be compared to co-ordinates of the actors at the extremities of each interface. The formula to put a numerical value to both FOV metrics to allow this comparison is outlined in Fig 3.5.

Equation 2: FOV calculation formula

Because the camera is in a fixed position, with only rotations allowed and not transformation, the Z-Axis (depth) values are always determined by which actor in the scene the participant is looking at. For this reason, the Z-Axis values did not need to be included as part of the calculations.

The output for the FOV differential is the percentage difference between the number for the FOV viewed by the participant and the FOV utilised by the interface. It is calculated by dividing the difference between the two values stored for the FOVs by the value of the Interface FOV. For example, if the FOV utilised by the interface was set at a value of 100, if the user viewed a FOV of either 95 or 105 the percentage differential would be 5%.

As for the other two metrics - time and clicks - neither of these required any algorithms or formulae to have their values calculated. A timer begins as soon as the party observing the experiments and is automatically stopped when the application detects that its “win condition” has been met. Regarding the clicks metric, a variable with a default value of 0 is instantiated whenever a task is started by the observing party and its value is incremented whenever the application receives a certain input.

To prepare the objective data for the triangulation process, an overall score had to be generated. This was done so using the formula devised below, in Figures 3.6 and 3.7

Equation 3: Formulae for calculating the objective variables Time, Clicks and Mistakes

Equation 4: Overall objective score calculation formula

In the above formula, T represents the time taken by the individual participant for that particular task performance, whereas t represents the average time taken to complete than task on that interface. Likewise, C represents the number of clicks performed during that task attempt with c representing the average clicks required to complete the task on that interface. M represents the individual’s mistakes with m representing the average number of mistakes.

For each of the variables, the actual times, clicks are mistakes of each participant are divided by the average of the relevant metric. This number is then subtracted from 100.

This creates a metric on a 100-point scale (although it is possible for a task performance to receive a negative score). The average of each of the three variables is calculated to provide the objective score.