Study methodology

5.2.2.1 Experimental design

The study was constructed with a within-subjects repeated-measures design. Each pair of participants completed three Lego model assembly tasks, in each of three different gesture orientation conditions (the independent variables – see figure 5.1). These conditions were Lateral (e.g. gestures projected onto Worker‟s task space at 90° to Worker‟s seated position – the Helper‟s hands always entering from the right hand side), Face-to-face (e.g. Helper‟s hands projected onto Worker‟s task space as if participants were sat facing each other on either side of a table) and Overlaid (e.g. as per standard technology set-up, see section 3.5.1, similar to a side-by-side orientation, but more overlapped). Measures were recorded (the dependent variables) of time taken to reach a certain stage of each model, the final stage of completion of each model at the end of 10mins and whether any mistakes were made with the model up to the last completed stage (results were coded as either minor mistake(s) made or major mistake(s) made). A questionnaire was also administered at the end of all three trials to assess participant preferences amongst the three tested orientations. Models assembled were alternated between trials to limit task learning bias and presentation of models and experimental conditions was counterbalanced across pairs.

5.2.2.2 Participants

A total of 36 participants took part in the study, 18 self-selected pairs (15 females and 21 males). Participants‟ ages ranged from 18-34 years (mean 21.2 years, standard deviation 3.74 years), and they were mostly University undergraduates. Participants all had normal or corrected to normal vision and were paid £5 each for taking part in the study.

5.2.2.3 Equipment

The equipment used for the study was as per the technological set-up described in section 3.5.1, with some minor modification. To alter the relative orientation of gestures between trials the video camera above the Helper‟s desk was rotated by 90° or 180° as appropriate, so that the projection of gestures would show the Helper‟s hands entering the Worker‟s task space from the desired alternative angle. To ensure usability of the system and to keep gesturing effort consistent for the Helper between trials the monitor used by the Helper to view the remote task space was appropriately rotated between trials. This ensured that the Helper‟s view of their own hands was always such that their hands were presented going up the screen, effectively maintaining their natural orientation to their own gestural actions.

5.2.2.4 Materials

Three Lego model kits were used for the study (model numbers 8441, 7113 and 1354), accompanied by the three relevant sets of assembly instructions. A bespoke questionnaire was also provided (see appendix 5.1) for completion at the end of the study.

5.2.2.5 Procedure

Participants volunteered in pairs. They were then invited to the lab. Prior to the trials starting they were asked to read an information sheet outlining the structure of the experiment (see appendix 5.2) and were also asked to sign a consent form (see appendix 4.5). Once this was completed they were shown the experimental equipment and were shown how it works utilising gestures in whichever orientation was to be used in their first trial. Participants were invited to decide amongst themselves as to who would be the Helper and who would be the Worker (in the event that a decision could not be made the Experimenter chose randomly). Once this decision was made the participants briefly trialled the equipment, using a three piece simple toy assembly. Once the Experimenter was satisfied that the participants understood the nature of the task to be completed and the way in which the trials would differ, the experiment began. Participants worked through three trials in turn, being allowed 10mins for each trial, and being told to complete as much of the model as they could in that time. Participants were given a short break between each trial as the video camera and VDU were rotated to construct the environment for the next trial. They were not allowed to leave their seats between trials. After the final trial participants were given the bespoke questionnaire to fill out.

5.2.2.6 Problems encountered

The only minor difficulties encountered occurred in a small number of trials where the VDU unit used (a TV) suffered difficulties through operating on its side or when fully inverted. Such motion occasionally altered the colour balance of the screen, making all images (as seen by the Helper) tinged green. The problem was rectified after it occurred in the first three trials by giving the VDU longer to rest in its new orientation between trials. When the problem did occur the experiment was paused briefly, the participants informed of the nature of the problem and were then instructed to continue. In no cases did the participants feel that the VDU difficulties impinged on their ability to perform the task.

5.2.2.7 Statistical Analysis

Results were analysed as appropriate with a one-way repeated measures ANOVA, comparing the three orientation conditions. Responses to the questionnaire were categorised and analysed using Chi-squared tests.

5.2.3 Results

The first stage of analysis was to compare the final stages of completion for each pair of participants‟ three models. After each trial the model they had been assembling was inspected and the final the stage they were currently working on was recorded. Table 5.1 below illustrates the average final stage of assembly for each of the three orientation conditions.

Lateral Face-to-face Overlaid Stage of Assembly 8.39 (2.3) 8.61 (3.2) 9.22 (2.8)

Table 5.1 Average final stage of assembly for each of the three orientation conditions (N=18 pairs) (Standard deviation in number of stages is shown in brackets)

The results have a clear trend suggesting that in the Overlaid orientation more stages of the models were being completed. This finding was statistically analysed using a one-way repeated-measures ANOVA, the finding however, suggested that the difference between the conditions was not statistically significant (F (2, 34) = 0.792, p= n/s), presumably due to the relatively large standard deviation caused by variability in performance.

Having analysed the final stage of completion and being concerned that natural differences in the ability to complete each of the models (due to varying model complexity) might have affected the results a further analysis of performance was conducted. This analysis focussed on the time to complete a specified stage of each model. To counter model complexity differences the time to complete different stages was recorded, varying by model. The correct stage to use for each model‟s measure was calculated by assessing what the average stage of progress was at 5mins for each of the models. This average stage was then taken as the point to which timings should be made. This weighting meant that measures were taken to the completion of stage 4 for 1 model (serial no. 8441) and stage 6 for the remaining models (serial no.‟s 7113 and 1354).

Table 5.2 below presents the average time in seconds for the pairs to complete the required number of stages of each model, split out by gesture orientation condition.

Lateral Face-to-face Overlaid Time to reach required stage 374.94 (149.82) 357.33 (145.72) 345.67 (154.42)

Table 5.2 Time to reach required stage of assembly (in seconds) for each of the three orientation conditions (N=18 pairs) (Standard deviation in number of stages is shown in

brackets)

Again a clear trend in the data was present suggesting that the overlaid orientation was leading to faster performance, with the face-to-face orientation coming second and with lateral

presentation of gestures leading to the slowest performance. The results of this analysis were again analysed using a one-way repeated-measures ANOVA, the finding however, again suggested that the difference between the conditions was not statistically significant (F (2, 34) = 0.435, p= n/s). From an inspection of the raw data it was obvious that a ceiling effect had been present. Table 5.3 below illustrates the number of pairs who managed to complete the required number of stages for the above analysis.

Lateral Face-to-face Overlaid

Number of pairs 14 15 16

Table 5.3 Number of pairs to complete (within 10mins) the required stage for analysis in each of the three orientation conditions (N=18 pairs)

The results shown in figure 5.3 would suggest that pairs completing the lateral orientation assembly were less likely to complete the required number of stages. This meant that for the lateral orientation condition there were 4 pairs whose time was fixed to 600 seconds, when in reality they may have required much longer. All scores that were at the ceiling of 600 seconds were therefore removed from the data. Re-analysis of this new data sample proved inconclusive, and given that the samples for each orientation condition were now of very different sizes, it was felt that the analysis would be unlikely to be able to find any new significant results.

Having observed only trends thus far and no firm statistical support for differences between orientation conditions, attention was turned to the numbers of mistakes made during completion of the assembly. Each model was inspected after assembly was ended at 10mins. The models were classified according to whether they were currently Correct, they suffered from a Minor mistake or a Major mistake, Table 5.4 below, highlights the differences in accuracy for model making as a product of gesture orientation condition.

Lateral Face-to-face Overlaid

Correct 9 9 10

Minor Mistake 4 4 5

Major Mistake 5 5 3

Table 5.4 Number of pairs to assemble their model (up to last completed stage) correctly or with mistakes in each of the three orientation conditions (N=18 pairs)

The results suggested little difference between the orientation conditions. There appeared to be a mild suggestion that in the overlaid condition, models were more likely to be correct, and if mistakes were made they were more likely to be minor mistakes than major mistakes. The high similarity between scores however, suggested quite strongly that statistical differences would not be found between the conditions, so a statistical analysis was not required.

The final stage of analysis for the orientation experiment concerned the responses to the final evaluative questionnaire that participants received after completing their final trial. The questionnaire posed two questions (supported by accompanying diagrams for clarification), firstly, which of the three orientations did the participants prefer using? And secondly, which orientation caused the most confusion?

Total responses to the first question can be seen in Table 5.5 below, with these results split out by Helper and Worker shown in figure 5.2 (also below).

Lateral Face-to-face Overlaid

Number of pairs 6 13 17

Table 5.5 Number of respondents to choose each of the three orientation conditions when asked „Which orientation was easiest to use?‟ (N=36 respondents)

Preferences when asked ‘Which orientation was easiest to use?’ 0 2 4 6 8 10

Lateral Face-to-face Overlaid

Orientation condition N u m b e r c h o o s in g Workers Helpers Figure 5.2

Figure 5.3 clearly demonstrates that the use of a lateral orientation for the presentation of gestures appears to be particularly frustrating for the Helpers (i.e. the producers of the remote gestures), and generally there is some preference for overlaid orientations. The numbers reported in table 5.5 were subjected to statistical analysis with a Chi-squared test, the result ( 2

(2) = 5.167, p=0.076), suggests that the difference is approaching significance, but does not quite meet the criteria for acceptance as statistically significant. The trend however, remains firm, in that a preference is observed for the overlaid orientation.

Responses to the second question „Which orientation did you find most confusing?‟ are presented below in table 5.6 and figure 5.3.

Lateral Face-to-face Overlaid Number of

pairs

13 10 11

Table 5.6 Number of respondents to choose each of the three orientation conditions when asked „Which orientation did you find most confusing?‟ (N=34 respondents – due to 2

Preferences when asked 'Which orientation did you find most confusing?'

0 2 4 6 8 10

Lateral Face-to-face Overlaid

Orientation condition N u m b e r c h o o s in g Workers Helpers Figure 5.3

Comparable with other results the lateral orientation was perceived to be the most confusing by the largest number of respondents. There were very little differences between the ratings of face-to-face and overlaid orientations, with each receiving largely the same number of votes. Chi-squared analysis showed the differences between the three conditions, however, to be non- significant. Analysis of the reasons behind these decisions though are of some interest (full transcripts can be seen in Appendix 5.3). For the Workers, all of those who chose the overlaid orientation as more confusing claimed that this was so, because the representation of hands presented to them occasionally obscured items on their desk. It is interesting however, that this obscuring of task artefacts does not appear to have had any significant impact on performance. One participant in particular claimed that the projected hands had at times forced him to stop what he was doing and pay specific attention to the gestural movements of the Helper. The hands therefore appearing to, in some cases, actually enforce turn-taking and attention apportioning behaviours (by the participants own admission).

For those Helpers who chose the overlaid orientation as the most confusing, all of the reasons given centred on the arguments that the orientation seemed in some way „unnatural‟. None of the Helpers claimed that it had hampered their performance only that it did not seem like a form of interaction which would be physically possible, unless collaborators were sat on one another, and therefore they argued that it must be the most confusing. It could be argued that this is relatively weak criticism, and is a rejection based on the novelty of the technology, rather than any actual discrimination based on performance impact.

5.2.4 Results summary

In summary, the raw data appeared to offer some promising trends. On multiple occasions the data suggested that there was a possible advantage, in terms of speed of performance, progress, and accuracy in using an overlaid gestural orientation, as had been adopted in the mixed ecologies inspired remote gesturing prototype (although this was admittedly trend-line data

and was not statistically reliable). A key factor to focus on is the strength of opinion given by the users who appeared to claim they preferred the overlaid orientation. Of those critiques of the overlaid orientation that were given all appeared to stem from an uneasiness concerning the novelty of the arrangement, rather than an outright dissatisfaction with the usability of the technology. And measures of performance were at least, unaffected by using an overlapped orientation, suggesting that it could have no negative impact on performance. Whilst arguments for the merits of choosing between face-to-face and overlaid orientations are perhaps a little tricky, it is apparent that the data is giving a good clear indication that a lateral orientation is not liked, and is likely to impinge on performance, at least in this form of task.

5.3 Gesture Format and Location