Summary of the evaluation

This chapter has presented an evaluation of the developed Conductor Interaction Method and Presentation Conductor implementation. The evaluation aimed to assess the Conductor Interaction Method through the Presentation Conductor, its usefulness and its application.

A formative and qualitative evaluation approach was used, due to the prototype nature o f the developed tool, and consisted o f two stages. To provide a quantitative view o f the model and tool the NASA Task Load Index was also used. This allowed workload comparisons to be made across the different user groups and experiments.

In section 6.2 a number o f issues were identified that the evaluation would seek to address. A summary o f these issues and the relevant feedback that was obtained is shown in table 6.3.

Issues Feedback

If users find it easier to interact with computers using the Conductor Interaction Method, than with existing interaction techniques

Users found the Conductor Interaction Method to be complex, especially when first using it. Despite this, the two-phased interaction used by the conductor method was received particularly well. The experienced users were less favourable about the method, (cf. section 6.4.2.1)

If the metaphors that comprise the method are useful and beneficial

Both metaphors were found useful by the majority o f users, although some experienced users misinterpreted the Orchestra metaphor. Users found the Conductor metaphor to be very engaging and helped in their understanding o f the Conductor Interaction Method. Both, however, could still be further refined (cf. section 6.4.2.2)

If the learning overhead for the method is lower than with existing interaction techniques

Experienced users found the learning overhead for the Conductor Interaction Method to be greater when compared to existing interaction techniques. Inexperienced users found the learning overhead to be no different from any other interaction technique they had used. (cf. section 6.4.2.3)

If users would prefer to use such an approach over existing ones

Experienced users were split on whether they preferred the Conductor Interaction Method to existing techniques, while the inexperienced users in general preferred it. (cf. section 6.4.2.1)

If the technology used can influence the effectiveness o f the Conductor Interaction Method, in particular gaze versus head pointing, and small screen versus large screen

It was observed that the effectiveness o f the Conductor Interaction Method could be significantly influenced by the technology used. This was particularly the case with the eye tracker, which repeatedly required calibration. The large screen set­ up was generally more effective, with almost no problems (cf. sections 6.5.2.1 and 6.5.2.2)

If the Conductor Interaction Method is appropriate for this type o f application, and whether users would prefer to use such an interaction approach for such an application

Users found the Conductor Interaction Method was appropriate for this type o f application. Experienced users were split on whether they would prefer to use such an interaction method, while the inexperienced users preferred it. (cf. section 6.4.2.1)

T able 6. 3 - Summary o f evaluation issues and feedback

Feedback on the method was mixed. As expected, it was less well received by the experienced users, most likely due to their familiarity with, and bias towards, alternative interaction methods (i.e. WIMP). However, the inexperienced users found it an enjoyable experience and were open to the idea o f using it rather than other interacting techniques. This was more positive as one of the initial objectives for the method was that it should be most beneficial to users with little or no computing experience. Although the feedback was not overwhelmingly

p o sitiv e , it is p ro m isin g th a t an a lte rn a tiv e in te ra c tio n m e th o d , w h ile a t th e p ro to ty p e sta g e , w as g e n e ra lly w ell re c e iv e d .

A sse ssm e n t o f th e d iffe re n t te c h n o lo g ie s in d ic a te d th a t th e e ffe c tiv e n e s s o f th e m e th o d c o u ld be s ig n ific a n tly a ffe c te d by te c h n o lo g y th a t is u sed . In p a rtic u la r, te c h n o lo g y th a t is re s tric tiv e o r p ro n e to te c h n ic a l p ro b le m s (fo r e x a m p le , re c a lib ra tio n ) can be a m a jo r im p e d im e n t to sm ooth in te ra c tio n . It w o u ld be w o rth w h ile to e v a lu a te o th e r a v a ila b le te c h n o lo g ie s w ith th e C o n d u c to r In te ra c tio n M e th o d to fu rth e r a s s e s s th e im p a c t te c h n o lo g ie s c a n h a v e o n th e m ethod.

T h e ta sk w o rk lo a d a s s e s s m e n t th a t w a s c a rrie d o u t a lso illu stra te d th e e ffe c t th e te c h n o lo g y could h av e on th e m e th o d , as w ell as h ig h lig h tin g th e p e rc e iv e d d iffe re n c e s b e tw e e n e x p e rie n c e d a n d in e x p e rie n c e d g ro u p s. F ig u re 6.6 s u m m a rise s th e ta sk w o rk lo a d fo r all th e e x p e rim e n ts. T h e g ra p h a lso h e lp s to illu stra te h o w th e ra tin g s p ro v id e d by th e in e x p e rie n c e d g ro u p fo r th e P o w e rP o in t an d larg e sc re e n e x p e rim e n ts w e re on a s im ila r sc ale .

OVERALL TASKLOAD INDEX

18 16 14

12

10

Perform ance E ffort

Mental Demand Physical Dem and Temporal Demand

Frustration EYE TRACKER-CNTR EYE TRACKER-EXP

PPT-EXP PPT-INXP

LARGE SCREEN-EXP LARGE SCREEN-CNTR

EYE TRACKER-INXP LARGE SCREEN-INXP

Figure 6.6 - T a sk lo a d fo r all e x p e rim e n ts by all g ro u p s

T h e P re se n ta tio n C o n d u c to r its e lf w a s g e n e ra lly w ell re c e iv e d a n d d e e m e d as b e in g an a p p ro p ria te a p p lic a tio n fo r th e m e th o d (a g a in , th e e x p e rie n c e d u se rs w e re le ss d is p o s e d to w a rd s it). O th e r p o s s ib le a p p lic a tio n d o m a in s su g g e ste d by th e u se rs in c lu d e d b ro w s in g , m u sic m ix in g , g a m in g a n d d e s ig n in g . In te re s tin g ly , th e s e d o m a in s sh a re th e fa c t th a t in a n o n ­ c o m p u tin g e n v iro n m e n t th e y w o u ld all ty p ic a lly in v o lv e th e u se o f h a n d -b a se d m a n ip u la tio n s . T h is h ig h lig h ts th e fac t th a t th e C o n d u c to r In te ra c tio n M e th o d is p r o b a b ly m o st s u ita b le fo r

applications where a hand-based interaction approach would be central, and possibly where an equivalent or similar interaction is already carried out in a non-computing environment.

One issue that was raised was the amount o f training required in order to use the application (and method). Better training support was suggested as a possible solution. It was observed, however, that once the users had begun using the application, the interaction would become easier and the users would more readily recall the appropriate gestures based on the context.

It would have also been interesting to see what effect the order o f the tasks within the evaluation, could have on the results. This would illustrate whether the user’s experience and preference o f technology is influenced by the order that the experiments are conducted and whether they can adapt to using the eye tracker after using the optical tracking tower. Unfortunately, as it was difficult to recruit larger numbers o f inexperienced users, this could not be investigated. Three additional groups o f each type, would be needed to perform to the evaluation in reverse order, i.e. large screen experiment first and eye tracker last. Future evaluations could place more o f an emphasis on investigating this, however.

Overall, the evaluation studies indicated that in general the Conductor Interaction Method can be consider as a viable alternative for certain application domains, though there are still areas that need to be further refined (for example the metaphors, technologies used and training). These areas are further expanded upon in the next chapter.