The strengths of hand-based gesturing

Research evidence has suggested that gesturing with unmediated representations of hands during collaborative physical tasks, may lead to better performance than when gesturing is achieved through other means. Some of the problems associated with the use of sketched objects have been discussed above, as a partial explanation of this effect, but clearly for a full understanding the comparative strengths of the hand gesturing approach should be considered. The strengths of unmediated views of hands start with the fact that there are two hands available (usually) for gesturing). Whilst it is acknowledged that the majority of gesturing only really requires one point source, either an indexical finger or a pen tip or a cursor, for the performing of pointing gestures, at the point at which a complex gesture demonstrating relative

orientations is required, the ability to use two hands is a significant advantage. People are used to manipulating an object with one hand and pointing at it or gesturing toward it with another. However, pen use reduces the number of gestural information points available down to one. Coupled with this is the fact that hands are in themselves highly complex features. They have multiple degrees of freedom of movement, with individual digits available, each offering separate gestural ability, multiple fingers can be used to point at multiple objects or just as easily be re-shaped and combined to model the interrelationships of a complex assembly piece. As discussed previously, hand gestures are less likely to be interpreted in an overly literal sense (compared to sketches) as such latitude is given to their interpretation. Although this potentially poses a comparative problem in that even the best hand gestures may well at times be insufficient to model a particular complex shape which may well be much more easily interpreted from a quick sketch. Hands do however have the advantage of being able to be re- used quickly and formed into multiple representations at varying levels of abstraction with relative ease. It is easy for a hand to represent a piece of Lego in one second and then be switched to represent the Worker‟s hand in the next. Sketches don‟t have this facility, and have to be re-drawn, demonstrating a reduced fluidity of multiple gesturing and sometimes leaving a temporal residue of un-required old sketches littering the work-space.

Consideration of the fluid nature of gesturing leads also to the understanding that hand-based gestures are also performed in an animated fashion. They effectively animate instructions. A sketch must convey the relative spatial orientations of pieces by showing the resultant end images of how combined pieces should look, unless an extremely complex gesture is sketched with multiple stages of interaction and functional descriptions of movement added with the use of arrows. Such an endeavour is a complex process, hands being much quicker as a way to show a real-time animation of an interaction in progress.

It was believed initially that the ability to refer to established sketched objects, which had been developed in the cultural-historical context of the collaborative task, would be a particularly powerful tool, but this was not the case. It may be that having such objects added to the space which can only be manipulated by the Helper and are therefore asymmetric elements of a shared space, creates fractures within the interaction. Hand gestures alternately, whilst not having the ability to develop highly complex objects, may satisfice the key requirements of the communication element of the task in that they can model complex spatial relationships enough to aid understanding of description.

The construction of a mixed ecology tries to develop a working environment in which collaborators share equally, and which is as close to the presumed optimal standard of face-to- face communication as possible. In a face-to-face situation a Helper would be unlikely to draw a complicated diagram and continually refer and gesture toward that rather than the pieces to be interacted with. When available however, the desire to sketch rather than explain can be quite powerful, but sketches can be easily misinterpreted. By using unmediated representations

of hands (as suggested by designing from a mixed ecologies perspective) collaborators are forced to focus their gestures more on the model as it is assembled, they do not have the ability to create a separate plane of information that the Worker must shift attention towards and collaborators are more likely therefore to stay task-object focussed. Gesturing with hands will also allow Helpers to perform complex embodied gestures conveying detailed implicit information regarding movements which are as discussed difficult to represent in a static sketch. Where results have demonstrated the superior ability of hand gestures to convey information over sketches in collaborative physical tasks, it is potentially due to these considerations.

6.7 Chapter Summary

Previous chapters have stated that gesturing with hands improved performance and that using hands in particular was best, but the analysis contained therein didn‟t explain why. To increase knowledge of how to develop remote gesture technologies appropriately, this „why‟ must be fully understood. Knowledge of why remote gesture improves interaction helps to distil what the key aspects of interaction are, and understanding these key aspects suggests which features future technologies should be focused on supporting.

The work of this chapter therefore detailed the common practices and cycle of activities of remote gestural use during collaborative physical tasks. Within an identified common structure, or cycle of activities, including search, select and direction of manipulation gestures, observation was made of what gestures were used in each different gesture format and importantly how they were used and the impacts that this had on collaborative performance. Through the development of an understanding of the praxiological character of remote gesture, insight has been derived which has articulated the way in which representations of gesture must be commonly understood between collaborators, they must be fast and fluid and ideally leave only as much temporal residue as is absolutely necessary (to keep the workspace clear). Gestures also directly benefit from occurring in three-dimensions and in being presented in a format such that the appropriate temporal course of an action can be determined. The analysis also suggested the benefits of gestures being presented in a format which avoids over literal interpretation, to allow for inconsistencies in understanding and to implicitly prompt the gesturer to keep trying to relay their intent until they are satisfied that the recipient has understood. This final observation in particular, has helped to further understanding of how interaction might become fractured through processes of misdirection. Put together these observations further suggest why the unmediated representation of hands as a remote gesture format should be considered an integral aspect of a mixed ecology communication device.