Cooperative Object Manipulation

Researchers have also explored the combination of input from multiple users in vir- tual reality scenarios – often following the example of joint object handling in the real world. Similarly, users can pick large virtual objects at multiple points to sim- plify their handling while maintaining the accuracy of the larger scale (e.g. [90, 101].

In the real world, however, the object provides a physical link between all involved hands which helps to coordinate their individual motion through haptic guidance. The physical link also constrains the motions of each hand. During the manipulation of virtual objects such a physical link is missing. Interaction techniques need to map the unconstrained movement of the involved hands to the available degrees of free- dom of the object. There is no simple solution to this problem that is intuitive to the users and also results in a controlled object motion. A pseudo-physical or physical simulation of the object manipulation can improve the sense of control by providing feedback for the deviation of the hands from their initial grip positions with rubber- band visualizations [5, 100] or bending input widgets [282]. However, Salzmann et al. [297] and Aguerreche et al. [6] found that a shared tangible input device for mul- tiple users can be preferable.

Hornecker and Buur suggested further benefits of tangible interaction for collabora- tion [148]. Besides the already mentioned benefits of physically perceived coupling through tangible manipulation, their conceptual framework includes the themes of spa- tial interaction, embodied facilitation and expressive representations. Spatial manipulation emphasizes the relevance of a shared interaction space in which the movement of objects and one’s body has a comprehensible meaning. Support for full body in- teraction encourages performative action and may thereby leverage body language. User interfaces should ensure that everybody can continuously follow the interac- tion process and avoid fragmented visibility as much as possible. Embodied facilitation highlights that tangible interaction devices embody physical constraints and provide multiple access points, both of which can balance the involvement of participants. The physical characteristics of environments and objects afford particular usage patterns. Moreover, tailored representations of interaction devices can better fit the users’ skills and experiences. The latter is closely related to the concept of expressive representa- tion which involves meaningful and long lasting representations of functionality and content as well as comprehensible interrelations between physical and digital arti- facts (representational significance and perceived coupling). The authors further argue that tangible objects can facilitate the immediate externalization of ideas and pro- vide unambiguous references for the communication among group members. Three case studies confirmed the benefits of the specified features of tangible interaction for multi-user cooperation.

The cooperative manipulation of objects generally involves two or more contact points, which implies the simultaneous manipulation of position, orientation and, if permitted, also scale. Such highly integrated control may facilitate coarse approx- imation, but it complicates accurate adjustments of individual degrees of freedom. The required coordination for co-operating users acting on the same degrees of free- dom of a single object is challenging. Clever input integration schemes could allevi- ate this issue. Ruddle et al. explored the effect of symmetric and asymmetric input integration on the performance in a collaborative 3D manipulation task [293]. In the symmetric case, only motion input that is induced by both users will be applied. Asymmetric input integration, instead, applies the mean of both actions. They ob-

Multi-User Cooperation 75 served that symmetric input integration is preferable if both users aim for the same motion trajectory, but that the asymmetric conditions led to better results during peri- ods when the task required both participants to move in different directions. In both conditions, however, they observed a significant cooperation overhead.

In the context of a similar 3D manipulation task, Pinho et al. suggested to distribute the control of multiple degrees of freedom (DOF) among participants [269]. More specifically, they assigned the control of position and orientation to different users and also experimented with a separation of motion in depth from motion along the image plane. This approach of distributing DOF among users had earlier been pro- posed for cooperative object manipulation in 2D user interfaces [46]. In both cases, the distribution of DOF enforced collaboration. Pinho et al. reported that the separa- tion of input parameters can increase manipulation accuracy, and that the cooperative operation allows users to control more degrees of freedom simultaneously. They also observed, though, that the distribution of DOF negatively affected the comprehensi- bility of the interface.

Benford et al. proposed to encourage collaboration with a variety of simultaneously available tools that can be used in combination to realize additional functionali- ties [27]. Instead of enforcing collaboration, this approach is promising increased efficiency, fluency or fun by means of collaboration. They developed collaborative storytelling applications for children that were offering a variety of digital tools for simultaneous operation by multiple mice. With initial implementations, a tendency towards individual action was observed among their test users as well as competition instead of collaboration. As a result of this observation, the researchers implemented extended functionalities that could only be achieved collaboratively, e.g. mixing col- ors from two differently colored drawing tools, to encourage more cooperation. Chil- dren that were testing the revised applications appreciated the collaborative tool be- havior. It still seemed difficult, though, to discover the combined features and use them for a common goal. Benford et al. concluded that opportunities for cooperation should be clearly visible and indicate concrete benefits.

Morris et al. later explored cooperative touch gestures using a multitouch tabletop display with user identification [234]. Their collaborative drawing application pro- posed cooperative actions to realize 1. implicit agreement on global state changes, 2. simplification of effortful actions like reaching over the table, and 3. playful manip- ulation of larger parameter sets through simultaneous actions. Test users endorsed cooperative gestures to express their agreement to global changes but expressed little understanding for required cooperation if the same result could be achieved alone with only slightly more effort. As an example for the simultaneous manipulation of parameter sets, the test application included an input gesture to manipulate the in- tensity and thickness of a stroke while it is drawn by another user. The users disliked this distribution of control over the appearance of the stroke. They complained about confusion, inefficiency, and tedium due to the artificial separation. Moreover, Mor-

ris et al. reported that several test users felt uneasy about cooperative gestures that required intimate proxemic distances in order to express agreement.

Collaborating people frequently switch between tightly and loosely coupled cooper- ation [20, 161]. These dynamics can be reflected with a flexible workspace structure that fluently adapts to varying needs [154, 333]. Scott et al. observed the pattern of territoriality in collocated collaboration. Territoriality is a behavioral patterns of hu- mans and animals to establish affiliations with particular areas or spaces through reg- ular usage or occupation. In the tabletop setting Scott et al. observed the emergence of distinct interaction areas for personal interaction and group exchange as well as the establishment of further subspace for storage [311] (Figure 5.5). Supporting this user behavior can reduce input interferences and increase the effectiveness of collab- orative work [156, 189, 336, 340]. Multiple separate display and interaction devices like smartphones and tablets inherently serve as personal and storage territories, but effective means for exchange with a shared group territory are required to support collaboration (e.g. [227, 243, 365]).

Figure 5.5: Emergent territoriality in a tabletop collaboration setting. Partici- pants in collaborative interaction implicitly establish separate areas for private activities (green) and group exchange (blue). Also the emergence of storage ar- eas (yellow) can be observed. The latter are often subspaces of private or group territories.

5.4

Conclusion

This chapter reviewed research and interface developments in the fields of biman- ual interaction, multimodality, and multi-user collaboration. The review highlighted the potential of cooperation as a general principle towards more expressive human- computer interaction. Combining activities of two hands, multiple modalities, or several users promises considerable advantages.

Conclusion 77 Among existing interaction systems and prototypes, bimanual interfaces revealed the most obvious performance benefits. Unfortunately, only few available sensing tech- nologies can distinguish input from different hands. Technologies to distinguishing dominant from non-dominant input roles in asymmetric bimanual division of labor seem to be required for further progress in that direction.

The benefits of multimodal interaction appear to be obvious at first sight. So far, however, prototypical systems do not live up to the expectable leap in performance. Cognitive overheads of suggested multimodal input combinations may constitute a bottleneck. Moreover, we argued that the design of multimodal interfaces should put a stronger emphasis on their applicability in social contexts.

With respect to multi-user cooperation, the literature review focused on real-time in- teraction in collocated settings and telepresence applications, where the coordination overhead often seems to undermine many potential advantages of cooperative inter- face prototypes. The success of social web applications, on the other hand, clearly shows many advantages of cooperation with others. The latter enable social infor- mation exchange based primarily on asynchronous communication with remote cor- respondents – partially at the expense of awareness for local situations and collocated peers. It can be argued that the next step in interfaces for social cooperation requires the seamless integration of local and remote peers in joint action.

The motivations, the application setups, and the involved processes differ between the three reviewed fields of research. However, several interaction patterns and re- quirements seem to be very similar. The following chapter discusses these similarities and suggests common cooperation patterns to derive essential cooperation require- ments and high-level design principles for cooperative user interfaces.

79

Chapter 6

Cooperation Patterns and

Requirements

In the last chapter we reviewed a variety of user interfaces that promise expressive user input through the integration of multiple simultaneous input streams. From the comparison of these different approaches we can derive common situations and requirements. As a basic structure for this analysis, we adopt Martin’s typology of multimodal interaction patterns [61]. In the following sections, we discuss its ex- tended applicability to cooperative user interfaces in general and the resulting inter- face requirements in terms of workspace characteristics and the cooperative coupling of participants. Moreover, we suggest high-level design principles that promote ex- pressive user input across various platforms and settings, including the implications of social settings and multi-user interaction.

6.1

Cooperation Patterns

Martin proposed a typology of cooperation patterns for multimodal interfaces [61]. The types of cooperation he identified, namely specialization, equivalence, redundancy, complementarity, and transfer, can also be observed in bimanual interaction and the collaboration of multiple users. We therefore adopt this classification and extend it where necessary. From such a broadened perspective we can identify relations between these concepts and other cooperation patterns.