Joint Action

The research reviewed so far has concerned almost solely spontaneous coordination in simple tasks like finger tapping and target acquisition with few degrees of freedom. The temporal synchronization of involved actors could be explained by the earlier mentioned model of coupled oscillators – a process known as entrainment [170, 303]. These couplings may be largely dependent on interlimb synergies that result from attention rhythms, e.g., eye movements while tracking a moving object [194, 303]. The question remains, how entrainment affects more complex cooperative action. Knoblich et al. recently published a comprehensive review of research results on in- terpersonal coordination and joint action [180]. They distinguished between emer- gent and planned coordination. Low-level perception-action couplings like entrain- ment contribute to emergent coordination. Planned coordination, instead, specifies higher-level interaction goals through common task representations and perspective taking. Complex cooperative performances clearly involve coordination on both lev- els. Knoblich et al. argued that planned coordination builds on the fast perception- action couplings, which in turn are affected by one’s goals and interests through con- scious attention.

Besides entrainment, Knoblich et al. suggested three further sources of emergent co- ordination: joint affordance, perception-action matching, and action simulation [180]. Joint affordance recognizes that a group of people has extended capabilities and dif- ferent action requirements, which affects the affordances of objects, environments, and situations. An open door to a building, for example, invites a single person to enter directly, while it does not afford for a whole group to pass simultaneously (Fig- ure 4.6). Perception-action matching and action simulation are closely related. The former describes the activation of action representations corresponding to perceived ones. The represented actions must not necessarily be performed, but apparently, one is generally better prepared to perform the same actions as those observed in the behavior of others. The behavior is thus also termed action imitation. The inter- nal representation of perceived actions provides the basis for mutual action simula- tion. Predictions of each other’s actions and their effects can be derived that facilitate appropriate reactions, mutual error compensation and the planning of higher level sequences.

The effects of perception-action matching are very similar to those described by stimulus-response compatibility. Studies have demonstrated that people could start a particular movement more immediately if they have just been observing a similar one (e.g. [45, 330]). The conceptual difference between both theories is the dimension

Figure 4.6: For individuals, an open door affords going through. Its joint affor-

dance is rather to negotiate an order of passing.

of similarity. Stimulus-response compatibility considers geometrical relations, while perception-action matching relates to ideomotor compatibility, i.e., the similarity of movements. Bach and Tipper showed that watching someone kicking a ball led to faster responses with the foot, while the observation of a typing person prepared for rapid responses with finger motion [17]. Stimulus-response compatibility could not explain this effect, while perception-action matching predicts the general activation of limbs involved in an observed action. According to stimulus-response compati- bility it should make no difference, whether we are stimulated by a moving object or a hand following the same trajectory. Brass et al., however, showed that videos of finger movements affected the corresponding responses more strongly than ab- stract representations [45]. Apparently, both effects can reinforce each other. They also found that, in case of conflict, ideomotor compatibility seems to be the stronger cue.

According to Knoblich et al., planning is required to prepare for joint action and accommodate to changing situations as well as to exchange with collaborators and distribute subtasks. It involves the development of shared task representations, an understanding of each other’s capabilities and considerations of potential differences between the collaborators’ goals and perspectives. The low-level behavioral patterns described as emergent coordination, on the other hand, enable the accomplishment

Conclusion 59 of joint action plans in real time. These seem to be rooted in perception-action cou- plings, and therefore depend on one’s attention, which follows higher level goals.

4.5

Conclusion

Cooperative action occurs on various levels. This chapter has reviewed psycho- logical research on the synergies of connected body joints, bilateral coordination, and joint action of multiple people. Very similar behavioral patterns were found in these apparently different cases. Considering the generality of the coupled-oscillators model [170], it seems reasonable to expect similar results from observations at further scales, e.g. individual muscles, cells, multiple groups, and crowds. In fact, entrain- ment effects have also been shown for swarms of insects (e.g. [127]). For the purpose of the present work, we are most interested in the coordination of multiple limbs and persons. The requirements of the involved processes may inform the design of versatile and expressive user interfaces.

Cooperation involves multiple interdependent actors with variable contributions. The more actors that are involved, the more degrees of freedom that must be co- ordinated for a joint performance. The movement flexibility of the human body is a prime example of this issue. Bernstein observed that the interconnected limbs, combine to a complex system involving more degrees of freedom than necessary to describe the resulting motion trajectories of end effectors. This movement flexibility seems to challenge motor control, but apparently, synergies between the limbs enable even higher accuracy, i.e., the variance of individual limb motion is larger than that of their combined effect. The involved limbs compensate each other’s deviations. Such synergistic behavior could also be demonstrated for bimanual and interpersonal co- operation.

Temporal synchronization (entrainment) seems to be among the primary organizing principles of cooperative action. Interacting at conflicting rhythms is indeed very dif- ficult. The robust coupling behavior corresponds to other dynamic systems and can be predicted by models of coupled oscillators. The internal mechanisms of this in- teraction are not entirely understood. The coupling can potentially occur at the level of motor control in the nervous system, e.g. through synchronized afferent signals. However, entrainment and other effects of dynamic coupling have also been demon- strated between different people that coordinated their actions on the basis of mutual visual perception, hence entrainment also occurs as an effect of perception-action coupling.

The coordination of more complex cooperative actions involves planning and the distribution of subtasks. Consequently, joint action is considered a combination of planned and emergent coordination processes. Conscious action planning sets the goals, negotiates subtask distribution, and guides our attention, while the ac-

tual cooperative performance is implicitly controlled through perception-action cou- plings. Similar hierarchical control models have been suggested for bilateral coordi- nation [210]. Complex actions seem to be further facilitated by an asymmetric divi- sion of labor. In the reviewed examples of cooperative action, however, the respective responsibilities are rarely predefined or fully separated. Synergistic error compensa- tion may only occur, if the involved parties can affect the same parameters. The different roles emerge and alternate as required in dynamically changing situations. The similarities of bilateral and interpersonal coordination indicate that both cases do not need to be considered independently in the design of user interfaces for coopera- tion. Instead, a workplace that offers a variety of access points for the cooperation of multiple users, may also facilitate bimanual interaction. From the observations dis- cussed above we can derive several requirements for such workplaces. Entrainment, a basic building block of coordination, requires the correct spatiotemporal perception of involved actors. Mutual error compensation, an essential characteristic of syner- gies, requires that the involved actors may somehow affect the same parameter space. Action imitation and prediction, two further sources of emergent coordination, de- pend on mutual perception and knowledge on the action capabilities of co-actors. This is also true for joint affordance and joint perception, two sources of emergent and planned coordination, which additionally require coherence of the shared inter- action space. Last but not least, subtask distribution or division of labor requires that cooperators can assume different, and potentially independent, roles.

61

Chapter 5

Related Work on Cooperative

User Interfaces

Several established user interfaces and many research prototypes support coopera- tive input from multiple hands, through different modalities, and also by multiple users. In the following we review a broad range of examples from commodity prod- ucts to research prototypes. The described interfaces are very diverse and may appear unrelated, but, all of them promise increased expressiveness through the involvement of multiple simultaneous activities.

5.1

Bimanual User Interfaces

Bimanual input is an established pattern of current user interfaces. The combina- tion of manipulation input with the mouse or a pen and symbolic mode switching through keystrokes can be considered an example of asymmetric cooperation1. Hold- ing a tablet computer in one hand while operating the touch interface with the other involves an asymmetric division of labor similar to handwriting in a paper notebook. The popular multitouch interface for integral manipulation of position, orientation and scale involves simultaneous symmetric actions of two fingers. The most impres- sive example of skilled bimanual input is probably touch typing, where 2-10 fingers perform symmetric tapping tasks in phasic alternation.