System Features

In the traditional physical workspace, only physical medium is present, so the collaborators’ interactions are restricted by the laws of physics. However, digital technologies have enabled more flexible interaction mechanisms. The users’ interactions with the shared workspace in co-located technologies can be broken down based on how the following three key spaces coincide: physical input location, virtual input location, and virtual impact location.

Physical Input Location (Control Space). The physical input location represents where the physical input is occurring, i.e., the physical location of a user’s body and hands in direct-touch

(Pinelle et al., 2003). However, users may not always be able to observe each other’s’ physical body given their seating arrangement and distances in the co-located workspace.

Virtual Input Location (Display Space / Cursor). The virtual input location represents where the input is in the virtual world, and it is not always limited to the user’s physical location. For example, users may be using a mouse or phone as an input device to a large display. In this case, their physical location is different from where their pointer (virtual input) is in the virtual world. Virtual

embodiments that are used to represent users’ virtual input locations provide a way for collaborators to observe consequential communication of their actions in the virtual space.

Virtual Impact Location (Display Space / Animation / Feedback Location). The virtual impact location represents the location of the resulting action in the virtual world. For example, users may press a command button on the border of a tabletop interface, and the changes are reflected in the shared tabletop area. In this case, the physical input location happens at the border, and the virtual input location is in that same border space. However, the virtual impact location is at the centre of the tabletop. When virtual impacts take place, the changes being reflected in the system allow for

feedthrough. The observation of changes in the share artifacts helps a user to understand collaborators’ manipulations in the workspace.

See Figure 6-1 for the three types of interaction derived from the different coinciding relationships of these locations, and see Table 6-2 for examples of each type. The proposed interaction types and the classification are by no means complete, but they act as a framework to contextualize the different types of interactions on tabletop and multi-device environments. Further research is needed to map out the design space and investigate a complete taxonomy of interactions in co-located technologies.

Figure 6-1: Three types of interaction in co-located technological environments. Three interaction types emerge based on how physical input, virtual input, and virtual impact locations coincide: direct interaction, indirect interaction, and propagation of action.

Tabletop Systems Multi-Device Environments Direct

Interaction

Photo browsing (Otmar Hilliges, Dominikus Baur, 2007; Scott et al., 2005)

Collaborative browsing and editing (Morris et al., 2010)

Pour data from phone to tabletop (D. Schmidt et al., 2012)

Use tablets to view slices of data on tabletop (Seyed et al., 2013)

Indirect Interaction

Mouse input (Hornecker et al., 2008)

Laser beam (Nacenta et al., 2007)

Personal devices as pointers to public displays (Masuko et al., 2015)

Propagation of Action

Replay animation in timelines; Photo tagging with replicated

control (Morris et al., 2006)

Student edits on their devices are reflected on other devices

Direct Interaction. Direct interactions are when all three spaces coincide, where the users’ physical input, virtual input, and resulted effects are at the same location. Direct interactions are common in many tabletop applications such as photo browsing (Otmar Hilliges, Dominikus Baur, 2007; Scott et al., 2005) and collaborative browsing and editing (Morris et al., 2010). In multi-device environments, each device is perceived as an entity. Examples of direct manipulation techniques with multiple devices include using a personal device to pour data onto a tabletop (D. Schmidt et al., 2012) and to view slices of data on a tabletop (Seyed et al., 2013). Direct interactions require more explicit body movements such as reaching out to the objects in the group tabletop territories (Scott &

Carpendale, 2010) or performing gestures and device movements in multi-device environments. Since the users’ actions are more observable, direct interactions facilitate consequential communication (observation of collaborators’ bodies) and feedthrough (observation of the shared artefacts). This is consistent with findings comparing direct-touch interactions and indirect multi-mouse interactions on tabletop systems (Hornecker et al., 2008).

Indirect Interaction. When only the virtual input and impact locations coincide, interaction techniques allow for indirectly manipulating the content. For example, there has been work

examining using mice (Hornecker et al., 2008) as inputs for tabletops, rather than direct touch of the virtual objects. Several interaction techniques allow users to reach the entire workspace through interaction in their personal space such as laser beam, radar view, and virtual arm embodiments (Doucette et al., 2013; Nacenta et al., 2007). These interaction techniques typically allow users to interact at the border of tabletop systems. For multi-device environments, indirect interactions may involve individual devices acting as pointers or controllers to the public large displays, for instance Masuko et al. (2015). Observing consequential communication may be more challenging for indirect interaction than for direct interaction since users may have difficulties observing collaborators’ actions in their personal territories (Scott & Carpendale, 2010) or on their individual devices.

Propagation of Action. When only physical input and virtual input locations coincide, this type of interaction represents a propagation of action. A user’s action in one location can result in changes in other parts of the system. The replay animation triggered by timeline interactions is an example of propagation of action. When users tapped on events of interest on the timeline, they also invoked highlights in other parts of the system, e.g., the game map in the shared workspace. For the multi- device classrooms studied, a student’s edits on their device were reflected in other locations (i.e., other students’ devices), which is another example of propagation of action. Propagation of action

presents challenges both in terms of consequential communication and feedthrough. For

consequential communication, the actions conducted in users’ personal territories and individual devices are difficult to observe. Without feedthrough support in the system, the propagated changes invoked by users can be confusing or may be completely missed.

As the three locations coincide less, observing awareness information without system support becomes more difficult, and thus raises higher awareness demands for the system. For propagation of action on tabletops, while the small gestures for invoking replay animation through timelines required less physical effort to carry out relative to physically pointing at the board, it provided little

information for consequential communication. The users’ confusion confirmed what previous work found about the trade-off between physical effort and workspace awareness (Ha et al., 2006; Hornecker et al., 2008; Pinelle et al., 2008). While propagation of action allowed for minimal physical effort to reach more of the tabletop interface, maintaining workspace awareness became more difficult than with direct interactions. Furthermore, Hornecker et al. (2008) found that

collaborators using direct touch on tabletop interfaces experienced more interference, but were able to resolve interference more quickly relative to groups using mice (indirect interaction). The latter groups reported lower workspace awareness. Similarly, this thesis investigation found that the lack of information about who changed workspace awareness elements in the replay animation caused users to spend more effort coordinating and resolving confusion verbally.

Much existing work has focused on comparing direct and indirect input methods (Ha et al., 2006; Hornecker et al., 2008). Little work has examined the awareness needs for propagation of action. While the investigations conducted in this dissertation did not explicitly compare the different types of interactions, they provided valuable lessons in informing the awareness needs. The next section presents recommended awareness elements to support tabletop systems and multi-device

environments, specifically for direct interaction and propagation of action.