Dragging Content across Displays

Besides moving items on one display using touch input, users are also able to drag content across displays. To do so, they touch an item on the originating display and start dragging it as described before. Simultaneously, they move the device to the destination display and can then position the

7.1 Continuous Input on and Across Displays 139

item on its final location. As the mobile device tracks its spatial relationship with respect to the displays (and thus partially to the environment), common problems known from relative pointing in display-less space do not exist [NMG08]. Users decide the path they take on their own which does not introduce inconsistencies between the motor space and the feedback. However, while traversing from one display to another, users do not get feedback with the original idea of having the mobile device as a purely transparent one. This limitation can be sidestepped by showing a thumbnail of the dragged item whenever the device is not pointed at an external display (see figure 7.3). As soon as another display is in sight and shown in the viewfinder respectively, the thumbnail is removed and the item is shown on the corresponding display. The mobile device acts as a display in the environment if needed. Overall, users can always track the dragged item – either on a display shown in the viewfinder or on their personal device (see figure 7.4).

Figure 7.3: Visualizing dragged content in display-less space: a thumbnail is shown when-

ever the mobile device’s camera is not pointed at a display in the environment.

To allow for such drag operations, the environment manager needs to know when a touch point that currently drags an item leaves a display. Whenever an event of this type occurs, the manager needs to send the visual representation of the dragged content to the mobile device. At the same time, the content needs to be removed from the display to avoid the content being shown twice. Before transferring the visual representation of an item to the mobile device, it needs to be transformed according to the current device orientation to allow for a seamless transition between external screen and mobile display. This can be done by using the inverse of the current transformation matrix between both devices’ image planes. Similarly, when reaching another display, the item’s local geometry needs to be translated into the screen’s coordinate system. Each corner point is transformed again as described earlier. The content’s thumbnail shown on the mobile device can be manipulated while moving the device. For example, users may drag the item on the mobile display towards the device’s edges to allow for greater visibility of potential target displays through the viewfinder. To allow the aforementioned translation back into the external display’s coordinate system, the environment manager needs to be aware of such manipulations.

We decided to send local touch events to the environment manager allowing the manager to trace the actions while the device is not pointed towards a display. This information allows for a correct transformation of the dragged item once an external screen is in the viewfinder.

Besides handling the visual output, the input also needs to be handled during cross-display op- erations. In our architecture, touch points are and their corresponding actions respectively are directly performed on the target display. Hence, if a touch point leaves the current target display shown in the viewfinder of the mobile device, the environment manager has to notify the screen as well, so that it can remove it. This point is then temporarily active on the mobile device as long as no display is shown underneath the touch point in the viewfinder. Similarly to the dragged item, the point needs to be removed from the local layer and then added to the respective external screen, when the point enters a display in the live video. Users can then continue to drag directly on the external screen. Figure 7.5 denotes the input and output handling process. Naturally, if a second touch point is added or removed on the mobile device while the input is performed locally (i.e., no display is shown in the viewfinder), the point needs to be added as well when the device reaches another screen. Such events may occur when users want to scale the currently dragged item to fit the destination display (i.e., downsizing it if the screen is smaller than the originating one). In our architecture, the environment manager is aware of input on the mobile device at all times to allow for the aforementioned input handling.

a

b

c

Move to destination

d

e

f

Arrive at destination

Figure 7.4: Dragging content across displays with the original metaphor: The user aims

at a display (a) and touches the item of interest (b). When moving the device off-screen, a thumbnail of the item is shown (c). After reaching the destination display (d), the item can be positioned precisely (e). When the finger is released, the item has been transferred (f).

7.1 Continuous Input on and Across Displays 141

Users can also (accidentally) remove the last touch point when they drag an item in display- less space. The logical operation is then to drop the item at this location. Since we do not track the device with respect to the environment but rather with respect to each display, the item would be lost and is not detectable anymore. In this case, we decided to return the item to its original location before the drag started. To do so, the environment manager stores the necessary information by keeping an exact copy of it. It is then able to return the item (i.e., adding the copy to a display) once it is released in display-less space.

“Touch down” “Remove Point” “Add Point”

a

b

c

Figure 7.5: Touch event handling when interacting across displays: (a) A touch event is

projected onto the current screen. (b) Leaving the screen requires a removal of the point. (c) Reaching another screen then requires the manager to add a new touch point to the display.