Absolute Pointing as Promising Alternative

Two characteristics of displays – namely the number of users as well as the flexibility of the environment – seemed to be limited by relative and indirect pointing. As pointers on the remote display consume a certain amount of screen space, the number of users is restricted depending on how much screen space is still needed for the application to be usable. On the other hand, the flexibility of the environment in cross-display operations can only be increased if either the environment is known a priori or users disconnect from one screen and reconnect to another. Besides these factors, bystanders are aware of the user’s actions since a personal pointer is always shown on the remote display. However, this issue is beyond the scope of this thesis. Although the uniform approach (i.e., having the same type of input on various displays) is already ensured

by relative and indirect pointing, it appears that some of the limitations cannot be overcome

by this type of interaction. In the following we discuss whether absolutepointing (as an input alternative) can overcome the aforementioned limitations.

In contrast torelativepointing that almost necessarily is in anindirectfashion,absolutepointing can be performed either in adirect or in anindirect way. Theindirect approach still requires a pointer to be shown on the external display causing similar issues compared to relative pointing.

Direct interaction on the other hand requires that the user’s action and the system’s response occur at the same location. One prominent example for such input is a direct touch interface. As stated before, public displays either are not always reachable directly or users do not want to touch them directly due to security (e.g., the visibility of the interaction in public) or hygienic reasons (e.g., a layer of grease on the display). The solution needs to incorporate the personal device as users trust them at all times. In chapter 3 we already showed how the personal device can be used to interact with remote content in adirectandabsoluteway. If the display is unreachable, however, the presented approach needs to be changed. One solution is to transfer Tani’s idea to mobile devices [TYT+92]. The local video shown on the mobile device then turns the mobile device into asee-throughinterface [WFB+07].

Absolutepointing further requires users to point towards the display they intend to interact with.

By doing so, they implicitly identify the target display. Combined with the use of the mobile device’s video, this approach can be used to connect to an external display by analyzing the visual content. In this case, the aforementioned interaction phases can be reduced toacquiring

content andmanipulatingcontent leaving out theconnectionphase. From the user’s point of view the phase of acquiring the screen will hence become a part of the content acquisition. This would further improve cross-display interactions: moving the pointer from one display to another now works without the need of disconnecting from one display and reconnecting to the other one. In the remainder of this work, we present an approach and its architecture allowing for suchabsolute

III

THROUGH THE

DISPLAY

Chapter

5

Shoot & Copy:

Discrete Pointing through Live Video

Discovery consists of seeing what everybody has

seen and thinking what nobody has thought.

As shown in the previous chapter, relative and indirect pointing allows for interaction on large and possibly distant displays. However, such techniques are limited due to personal pointers shown on the external screen. First and foremost, users have to explicitly select the display they want to interact with a priori. With an absolute approach, on the other hand, users implicitly select their target display. For example, pointing on content on an external display using an indirect device such as a laser pointer immediately reveals the target display as well. These indirect approaches still show a personal pointer on the external screen leading to the problems identified in chapter 4. In this chapter we1present Shoot & Copy, a system that allows pointing at external displays in an absolute and direct fashion. At the same time, users aim their mobile device at the desired content which is then shown in their mobile device’s viewfinder. Upon content selection (i.e., the item underneath a crosshair), the mobile device takes a picture which is then analyzed by a centralized component within the environment to identify the targeted content. This means that from the user’s point of view, the connection process is embedded into the selection phase. The mobile device maintains a permanent connection to this centralized instance which is established automatically during the application launch. This setup further allows the selection of content

1 _{The work presented in this section has been published in a scientific paper [BAB}+_{07]. The scientific plural refers}

to all authors of these publications – namely Sebastian Boring, Manuela Altendorfer, Gregor Broll, Otmar Hilliges, and Andreas Butz.

independently of the target display without the need for users to connect to it in a modal way. In a qualitative evaluation we found that users liked the idea of selecting content by taking a picture although the overall selection time (i.e., the time between taking the picture and actually receiving the content) in our implementation was rather long.

The remainder of this chapter is structured as follows: We start by motivating the approach of content selection by taking images (see section 5.1). Subsequently, we describe the necessary steps of computer vision to identify an external display purely based on the mobile device’s viewfinder content (see section 5.2). Based on this architecture, we present a proof-of-concept implementation and its performance and qualitative evaluation (see section 5.3). We then extend the infrastructure to allow the detection of multiple independent displays (see section 5.4). We conclude this chapter with a summary and a discussion (see section 5.5).