Managing the Environment

In environments with several screens, each display is usually driven by its own computer (or processing unit). From the technical point of view, a connection has to be established between the mobile device and the target display. Several solutions exist depending on the task. If the environment only allows for accessing information on a certain display, the user can take a picture through the mobile device which is then sent as multicast package to displays connected to the same network. Displays can then evaluate whether the picture represents a portion of their content or not. If the picture matches a display’s content, the screen can respond to the multicast. Due to potentially wrong detections, multiple displays may answer positively to a single multicast. The mobile device then does not know which (or if even one) of these displays is the currently targeted one. While being a promising solution, a decentralized approach may fail if the displays act as autarkic devices not communicating to each other.

To allow the idea of having an entire environment acting as a single device and display respec- tively, a centralized managing component can be installed. This approach is similar to the one described in chapter 4. Thisenvironment manager is the gateway for traffic between the users’ mobile devices and any present display (and its computer respectively). Pictures taken by any mobile device can be sent to the environment manager. The question is then at which component (i.e., each display or the environment manager itself) the pictures are analyzed. On the one hand, each display can determine on its own whether a picture matches its content. This leads to the same problems such as multiple positive responses by several displays. Besides being a gateway, on the other hand, the environment manager can also evaluate the images coming from mobile devices (see figure 5.12). This decreases the computational resources needed on each display by having a high-performance machine as analyzing instance. The displays then require less processing power as they simply display content.

Connect 3 4 Discover Manager 7 Detect Display Send Picture 6 5 Take

Picture ₁ Discover &

Connect

1 Discover &

Connect 2 Send Content

2 Send Content

Figure 5.12:Architecture for environments with multiple displays: On startup, each display

discovers theenvironment managerand connects to it (1). They also send their content for image analysis (2). Mobile devices also discover the environment manager (3) and connect to it (4). When users take a picture (5), the image is sent to the managing instance (6). There it is analyzed based on the content of each and every display (7).

To enable the image processing on this centralized server, the managing component has to know each display in the environment. This means, that the content and its constellation on each display needs to be present at all times. On startup, displays are discovering the environment manager in their public space. Subsequently, they connect to it and deliver content information. A simple screenshot is sufficient as the same image processing methods can be used to separate content from background. A more reliable solution is to transfer the content combined with geometrical information directly. By knowing the content, the environment manager is now able to associate incoming pictures from mobile devices to displays in the area. Furthermore, it always results in an unambiguous result: either one display has been found (this has not necessarily to be the correct one) or no screen has been detected. Obtaining the content of each display only once (during startup) may not be sufficient for longer time periods. For example, if the display changes its content over time (e.g., an animation is shown or users influence the screen’s content), it is not possible for the environment manager to either (1) identify the display at all or (2) calculating the correct transformation between the display and a mobile device. Updates have to be communicated to the environment manager whenever they occur. In theory, this can again be done by sending a screenshot. Only sending position updates for items already known to the environment manager may decrease the necessary bandwidth. This approach is theoretically also possible in distributed scenarios without having a centralized instance. However, for testing purposes we chose to rely on the environment manager.

If users want to interact through their mobile devices, the underlying application discovers the environment manager in the same network and connects to it. Whenever users take a picture indicating a selection, the mobile device sends this picture to the environment manager which

5.4 Allowing Multiple Displays in the Environment 107

then evaluates the image and identifies the display and its portion respectively (see section 5.4.2). In theory, the environment manager could send the connection address of the corresponding display to the mobile device for a direct connection. As this would require the mobile device to reconnect to the environment manager for each selection, we decided to keep a permanent connection between the mobile device and the managing component. If each display registers with full information instead of sending a screenshot only, the managing component becomes a machine with multiple displays attached. From the users’ point of view, each display in the environment presents a portion of the information in the area than acting as a singleton.