Collisions and shadows

Collision detection is routine in computer graphics and virtual reality. The bounding volumes of objects are checked against each other, and the application prevents objects from overlapping. Collision detection between a real and a virtual object in augmented reality is more complex than between virtual ones. The application does not know the boundaries of real objects. Instead, the system needs first to use some 3D reconstruction or computer vision method to detect and analyse the real object. This is challenging and time consuming, and therefore AR applications seldom have any mechanism for collision prevention and users may for example move virtual objects through real ones.

Occlusion detection methods, which we discussed earlier, are usually limited to detecting the order of the objects’ facades. This information is insufficient for colli-sion detection.

Augmented reality systems that are capable of detecting and preventing colli-sions with static environment often require offline processing for 3D reconstruc-tion. It is also possible to use a model-based approach as proposed in [238]. The model-based approach is able to handle both occlusion (masking) and collisions.

However, it requires a priori information of the environment (models) and registra-tion technique to align the model and the corresponding real object.

Systems that are able to detect and prevent collisions with moving physical ob-jects typically have complex hardware systems (e.g. multi-camera systems [239], TOF cameras or a complex stationary setup [240]). To our best knowledge, there is no lightweight AR system capable of handling collisions. However, there are application areas where collision prevention is an important issue and there is a clear need for development and further research in this area.

Preventing physical collisions is a big issue in robot operating, which AR can help in programming collision free robot movements. For instance, an interactive AR system is used for planning collision free paths for maintenance robots [241].

Interior design is another area where collision detection would enhance the AR system. In our ongoing work [16] we interviewed professional interior designers, consumers and interior design bloggers among other players in relation to AR interior design. Our interviewees considered the lack of collision prevention in current AR interior design applications as one of the bottlenecks preventing seri-ous or professional use of them.

A typical question in interior design is whether a specific piece of furniture will fit in the space available. However, current VR/AR systems let the user move virtual furniture through physical walls and furniture. Interior design plans are supposed to illustrate by default how furniture will fit in a room. Therefore, to be able to use the application for making an interior design plan, the application should ensure that objects do not overlap, and that the user is not able to move them through the wall accidentally. Naturally, the user can visualise furniture with the current sys-tems and use them for daydreaming, as our interviewees mentioned.

Marker-based AR is able to get the floor plane coordinates from a marker posi-tion and thus keep objects on ground level. In straightforward manner, markers can be used to mark walls and prevent collision with them, i.e. to keep furniture inside a room. In our earlier work [18], we learned that automatic detection of walls or simple user interaction to point out walls is an essential function in a renovation and interior design application. Once the application detects the walls, it can also change the wall coatings virtually.

Figure 87. Our gesture detection user interface in action [7].

In some games and applications using gesture interface, collision detection reduc-es to 2D. In this kind of selection, it is sufficient to detect when objects (e.g. a hand and an icon) overlap on an image plane. For example, in our augmented assembly demonstration with multimodal user interface [7], the gesture recognition works this way. The user selects a menu item by moving a hand on desired icon, then forward and backward icons appear, and the user selects the desired action (see Figure 87). In some games the user may hit virtual objects, which then bounce from the player’s hand. This kind of collision detection and prevention can be implemented for example using the 2D sphere-to-sphere collision estimation pre-sented in [242].

Another approach for collision prevention is marking collision-free areas. Typi-cally, this is done by sweeping collision-free areas with a marker (as in [241]) or other detectable object. This normally requires user interaction and is usually done in separate initialisation phase.

A simple robot equipped with a camera could be used for creating a 3D model of a space. Modern robot-hoovers such as Samsung VCR8845 have an integrated camera and are capable for 3D reconstruction of the free space. Some robot-hoover manufacturers (e.g. iRobot for Roomba) provide an Application Program-ming Interface (API) for making individual enhancements. This indicates that in the near future, feasible (i.e. cheap and easy to use) “scanning robots” will be availa-ble for defining collision-free areas and other 3D construction tasks, for example.

In Section 6.1.3, we considered shadows that virtual objects cast on real envi-ronment. However, to perceive seamless integration of the virtual and real world a system should also consider real shadows on virtual objects. Couple of research papers have addresses this issue and proposed methods for recasting real shad-ows on virtual surfaces.

For example, [193] uses an approach similar to template matching for detecting predefined surfaces and its lighting conditions in the presence of partial occlusion.

The surface texture is then replaced with a virtual one. This paper proposes a visibility map based on the work of [243], but with the difference of using very local similarity measures instead of global ones. Thus, the method is able to distinguish

between occluding objects and shadows. Occluding objects are then masked to foreground and shadows are recasted on top of augmentation.

As mentioned earlier, the method presented in [224] recasts shadows of real foreground objects on virtual ones. This paper uses simple spherical geometry together with spherical vision camera for estimating the foreground depth, height and directional light (shadows are caused by the sun), which gives enough infor-mation to recast shadows on virtual objects.