There is a long history of research into the use of tangible props as user interfaces. It is now well established in the 3D user interfaces and human-computer interaction communities that even passive haptic props typically provide great advantages over freehand interfaces in terms of spatial understanding and control over 3D operations.
Early work in this area focused primarily on the use of pens and palettes [39, 40]. For example, the 3-draw system [39] enabled an artist to draw 3D curves using a tracked pen. Others (e.g. [41]) have looked at combining 2D and 3D pen based input in VR environments.
More recently, the ModelCraft framework proposed by Song et al. [42, 43] combined folded paper props and a pen interface to capture digital annotations on the prop surface. While this paper-based system worked well for annotating architectural models with mostly flat surfaces, it is less well suited to the complex organic geometry intrinsic to many scientific datasets.
Using this work as a foundation, new research explores the potential of prop-based interfaces that are made possible (and increasingly practical) by emerging rapid proto- typing technologies. Accurate physical 3D data printouts from rapid prototyping ma- chines have been used as interaction props for 3D data visualization (e.g. for molecular biology visualization [44], cartographic GIS visualization [45], and infovis [46]).
Others [47, 48] have looked at combining spatial pen-based input with rapid pro- totypes. For instance, Kruszyn´ski and van Liere [48] reported on requirements for tracking, calibration, latency, and printing for an application that uses a tracked pen to interface with tangible props printed from 3D coral data. This system was motivated in part by the need to analyze the precise 3D shape of specimens collected from coral
reefs which are too delicate to handle extensively. The 3D shape of the corals are par- ticularly complex, thus, visualizing them as a physical rather than virtual object aids in understanding. The particular interface developed in this work was an interactive measurement system. A coral prop was held in one hand while a stylus held in the other was used to indicate precise locations on the surface of the coral, from which distance and thickness calculations were made.
While these rapid prototypes can provide a very realistic display of the data for interaction, they are not transferable to different datasets. New models must be printed for each set of data, which can slow down the scientific workflow. Motivated in part by this limitation, many tangible interaction props are more abstract (e.g. bricks as a graspable proxy for various objects [49, 50]). A study [51] by Colin Ware showed that the prop does not need to have the exact shape as the virtual object to facilitate interactions.
This finding is apparent in Hinckley’s seminal work in this area, applied to scientific visualization. Hinckley used simple props tracked in space (a doll’s head, a clear square of plastic) to help doctors fluidly explore and slice through brain imaging data [52].
The cubic mouse [53] was also used for navigating visualizations. Shaped like a cube with three cylindrical rods of a thickness similar to a pen, it was held in a user’s hand. By twisting or translating one of the rods relative to the center of the device, the user could navigate and manipulate the virtual world.
In addition to visualization, tangible props have also been used for artistic inter- faces. Balakrishnan et al. [54] introduced a flexible bend and twist sensitive strip called ShapeTape. By manipulating the strip, a user could create precise curves and surfaces through extrusion, lofting, and revolves. Although precise shapes could be created by using spring steel rods and jigs to constrain the movement, users found that the inter- face suffered from the “iron horse” effect. Named for the first automobiles that were
controlled and even shaped like a horse, the iron horse effect occurs when new designs mimic the properties of an analogous physical object too closely. ShapeTape suffered from being unable to create some input curves because the inherent properties of the material limited the bend curvature. It also was difficult to move from one workstation to another. While the tangible input gave the users increased control, it limited some of the advantages of working with a computational tool, such as the ability to ignore physical properties like gravity.
Similar physical interfaces have been used for modeling. Sheng et al. [55] presented a clay-like modeling system that uses a foam sponge as a physical proxy for the model. By tapping, twisting, sliding, and pressing on the surface of the sponge, the artist could create models. Unfortunately because of the limited tracking resolution and the way that the sponge responds to input, most of the models created using the system have a blobby topology that is characteristic of many of the modeling systems in this style.
Tangible props for visualization are not just limited to changing the physical input device, but also changing the physical display. For instance, Konieczny et al. [56] used a flexible tracked projection screen that the user could move and flex to see a virtual slice of 3D volumes.
Touch-sensitive display interfaces also share some similarity to our work in that they enable fluid, natural styles of interaction with data [57]. For example, CubTile [58] uses a cube with five out of the six sides covered with a multi-touch surface to leverage the 3D spatial layout in overcoming the 2D limitations of traditional touch screens. Other work has looked at changing the touch surface into a sphere. In particular, Grossman et al. [59] define several interaction techniques where the angle the finger makes with the surface, for instance pointing at the sphere vs. parallel to the display, can be used for different modes. This is similar to our approach for Anchored Gestures in Chapter 3.