Frustrated Total Internal Reflection

A popular method for camera-based touch sensing relies on a concept called frustrated total inter- nal reflection (FTIR). A light wave propagating in a medium with a higher refractive index than an adjacent medium at a sufficient angle of incidence is entirely reflected, known as total internal reflection [46,61]. Another medium located sufficiently close to this boundary can disrupt (or frus- trate) the internal reflection, causing the light to escape. Such light can be captured by cameras or other light sensors and interpreted to perform touch sensing [79, 135, 151]. However, FTIR does pose certain drawbacks. Most importantly, touch applications relying on FTIR are limited to either flat surfaces or surfaces with minimal curvature [125]. The traditional FTIR approach is not ca- pable of detecting hovers, though this may be a benefit in some scenarios. Other IR light sources, such as sunlight or other optical tracking systems, can interfere with FTIR-based routines [44]. Additionally, fast hand movement can lead to intensity decreases in the camera imagery, which can make contact detection more difficult [74].

FTIR has been used to create images of fingerprints [53], including even as early as 1965 [155]. As a natural extension to this idea, Johnson and Fryberger obtained a patent in 1972 describing the use of FTIR to support touch input on a cathode ray tube (CRT) [79]. Other FTIR touch-sensitive CRT systems have been developed by Mallos [100] and Kasday [83], and similar methods have been designed to support CRT drawing applications via fingertip, brush, and pen input, either using photodetectors [109] or cameras [58] to capture the frustrated light. Han repopularized the use of FTIR-based touch sensing in 2005, noting several benefits of the technique: low cost, high resolution, high accuracy, and the ability to integrate with dynamic rear-projection graphics [61]. Davidson and Han later integrated touch pressure information into this approach, allowing users to “tilt” graphical objects by selectively applying pressure to place them on top of or below other ones [40].

Some researchers have investigated ways to augment traditional FTIR-based touch sensing ap- proaches. Echtler et al. combined a standard FTIR multi-touch table with an overhead light source that allows for the detection of shadows as an additional input modality [46]. The traditional side- mounted IR LEDs used for touch sensing and the ceiling-mounted IR light for shadow tracking are enabled in an alternating fashion; as such, a single input comprises a contact image and a shadow image taken on consecutive frames. When the system detects a shadow, it places a cursor at its peak, shifted by some amount to prevent occlusion from the user’s hand. Once the user’s finger contacts the surface, a click is triggered at the cursor’s location. This is analogous to mouse input in computer interfaces, where placing the cursor over an object and selecting that object by clicking are two separate events. In a human-subject study, participants touched squares displayed on the table, either with or without the shadow-based cursors. With the cursors, participants were more accurate in touching the squares, but this doubled the amount of time required.

Iacolina et al. also experimented with shadow detections in an FTIR setup on a flat tabletop [74]. However, their prototype relies on natural, uncontrolled IR light in the user’s environment, such as

sunlight, rather than additional controlled IR light sources. The lack of control does not cause ma- jor issues: the standard IR tracking works better when there is less sunlight, and the supplemental shadow tracking performs better when there is more. As in similar setups, a primary motivation is the ability to detect proximity to the touch surface.

Similarly, Dohse et al. added an overhead camera to a traditional FTIR setup to perform hand tracking [44]. They had two primary motivations: assigning touch ownership to the appropriate user in a multi-user environment and making the system less susceptible to IR noise due to poor lighting conditions. The hand-tracking camera could additionally be used to recognize gestures above the surface, though the authors do not explore this.

Sch¨oning et al. describe interscopic multi-touch surfaces (iMUTS), which leverage both 2D and 3D interaction techniques with both monoscopic and stereoscopic visual content [130]. Their prototype comprises an FTIR-based multi-touch wall. They describe two interscopic interaction techniques. The Windows on the World metaphor presents a 2D overlay over 3D content; users can navigate in three dimensions throughout the virtual world via various touch interactions on the window. The second interaction technique allows users to cut and subsequently deform the 3D volume of the presented data by selectively dragging on certain parts of the wall. To simplify the interface, predefined cutting templates are provided.

While FTIR-based approaches are generally limited to flat surfaces, Weiss et al. designed Bend- Desk, an interactive hybrid desk featuring a horizontal tabletop and a vertical board connected by a curve [145]. The imagery from three cameras is interpreted to detect touches across the entire surface, including the curve. The authors’ findings indicate that users tended to think about the sur- face in terms of its three constituent components rather than as a single one, and they often avoided interacting “across” the curve. Likewise, Villar et al. created a curved multi-touch mouse based on FTIR [143]. The surface of the FTIR Mouse is a smooth arc that still supports total internal

reflection. However, due to the shape and camera placement, the mouse is only capable of sensing touch in a small area toward the front.

Using FTIR, Roudaut et al. investigated how surface convexity and concavity affect touch in- put [125]; due to light bleeding, they made a few modifications to their curved surfaces, including applying silicone spray and manually smoothing the surfaces to reduce IR hotspots. They found that pointing accuracy increases as surface convexity increases and that users were less accurate when touching targets on concave surfaces.