Interacting in Smart Homes

In the context of the smart home, several researchers are investigating tangible interaction and gesture interaction as solutions for controlling the infotainment system or other household devices. Although knobs, buttons and traditional remote controllers are still the most popular solutions, gesture interaction and tangible interaction could offer richer experiences to the user, exploiting spatiality and benefiting of more human skills than the few ones required by traditional interfaces.

There are several tangible and gesture solutions for controlling the smart home. In this chapter, I will present only those who closely influenced my research, starting from the systems developed by the research groups to which I am affiliated. Accessing digital media in traditional interfaces generally requires browsing through a list of several files relying on their names or through visual previews. The Memodules project [146], developed at the HEIA-FR&UNIFR, boasts a tangible interface based on RFID-tagged objects as shortcuts for digital media, such as photos or videos. Memodules takes advantage of physical objects, e.g., souvenirs bought during the holydays, as tangible cues for the user’s memories and physical links to access the related digital content, e.g., photos from the holydays where the souvenir was bought. Following the idea of Hoven and Eggen [212], the system exploits personal objects that have a special meaning for the user as graspable cues to recollect memories.

Indeed, Hoven and Eggen [212] distinguished personal objects according whether they have a particular meaning for the user or not. The first category is very common in home environments and many objects can assume different meaning or recall different memories depending on the user.

Moreover, personal objects can assume several forms: Nunes et al. [148] distinguished between individual or group collectibles (such as items collected to represent events or a places either personally visited or visited by the friends), worn/consumed objects (such as clothes, accessories and food that are representative of a particular place), objects to commemorate personal accomplishments (such as sport or completion trophies) and trip output objects (such as tickets, maps and pamphlets used during a trip).

Although Memodules [146] exploits the physicality of the objects and leverages the human cognitive skills that associate memories to the image of a physical object, it still constrains most of the interactions with digital media to a dedicated console. Indeed, the console integrates an RFID reader to recognize the tagged object and the users can browse the media content associated to each object through touch buttons integrated in the console. Therefore, the interaction area (cf. also Section 6.1.4) is limited to this console and the user cannot fully exploit the spatiality of their everyday environment.

Conversely, another system developed at the HEIA-FR, explored spatial interaction to control household devices with simple gestures such us pointing the device to turn it on or off [39]. The system relies on 3D cameras to track the user skeleton and to map the direction of the pointing arm to the position of the objects. Carrino et al. [42] suggest a reduced vocabulary of functional gestures to interact with household devices through a similar system: the same gesture could be used for similar commands on different objects, thus reducing to the user the time required to learn gestures. This system requires that the user registers the position of the objects in the system in order to allow the recognition of pointing gestures, limiting the flexibility over time of the interaction. Moreover, the system has a technical limitation in terms of the spatial resolution needed to distinguish objects positioned one next to the other. Therefore, while the system is particularly interesting for the interaction with bigger household devices, it is less suitable for interacting with smaller objects, such as souvenirs positioned on a shelf. In this context, wearable solution can overcome this limitation.

In literature, several wearable devices to interact with the smart home through gestures have been developed. Starner et al. [195] used a pendant to recognize free-hand gestures. The Gesture Pendant is probably the first system that used a personal worn device to recognize hand-free gestures for interacting in a smart environment. The authors stressed the advantages of portability and privacy for the user in comparison to an approach based on several cameras distributed in the environment, which often suffers from the problem of occlusion. Moreover, they showed that such system could significantly reduce the number of gestures necessary to interact in the home, exploiting the user location information. The ReachMedia project [63] has been the first attempt to benefit from the specificity of the objects for gesture interaction through a wearable device. The authors proposed a bracelet equipped with an RFID reader and an accelerometer, which allow recognizing, respectively, RFID tagged objects and gestures. In their system, when the user grabs an object, an audio menu is presented to the user, who can browse through it using gestures.

An alternative approach to interact with the smart home is performing gestures through a handheld object. Künhel et al. [123] explored 3D gestures performed with a smartphone in the hand to

control different household devices. In a survey, they individuated the most common devices that users would like to control in the home: the television, lamps, the hi-fi system and blinds. Then, they conducted a gesture elicitation study to discover which gestures they would like to perform to control the aforementioned devices. The study shows that users preferred predefined gestures for popular commands where the users obtained a large agreement, while they preferred to have the possibility to define their own gesture for more uncommon commands.

Other studies explored novel designs for remote controllers, exploring tangible interfaces.

Spinning in Control [115] is a remote that exploits four rotating elements to control circular menus and data in media-center interfaces. The Peppermill [218], used a similar gesture on a peppermill-shaped device to select items in a video browser; in particular, the device was powered harvesting the energy from the same user’s gesture. Slurp [237] is a device that exploits the interaction metaphor of an eyedropper to pick-up digital media elements from a display or an object in the physical environment and to transfer it to another device or object in the environment, by squeezing out the media in the desired location. Slurp allows exploring spatiality, seamlessly interacting in the ubiquitous computing thanks to the tangible affordances provided by the slurp device and to interactive locations equipped with simple IR sensors.

As a result of the literature review in the context of gesture and tangible interaction in the smart home, I can summarize the following findings:

• Tangible interaction allows exploiting the spatiality of the environment and, therefore, users’ spatial cognition skills

• The physical objects that can be found in the environment can be used as tangible cues for digital information

• Personal objects can be used as cues for personal memories

• The users generally appreciate the possibility to define personal gestures

• Having a small gesture vocabulary is important to reduce gesture learning times and cognitive effort for remembering gestures while interacting with the system

• People are used to remote controllers to interact with the home, but having a different controller for each device is cumbersome

The results of the investigation in this domain will be exploited during the design and development of a smart watch to recognize gestures with objects. This system is presented in Section 7.5.