Implicit Interaction With Smart Objects

We will use the term implicit interaction to subsume interaction paradigms that involve the use of situational context information, such as the location and state of smart objects and users. Implicit system inputs are therefore actions of the user, which are not perceived as such, instead they are real world actions with an independent purpose. Albrecht Schmidt has coined this term in 2000 and proposed the use of context in order to improve user interfaces [Schmidt, 2000]. The application ContextNotePad was introduced to exemplify this idea and included context adaptive features, such as hiding the display content, when the device was not in use and other people are nearby.

This approach is further picked up in [Schmitz et al., 2007a] and conceptually applied to the shopping domain: In this scenario a user in a supermarket might take a product out of the shelf, turning the product in order to read what is written on the outer packing, gathering information about the product, instructions to use, additionally needed accessories or in case of groceries the nutrition facts and serving suggestions. Actions like picking up/putting back products from/to the shelf or shopping basket can for instance be detected using RFID tech- nology. Interactions with the products outer packing can be monitored using sensor boards, which are able to provide sensing capabilities e.g. for light, temperature, sound and accelera- tion. Such observed interactions could trigger events in an UbiComp environment the user is not aware of. In the case, where the user in our scenario puts the product back to the shelf, it would for example be possible for such a value-added service to compile a personalised prod- uct catalogue, that is somehow (e.g. via e-mail, bluetooth transmission, or printed version) handed over to the user at the end of his shopping process. Or the system might automatically check the nutrition facts against the user’s allergy profile and warn him, whether the product contains ingredients that might cause allergic reactions. As these examples illustrate, the in- teractions are typically unnoticed and the design of the interface does not intend a designated explicit interaction - as opposed to the concept TUIs.

3.1. SMART OBJECT USER INTERFACES 39

3.1.1.1 The Smart Shopping Assistant

In this fashion Schneider has developed a prototype of a smart shopping assistant [Schneider, 2004], which reacts on certain actions of the user: Taking a product out of the shelf and placing it into the shopping cart. Upon the first action the system will display general product information on a display attached to the cart and additionally compare two products if another item is taken out of the shelf at the same time as shown in Figure 3.1. The second action will cause the shopping assistant to evaluate the current set of products inside the cart in order to infer, whether the user is going to prepare a certain recipe. The system will provide a list of recipes, which require products that have already been placed into the cart and orders them according to the probability that the the user is going to prepare this dish.

Figure 3.1: The Smart Shopping Assistant in use

Clicking on one of the recipes will display the list of required items, indicating which of them are still missing. This example of implicit user interaction also shows the potentially blurred boundaries to explicit interaction: As soon as the user believes that the interaction pattern of the system has been completely induced, she or he can attempt to exploit its be- haviour to actively trigger desired responses, which might turn an implicit interaction style into an at least partially explicit one. For example, a user might place a couple of products into the cart in order to get a certain recipe displayed on the list, just to be able to check the required preparation time.

3.1.1.2 The MediaCup

Some of the first prototypes illustrating the potentials of implicit interaction based on context gathered by augmenting everyday artefacts with sensing and communication abilities was developed within the MediaCup project [Beigl et al., 2001].

Figure 3.2: The augmented MediaCup (source: [Beigl et al., 2001])

The MediaCup itself is still a prominent instance of these efforts: An ordinary coffee mug with hardware embedded into the base, containing both hardware and software for sensing, processing and communicating the state of the cup (see Figure 3.2). A coffee cup was chosen as the central item, because it constitutes a typical everyday object that is frequently used while remaining in the background of the user’s attention. While a focus was on communi- cation infrastructure and technical issues, several application scenarios developed over time. The most obvious step was to detect whether the cup is filled with hot coffee or not and to let the coffee machine start brewing replenishments if appropriate. Further, so called Smart Doorplates, displays with dynamic contents attached to the outside of office doors, would be able to detect whether there is a meeting situation and to show a ”meeting” warning to potential visitors. Such a meeting situation is basically assumed when more than one cup are present in an office or meeting room and being used.

The exactly same approach is taken by the Sentient Artefacts project [Kawsar et al., 2005], although a the name of their concept does not pick-up the preva- lent terms. Again, everyday objects are augmented with sensors to enable unobtrusive and implicit value-added services to the user. The smart artefacts here are also seen as aggregator of context information, including the state of the object and user activities. Several applications for home environments were developed, for example the AwareMirror [Fujinami et al., 2005], a mirror with display capabilities installed in a washroom, showing various information such as the schedule of the day or weather forecasts in addition to the person’s reflection. A toothbrush is augmented with two-axis accelerometer and actually

3.1. SMART OBJECT USER INTERFACES 41

used as sensor that allows to infer the use of the bathroom, which initiates the mirror interaction.

3.1.1.3 Smart Mats, Tags and Clips

Preceding the development of our prototyping environment (see Section 5.2) we have con- structed a series of smart objects by integrating different kinds of sensor technologies partly into existing objects and partly into devices specifically created for this purpose. The expe- riences of this project were incorporated in the development of our final prototyping testbed and also helped us to develop industrial design guidelines presented in Sections 4.1.5 and 4.2. In the following we will describe selected products that have been created in the context of an interdisciplinary project involving students of computer science and product design, who were given the task to realise fully functional TUIs under our technological and conceptual guidance.

Figure 3.3: The interactive beer mat.

In this project we hoped to bridge these two worlds of design and technology and to achieve a design process, which is informed from both sides. While the product design students could still do their original work of finding and developing forms and shapes, they received permanent feedback about how well their designs were suited to being turned into a functional prototype with the technology at hand. They learned to pay attention to this aspect early in their designs. Conversely, the computer science students learned a lot about the restrictions implied by mechanical design and were forced to be more creative in their choice of technology. We expected that the emerging designs would gain substantially from this constant dialog between the free flow of ideas and the necessity to produce a fully functional

prototype in the end.

The first product we want to introduce is the interactive beermat, which is also an example for interfaces that deliberately provide both implicit and explicit interaction modes [Butz and Schmitz, 2005]. The device is basically a reusable beer coaster made of plastic with integrated weight and acceleration sensors (see Figure 3.3) .

The implicit part is realised with the weight sensor, which detects whether a glass on top of the coaster is empty and in that case sends that information to the desktop computer, which would theoretically notify the waiter. The beer mat is further also intended to support entertainment activities in pubs, by using acceleration sensors to sense orientation in space, which can be used for voting processes. While simply raising the glass is usually associated with a positive reaction, a negative vote can be given by raising the glass and explicitly turning the mat upside down. Voting can be used in song contests to determine the vote of the audience or in karaoke bars to give immediate feedback to the singer already during performance. In sports bars, the decisions of the referee on the big TV screen are often cause of discussion, and the collective voting can here convey the average opinion of the local pub crowd, thus creating an additional level of interactivity for watching sports in a group.

Figure 3.4: The smart security tag.

The smart security tag is an extension of conventional security tags that are attached to clothes and also other products, in order to trigger alarms when carried through the exit [Kaiser, 2004]. The basic idea is to have such conventional tags extended by the means of generating simple visual signals that indicate whether that particular piece suits a given cus- tomer profile or not. This profile could contain information about age, gender, size and price range, depending on the final scenario and available information. The developed prototype applies a fixed set of selected attributes and matches them against available items. A subset of tags which is associated to the items that are potentially interesting for the customer, will glow as long as the user is present and ”logged in” at the shelf, which was realised by an RFID-based recognition of a personal item, such as a customer card (see Figure 3.4).

3.1. SMART OBJECT USER INTERFACES 43

cating with the i-Port III RFID reader. The communication of between the RFID tags and the reader is divided into two steps:First, the reader scans on the connected antennas of the i-Port to determinate which tags are within range. Second, a blink command is sent to the corresponding RFID tags. The first design concepts involved Smart-Its sensors (predecessors of the pPart Particles, see Table 5.1.1), but since the final version did not require additional sensors, the technology requirements have changed such that the size of the device could also be reduced.

Figure 3.5: The smart clip.

The smart clip is a brooch that supports gymnastic lessons by tracking movements with acceleration sensors and sending the data to the main application which displays feedback on a nearby screen [Berwanger, 2004].

The brooch itself has to be fixed to a particular position at the user in order to facilitate the recognition of the movements (see Figure 3.5). The requirements of the case therefore include the possibility to easily (re-)attach it to clothes and certainly to hold the sensor device and to provide breakouts for a recharger and a power switch. At the same time the size has to be minimised to make it as unobtrusive as possible.

The main software is located on a common PC and mainly analyses incoming accelera- tion data in order to detect whether the movements were performed in the intended manner. The actual feedback was performed by displaying video clips that contain either affirmations or corrections of a few selected movements.