Responsive Floors Technologies and Software Systems

From the technical point of view responsive floors are subdivided in two main groups. The first employs force sensors surfaces to detect users presence and position, whereas the other uses various camera-space systems and algorithms to detect not only presence and position but also users gestures and movement features. The short review provided below summarizes on technology progress and present state of art.

2.3.1.1 Sensing floors systems

A grid of 1,000 force sensors has been the first responsive floor in Krueger’s Psychic Space in 1971. Later on, Paradiso for Magic Carpet (Paradiso et al., 1997) employed a grid of piezoelec- tric wires for user position and a pair of Doppler radars to monitor user’s movement to increase the sense of immersiveness. In the same years Litefoot, a system aiming at studying dance move- ments, was designed at the University of Limerick (Griffith and Fernstr¨om, 1998). In spite of force sensors good responsiveness, both systems suffered of signal confusion in time or space. To avoid this, a new collaborative project called Z-Tiles started in 2001 under the auspices of MediaLab Europe in Dublin. The design is grounded on a multiple sensors tiled surface employ- ing blob detection algorithms for user’s feet localization (Richardson et al., 2004). From 2000 a playware system employing tangible tiles was developed at the Technical University of Den- mark. The tangible tiles are new play elements which function as building blocks by containing processing power, sensors, actuators, and communication capabilities. They provide the oppor- tunities for creating new kinds of play and games. It is possible to have more than one game in the microcontroller and make different physical configurations of the tiles so the users can play different types of games (Lund et al., 2005). Contact sensing techniques, developed in 2010 allow to resolve the locations of the touch point, measuring the force applied on the interface. This method can be viewed as an efficient alternative to prior sensing techniques, as it requires far fewer sensors (Visell et al., 2010). A more recent floor prototype, GravitySpace employs a camera system located below the floor to sense pressure generated by people or room’s furniture. While force sensors can track only what happens on the floor surface, this system can sense also above the floor e.g. allowing user’s pose detection (Br¨anzel et al., 2013).

2.3.1.2 Camera-space systems

The first camera-space systems were realized by Krueger’s in the Videoplace application, where video projections were employed; or in the Metaplay installation, where live drawing interaction was performed by an artist on a data tablet and then projected on a wall screen. Starting from the ALIVE project (Maes, 1995), implemented at MIT Media Lab in 1994, many improvements were introduced, using computer vision algorithms for the tracking not only of the position of a person but also of a person’s hands and head. Kidsroom (Bobick et al., 1999) further improved this technique, providing to recognize by the story a dozen of simple individual and group ac- tions in specific contexts. The most popular and simple way to create a responsive floor is to place the floor area under the range of a ceiling-mounted camera at a convenient height, and to employ video streams data to feed real-time computer vision algorithms. In order to manage the

14 Interactive Spaces: Models for Motion-based Music Applications

noisy camera data, efficient algorithms must be implemented for data stream filtering and further processing. An important drawback of such systems is the dependence on environmental light condition which can alter the camera input and prevent efficient algorithm working. The Danish centre Interactive Spaces has employed two different responsive floor systems. One is the iFloor system which is and interactive floor which receives messages from SMS or emails. The floor interaction is based on a ceiling-mounted camera which tracks the people inside one meter band at the rim of the surface. People’s position and movement are interpreted as magnetic forces at- tracting a cursor with its home position at the center of the floor display. The force is proportional to the size of the shadow blob generated by a person moving under the projector. iFloor main- tains precise tracking of up to 10 people at one time in a 4x5 meters rectangle. The coordinates of the tracked persons are employed to calculate the movement of the cursor for application’s content selection (Krogh et al., 2004). But, this technology is not enough efficient when a higher control level of many users movements is required. Thus, an interactive glassy floor platform called iGameFloor was designed and placed in a school department at Aarhus (Denmark). The system is based on four cameras put in a square box hollowed below the floor level. The users limbs can be tracked using webcams data to recognize the contact points created with the surface under suitable light conditions. It is thus possible to hit a button in an application even though other users are standing close (Grønbæk et al., 2007).

2.3.1.3 The Zone Tracker application

A much simpler system for button function activation is the Zone Tracker application, imple- mented at the University of Padova (Amico, 2012). The video analysis algorithm analyzes the input images in three steps:

1. Background subtraction. The background is subtracted following the averaging back- ground method proposed by (Jabri et al., 2000). The method used is based on statistical modeling of the background for each channel color and detection with adaptive threshold. The background model is obtained by calculating for each channel the average and the standard deviation of each pixel in the course of a certain number of frames acquired from static shots of the background. After building a statistical model of the background, the new frames are compared with it, thus producing what is called a confiance image, that is a grayscale image, where the value of each pixel represents the probability that it forms part of a region of the image not belonging to the background.

2. Morphological image processing. The images are processed by means of morphological transformations (Vincent, 1994). Dilation and erosion are the two basic mathematical op- erations which correspond respectively to the expansion and thinning of the black region. A combination of the two leads to opening and closure morphological operations, aiming at defining well shaped silhouettes blobs.

3. Blob tracking. The blobs moves are tracked and the two-dimensional barycenter of each blob is calculated.

Chapter 2. Responsive Floors 15

Figure 2.4: Users tracking results obtained with a six sensors OpenPTrack system. Source https://vimeo.com/channels/864928

The button function is activated by comparing the blob position with a customizable mask which partitions the floor surface for content synchronisation. Also if the application allows the tracking of more than one users, when people are close the two blobs may merge, thus generating some confusion in the tracking.

2.3.1.4 The OpenPTrack project

Launched in 2013, the open source software OpenPTrack6 is a scalable, multicamera solution for person tracking. OpenPTrack aims at supporting creative coders in the arts, culture, and educational sectors who wish to experiment with real-time person tracking as an input for their applications. The system allows to use a network of imagers to track the moving centroids (center of mass) of people within a defined area. It performs users detection in the machines connected to each sensor, whereas tracking is executed by a single node receiving data from all the network. These data can be incorporated into creative coding tools like Max/MSP, Processing, as well as a variety of other software languages and environments. (Munaro et al., 2016). It is very efficient for users tracking in every light condition and allows many users to be followed without any problem of occlusion.