Design of a serious game for learning vibrotactile
messages
Bob-Antoine J. Menelas
Mathematics and Computer Science Department University of Quebec at Chicoutimi
bamenela@uqac.ca
Martin J.-D. Otis
Applied Science Department University of Quebec at ChicoutimiMartin Otis@uqac.ca
Abstract—To prevent accidental falls, we have designed an augmented shoe aiming at assisting a user when walking. To this, the risk level (low, medium, high and very high) represented by the current situation is conveyed to the user through vibrotactile messages. In this paper, we describe the design of a serious game dedicated to learning of these signals. The game is centered on a virtual maze, whose parts are associated with the four risk levels. To explore this maze, fitted with a pair of the augmented shoes, the user is invited to walk in a room, completely empty, whose dimensions are mapped to those of the virtual maze. When moving, for each area explored the corresponding signal is delivered to the user through the augmented shoes. An initial experiment confirmed the idea that vibrotactile messages can serve for communicating the level of risk.
I. INTRODUCTION
According to statistics, over one third of those over 65 fall at least once a year, causing 65% of injuries in this age group [1]. In this category, one observes a significant decrease in mobility, due to the condition of the pavement, especially because of seasonal factors (ice, snow etc.) [2]. Considering such observations, we have recently developed an intelligent system to provide continued assistance to vulnerable users (elderly or those with balance disorders). Specifically, this product is designed to prevent accidental falls related to conditions of the physical environment of the person (slippery, abrupt soil etc.), or abnormalities of its gait. This system is centered on an augmented shoe (see left part of Fig. 2).
This device counts on one side, a set of sensors that serves for characterizing the dynamics of walking, the posture of the user and the physical properties of the environment. On the other, it brings together several actuators aiming to transmit vibrotactile signals to the user. These signals intend to attract the attention of the user towards situations deemed dangerous by the control system running on a smartphone. Therefore, these signals appear as an aid designed to assist the user.
Beyond aspects related to the computation of a falling risk, to be useful, rendered signals must be properly assimilated by the user. In other words, in the case of a situation representing a high level of risk in addition to correctly interpreted the rendered signal (identify the correct level of risk), the user should not be alarmed by the identified risk level since the stress occasioned by that can lead to a fall. Similarly in the case of a situation presenting a low risk level, the user should not be less attentive since lack of attention can also lead to
a fall. For all these reasons, it seems important to us to pay particular attention to the process of familiarization with the system within a fully controlled environment. For this, in this paper we present a serious game dedicating to the learning of vibrotactile signals.
By experimenting this game, we want the user to be able to identify easily the four signals associated with risk levels: low, medium, high and very high. The game is designed in order to be usable for the realization of home exercises without using a third person. The central idea of the game is defined around a virtual maze, whose parts are associated with the four risk levels (see Fig. 1). To explore the virtual maze, fitted with a pair of the augmented shoes, the user is invited to walk in a room, completely empty, whose dimensions are mapped to those of the virtual maze. When moving, for each area explored the corresponding signal is delivered to the user through the augmented shoes. At each time the user is prompted to identify the level of risk associated with this signal. In the same occasion, various measures relative to the users gait and its heart rate are mesured in order to check whether the rendered signal does affect the user or not.
The paper is organized as follows: motivation and related work are presented at section II, a brief overview of haptic and tactile icons perception is given in section III. The developed serious game is detailed in section IV. The initial experimentation is presented in section V, followed by the conclusion in section VI.
II. MOTIVATION ANDRELATED WORK
A. Motivation
To prevent accidental falls, we designed a system centered around an augmented shoe aiming to assist vulnerable people (elderly or those with balance disorders) during their dis-placements. The sole located in the shoe is shown in Fig. 3. Specifically we want to be able to alert in real-time the users about a potential fall hazard. To do so, the shoe must be capable of communicating with the person without preventing him of being fully aware of his external environment. For this, we want to investigate possible solutions that can exploit a channel different than audio or visual. Furthermore, it is known that elderly often do have hearing and/or visual problems. Because of all that, we want to investigate the use of vibrotactile feedbacks for the communication. We thus intend
Fig. 1. Vibrotactile signals (tactons) associated with four risk levels.
to transmit the messages via the actuators distributed in the shoe. Knowing that humans are not used to exploit these kinds of communication, it will be important that the user fully understand transmitted messages before any exploitation. This aspect is especially important because we want to avoid all kind of misinterpretation of rendered messages as well as all kind of stress etc. Therefore it is essential to create a suitable environment that will help users at learning transmitted mes-sages. This is the goal that we target through the proposed serious game.
B. Serious Game
Origin of serious games goes back to 1997 with the publi-cation of the report entitled:Modeling and Simulation Linking
Entertainment and Defense. Today, serious games concern
any meaningful use of computerized game or game industry resources whose chief task is not entertainment [3]. Thus, according to Michael and Chen, serious games may be related to all domains ranging from advertisements to military training exercises [4]. As part of this brief review, we will focus on aspects related to the potential of serious games for transfer of knowledge. These types of game present a learner approach; while creating a meaningful environment for learning [5]; by doing so, they can address both cognitive and affective dimensions of learning [6]. As a result, in this decade several games have focused on the exploitation of serious games for learning, creating what is callededutainment,infotainmentor
entertainment-education. Among the most famous developed
games, one countsSuperchargedwhere in a three dimensional world, players are invited to take advantage of electromagnetic laws in order to navigate a spaceship [7]. In the same way, in River City, students were learning biology notions while investigating reasons why residents of a virtual town were getting sick [8]. Similar results have been demonstrated in [9] where studies have showed that serious games have the potential to improve student learning and attitude toward digital design and electrical engineering. In a recent study regarding effectiveness of serious games compared to other learning approaches, Wounters et al. concluded that these
new generations of serious games support the acquisition of knowledge [10].
Considering the promising results of cited studies, we have decided to realize a serious game that will help users at learning vibrotactile messages.
III. PRELIMINARIES ON TACTONS
Besides the visual modality, various studies have shown that haptic and audio channels could be exploited to convey information. As a result, as their visual counterpart, haptic icons have been designed. This group includes:
• Haptic iconsare complex vibrations which reproduce the impulse response resulting from the contact of physical objects, such as vibrations that occurs when a hammer is tapping on wood, iron or rock. MacLean et al. define them as computer generated signals display through force or tactile feedback to convey information such as event notification, identity, content or state [11]. Guidelines for their design are presented in [12].
• Tactonsare composed of a sum of sinusoids with a decay
exponential envelope similar to a musical note. Brewster et al. describe them as structured abstract messages that can be used to communicate messages non-visually [13]. Guidelines for their design are presented in [14]. Most work of the literature has focused on evaluating discrimination and identification capabilities of haptic and tactile icons via the hand. About haptic icons, studies of MacLean et al. showed that frequency should be the dominant parameter and should range from 4 to 20 Hz [11]. In such a condition they noted that users can clearly distinguish a set of 10 haptic icons. In work described in [15] they have combined various haptic icons in order to form messages. They found that users could recall an arbitrary association between a haptic stimulus and its assigned arbitrary meaning for a period up to 45 minutes. In regard to tatcons studies, Hoggan and Brewster showed that the use of waveforms (sine, square or triangle wave) as texture parameter intacton design offers better discrimination rate than frequency or amplitude modulation [16].
On the other hand, less work has explored perceptual capabilities of human in differentiation, identification and recalling haptic or tactile icons presented via the foot. One of the few studies conducted on this subject has concerned the used of an instrumented tile to render structured vibrotactile signals [17]. Using similar haptic tile, Okamoto et al. have proposed a vibrotactile display method to capture the crinkle of fragile structures [18]. In the same way, Megard et al. have investigated the possibility of using the floor of a car as potential locus of vibratory for collision warnings [19]. Other studies have concerned the rendering of haptic stimuli via instrumented shoes. Several prototypes have been developed towards the creation of a shoe or a sole enhanced with data transmission capabilities through vibrotactile feedbacks as shown in [20] and [21]. For example, using a 16-point array of actuators Velazquez et al. have stimulated the plantar surface by inducing different vibration frequencies [20]. With
Fig. 2. A player is experimenting the game. The augmented shoe is represented on the left whereas the first maze of the game is on the right side.
the realized experiment, they concluded that some shapes were discriminable. However, memorization and identification aspects have not been studied.
As seen through this section, although some guidelines exist, we are still far from establishing a standard for the design of vibrotactile icons. Therefore in order to ensure that users will be able to correctly interpret the messages transmitted via the instrumented shoe, it seems appropriate to develop a game that will help these users at learning easily the signals. Moreover, knowing that the messages will be sent to the user while he will be walking, employment oftactonsseems more reasonable rather than haptic icons in order to avoid confusing the sent message with a natural phenomenon.
IV. DESIGNEDGAME
Our research project aims at using multimodal feedbacks mainly vibrotactile ones, through an augmented shoe in order to prevent accidental falls. To be effective, these feedback must be properly assimilated by the users; particularly they should not stress the user or initiate a casual behavior. To face that and considering that little studies have been done regarding the design of effective vibrotactile feedbacks that should be interpreted through the foot, we report here the design of a serious game dedicated to familiarization of the user with these icons.
In this game the goal of players is to browse a virtual maze as fast as possible and identify the level of risk associate with areas of this maze while maintaining a normal gait. For this, equipped with the augmented shoe, the player is invited to walk in a real room whose dimensions are mapped those of the virtual maze. This virtual scene is displayed on the screen of anAndroid tablet hand-held by the user (see Fig. 2). When walking, the player’s position in the virtual scene is updated according to his displacements in the real room.
A. System overview
One of the principal components of the system is the aug-mented shoe developed in our laboratory. This shoe contains several sensors and actuators as shown in Fig. 3. In the context of this game, sensors used are an accelerometer, force sensors and a bending variable resistor. The bending sensor measures deformations of the shoe sole during the propulsion phase of
Vibrat ing motor
Fig. 3. Location of the sensors and actuators in the sole of the augmented shoe prototype.
the gait whereas force sensors inform about distribution of the user’s weight. These sensors serve for the computation of the stance phase andswing phase of the gait. Based on the approach proposed in [22], the accelerometer inserted in the heel lets us to compute the position of the user in the real room. These positions are mapped to virtual maze in a way that emplacements of physical walls do correspond to external walls of the virtual world. This design choice has been made in order to insure that the user has the possibility to play the game in a safe environment without the presence of a third person. Moreover, three types of actuator are included in the shoe to transmit the vibrotactile feedbacks. They are: a vibrating motor (low and single frequency), piezoceramic discs: PZT (high and multi-frequencies) and a haptuator [23] which enables rich vibrotactile signals.
Another sensor used in the system is a portable wireless device that helps to measure the electric activity of the heart. This heart rate monitor is designed for cardiovascular training - running, jogging, or any kind of strenuous exercise. Its main component is a heart rate monitor strapped over the chest with bluetooth capabilities. This device reads and transmits the heart rate to the Androiddevice.
B. Playing of the game
When starting a session, several tactons are presented to the player. At this first step he has to select four of them that will be associated with the four levels of risk (low, medium, high and very high). This design choice is based on studies described in [24] where it has been showed that there was a strong, positive correlation between preference and successful identification of auditory notifications.
After the selection of the desired tactons, the user has to browse a virtual maze as fast as possible. When moving, for each area explored the corresponding signal is delivered to the user through the augmented shoes. At each time the user is prompted to identify the level of risk associated with this signal. At current level of the game ten types of maze are proposed with different level of complexity.
The complexity of the game could be described by two components: the maze it-self (number of paths, size of the maze and number of dead end) and the information transmitted
to the user (the tacton). At the current version of the game, we do have only rectangular mazes: proposed route are only vertical or horizontal. Each little rectangle described by the route defines a particular zone and is therefore associated with a particular tacton. As a result, in the current version of the game, the complexity is mainly determined by the complexity of the maze itself.
In the first maze, in order to strengthen the meaning attached to selected tactons, the risk level associated with a zone is directly proportional to the distance of this zone to the path leading to the exit. When the user chooses the wrong direction (a path with a dead end or longer than necessary), tactons representing higher risks are transmitted. Conversely, shorter paths are associated with safertactons. However, in the other nine mazes the distribution of the risk levels in the mazes is done randomly at each session. In this way, the player may experiment several times the same maze without having a common distribution of risk levels.
To maintain the engagement of the player in the game, we consider all the factors described previously to evalu-ate a score. The score displayed in real-time is computed proportionally to the number of right answers G about the risk level multiplied by the maze complexity Cand inversely proportional to the time t with a scale factork, as described by equation (1) :
Score=G∗C/(kt) (1) This score aims at evaluating the level of memorization of select tactons by the player. Moreover, considering that another purpose of the game is about the acceptance of renderedtactons(do they stress or initiate a casual behavior of the user), data recorded by the ECG and those concerning the gait of the player are also analyzed at the end of the session. From this analysis, prerecorded texts are displayed. These texts were proposed by a physiotherapist specialist. They are advises regarding the gait and the anxiety level.
C. Safety issue
One of the main advantages of using a serious game for learning vibrotactile messages resides in the fact that the game can be experimented at home without the presence of physiotherapist. However, this raises a number of problems related to players safety. For this, several design choices were made in order to guaranty the safety of the player. For example, previously we mentioned that the experimental room must be completely empty in order to avoid accidents. Furthermore, the dimensions of this room will be mapped to those of the virtual room to avoid players banging on the walls. Beyond these aspects, in order to prevent potential injuries, at the end of the analysis performed on the data logged from the ECG and those concerning the gait of the player, if a serious trouble is detected, the game is locked and invites the player to contact the costumer service because the game seems not be appropriate to his profile.
Tacton W1 Tacton W2 Tacton W3
Tacton W4 Tacton W5 Tacton W6
Identifier les quatre meilleurs Tactons. Appuyer sur Enter ensuite.
Fig. 4. Waveforms of proposed tactons.
D. Proposed tactons
This game uses four tactons for an association with a risk level of falling using these main parameters: main frequency at 65 Hz with the same rythm signal and a duration of 1 second. These tactons are shown in Fig. 4. They are designed according various studies of psychophysical perception [17] and [25]. The waveforms of each tactonare described by equation (2):
W1=asin(2π121t) W2=asin(2π60t) sin(2π121t) W3=asin(2π3t) sin(2π121t) W4=asin(2π31t) sin(2π53t) W5=asawtooth(2π80t) W6=ae− (x−b)2 2c2 , with t= [0 : 1/9600 : 1]sec. (2)
Based on the study described in [26], W1 defines a pure
sinusoidal wave, resulting in a continuous vibration of 121 Hz.
W2 is an amplitude modulation of W6 by a 60 Hz sinusoid
producing the sensation of a pulsing vibration. W3 is a
modulation ofW1by a 3 Hz sinusoid, producing the sensation
of rapid impulse vibration.W4is a 53 Hz sinusoid modulated
by a 31 Hz whose combination resulted in a somewhat rough vibration sensation. W6 is a Gaussian function where a is
the amplitude of the signal, e is the Euler number, b is the position of the center of the peak, andc adjust the bandwidth of the function. The tactons used for the current design already satisfy a little cognitive demand for the association between the tacton and a risk level. If a good mapping between a source of vibration and a source of data can be found, the meaning of a tactile icon should be easily learned and remembered.
V. PRELIMINARY STUDY
In the first step of, based on its preferences and ease of learning, the game the player has to select fourtactonsamong six others. In a preliminary experiment, we have evaluated whether users were able to memorize selected tactons. For this, this test analyzes performances regarding the association of a risk to a tacton while minimizing the cognitive demand for learning the relationship between the properties of selected vibration and a particular risk level.
A. Hypothesizes
Two hypothesis need to be evaluated. First, it appears that the augmented shoe can be used to warn the user about the presence of a risk in the environment. We want to measure the learning ability oftactonsbased on user preferences. Second, we hypothesis that sometactonscan be easily associated with a particular level of risk. To this end, this assessment proposes to study the association of a particular tacton to a risk level. B. Participants
A total of 10 participants (8 males and 2 females) aged between 19 and 32, were asked to participate. Among the sub-jects, 6 were undergraduate and graduate students in computer related field. Based on our pre-experimental questionnaire; 4 claimed being familiar with haptics.
C. Experimental tasks and procedure
For this experiment, in a first time participants have to select, among the sixtactonsproposed at subsection IV-D, four that will be associated with four level of risk. No time limit was imposed for this task. For this stage, we record selected tactons and their associated risk level as well as the time spent for the learning. Thereafter this step, in order to validate the association, users were invited to rank selected tactons according their respective associated level of risk. Here, we record if the participant manages to identify correctly the risk level associated to each selected tacton. Finally, ten tactons are alternately and randomly presented; the user is prompted to indicate the level of risk associated with each of them. At this stage, the participant has only one chance to indicate the correct level of risk associated with the tacton.
D. Results and Analysis
The results presented in Table I show that the process is quite fast with a mean time of 14 minutes and a standard deviation of 9 minutes. One observes that only three persons have spent over 20 minutes, whereas 6 were under 9 minutes. All the participants have enjoyed this test because it was fast and it required little cognitive demand. The mean score (recognition rate) of the final test is 5.9/10 with a standard deviation of 1.9. This observation underlies the idea that without any training, people are able to associate a tactonto a risk level with a score of at least50%.
Regarding the step of tactons selection, we observed that different choices have been made. 3 participant (30%) have eliminated tactons W2 and W5; whereas another 3 have
eliminated tactons W1 and W6. The four others have made
completely different and unique choices. These observations support the idea that various tactons should be proposed. This is in line with analysis made in [24]. Therefore, it seems that if a player chooses his tactons according to his preferences and his ability to differentiate them, the association should be easier to learn. This suggests that time spent to using the game will promote adequate learning of tactons. As a result, using the game should improve the recognition rate of messages associated with tactons. Therefore, this game can be used as an effective environment for learning the tactons.
TABLE I
RESULTS OF THE PERFORMANCE EVALUATION
User Eliminated Score Time Tactons (min.) 1 1 6 6 4.3 2 1 6 6 24.6 3 1 6 5 12.8 4 2 5 9 5.4 5 4 6 4 25.1 6 1 2 7 8.2 7 2 5 7 7.5 8 5 6 3 8.7 9 2 5 8 8.9 10 1 4 5 28.1 VI. CONCLUSION
In this paper we described the design of a serious for learning vibrotactile messages. In this game, in a first time players have to select, among six tactons, four that will be associated with four risk level. Thereafter, fitted with a pair of augmented shoes, the player is invited to explore a virtual maze by walking in a real room. During the displacement, for each area explored the corresponding signal is delivered to the user through the augmented shoes and each time the user is prompted to identify the level of risk associated with this signal. Currently, a first prototype of the game is completed. The idea that vibrotactile messages can be associated with a risk level has been validated through an initial experiment.
In a near future, a group a ten players (aged from 20 to 40) is expect to experiment the game on a daily basis. Thereafter, we will run a formal evaluation where this trained group will be compared to a control one. After this evaluation, and maybe eventual improvements of the game, evaluations with elderly will take place.
REFERENCES
[1] V. Scott, M. Pearce, and C. Pengelly, “Injury resulting from falls among canadians age 65 and over,” inReport on seniors falls in Canada. Public Health Agency of Canada, 2005, pp. 1–16.
[2] H. Wennberg, A. Stahl, and C. Hyden, “Older pedestrians’ perceptions of the outdoor environment in a year-round perspective,”European Journal of Ageing, vol. 6, no. 4, pp. 277–290, 2009.
[3] B. Sawyer, “The ”serious games” landscape,” inIn Proceedings of the Instructional & Research Technology Symposium for Arts, Humanities and Social Sciences, 2007.
[4] D. R. Michael and S. L. Chen, Serious games: Games that educate, train, and inform. Course Technology PTR, 2005.
[5] A. Stapleton, “Serious games: Serious opportunities,” inIn Proceedings of Australian Game Developers’ Conference, Academic Summit, 2004. [6] H. F. O’Neil, R. Wainess, and E. L. Baker, “Classification of learning
outcomes: Evidence from the computer games literaturend,”The Cur-riculum Journal, vol. 16, pp. 455–474, 2005.
[7] K. Squire, B. Barnett, J. M. Grant, and T. Higginbotham, “Electromag-netism supercharged! learning physics with digital simulation games,” in Proceedings of the international conference on Learning sciences, vol. 6, 2004, pp. 513–520.
[8] C. Dede, J. Clarke, D. Ketelhut, B. Nelson, and C. Bowman, “Students’ motivation and learning of science in a multi-user virtual environment,” inProceedings of AERA 2005, 2005.
[9] V. Srinivasan, K. Butler-Purry, and S. Pedersen, “Using video games to enhance learning in digital systems,” in Proceedings of the 2008 Conference on Future Play: Research, Play, Share, ser. Future Play ’08. New York, NY, USA: ACM, 2008, pp. 196–199.
[10] P. Wouters, E. van der Spek, and H. van Oostendorp, “Current practices in serious game research: A review from a learning outcomes perspec-tive,” inGames-Based Learning Advancements for Multisensory Human Computer Interfaces: Techniques and Effective, 2009, pp. 232–255. [11] K. Maclean and M. Enriquez, “Perceptual design of haptic icons,” inIn
Proceedings of Eurohaptics, 2003, pp. 351–363.
[12] M. J. Enriquez and K. E. MacLean, “The hapticon editor: A tool in support of haptic communication research,” in HAPTICS’ 03. Los Angeles, CA, USA: IEEE Computer Society, 2003, p. 356.
[13] S. Brewster and L. M. Brown, “Tactons: Structured tactile messages for non-visual information display,” inProceedings of the Fifth Conference on Australasian User Interface. Australian Computer Society, 2004, pp. 15–23.
[14] S. A. Brewster, P. C. Wright, and A. Edwards, “Evaluation of earcons for use in auditory human-computer interfaces,” inConference on Human Factors in Computing Systems, 1993, pp. 222–227.
[15] M. Enriquez, K. Maclean, and C. Chita, “Haptic phonemes: Basic build-ing blocks of haptic communication,” inProceedings of the International Conference on Multimodal Interfaces. ACM Press, 2006.
[16] E. Hoggan and S. Brewster, “New parameters for tacton design,” in
Human factors in computing systems. New York, NY, USA: ACM, 2007, pp. 2417–2422.
[17] Y. Visell, A. Law, and J. Cooperstock, “Touch is everywhere: Floor surfaces as ambient haptic interfaces,”IEEE Transactions on Haptics, vol. 2, no. 3, pp. 148–159, 2009.
[18] S. Okamoto, S. Ishikawa, H. Nagano, and Y. Yamada, “Spectrum-based vibrotactile footstep-display for crinkle of fragile structures,” inROBIO. IEEE, 2011, pp. 2459–2464.
[19] C. Mgard, M. Anastassova, and D. Repain, “Exploratory investigation of vibration floor as potential collision warning,” inHuman Modelling in Assisted Transportation, P. C. Cacciabue, M. Hjlmdahl, A. Luedtke, and C. Riccioli, Eds. Springer Milan, 2011, pp. 275–281.
[20] R. Vel˜azquez, O. Baz´an, and M. Maga˜na, “A shoe-integrated tactile display for directional navigation,” in Proceedings of the IEEE/RSJ international conference on Intelligent robots and systems, ser. IROS’09. Piscataway, NJ, USA: IEEE Press, 2009, pp. 1235–1240.
[21] A. A. Priplata, J. B. Niemi, J. D. Harry, L. A. Lipsitz, and J. J. Collins, “Vibrating insoles and balance control in elderly people.”Lance, vol. 362, no. 9390, pp. 1123–1124, 2003.
[22] C. Smith and H. I. Christensen, “Wiimote robot control using human motion models,” inProceedings of the IEEE/RSJ international confer-ence on Intelligent robots and systems, ser. IROS’09. Piscataway, NJ, USA: IEEE Press, 2009, pp. 5509–5515.
[23] H.-Y. Yao and V. Hayward, “Design and analysis of a recoil-type vibrotactile transducer,”Journal of the Acoustical Society of America, vol. 128, no. 2, pp. 619–627, 2010.
[24] S. Garzonis, S. Jones, T. Jay, and E. O’Neill, “Auditory icon and earcon mobile service notifications: Intuitiveness, learnability, memorability and preference,” inCHI2009: Proceedings of the 27th Annual CHI Confer-ence on Human Factors in Computing Systems, Vols 1-4, S. Greenberg, S. E. Hudson, K. Hinkley, M. RingelMorris, and D. R. Olsen, Eds. Association for Computing Machinery, 2009, pp. 1513–1522. [25] E. Hoggan and S. Brewster, “New parameters for tacton design,” in
Human Factors in Computing Systems, 2007, pp. 2417–2422. [26] B. M´en´elas, L. Picinalli, B. F. G. Katz, and P. Bourdot, “Audio haptic
feedbacks for an acquisition task in a multi-target context,” in3DUI. IEEE, 2010, pp. 51–54.
