Implementation

Our application ran on the Horde3D GameEngine2_{. Additionally, we used}

SceneMaker 3 [124] to model and execute the story as a hierarchical finite state machine extended with multimodal scene scripts. The scene scripts consisted of the text to be spoken and embedded additional commands such as the playback of animations or sounds.

Unlike the book, our application was designed for two users listening to a virtual narrator and interacting at specific points to influence the story. As in the book, the users had to decide on how to go on at specific points in the story as described in the preceding section. We integrated those decisions in a Wizard-of-Oz design with speech commands and evaluated it in a user study. As this does not concern the topic of this dissertation, I will omit this part here, however the reader can look it up in the original publication [98]. Nevertheless, a comparison of speech and gesture input with real-time recognition will follow in Section 3.2.

As a second type of interaction, we added so called quick time events (QTEs) that are frequently used in current video games. To our knowl- edge, there had been no scientific studies about the application of QTEs,

1_{Re-translated to English from the German version [137]} 2

until the publication of this work [98] . However, good examples of games, which make extensive use of QTEs were Fahrenheit (Indigo Prophecy) and Heavy Rain from the games developer Quantic Dream3_{. Whenever a QTE}

occured in a video game, a symbol representing a specific action on the control device appeared on screen. The user then had a limited amount of time to perform that action in order to complete the QTE successfully. Most times, successful performance of the QTE resulted in a particular action by the player avatar, while unsuccessful performance caused the player avatar to fail in this action. In general, the utilization of QTEs can range from enriching cutscenes with interactivity to using QTEs as the main game- play mechanic. In opposite to the video game examples, we did not use traditional control devices, but full body interactions.

Some passages of the book already contained situations that were well- suited for the application of QTEs. One example in the original text (p. 13)4 _{read as follows:}

You start to climb the steep hill. It is highly exhausting and one time you loose your grip and almost fall down a rock face. But finally you arrive at the top.

The modified text part was (modifications marked with bold font):

You start to climb the steep hill. It is highly exhausting and one time you almost loose your grip.

The QTE started immediately after this text, and when it was solved, the following message was narrated:

You manage to hold on just in time and finally you arrive at the top..

Otherwise, the text was:

You fall down a rock face but you are lucky that you did not get hurt too badly.

If the QTE was solved, the story continued as in the original version (here: page 20), but if not, it jumped to a different, but appropriate, page of the story (here: page 14).

As soon as a QTE started in our application, a countdown appeared

3

http://www.quanticdream.com(accessed 2015-9-15)

centrally in the upper part of the screen with a symbol shown for each user representing the action requested (see Figure 3.1). At the moment one user solved a QTE, a mark was shown on top of the symbol, providing immediate feedback. The full QTE was solved if both users successfully completed their action before the countdown reached zero.

Figure 3.1: Screenshot of the quick time even (QTE) “Run” in Sugarcane Island

We employed two different modes to carry out of QTEs, both using full body interaction: In the first mode (i.e. the button mode), each user had to press a randomly-positioned and -sized button on screen, using a cursor controlled by moving the hand. In the other mode (i.e. the gesture mode), users needed to perform gestures that were indicated on screen via one of the symbols shown in Figure 3.2.

Figure 3.2: QTE symbols in Sugarcane Island

The button mode was implemented as freehand GUI interaction as de- scribed in Section 5.3. The user had two ways to select a button: by hovering a button for more than 1.5 seconds with the cursor, or by per- forming a push gesture in direction of the screen. As soon as the QTE started, a button containing text for the requested action was shown for

each user. The buttons were positioned randomly: on the right half of the screen for the right user, and on the left half for the left user. In addition, they randomly had a slightly different size for each QTE. Once a button was activated, it disappeared and a tick appeared to inform the user about the successful action. Overall, the button mode represented the action of the QTE in an abstract way. The meaning was provided by the text string, but the actual interaction task depended on random parameters only.

In the gesture mode, the requested gestures represented the QTE ac- tions more directly. Figure 3.2 displayed the symbols used to visualize the requested user actions (from left to right):

Balance: Hold hands out at shoulder-height; Kick : Perform a kick with one leg; Catch: Put the hands together in front of the body; Climb: Move hands up and down in front of the head, as if climbing; Left and right Hand : Raise the left or right hand; Run: Move the feet up and down like running, but without moving.

Figure 3.3: QTE gestures performed by two users in Sugarcane Island

Figure 3.3 shows two users in front of a screen, performing the QTE

centrally below the screen. The left user has already succeeded in the QTE “Kick”, so a green tick has appeared over the corresponding symbol.

The recognition for the gestures shown in Figure 3.2 was implemented according to Section 5.4 using an early version of the FUBI framework. For example, “left hand” was implemented using the position of the left hand relative to the left shoulder and testing whether the left hand was currently above the shoulder (y-coordinate of the hand greater than y- coordinate of the shoulder). Other more complex gestures were modeled as simple finite state machines concatenating postures to sequences with specific time constraints in an early version of the combinations described in Section 5.4.