Evaluation

The evaluations and analyses conducted within the scope of this thesis aimed to consider as many aspects as possible. Therefore, not one single study was conducted, but several kinds of studies and evaluation methods were combined. They can be divided into two groups: developer-centered evaluations, i.e. the evaluation of the dialog framework, and the end-user-centered evaluations, i.e. the evaluation of the HRI scenarios in which the dialog framework was deployed. Table 2.3 lists the different aspects that were evaluated. One aspect of the developer-centered evaluation is the usability of the framework. This evaluation provides information about the ease with which developers can work with the framework, by observing them performing a given task. It also gives information about

2.4 The Adopted Approach 29

Field Aspect Study type Method Chapter

Developer- centered

Usability Usability test Descriptive statistics, observation

4.1

Efficacy Comparative case study

Observation 4 End-user-centered Does robot

initiative facilitate learning? Explorative test run Video analysis 6.1.3 Does robot initiative facilitate interaction? Explorative video study Descriptive statistics 6.2.3 Speech understanding performance Explorative user study Descriptive statistics 6.3.2 User strategies in object teaching

WOz study Qualitative analysis 7.1.2 Speech recognition performance Explorative test run Descriptive statistics 7.1.3 Factors relevant to user satisfaction

User study PARADISE approach

7.4, 7.4.2 Influence of robot

initiative

User study Descriptive statistics

7.4, 7.4.2 Recurring

phenomena

User study Qualitative analysis

7.4, 7.4.2

Table 2.3: Aspects of dialog modeling evaluated in this thesis.

possible drawbacks in the API. Another aspect is framework efficacy, i.e. the question how well suited the framework is for its intended purpose. This question is investigated by analyzing a comparative case study, wherein a given target scenario was reimplemented by using different state-of-the-art dialog frameworks.

The end-user centered evaluation address different levels: (i) initially, new dialog strategies were explored, (ii) concrete system parameters were optimized during an iterative design process, and (iii) a user-study was conducted that promises generalizable results. On the dialog strategy level, the focus was placed on exploring mixed-initiative interaction strategies. Two facets of mixed initiative were investigated: can it facilitate learning, and can it facilitate interaction? The first aspect was evaluated by analyzing an explorative test run, the latter by analyzing a video study.

During the iterative design process of our main scenario, the Curious Flobi, several parameters were optimized. First, the speech understanding performance of the predecessor scenario was measured by means of a user study. This analysis helped to derive

strategies how to reduce speech understanding problems in future iterations. Second, a Wizard-of-Oz (WOz) study on object teaching was analysed in order to gain an impression of how users demonstrate objects to a robot and of the interaction strategies they use. This analysis formed the basis for the design of the speech recognition grammar and the dialog system configuration. To exclude that the resulting speech recognition grammar is too complex, which would reduce recognition accuracy, a pre-test of speech recognition performancewas conducted.

With the result, a large-scale user study became possible. The study followed an integrated approach, focusing not on single aspects of the system, but rather on the overall system. One objective of the study was to identify the factors that contribute to user satisfaction. This was evaluated by means of a PARADISE-style regression analysis. Further, the mixed- initiative dialog strategy was evaluated in this scenario, too. As a three-level between- subjects factor, the degree of robot initiative was varied. Moreover, a qualitative analysis revealed interaction phenomena, whose causes were then traced back to the responsible system components in order to identify possible solutions.

3 Specifying the External Interface: The Task State

Protocol

This chapter takes an external view on the suggested approach and focuses on the external interface of the dialog framework and its role within the robot architecture. Section 3.1 states the importance of a well-defined component interface and identifies its requirements, which partly result from the general requirements of HRI as discussed in section 2.2.1. Architectures for dialog systems will be discussed in section 3.2, with particular focus on their integration with the domain back-end. Section 3.3 presents an approach for a component interface that relies on the concept of tasks and their states. Section 3.4 illustrates its benefit for HRI and describes how it supports crucial features of HRI, such as mixed task initiative, tight integration of action execution and interaction, multitasking and interactive learning.

3.1 Motivation

A dialog system for human-robot interaction is not a stand-alone component, but em- bedded in the robot control architecture that consists of several diverse components and may be highly complex. The components of the architecture have to communicate with each other to cooperate. In this process, the dialog system plays a key role: it can be seen as a mediator between the user and the robot’s functionalities. For instance, if the human gives an instruction, the dialog system will ensure that it is translated into the corresponding system command so that it can be handled by the responsible component. Conversely, the dialog system will keep the human informed about the current system state. Out of the characteristics of human-robot-interaction that were discussed in section 2.2.1, several requirements for the communication between the dialog system and the robot subsystem arise. First, because of the situatedness of human-robot interaction, on-going activities and visible objects will be the topic of conversation. Thus, to achieve real-world integration, the communication needs to support close collaboration between the dialog system and the respective robot components to integrate their results into the interaction. Second, human-robot interaction requires the coordination of joint activities, and there may be several concurrent activities. Also, they may be longer running and need to be monitored over time. On the other hand, due to the dynamics of the environment, situation changes may happen at any time, and require human intervention, or at least notification. This means that the coordination of the activities needs to be asynchronous, i.e. the dialog

system does not wait until a specific activity is finished, but continues computation and still remains responsive to status updates of ongoing activities. Third, human-robot interaction involves a mixed-initiative interaction style. For the communication between components, this means that the communication flow needs to be bidirectional. Interaction episodes will be initiated not only by the dialog system, upon a human’s initiative, but also by other system components, if human intervention or clarification is required. Accordingly, in order to equip a robot with the ability to take task initiative autonomously, system activities will not only be triggered by the dialog system, on the human’s instruction, but may be triggered by other system components as well. As a consequence, the responsibility for the interaction flow will not lie exclusively with the dialog system any more (cf. 3.8.2 for further discussion).

The way a component interacts with other components, i.e. its external interface, greatly affects its degree of integration with the rest of the system, and the effort integration takes. A design principle to achieve good interoperability is a contractually defined component interface that specifies the structure and semantics of the information exchanged between components, as well as an agreement how to use it [BS09]. This chapter will focus more on the information that is exchanged, while the following chapter will detail how the information is used, i.e. how the dialog manager internally takes decisions based on it. This chapter will present the Task State protocol: an approach for a well-defined dialog system interface that i) meets the above described requirements for communication between the dialog system and the robotic subsystem and ii) allows the dialog system to treat all components in a uniform manner, instead of relying on individual back-end interfaces to the different components.

The Task State protocol, or Task State communication pattern, has been developed and investigated within several projects, and several colleagues have contributed to it, namely the Home-Tour team (consisting of Marc Hanheide, Frederic Siepmann, Torsten Spexard and myself) and the Curious Robot team (consisting of Ingo Lütkebohle and Robert Haschke and myself). A very first draft was used in the Home-Tour scenario (see chapter 6.1), for the communication between the then used Sunshine Dialog system (see section 5.1) and a room classification component. Later, within the Curious Robot scenario (see chapter 6.2), more use cases were identified, and the communication pattern was further refined. Lütkebohle investigated its role as a general coordination pattern and provided a toolkit that supports request, monitoring and revision of tasks [LPP+_{11]. On}

the way towards concepts for a reusable dialog framework, this motivated me to rely on the Task State protocol as the exclusive interface to the dialog framework.