Foundational Work

3.2 Foundational Work

In traditional architectures for dialog systems, components are structured in a pipelined manner. A speech recognizer recognizes the words the user said, a natural language understanding component assigns a meaning to these words, the dialog manager then determines how utterances fit into the dialog context so far and decides the next system action (e.g. clarifying, confirming, asking for further information, or making a lookup in an information source like a timetable or a flight schedule), and a language generation component chooses the words and phrases used in the response, which is then verbalized by the text-to-speech synthesis [McT04]. Pipeline architectures can typically be found in information access domains, for example in the Ariadne system [Den02], in the LIMSI system [LRGB99], or in the MALIN dialog system, which is shown in figure 3.1. In such kinds of architectures, the dialog manager tends to be a monolithic component that controls everything in the system [OM00]. This makes it hard to build modular systems that comply with the requirements of robotics, such as real-world integration, concurrent activities and mixed initiative. Also for building more advanced conversational systems that approach human-like behavior and account for phenomena such as back-channeling, turn-taking, grounding, interruptions and incremental processing, this type of architecture is not flexible enough [All01].

Dialog manager Interpreter Generator Dialog history Domain knowledge manager Domain task model Knowledge

module 1 Knowledgemodule n

System task model Dialog

model

Figure 3.1: The MALIN dialog system architecture: a typical pipeline architecture (af-

ter [EJ00]).

Accordingly, research in the spoken dialog system community has been shifting towards more modular, distributed architectures, where dialog management is divided into smaller components. Nowadays, most architectures follow a client-server model with a central component that regulates the interaction between the other components. The logical information flow still mostly follows a strictly sequential pipeline, though. Frameworks and infrastructures for distributed dialog system architectures have been developed, which

entails the question of inter-component communication and component interfaces. In the dialog community, this issue has been addressed in two different ways: components communicate either through user-defined interfaces, or through standardized interfaces based on agent communication languages.

HUB RAVENCLAW Dialog Manager ROSETTA Language Generation KALLIOPE Speech Synthesis TTYServer Text I/O SPHINX Speech Recognition PHOENIX Parsing Application Back-end

Figure 3.2: The RavenClaw dialog system architecture (after [BR09]), which relies on the

Galaxy-II infrastructure. The lines denote the communication connections, whereas the dashed arrows denote the logical information flow. Some components (such as the Date- Time server) have been skipped for the sake of clarity.

The most prominent example of the first approach is the Galaxy-II Communicator architec- ture, a distributed hub-and-spoke infrastructure for constructing dialog systems [SHL+_98].

The RavenClaw framework, which was introduced in section 2.1.2, is based on the Galaxy-II infrastructure. Figure 3.2 shows the architecture of the RavenClaw dialog framework. In this client-server architecture, components are connected via a central hub component and communicate through frame-based messages. The hub component handles all communica- tion between the components, based on sequential rules for routing messages, specifying basically the operation to perform and the resulting input and output variables. Figure 3.3 shows a result frame from the Curious Robot case-study described in section 4, where the Curious Robot scenario was re-implemented based different dialog frameworks, including

U s e r Q u e r y : { t y p e : g r i p q u e r y l a b e l : a p p l e g r i p : p o w e r _ g r i p t a s k s t a t e : c o m p l e t e d }

Figure 3.3: A Ravenclaw result frame from the Curious Robot case study, in which a grip

3.2 Foundational Work 35

RavenClaw, which relies on Galaxy-II.

From robotics point of view, the Galaxy-II architecture has two drawbacks. First, the semantics of the frames is not specified, i.e. it needs to be defined by the respective component developer. This means more freedom, but also more effort for the developers. Particularly in robotics, where the application back-end consists not of a single database, but of several cooperating components that are implemented by different developers, a common standard would facilitate their integration with the dialog system. Second, the Galaxy-II framework does provide both blocking and non-blocking function calls, but it lacks provisions for monitoring the execution of non-blocking operations. This makes it hard to implement concurrent activities with it. These drawbacks are not specifics of the Galaxy-II framework, but are widely found in similar architectures for dialog systems, e.g. in the CSLU toolkit [SCd+_{98], the WIT dialog framework [NMY}+_{00], and the iHUB}

integration framework [RS05].

The above problems were further investigated in the Curious Robot case study. Addi- tionally, several other drawbacks were identified in this case study. For example, as the logical information flow follows a pipeline (cf. figure 3.2) where the back-end actions are triggered by the dialog system, it is hard to realize system task initiative and to integrate real-world events. Also, RavenClaw does not support reusability of interaction episodes. These drawbacks however are rather related to the RavenClaw dialog manager than to the Galaxy-II architecture itself.

OAA2 Facilitator Festival TTS Agent GUI Interactive Map Display Robot

Control and Report Interface Dialog Manager Gemini NL Agent Nuance SR Agent

WITAS UAV

An example of the second approach, where communication between components is based on agent communication languages, is the Open Agent Architecture [MCM99]. The OAA has been designed for distributed agent systems in general, but provides several agents specifically for the purpose of building dialog systems, e.g. for speech recognition,

natural language parsing and generation, text-to-speech synthesis and database queries. Several dialog frameworks rely on it, for instance TrindiKit [TL03], Dipper [BKLO03] and WITAS [LBGP01], whose architecture is shown in figure 3.4.

The OAA is a framework for integrating software agents in a distributed environment. In the context of the OAA, an agent is defined as a software process that provides services or, in the OAA terminology, capabilities to other agents, registers these with the framework and is able to “speak” the Interagent Communication Language (ICL). The ICL is a logic- based language similar to Prolog, used to perform queries, execute actions, or exchange information. ICL messages have the form solvable(Goal, P arameters, P ermissions). The solvable, which corresponds to an agent capability, specifies what action should be performed. The facilitator forwards it to the agent which is able to solve it. The responsible agent will then try to solve the given goal by unifying the variables contained in the goal expression. For example, in the case study described in section 4, the goal listObjects(Result) was used to request a list of all visible objects from a vision agent. In its response, the vision agent binds the list of object names to the variable Result, e.g. listObjects([apple, [banana, lemon]]). Besides the goal itself, a message may contain additional control parameters (P arameters and P ermissions) that specify how the task should be routed, or which agent should perform it.

As for asynchronous processing, the OAA provides more flexibility than the Galaxy-II architecture. All agents are independent processes that run asynchronously and may be written in different programming languages. The OAA can handle asynchronous delivery and triggers, which enables the dialog system to keep track of multiple parallel activities and to react to real-world events. However, similar as with the Galaxy-II architecture, the OAA does not provide a clearly defined component interface, but leaves specification of component communication to the individual developers.

The influential TRIPS architecture, which was introduced in section 2.1.3, uses an alter- native approach for communication between different agents. It makes use of the agent communication language KQML, which is based on ideas from speech act theory [FFMM94]. KQML defines a fixed set of performatives, that correspond to the illocutionary force of a speech act in natural language. For example, ask-all is a directive performative meaning that “A wants to know all B’s responses that would make the content true of B”, and tell

t e l l : l a n g u a g e P r o l o g : o n t o l o g y o b j e c t s : in - reply - to q1 : s e n d e r V i s i o n : r e c e i v e r D i a l o g : c o n t e n t ’ ’ l i s t O b j e c t s (([ apple , [ banana , l e m o n ]])) ’ ’)

Figure 3.5: Example KQML message from a vision component to the dialog component,

3.2 Foundational Work 37

is an assertive performative meaning that “A states to B that A believes the content to be true” [LF94]. Figure 3.5 shows the above mentioned example message from a vision agent that knows about present objects in the KQML message format.

Like other agent communication languages, KQML adopts a cognitive-state approach to communication, which assumes that a conversation emerges from the agents’ beliefs and intentions. Accordingly, while syntax and semantics of single performatives are detailed in technical specifications, no explicit agreements have been formulated that regulate the sequences in which messages may occur. The correct interpretation of a message, and a meaningful reaction to it, is left to the individual agents. However, as the semantics of the performatives is specified in natural language, the semantics may be interpreted slightly different by different developers. Also, the semantics of a message may depend on the context. For example, an inform performative should sometimes be interpreted as a suggestion but in another context, as a command [EH99]. Thus, interoperability of agents can not be guaranteed.

This shortcoming has long been recognized in the intelligent agent community. As a con- sequence, conversation policies have been suggested that constrain the messages that are sent by specifying shared conventions about what messages may follow each other1_{, either}

in form of graphs (e.g. [LF94, Par96, Kön09]), or in form of constraints to the messages (e.g. [MB95, EH99, HHB99]). There is no consensus on the appropriate form of conversa-

tional policies, let alone their semantics [HHB99], and they have not been included in the specifications of the agent communication languages. Those dialog management approaches that base communication on agent communication languages, such as TRIPS, have not adopted the concept of conversation policies, but rely on more specific approaches for the interpretation of agent messages, e.g. based on Allen’s problem solving model [ABF02]. To sum up, in robotics the issue of inter-component communication – and in particular the communication with the back-end – is crucial for the integration of a dialog system with the robotic subsystem, as detailed in section 3.1. In the spoken dialog community, on the other hand, this issue has often been neglected. Asynchronous coordination is provided by some, but not all, architectural frameworks for dialog systems. Dialog systems rarely provide well-defined component interfaces, but often rely on component-specific solutions, e.g. based on user-defined result frames. Alternatively, communication between components in distributed dialog system architectures is often based on agent communication languages, which do specify a message format but lack a definition of valid message sequences. Thus, from robotics point of view, existing dialog systems lack concepts for clearly defined interfaces to the domain level that allows for a detailed information exchange between the dialog system and the robotic subsystem.

1 The concept of conversation policies has influenced my idea of Interaction Patterns (cf. Chapter 3.4.4) that determine which dialog acts may follow each other.