GUI Input Interpreter

Since a concrete realisation of a GUI does not contain information about the semantic annotations in the GUI model, we need a component that complements the interpreta- tion of the syntactic representation of a user event with a semantic representation. The GUI input interpreter fulfills this task and is registered as a preprocessor for incoming GUI events. Together with the semantic annotations in the GUI model instance, which it retrieves from the display context manager, it reconstructs the annotations about the user’s intention. Listing 7.3 shows an example of the syntactic representation of a GUI input for the GUI model instance example in Figure 6.16. The GUI input interpreter preprocesses this message and gets the current display context for the device with ID car display from the display context manager. In the next step, it searches for the Ele- ment with ID “cinemaList”, which besides a semantic annotation of the ChangeEvent, contains a list of items with Cinema entities attached as semantic content. In a last step it adds the communicative function, the ChangeEvent is annotated with, as a hypoth- esis to the InputAct. The containing GuiContentReference refers to an entity of type Cinema. This is taken from the ListItem with index 0 and added to the communicative function. The result is shown in listing 7.4.

<i o : I n p u t A c t>

<r e p r e s e n t a t i o n d e v i c e=” c a r d i s p l a y ” m o d a l i t y=” \ g l s { g u i } ” x s i : t y p e=” g u i : \ g l s { g u i } Event ”>

<e v e n t D a t a i n d e x=” 0 ” t a r g e t I d=” c i n e m a L i s t ” v a l u e=” C i n e s t a r ” x s i : t y p e=” g u i e v e n t s : C h a n g e E v e n t ” />

</ r e p r e s e n t a t i o n> </ i o : I n p u t A c t>

Listing 7.3 – Example for a GUI input event

<i o : I n p u t A c t>

<r e p r e s e n t a t i o n . . .

<h y p o t h e s e s c o n f i d e n c e=” 1 . 0 ”>

<c o m m u n i c a t i v e F u n c t i o n x s i : t y p e=” c o m m u n i c a t i v e f u n c t i o n s : I n f o r m ”> <s e m a n t i c C o n t e n t r e s o l v e d=” t r u e ” v a l i d i t y=”−1”>

<c o n t e n t name=” C i n e s t a r ” x s i : t y p e=” sab:Cinema ” /> < r e f e r e n c e x s i : t y p e=” r e f e r e n c e s : G u i C o n t e n t R e f e r e n c e ”> <r e f e r e n c e P a t t e r n i n s t I d x=”−1”> <t y p e h r e f=” h t t p : //www. d f k i . de / i u i /mmds/ sab demo / model#//Cinema ” /> </ r e f e r e n c e P a t t e r n> </ r e f e r e n c e> </ s e m a n t i c C o n t e n t> </ c o m m u n i c a t i v e F u n c t i o n> </ h y p o t h e s e s> </ i o : I n p u t A c t>

7.9 Fusion & Discourse Resolution

The SiAM-dp platform contains a component which can be used for handling multi- modal and dialogue phenomena, the Fusion and Discourse Engine (FADE). Our FADE component is inspired by the work of Pfleger (2007) and supports the core function- alities described by him. Section 5.5 already describes how his ontological modelling approaches for dialogue acts and referring expressions have been adopted to the general modelling environment of the SiAM platform.

FADE provides generic and reusable implementations for the resolution of missing con- tent that arises from multimodal integration and phenomena in dialogue. It monitors the incoming and outgoing dialogue acts during discourse and becomes active in three situations:

1. An input act contains unresolved semantic content. 2. An input act is marked as ellipsis.

3. An output act contains a referring expression to the long-term memory.

Furthermore, it is responsible for updating the discourse context with the monitored information and works closely together with the discourse manager. Hence, in the next subsection the internal representation of data in the discourse context is described.

7.9.1 Managing the Discourse Context

The discourse context provides the required information for resolving context dependent phenomena in the FADE component. Furthermore, it can be used by other components for arbitrary analyses on the discourse since it keeps track of the complete dialogue his- tory. Internally, three different types of data are managed in their own data structures:

Discourse History: In the discourse history the complete communication between the dialogue platform and the CPE is logged. Every input and output act is saved in a list ordered by the time point when the action occurred.

Long-Term Memory: According to Pfleger (2007) the Long-Term Memory (LTM) is used for representing the knowledge about the world independently from the knowledge that has implicitly been mentioned during the discourse. This has two reasons: First, it is possible to reference to entities in the world that have never been mentioned during the previous discourse. Second, it serves as source for the real world context since content that is introduced by discourse not necessarily de- scribes the real world state but rather the subjective point of view of the user. The long-term memory is not implicitly modified by the FADE component. Moreover, entities must be explicitly modified by, e.g., external services that collect environ- ment information or the dialogue management. For this, every entity in the LTM is

7.9 Fusion & Discourse Resolution 171

Figure 7.10 – Structure of the working memory. The resolved semantic content entries of every communicative act are saved in an own entry. Newer contributions are saved on a higher level. The figure shows how the working memory is changed if new content is introduced into the system that updates the actual knowledge about Entity 4. Since the temporal difference between both working memory states is greater than 500 ms, the validity of Entity 5 is expired and the entity is removed.

addressable by a unique identifier. In SiAM-dp the Knowledge Manager is respon- sible for controlling the access to the LTM.

Working Memory: The Working Memory (WM) holds the entities of the semantic content that is transported within a communicative act. Only resolved content is added to this memory since a unresolved or ambiguous content does not provide additional contributions to the discourse. For every communicative act that contains resolved semantic content, a new entry in the working memory is created. The entries are organised in a list arranged by the time-point of content contribution. Semantic content contributions can manually be annotated with an expiration time by the property validity. If the assigned time span is elapsed, the semantic content is removed from the working memory.

Figure 7.10 shows how the WM is updated. The WM state at time point t0 contains

content from four different contributions. Entity 2 and Entity 3 have been jointly contributed by one communicative act. Entity 5 is annotated with a validity of 500 milliseconds. We assume that a new communicative act, which occurs 500 milliseconds or later after time-point t0, provides new semantic content that extends

the knowledge about Entity 4. For this, FADE searches the WM for entities that are unifiable with the new entity. If this happens, the old entity is updated with the new content by unification and pushed onto the top level of the memory. Since the validity of Entity 5 is expired, this entity is removed from the WM.