• No results found

Robust Interaction through Partial Interpretation and Dialogue Management

N/A
N/A
Protected

Academic year: 2020

Share "Robust Interaction through Partial Interpretation and Dialogue Management"

Copied!
5
0
0

Loading.... (view fulltext now)

Full text

(1)

Robust Interaction through Partial Interpretation and Dialogue

Management

A r n e J S n s s o n a n d L e n a S t r S m b ~ i c k * D e p a r t m e n t o f C o m p u t e r a n d I n f o r m a t i o n Science L i n k S p i n g University, S - 58183 L i n k S p i n g , S w e d e n

email: arj@ida.liu.se lestr@ida.liu.se

A b s t r a c t

In this paper we present results on developing ro- bust natural language interfaces by combining shal- low and partial interpretation with dialogue manage- ment. The key issue is to reduce the effort needed to adapt the knowledge sources for parsing and in-

terpretation to a necessary minimum. In the paper

we identify different types of information and present corresponding computational models. The approach utilizes an automatically generated lexicon which is updated with information from a corpus of simulat- ed dialogues. The grammar is developed manually from the same knowledge sources. We also present results from evaluations that support the approach.

1 I n t r o d u c t i o n

Relying on a traditional deep and complete analysis of the utterances in a natural lan- guage interface requires much effort on building g r a m m a r s and lexicons for each domain. An- alyzing a whole utterance also gives problems with robustness, since the g r a m m a r s need to cope with all possible variations of an utter- ance. I n this paper we present results on devel- oping knowledge-based n a t u r a l language inter- faces for information retrieval applications uti- lizing shallow and partial interpretation. Simi- lar approaches are proposed in, for instance, the work on flexible parsing (Carbonell a n d Hayes, 1987) a n d in speech systems (cf. (Sj51ander a n d Gustafson, 1997; Bennacef et al., 1994)). T h e interpretation is driven by the information needed by the background system a n d guided by expectations from a dialogue manager.

T h e analysis is done by parsing as small parts of t h e utterance as possible. T h e infor- m a t i o n needed by t h e interpretation module, i.e. g r a m m a r and lexicon, is derived from t h e database of the background system and infor- m a t i o n from dialogues collected in Wizard of

" A u t h o r s a r e i n a l p h a b e t i c a l o r d e r

Oz-experiments. We will present what types of information t h a t are needed for the interpreta- tion modules. We will also report on the sizes of the g r a m m a r s and lexicon and results from applying t h e approach to information retrieval systems.

2 D i a l o g u e m a n a g e m e n t

Partial interpretation is particularly well-suited for dialogue systems, as we can utilize informa- tion from a dialogue manager on what is ex- pected and use this to guide the analysis. Fur- thermore, dialogue m a n a g e m e n t allows for focus tracking as well as clarification subdialogues to further improve the interaction (JSnsson, 1997). In information retrieval systems a c o m m o n user initiative is a request for domain concept information from the database; users specify a database object, or a set of objects, a n d ask for the value of a property of that object or set of objects. In t h e dialogue model this can be modeled in two focal parameters: Objects relat- ed to database objects a n d Properties modeling the d o m a i n concept information. T h e Proper- ties p a r a m e t e r models the d o m a i n concept in a sub-parameter t e r m e d Aspect which can be specified in a n o t h e r sub-parameter t e r m e d Val- ue. T h e specification of these parameters in t u r n d e p e n d s on information from the user ini- tiative together with context information a n d the answer from t h e database system. T h e ac- tion to be carried o u t by t h e interface for task- related questions depends on t h e specification of values passed to t h e Objects a n d Properties parameters (JSnsson, 1997).

(2)

3 T y p e s o f i n f o r m a t i o n

We can identify different types of information utilized when interpreting an utterance in a n a t u r a l language interface to a database sys- tem. This information corresponds to the in- formation t h a t needs to be analyzed in user- utterances.

D o m a i n c o n c e p t s are concepts about which t h e system has information, mainly concepts from the database, T, b u t also synonyms to such concepts acquired, for instance, from the infor- m a t i o n base describing the system, S.

In a database query system users also often request information by relating concepts a n d objects, e.g. which one is the cheapest. We call this type of language constructions relation- al e~pressions. T h e relational expressions can be identified from the corpus.

A n o t h e r c o m m o n type of expressions are numbers. Numbers can occur in various forms, such as dates, object and property values.

S e t o p e r a t i o n s . It is necessary to distinguish utterances such as: show all cars costing less than 70 000 from which of these costs less than 70 000. T h e former should get all cars costing less t h a n 70 000 whereas the latter should uti- lize the set of cars recorded as Objects by t h e dialogue manager. In some cases t h e user uses expressions such as remove all cars more expen- sire than 70 000, and thus is restricting a set by mentioning the objects t h a t should be removed.

I n t e r a c t i o n a l c o n c e p t s . This class of con- cepts consists of words a n d phrases t h a t concern the interaction such as Yes, No, etc (cf. (Byron a n d Heeman, 1997)).

T a s k / S y s t e m e x p r e s s i o n s . Users can use do- m a i n concepts such as explain, indicating t h a t the d o m a i n concept is not referring to a request for information from the database, T, b u t in- stead from the system description, S.

W h e n acquiring information for t h e interpreter, three different sources of information can be uti- lized: 1) background system information, i.e. the database, T, and the information describ- ing the background system's capabilities, S, 2) information from dialogues collected with users of t h e system, and 3) c o m m o n sense and prior

knowledge on h u m a n - c o m p u t e r interaction and natural language dialogue. T h e various infor- m a t i o n sources can be used for different pur- poses (JSnsson, 1993).

4 T h e i n t e r p r e t a t i o n m o d u l e

T h e approach we are investigating relies on an- alyzing as small and crucial parts of the ut- terances as possible. One of the key issues is to find these parts. In some cases an analy- sis could consist of one single domain or inter- actional concept, b u t for most cases we need to analyze small sub-phrases of an utterance to get a more reliable analysis. This requires flex- ibility in processing of the utterances and is a further development of the ideas described in StrSmb~ick (1994). In this work we have cho- sen to use PATR-II b u t in t h e future construc- tions from a more expressive formalism such as E F L U F (StrSmb~ck, 1997) could be needed.

Flexibility in processing is achieved by one ex- tension to ordinary P A T R a n d some additions to a chart parser environment. Our version of P A T R allows for a set of u n k n o w n words with- in phrases. This gives general g r a m m a r rules, and helps avoiding the analysis to be stuck in case of u n k n o w n words. In the chart parsing environment it is possible to define which of the inactive edges t h a t constitute the result.

T h e g r a m m a r is divided into five g r a m m a r modules where each m o d u l e corresponds to some information requested by the dialogue manager. T h e modules can be used indepen- dently from each other.

D o m a i n c o n c e p t s are captured using two g r a m m a r modules. T h e task of these g r a m m a r s is to find keywords or sub-phrases in the expres- sions t h a t correspond to the objects and prop- erties in t h e database. T h e properties can be concept keywords or relational expressions con- taining concept keywords. Numbers are typed according to the property they describe, e.g. 40000 denote a price.

To simplify the g r a m m a r s we only require t h a t the g r a m m a r recognizes all objects and properties mentioned. T h e results of the analyses are filtered t h r o u g h t h e heuristics t h a t only the most specific objects are presented to the dialogue manager.

(3)

provides a marker to tell the dialogue man- ager what type of set operation the initiative requests, new, old or restrict. T h e user's utterance is searched for indicators of any of these three set operators. If no indicators are found we will assume t h a t the operator is old. T h e chart is searched for t h e first and largest phrase that indicates a set operator.

Recognizing

interactional

u t t e r a n c e s .

Many interactional utterances are not nec- essary to interpret for information retrieval systems, such as Thank you. However, Yes/No- expressions are i m p o r t a n t . T h e y can be recognized by looking for one of the keywords yes or no. One example of this is the utterance No, j u s t the medium sized cars as an answer to if the user wants to see all cars in a large table. T h e Yes/No-grammar can conclude t h a t it is a no answer and the property g r a m m a r will recognize the phrase medium sized cars.

S y s t e m / T a s k r e c o g n i t i o n . Utterances asking for information a b o u t a concept, e.g. Explain the numbers f o r rust, can be distin- guished from utterances requesting information acquired from the background system How rust prone are these cars by defining keywords with a special meaning, such as explain. If any of these keywords are found in an utterance the dialogue manager will interpret the question as system-related. If not it will assume t h a t the question is task-related.

5 A n e x a m p l e

To illustrate the behaviour of the system con- sider an utterance such as show cars costing less than 100000 crowns. T h e word cars indicates t h a t the set operator is new. T h e relational expression will be interpreted by the g r a m m a r rules:

relprop -> property :

0 p r o p e r t i e s = I p r o p e r t i e s .

r e l p r o p - > p r o p e r t y c o m p g l u e e n t i t y : 0 p r o p e r t i e s = 1 p r o p e r t i e s : 0 p r o p e r t i e s = 2 p r o p e r t i e s : 0 p r o p e r t i e s = 4 p r o p e r t i e s : 0 p r o p e r t i e s v a l u e a r g = 4 v a l u e .

This results in two analyses

[Aspect: price]

a n d

[Aspect: price, Value: [Relation: less, Arg:

100000]] which, when filtered by the heuristics,

present t h e latter, the most specific analysis, to the dialogue manager. T h e dialogue manager inspects the result and as it is a valid database request forwards it to the background system. However, too m a n y objects satisfy t h e request a n d t h e dialogue manager initiates a clarifica- tion request to the user to further specify the request. T h e user responds with remove audi 1985 and 1988. T h e keyword remove triggers the set operator restrict and the objects are in- terpreted by the rules:

o b j e c t - > m a n u f a c t u r e r : 0 o b j e c t = 1 o b j e c t .

o b j e c t - > m a n u f a c t u r e r * 2 y e a r : 0 o b j e c t = 1 o b j e c t : 0 o b j e c t y e a r = 2 y e a r .

This results in three objects

[Manufacturer:

audi], [Manufacturer: audi, Year: 1985]

a n d

[Manufacturer:

audi, Year:

1988]. W h e n filtered

the first interpretation is removed. This is in- tegrated by t h e dialogue manager to provide a specification on b o t h Objects a n d Properties which is passed to the background system a n d a correct response can be provided.

6 E m p i r i c a l e v i d e n c e f o r t h e a p p r o a c h

In this section we present results on partial in- t e r p r e t a t i o n i for a n a t u r a l language interface for t h e CARS-application; a system for t y p e d inter- action to a relational database with information on second h a n d cars. T h e corpus contains 300 utterances from 10 dialogues. Five dialogues from t h e corpus were used when developing the interpretation methods, the Development set, a n d five dialogues were used for evaluation, t h e

Test set.

6.1 R e s u l t s

T h e lexicon includes information on what type of entity a keyword belongs to, i.e. Objects or Properties. This information is acquired au- tomatically from the database with synonyms a d d e d manually from the background system description.

T h e automatically generated lexicon of con- cepts consists of 102 entries describing Objects

1 R e s u l t s o n d i a l o g u e m a n a g e m e n t h a s b e e n p r e s e n t e d

(4)

Table 1: Precision and recall for the grammars Yes/No S/T Set

Devel. set 100% 100% 97,5% Test set 100% 91,7% 86,1%

Objects

Fully Partial

Recall Precision Recall Precision Devel. set 100% 98% 100% 98% Test set 94,1% 80% 100% 85%

Properties

Fully Partial

Recall Precision Recall Precision Devel. set 97% 97% 99% 100% Test set 59,6% 73,9% 73,7% 91,3%

and Properties. From the system description in- formation base 23 synonyms to concepts in the database were added to the lexicon. From the

Development set another 7 synonyms to con-

cepts in the database, 12 relational concepts and 7 markers were added.

The five grammars were developed manually from the Development set. The object gram- mar consists of 5 rules and the property gram- mar consists of 21 rules. The grammar used for finding set indicators consists of 13 rules.

The Yes/No grammar and System/Task gram-

mar need no g r a m m a r rules. The time for devel- oping these grammars is estimated to a couple of days.

The obtained grammars and the lexicon of to- tally 151 entries were tested on both the Devel-

opment set and on the five new dialogues in the

Test set. The results are presented in table 1. In

the first half of the table we present the number of utterances where the Yes/No, System/Task and Set parameters were correctly classified. In the second we present recall and precision for Objects and Properties.

We have distinguished fully correct inter- pretations from partially correct. A partially correct interpretation provides information on the Aspect but might fail to consider Value- restrictions, e.g. provide the Aspect value price

but not the Value-restriction cheapest to an ut- terance such as what is the price of the cheapest volvo. This is because cheapest was not in the first five dialogues.

The majority of the problems are due to such missing concepts. We therefore added informa- tion from the Test set. This set provided anoth- er 4 concepts, 2 relational concepts, and I mark-

Table 2: Precision and recall when concepts from the test set were added

Properties

Fully Partial Recall Precision Recall Precision Test set 92,3% 79,5% 93,8% 90,6%

er and led us to believe that we have reached a fairly stable set of concepts. Adding these rela- tional and domain concepts increased the cor- rect recognition of set operations to 95,8%. It also increased the numbers for Properties recall and precision, as seen in table 2. The other re- sults remained unchanged.

To verify the hypothesis that the concepts are conveyed from the database and a small number of dialogues, we analyzed another 10 dialogues from the same setting but where the users know that a h u m a n interprets their utterance. From these ten dialogues only another 3 concepts and 1 relational concept were identified. Further- more, the concepts are borderline cases, such as mapping the concept inside measurement onto the database property coupd, which could well result in a system-related answer if not added to the lexicon.

As a comparison to this a traditional non- partial PATR-grammar, developed for good coverage on only one of the dialogues consists of about 200 rules. The lexicon needed to cover all ten dialogues consists of around 470 words, to compare with the 158 of the lexicon used here. T h e principles have also been evaluated on a system with information on charter trips to the Greek archipelago, T R A V E L . This corpus contains 540 utterances from 10 dialogues. The information base for TRAVEL consists of texts from travel brochures which contains a lot of information. It includes a total of around 750 different concepts. Testing this lexicon on the corpus of ten dialogues 20 synonyms were found. W h e n tested on a set of ten dialogues collected with users who knew it was a simulation (cf. the CARS corpus) another 10 synonyms were found. Thus 99% of the concepts utilized in this part of the corpus were captured from the information base and the first ten dialogues. This clearly supports the hypothesis that the relevant con- cepts can be captured from the background sys- t e m and a fairly small number of dialogues.

[image:4.612.63.297.54.205.2]
(5)

timated how many of the utterances in the cor- pus that can be analyzed by this model. 90,4% of the utterances can easily be captured by the model. Of the remaining utterances 4,3% are partly outside the task of the system and a stan- dard system message would be a sufficient re- sponse. This leaves only 4,8% of the utterances that can not be handled by the approach.

6.2 D i s c u s s i o n

When processing data from the dialogues we have used a system for lexical error recov- ery, which corrects user mistakes such as mis- spellings, and segmentation errors. This system utilizes a trained HMM and accounts for most errors (Ingels, 1996). In the results on lexical data presented above we have assumed a system for morphological analysis to handle inflections and compounds.

The approach does not handle anaphora. This can result in erroneous responses, for in- stance, Show rust .for the mercedes will interpret

the mercedes as a new set of cars and the answer

will contain all mercedeses not only those in the previous discourse. In the applications studied here this is not a serious problem. However, for other applications it can be important to handle such expressions correctly. One possible solution is to interpret definite form of object descriptions as a marker for an old set.

The application of the m e t h o d have only uti- lized information acquired from the database, from information on the system's capabilities and from corpus information. The motivation for this was that we wanted to use unbiased information sources. In practice, however, one would like to augment this with common sense knowledge on human-computer interaction as discussed in JSnsson (1993).

7 C o n c l u s i o n s

We have presented a m e t h o d for robust inter- pretation based on a generalization of PATR-II which allows for generalization of g r a m m a r rules and partial parsing. This reduces the sizes of the g r a m m a r and lexicon which results in re- duced development time and faster computa- tion. The lexical entries corresponding to en- tities about which a user can achieve informa- tion is mainly automatically created from the background system. Furthermore, the system will be fairly robust as we can invest time on

establishing a knowledge base corresponding to most ways in which a user can express a domain concept.

A c k n o w l e d g m e n t s

This work results from a number of projects on de- velopment of natural language interfaces supported by The Swedish Transport & Communications Re- search Board (KFB) and the joint Research Pro- gram for Language Technology (HSFR/NUTEK). We are indebted to Hanna Benjaminsson and Mague Hansen for work on generating the lexicon and de- veloping the parser.

R e f e r e n c e s

S. Bennacef, H. Bonneau-Maynard, J. L. Gauvin, L. Lamel, and W. Minker. 1994. A spoken lan- guage system for information retrieval. In Pro-

ceedings of ICLSP'9g.

Donna K. Byron and Peter A. Heeman. 1997. Dis- course marker use in task-oriented spoken dialog. In Proceedings of Eurospeech'97, Rhodes, Greece,

pages 2223-2226.

Jaime G. Carbonell and Philip J. Hayes. 1987. Ro- bust parsing using multiple construction-specific strategies. In Leonard Bolc, editor, Natural Lan-

guage Parsing Systems, pages 1-32. Springer-

Verlag.

Peter Ingels. 1996. Connected text recognition us- ing layered HMMs and token passing. In K. Oflaz- er and H. Somers, editors, Proceedings of the Second Conference on New Methods in Language Processing, pages 121-132, Sept.

Arne JSnsson. 1993. A method for development of dialogue managers for natural language interfaces. In Proceedings of the Eleventh National Confer- ence of Artificial Intelligence, Washington DC,

pages 190-195.

Arne JSnsson. 1997. A model for habitable and efficient dialogue management for natural lan- guage interaction. Natural Language Engineering,

3(2/3):103-122.

K£re SjSlander and Joakim Gustafson. 1997. An in- tegrated system for teaching spoken dialogue sys- tems technology. In Proceedings of Eurospeech '97, Rhodes, Greece, pages 1927-1930.

Lena StrSmb/ick. 1994. Achieving flexibility in uni- fication formalisms. In Proceedings of 15th Int. Conf. on Computational Linguistics (Coling'94),

volume II, pages 842-846, August. Kyoto, Japan. Lena StrSmb~ick. 1997. EFLUF - an implementa-

Figure

Table 1: Precision and recall for the grammars Yes/No S/T Set

References

Related documents

yields the D-in-D estimates of the impact of the 1990 EITC reform on low-skilled mothers’ mental health and subjective well-being.. In particular, the coefficient captures the

110. The mandate of the Office of Internal Oversight Services is to promote good governance and accountability in the Organization by providing independent and objective

A study to determine the diseases occurrence and the effectiveness in controlling infectious disease on the small poultry producers was carried out in

12 Assessment of comparative ionising radiation doses from the use of rapiscan secure 1000 x-ray backscatter security scanner, Health Protection Agency,.. Centre for

between markets and segments. Business conditions remained favorable in North America, most notably for Sandvik Machining Solutions. Demand in Europe fluctuated as the weaker

Three themes also emerged as motivators for the participants that enabled them to overcome the barriers they faced and experience resilience, these were perceiving the profession as

For example, Bellevue had the largest sinkholes in the state and is statistically distinct (CI test) from the other regions by perimeter, area (Figure 2), volume, major axis,

Stirring Speech (Su) 1/day, As a standard action,  Larazor can inspire greatness in her allies,