Case Role Filling as a Side Effect of Visual Search

CASE ROLE FILLING AS A SIDE EFFECT OF VISUAL SEARCH

H e i n z M a r b u r g e r

R e s e a r c h U n i t f o r I n f o r m a t i o n S c i e n c e

and A r t i f i c i a l I n t e l l i g e n c e U n i v e r s i t y o f Hamburg

M i t t e l w e g 179

D - 2 0 0 0 Hamburg 1 3 , F . R . Germany

W o l f g a n g W a h l s t e r

FBI0 - A n g e w a n d t e M a t h e m a t l k und i n f o r m a t l k U n i v e r s i t y of S a a r b r O c k e n

Im S t a d t w a l d

0 - 6 6 0 0 S a a r b r 0 c k e n 11, F.R. G e r m a n y

ABSTRACT

T h i s p a p e r a d d r e s s e s the p r o b l e m Of g e n e r a t i n g c o m m u n i c a t i v e l y a d e q u a t e e x t e n d e d r e s p o n s e s i n t h e a b s e n c e o f s p e c i f i c k n o w l e d g e c o n c e r n i n g t h e i n t e n s i o n s o f t h e q u e s t i o n e r . We f o r m u l a t e and

j u s t i f y a h e u r i s t i c for the s e l e c t i o n of o p t i o n a l

d e e p c a s e slots not c o n t a i n e d in the q u e s t i o n as

c a n d i d a t e s for the a d d i t i o n a l i n f o r m a t i o n con-

t a i n e d i n an e x t e n d e d r e s p o n s e . I t i s shown t h a t , i n a v i s u a l l y p r e s e n t d o m a i n o f d i s c o u r s e , c a s e r o l e f i l l i n g f o r t h e c o n s t r u c t i o n o f an e x t e n d e d r e s p o n s e can be r e g a r d e d as a s i d e e f f e c t o f t h e v i s u a l s e a r c h n e c e s s a r y t o a n s w e r a q u e s t i o n c o n - t a i n i n g a l o c o m o t i o n v e r b . The p a p e r d e s c r i b e s t h e v a r i o u s r e p r e s e n t a t i o n c o n s t r u c t i o n s used i n t h e German l a n g u a g e d i a l o g s y s t e m HAM-ANS f o r d e a l i n g w i t h t h e s e m a n t i c s o f l o c o m o t i o n v e r b s and i l l u s - t r a t e s t h e i r use i n g e n e r a t i n g e x t e n d e d r e s p o n s e s . I n p a r t i c u l a r , we o u t l i n e t h e s t r u c t u r e o f t h e g e o m e t r i c a l s c e n e d e s c r i p t i o n , t h e r e p r e s e n t a t i o n o f e v e n t s i n a l o g i c - o r i e n t e d s e m a n t i c r e p r e s e n t a - t i o n l a n g u a g e , t h e c a s e - f r a m e l e x i c o n and t h e r e p r e s e n t a t i o n o f t h e r e f e r e n t i a l s e m a n t i c s b a s e d on t h e F l a v o r s y s t e m . The e m p h a s i s i s on a d e t a i l e d p r e s e n t a t i o n o f t h e a p p l i c a t i o n o f o b j e c t - o r i e n t e d p r o g r a m m i n g m e t h o d s f o r c o p i n g w i t h t h e s e m a n t i c s o f l o c o m o t i o n v e r b s . The p r o - c e s s o f g e n e r a t i n g an e x t e n d e d r e s p o n s e i s i l l u s - . t r a t e d by an e x t e n s i v e l y a n n o t a t e d t r a c e .

1.

INTRODUCTION

F r e q u e n t l y a q u e s t i o n e r e x p e c t s more t h a n a d i r e c t , l i t e r a l r e s p o n s e a l t h o u g h he m u s t assume t h a t t h e a n s w e r e r i s n o t i n f o r m e d a b o u t w h a t p a r - t i c u l a r i n f o r m a t i o n he i s s e e k i n g . The q u e s t i o n e r

i m p u t e s to a c o o p e r a t i v e d i a l o g u e p a r t n e r the com- municative c o m p e t e n c e to r e p l y to a s i m p l e y e s - n o q u e s t i o n like (I) w i t h an e x t e n d e d r e s p o n s e (cf.

(12], (11]) l i k e (la) r a t h e r than w i t h a s i m p l e

Yes.

( I ) Are you going to t r a v e l t h i s summer? ( l a ) Y e s , to Sicily.

In the absence o f s p e c i a l i n f o r m a t i o n about the previous course o f the d i a l o g or the i n t e n t i o n s o f the q u e s t i o n e r (the unmarked case) an a n s w e r l i k e ( l a ) seems more a p p r o p r i a t e than ( I b ) or ( I c ) . ( I b ) Yes, w i t h an old school f r i e n d .

( I c ) Yes, by plane.

OF course, t h e r e are n u m e r o u s d i a l o g s i t u a t l o n s in

w h i c h ( l b ) os" ( l c ) c o u l d be g e n e r a t e d as a commun- i c a t i v e l y a d e q u a t e r e s p o n s e on t h e b a s i s o f a p a r . t ± c u l a r p a r t n e r m o d e l . B u t i t s t i l l m u s t b~ a s k e d why i n d i a l o g s o f t h e t y p e ' i n f o r m a t i o n , u p p l y ' t h e u n m a r k e d r e s p o n s e t a k e s t h e f o r m ( l a ) ~nd n o t ( l h ) o r ( l c ) .

In t h i s paper we w i l l present the r e s u l t s o f a computational study o f t h i s problem f o r the domain

R e s e a r c h on HAM..ANS Is c u r r e n t l y b e i n g s u p p o r t e d

by the German M i n i s t r y o f Research and Technology (8MFT) under c o n t r a c t 081T15038

' l o c o m o t i o n verbs' in d i a l o g s b a s e d on a v i s u a l l y p r e s e n t w o r l d of d i s c o u r s e . This q u e s t i o n is par-

t i c u l a r l y i m p o r t a n t for the c o n s t r u c t i o n of

c o o p e r a t i v e d i a l o 9 s y s t e m s , s i n c e , i n many a p p l i - c a t i o n s , no e x p l i c i t k n o w l e d g e a b o u t t h e d i a l o g g o a l s o f t h e q u e s t i o n e r i s a v a i l a b l e a t t h e o u t s e t . I f a , s y s t e m i s n e v e r t h e l e s s e x p e c t e d t o ' o v e r - a n s w e r , i . e . t o v o l u n t e e r i n f o r m a t i o n t h a t has n o t s p e c i f i c a l l y been r e q u e s t e d , i t m u s t com- mand a s e t o f h e u r i s t i c c r i t e r i a f o r s e l e c t i n g t h e a d d i t i o n a l i n f o r m a t i o n t h a t i s t o be v e r b a l i z e d [ 1 1 1 .

I t i s n o t e w o r t h y t h a t t h e t h r e e a d d i t i o n a l p o i n t s o f i n f o r m a t i o n i n ( l a ) , ( l b ) , ( 1 c ) c o r r e s p o n d t o f i l l e d d e e p c a s e s l o t s o f t h e v e r b u s e d i n t h e q u e s t i o n (GOAL, CO-AGENT and INSTRUMENT, r e s p e c - t i v e l y ) . T h i s s u g g e s t s t h a t t h e u n f i l l e d o p t i o n a l c a s e s l o t s i n t h e q u e s t i o n a r e c a n d i d a t e s f o r a d d i t i o n a l i n f o r m a t i o n . F o r a q u e s t i o n l i k e ( 2 ) , i n w h i c h a l l t h e d e e p c a s e s l o t s o f ' b r e a k ' a r e f i l l e d , o n l y a d i r e c t r e s p o n s e l i k e ( 2 a ) i s t o be e x p e c t e d as a p o s i t i v e a n s w e r , w h i l e i n ( 3 ) , w h e r e

only the o b l i g a t o r y d e e p case s l o t s are f i l l e d , an

e x t e n d e d r e s p o n s e l i k e ( 3 a ) can be e x p e c t e d . ( 2 ) D i d y o u b r e a k t h e w i n d o w w i t h y o u r s l i n g s h o t

y e s t e r d a y ? ( 2 a ) Y e s .

( 3 ) D i d y o u b r e a k t h e w i n d o w ? ( 3 a ) Y e s , w i t h my s l i n g s h o t .

S i n c e not e v e r y o p t i o n a l d e e p c a s e o f a g i v e n v e r b

u n s p e c i f i e d in the q u e s t i o n is s u i t a b l e for an

u n m a r k e d e x t e n d e d r e s p o n s e ( e . g . ( l a ) - ( l c ) ) we may d e f i n e t h e p r o b l e m more p r e c i s e l y by a s k i n g w h i c h o f t h e d e e p c a s e s l o t s u n s p e c i f i e d i n t h e q u e s t i o n a r e t o be c h o s e n as t h e u n m a r k e d v a l u e s .

F o r o u r d o m a i n Of i n v e s t i g a t i o n ' l o c o m o t i o n v e r b s ' l e t us c o n s i d e r q u e s t i o n s ( 4 ) and ( 5 ) , w h i c h r e f e r t o a v i s u a l l y p r e s e n t w o r l d o f d i s c o u r s e . I n e a c h c a s e p e r c e p t u a l p r o c e s s e s a r e assumed as a p r e r e - q u i s i t e f o r t h e a n s w e r .

4) Which v e h i c l e s t o p p e d ? &a) The b u s , on H a r t u n g s t r e e t .

4b) The b u s , b e c a u s e t h e d r i v e r s t e p p e d on t h e b r a k e .

5) Did the bus t u r n o f f ?

5a) Yes, from H a r t u n g s t r e e t onto S c h l u e t e r s t t e e t . 5b) Yes, t o g e t h e r w i t h the t a x i cab.

The i n s t a n t i a t i o n of the i o c a t l v e slot in a n s w e r

(4a) and the source and goal s l o t s in (Sa) i s p r e d i c t a b l e in c o n t r a s t to the c a u s a t i v e s l o t i n (4b) and the co agent s l o t i n (5b). As examples (4) and (5) demonstrate, the same o p t i o n a l deep case s l o t i s not always selected as the unmarked o p t i o n . The choice i s dependent upon the verb con- tained i n the q u e s t i o n . Moreover, { S a ) shows t h a t combinations o f d e e p cases are p o s s i b l e in

u n m a r k e d e x t e n d e d r e s p o n s e s ,

In the area u n d e r i n v e s t i g a t i o n here, the f o l l o w -

FAMILIAR WITH

SCENE BUT

CANNOT SEE IT

PDP-IO

NL DIALOG SYSTEM

H A M

A N S

IMAGE SEQUENCE

ANALYSIS SYSTEM

NAOS

MORIO

] IL

STREET INTERSECTION

Fig. 1: S i t u a t i o n a l c o n t e x t of the d i a l o g

s e l e c t i o n of the d e e p case slots for an um~marked

e x t e n d e d r e s p o n s e : Select the d e e p c a s e slots

w h i c h c o n t a i n the c o n c e p t s n e c e s s a r y for the p e r -

ceptual verification of the m o t i o n d e s c r l b e d by

the verb.

In o r d e r to v e r i f y a s t o p - e v e n t it is n e c e s s a r y to

d e t e r m i n e the end p o i n t of the m o t i o n (Cf. (4a))

but not the c a u s e (cf. (4b)). For a t u r n - o f f e v e n t a c h a n g e of d i r e c t i o n b e t w e e n s o u r c e and g o a l m u s t

be e s t a b l i s h e d (cf. (Sa)). It is not e s s e n t i a l to

d e t e r m i n e w h e t h e r o t h e r o b j e c t s m a k e this c h a n g e of d i r e c t i o n at the same time (cf. (Sb)).

H e n c e case role f i l l i n g f o r the c o n s t r u c t i o n o f an e x t e n d e d r e s p o n s e can be r e g a r d e d as a side e f f e c t of the visual, s e a r c h n e c e s s a r y to a n s w e r the q u e s - tion.

This a l s o a p p e a r s p l a u s i b l e w h e n seen in the l i g h t

of the b e l i e f s that the q u e s t i o n e r i m p u t e s to the

a n s w e r e r . The q u e s t i o n e r b e l i e v e s that the a n s w e r e r w i l l fill in the c a s e sluts n e c e s s a r y for

a n s w e r i n g the q u e s t i o n and that it is t h e r e f o r e

u n n e c e s s a r y to e x p l i c i t l y m e n t i o n these in the

q u e s t i o n . A d d i t i o n a l l y the q u e s t i o n e r b e l i e v e s

that the a n s w e r e r b e l i e v e s that the q u e s t i o n e r

e x p e c t s an e x t e n d e d r e p l y and fur this r e a s o n did not e x p l i c i t l y r e q u e s t the a d d i t i o n a l i n f o r m a t i o n . A c o o p e r a t i v e d i a l o g s y s t e m f u l f i l l s this user e x p e c t a t i o n by a p p l y i n g the h e u r i s t i c f o r m u l a t e d above.

A p r e r e q u i s i t e for the a p p l i c a t i o n of this h e u r i s - tic is that [he s y s t e m h a v e k n o w l e d g e about w h i c h

d e e p case slots are r e l e v a n t for the v e r i f i c a t i o n

OF a m o v e m ~ m t . This p r e r e q u i s i t e is not met by

m o s t n a t u r a l l a n g u a g e ( N L ) s y s t e m s since t h e y sim- ply r e p r e s e n t e v e n t s in t h e d o m a i n or d i s c o u r s e in fully i n s t a n t ~ a t e d Form u s i n g c a s e frames, e.g. as part of a s e m a n t i c net or frame h i e r a r c h y . In con- trast, the G,,rman l a n g u a g e d i a l o g s y s t e m H A M - A N S ( H a m b u r g a p p l i c a t i o n - - o r i e n t e d n a t u r a l l a n g u a g e

system) [6], w h i c h w e have d e v e l o p e d , can a p p l y

this h e u r i s ( ~ c b e c a u s e in a d d i t i o n to the case

f r a m e of each v e r b the s y s t e m i n c l u d e s a represen- t a t i o n of the r e f e r e n t i a l s e m a n t i c s of p r e d i c a -

tions associated w i t h t h a t verb which makes it

p o s s i b l e to ~valuate the ViSual i n p u t data f o r the movement i n question.

The g o a l of this a r t i c l e is to e l u c i d a t e the r e p r e s e n t a t i o n c o n s t r u c t i o n s for case f r a m e s and r e f e r e n t i a l s e m a n t i c s of verbs of m o t i o n u s e d in H A M - A N S and to i l l u s t r a t e t h e i r use in g e n e r a t i n g u n m a r k e d e x t e n d e d r e s p o n s e s .

2.

A SHORT OVERVIEW OF HA~-ANS

H A M - A N S is a l a r g e German n a t u r a l l a n g u a g e d i a l o g s y s t e m of b o t h c o n s i d e r a b l e d e p t h and b r e a d t h w h i c h p r e s e n t l y p r o v i d e s a c c e s s to t h r e e d i f f e r e n t a p p l i c a t i o n c l a s s e s , n a m e l y an e x p e r t s y s t e m ( h o t e l r e s e r v a t i o n s i t u a t i o n ) , a d a t a b a s e s y s t e m

( f i s h e r y d a t a ) and a s c e n e a n a l y s i s s y s t e m

( t r a f f i c s c e n e ) .

The c o m m u n i c a t i v e s i t u a t i o n s the s y s t e m h a n d l e s are c h a r a c t e r i z e d as follows:

In the h o t e l r e s e r v a t i o n s i t u a t i o n the s y s t e m

takes the role o f a h o t e l m a n a g e r , w h o tries to

p e r s u a d e the user to book a room. The c a l l e r is

a s s u m e d to h a v e the o v e r a l l g o a l of d e t e r m i n i n g w h e t h e r the r o o m o f f e r e d m e e t s his r e q u i r e m e n t s . The s y s t e m m u s t a t t e m p t to r e c o g n i z e the u s e r s

s p e c i f i c d e s i r e s c o n c e r n i n g t h e room as t h e y a r e

r e v e a l e d - u s u a l l y i n d i r e c t l y - in his u t t e r a n c e s and to m a k e use of the v a r i o u s d e v i c e s a v a i l a b l e in n a t u r a l l a n g u a g e that p e r m i t the r o o m in q u e s - tion to be p r e s e n t e d in a p a r t i c u l a r l y f a v o r a b l e l i g h t (e.g. the g e n e r a t i o n of t e n d e n t i o u s d e s c r i p - tions u s i n g h e d g e d r e l a t i v e a d j e c t i v e s ) .

In the d a t a b a s e a p p l i c a t i o n [8] H A M - A N S p r o v i d e s m a r i n e s c i e n t i s t s w i t h NL a c c e s s to a f i s h e r y d a t a b a s e , w h i c h c o n t a i n s d a t a c o l l e c t e d d u r i n g i n t e r n a t i o n a l e x p e d i t i o n s in the South A t l a n t i c and the I n d i a n Ocean. The c o m p l e x s t r u c t u r e d d a t a sets, w h i c h c o n s i s t of c r u i s e d a t a c o n c e r n i n g the

v e s s e l s ' v o y a g e s , k r i l l b i o l o g y d a t a , r e l e v a n t n e t h a u l s and o c e a n o g r a p h i c d a t a , a r e s t o r e d i n a r e l a t i o n a l d a t a b a s e (76 r e l a t i o n s . 1 1 M B y t e ) w i t h a PASCAL/R d a t a b a s e m a n a g e m e n t s y s t e m .

s t r e e t i n t e r s e c t i o n and s u p p l i e s on the p h o n e the

user, w h o is f a m i l i a r w i t h the scene but c a n n o t

see it f r o m his r e m o t e l o c a t i o n , w i t h i n f o r m a t i o n

a b o u t the t r a f f i c at that i n t e r s e c t i o n (cf. fig.

I). A g e o m e t r i c a l d e s c r i p t i o n of the o b s e r v e d

s c e n e Is p r o v i d e d by the i m a g e s e q u e n c e a n a l y s i s

s y s t e m NAOS/HORIO [ g ] . In this a p p l i c a t i o n our

research e f f o r t s are concerned w i t h the handling

o f time (tense and temporal adverbs) and the

semantics o f locomotion verbs ( c f . f i g . 2).

? How m a n y t r u c k s h a v e y o u s e e n ? TWO.

? W h i c h t r u c k s h a v e g o n e b y ?

A L I G H T - C O L O R E D ONE ON HARTUNGSTREET.

? Did a Mellow car and a green station wagon turn off first?

NO. A YELLOW CAR TURNED OFF H A R T U N G S T R E E T ONTO SCHLUETERSTBEET, BUT THE GREEN STATION

DID NOT TURN OFF F I R S T .

? W e r e a y o u n g man a n d an o l d woman w a l k i n g

down Schloeterstreet d u r i n g this time? YES. A N OLD WOMAN A N D A YOUNG MAN.

? Did the green station wagon stop on

S c h l u e t e r s t r e e t ? YES.

? How many station wagons altogether have

stopped?

ONE OF THE THREE ON SCHLUETERSTREET. ? What did I just ask you?

YOU ASKED: HOW M A N Y STATION WAGONS STOPPED. ? Did a station wagon drive off?

YES, ONE ON SCHLUETEBSTREET.

? By the way, the light-colored truck: Did

it drive down Schlueterstreet too?

YES.

How many cars are there in all?

SIXTEEN.

? Is one o f these cars blue 9

Y E S . TWO OF THEM ARE KIND OF BLUE.

? Did a blue car d r i v e down both s t r e e t s ?

YES, ONE BLUE CAB. So long!

Fig. 2: A t r a n s c r i p t o f an a c t u a l d i a l o g ( t r a n s l a t e d from G e r m a n )

3. REPRESENTATION FORMALISMS FOR THE SEMANTICS OF LOCOMOTION VERBS

3 , 1 . THE GEOMETRICAL SCENE DESCRIPTION

A basic requirement f o r answering questions about movements t h a t have occurred i n r e a l sequences o f scenes i s an adequate r e p r e s e n t a t i o n o f these

sequences. Not only the shape, the centers of

g r a v i t y , c o l , , r , e t c . o f o b j e c t s m u s t be represented, but also the t r a j e c t o r i e s o f moving o b ] e c t s .

Thls geometrical scene description c o n s i s t s of a

c o m b i n a t i o n o f a u t o m a t i c a l l y g e n e r a t e d o u t p u t s oF t h e scene a n a l y s i s p r o c e s s e s ( i n s o f a r as t h i s i s p r e s e n t l y p o s s i b l e ) and a number o f m a n u a l augmen- t a t i o n s .

The length i n time o f the s c e n e under considera- t i o n i s ca. 14 sec., which c o r r e s p o n d s . t o ca. 360

s i n g l e TV i m a g e s . From t h e s e 360 lmages 72

s n a p s h o t s a r e coded in a r e l a t i o n a l f o r m a l i s m , d e n o t l n g w h i c h o b j e c t s w e r e o b s e r v e d , t h e shape o f these o b j e c t s , t h e i r c u r r e n t center o f g r a v i t y and some o t h e r p r o p e r t i e s (e.g. c o l o r ) . The represen - r a t i o n o f the f i r s t snapshot contains i n f o r m a t i o n

about a l l o b j e c t s t h a t are v i s i b l e at t h a t time.

For the successive snapshots only changes w i t h

r e s p e c t to the p r e d e c e s s o r s are r e c o r d e d , i.e.

o b j e c t s and t h e i r d e s c r i p t i o n s are only e n t e r e d if

t h e y have changed l o c a t i o n or a p p e a r e d i n t h e

scene. A t r a j e c t o r y of an o b j e c t is d e t e r m i n e d by its d i f f e r e n t c e n t e r s of g r a v i t y r e l a t i v e to an

u n d e r l y i n g c o o r d i n a t e s y s t e m . I n c o n t r a s t t o t h e r e a l TV image s e q u e n c e t h i s r e p r e s e n t a t i o n i s o n l y 2 d i m e n s i o n a l and t h u s p r o v i d e s a b i r d ' s - e y e v i e w o f t h e s c e n e .

3 . 2 . THE REPRESENTATION LANGUAGES SURF AND DEEP

The l o g i c - o r i e n t e d s e m a n t i c r e p r e s e n t a t i o n l a n g u a g e s SURF and DEEP a r e t h e c e n t r a l r e p r e s e n - t a t i o n f o r m a l i s m s used i n HAM-ANS. T h e s e l a n g u a g e s a r e d e s i g n e d t o be d e c l a r a t i v e and e a s i l y e x t e n d - a b l e . SURF i s t h e t a r g e t l a n g u a g e o f t h e a n a l y s i s c o m p o n e n t s and s o u r c e l a n g u a g e f o r t h e g e n e r a t i o n c o m p o n e n t s and t h u s as c l o s e as p o s s i b l e t o NL u t t e r a n c e s , w h e r e a s DEEP i s b e t t e r s u i t e d f o r t h e e v a l u a t i o n o f u t t e r a n c e s on t h e b a s i s o f t h e s y s t e m ' s d o m a i n - s p e c i f i c k n o w l e d g e s o u r c e s . O r i g i n a l l y S U R F and D E E P w e r e d e s i g n e d t o

r e p r e s e n t t e r m and p r e d i c a t e s t r u c t u r e s w h i c h s e r v e as a r e p r e s e n t a t i o n f o r m a l i s m for s t a t e d e s c r i p t i o n s o c c u r r i n g t y p i c a l l y in the h o t e l

r e s e r v a t i o n s i t u a t i o n . For an adequate r e p r e s e n t a - t i o n o f the semantics o f questions c o n t a i n i n g verbs, the d e f i n i t i o n o f SURF and DEEP was aug- mented by m e t a - p r e d i c a t e s f o r m a r k i n g deep c a s e s , t e n s e and v o i c e a d a p t e d f r o m F i l l m o r e ' s deep c a s e

t h e o r y [3]. S i n c e e v e n t s can be e x i s t e n t i a l l y

q u a n t l f i e d as in (6) or e x p l i c i t l y q u a n t i f i e d as in ( 7 )

( 6 ) Did ]ohn f l y to Hamburg?

(7) Did John fly to Hamburg t h r e e times last week? SURF and DEEP p r o v i d e a means o f r e p r e s e n t i n g q u a n t i f i c a t i o n o f events. A s p e c i a l q u a n t i f i e r E-ACT denotes an e x i s t e n t i a l q u a n t i f i c a t i o n o f events. Other q u a n t i f i e r s l i k e those i n (7) are c u r r e n t l y not a v a i l a b l e but can e a s i l y be i n c l u d e d . Examples o f SURF and DEEP expressions are shown i n the annotated example ( c f . f i g . 8). In t h i s paper o n l y some o f the f e a t u r e s o f SURF and D E E P a r e d i s c u s s e d , see [ 6 ] f o r a more

d e t a i l e d d e s c r i p t i o n .

3 . 3 . THE CASE-FRAME LEXICON

The c a s e f r a m e s f o r v e r b s used i n t h e s y s t e m a r e s t o r e d i n t h e case-frame l e x i c o n [ 5 ] . Each e n t r y i n t h e w o r d l e x i c o n f o r a v e r b c o n t a i n s a p o i n t e r t o i t s a p p l i c a b l e c a s e f r a m e w h i c h d e s c r i b e s t h e s e m a n t i c s o f t h a t v e r b i n t e r m s o f c a s e r e l a t i o n s . A c a s e f r a m e i s r e p r e s e n t e d as a c o m b i n a t i o n o f deep c a s e d e s c r i p t i o n s s p e c i f y i n g f o r each deep c a s e i t s name, a m a r k e r , w h e t h e r t h e deep c a s e i s o b l i g a t o r y ( 0 ) o r o p t i o n a l ( F ) , and t h e s e m a n t i c r e s t r i c t i o n s w h i c h a r e r e q u i r e d f r o m a s y n t a c t i c s u b s t r u c t u r e t o f i l l t h e deep c a s e ( o f . f i g . 3 ) . This p o i n t e r technique permits the use o f a s p e c i f i c c a s e frame f o r s e v e r a l verbs during the a n a l y s i s phase w i t h o u t p r e d e t e r m i n i n g a s i n g l e p r o c e s s f o r t h e s e v e r b s d u r i n g t h e e v a l u a t i o n o f whole u t t e r a n c e s . For v e r b s w i t h d i f f e r e n t r e f e r e n t i a l semantics, e.g. ' t o a c c e l e r a t e ' and ' t o s t o p ' , a s i n g l e case frame, namely t h a t s p e c i - t y i n g an o b l i g a t o r y AGENT o f type ' v e h i c l e ' and a o p t i o n a l LOCATIVE o f type ' t h o r o u g h f a r e ' , i s a p p l i e d during the a n a l y s i s phase.

v e r i f i e d because o f the t r a n s i t i v i t y o f the super- set r e l a t i o n in the conceptual semantic net. I n t h e c a s e - f r a m e l e x i c o n t h e c a s e f r a m e s a r e n o t r e c o r d e d i n t h e f o r m shown i n f i g . 3. b u t r a t h e r a r e r e p r e s e n t e d as c o n s t r u c t o r c a l l s f o r b u i l d i n g

[rl-s: ageL~t:

[d-l: r o l o m m a r k e r : 0

r e s t r i c t i o n s :

(lambda: x l [af-a: ISA x l V E H I C L E ] ] ]

o b j e c t i v e :

SOUrce: l o c a t i v e .

( d - l : r o l A - - m a r k e r : F

r e s t r i c t i o n s :

[lambda: x l (af-a: I S A x l T H O R O U G H F A R E ] ] ]

g o a l :

time:

p a t h :

i n s t r u m e h t : ]

Fig. 3: Case frames f o r verbs o f type ' t o stop

a c a s e f r a m e a c c o r d i n g t o t h e a c t u a l s y n t a x d e f i n - i t i o n o f SURF, T h i s g u a r a n t e e s t h a t all p o s s i b l e

m o d i f i c a t i o n s o f SURF a r e i m m e d i a t e l y p r e s e n t i n t h e case f r a m e s .

3.4.

OB3ECT-ORZENTEB REPRESENTATION OF MOTION

CONCEPTS

In o b j e c t - o r i e n t e d p r o g r a m m i n g l a n g u a g e s p r o g r a m - ming i s more o r l e s s t h e a c t i v i t y o f c r e a t i n g a w o r l d o f e n t i t i e s c a l l e d o b j e c t s and o f s p e c i f y i n g a s e t o f g e n e r i c o p e r a t i o n s t h a t can be p e r f o r m e d on them• O b j e c t s can c o m m u n i c a t e w i t h each o t h e r by s e n d i n g and r e c e i v i n g m e s s a g e s . E s s e n t i a l l y , r u n n i n g a p r o g r a m means t h a t t h e o b j e c t sends a message t o ar, o b j e c t ( p o s s i b l y t o i t s e l f ) w h i c h i n t u r n sends a message e t c . , u n t i l t h e r e q u i r e d t a s k i s f u l f i l l e d . An i m p o r t a n t b e n e f i t o f t h e o b j e c t -

o r i e n t e d style is t h a t it lends i t s e l f to a par-

t i c u l a r l y s i m p l e and l u c i d kind of m o d u l a r i t y .

3 . 4 . 1 .

THE FLAVOR

SYSTEM

The F l a v o r s y s t e m [ 2 ] [ 1 3 ] i s an i m p l e m e n t a t i o n o f t h e l a n g u a g e f e a t u r e s t h a t s u p p o r t o b j e c t - o r i e n t e d p r o g r a m m i n g . Two k i n d s o f o b j e c t s e x i s t i n a F l a - v o r s y s t e m , n a m e l y one c a l l e d f l a v o r and t h e o t h e r

i n s t a n c e of a f l a v o r . A f l a v o r r e p r e s e n t s a g e n - e r i c object and an i n s t a n c e an i n d i v i d u a l r e a l i z a - t i o n of a ge,~eric o b j e c t . It is p o s s i b l e to s e n d

messages to b o t h k i n d s of o b j e c t s . F l a v o r s are o r g a n i z e d in ,, d i r e c t e d g r a p h c a l l e d the f l a v o r g r a p h • T h e r e is one d e s i g n a t e d flavor, the

v a n i l l a flays, r, w h i c h c o r r e s p o n d s to t h e t h i n g

f r a m e in FRL [I0]. S i n c e the h e r i t a g e of i n f o r m a -

tion f o r each f l a v o r i s provided by the f l a v o r

graph, i t zs necessary to s p e c i f y f o r each newly defined f l a v o r i t s l o c a t i o n in the graph by naming i t s d i r e c t predecessors ( i t s s u p e r f l a v o r s ) . The i n f o r m a t i o n contained in a f l a v o r i s a combination o f a l l the i n f o r m a t i o n i n h e r i t e d from i t s super- f l a v o r s and the added i n f o r m a t i o n given by i t s own d e f i n i t i o n . The added i n f o r m a t i o n can also o v e r -

ride, augment or m o d i f y the i n h e r i t e d i n f o r m a t i o n . This is one d i m e n s i o n of the i n f o r m a t i o n c o n t a i n e d

in a flavor: o w n e d or i n h e r i t e d . A n o t h e r is the

d e c l a r a t i v e / p r o c e d u r a l d i s t i n c t i o n . The d e c l a r a - t i v e knowle~tge of a Flavor is s t o r e d in v a r i a b l e s of d i f f e r e n t kinds w h e r e a s p r o c e d u r a l k n o w l e d g e is

encoded in so c a l l e d methods•

One kind of v a r i a b l e the i n s t a n c e v a r i a b l e - is

used to give i n s t a n c e s of the same g e n e r i c o b j e c t

t h e i r i n d i v i d u a l i n f o r m a t i o n . The o t h e r kind - the c l a s s v a r i a b l e is owned by a flavor, can be

' b e q u e a t h e d ' to o t h e r f l a v o r s , and a c c e s s e d by any o b j e c t i n t h e f l a v o r s y s t e m . H o w e v e r , a f l a v o r i s o n l y a l l o w e d t o c h a n g e a v a l u e o f a c l a s s v a r i - a b l e , if it o w n s t h i s v a r i a b l e .

M e t h o d s are f u n c t i o n d e f i n i t i o n s t h a t i m p l e m e n t the o p e r a t i o n s d e f i n e d for e a c h flavor. The c o m b i - n a t i o n of m e t h o d s f r o m d i f f e r e n t f l a v o r s is c a l l e d m i x i n g f l a v o r s .

I n c o m p a r i s o n w i t h FRL t h e F l a v o r s y s t e m has m a i n l y t h r e e d i s t i n g u i s h i n g f e a t u r e s :

The 'A k i n d o f ' s l o t i n FRL s e r v e s b o t h f o r e s t a b l i s h i n g an i n h e r i t a n c e h i e r a r c h y and f o r c o n n e c t i n g i n s t a n c e s t o s u p e r c l a s s e s , i . e . no c l e a r d i s t i n c t i o n i s made b e t w e e n g e n e r i c f r a m e s and i n s t a n c e s • On t h e o t h e r hand t h e f l a v o r g r a p h i s b u i l t by s p e c i f y i n g t h e s u p e r f l a v o r s f o r each f l a v o r , i n s t a n c e s a r e c r e a t e d by t h e m a k e - i n s t a n c e - m e t h o d .

B e c a u s e t h e d i s t i n c t i o n b e t w e e n g e n e r i c f r a m e s and i n s t a n c e s i s n o t made i n FRL t h e r e i s a l s o no d i s t i n c t i o n b e t w e e n i n s t a n c e v a r i - a b l e s and c l a s s v a r i a b l e s • I n t h e F l a v o r s y s - tem t h e s e m a n t i c s o f v a r i a b l e s i s more c l e a r l y d e f i n e d i n t h a t i n s t a n c e v a r i a b l e s can o n l y be m o d i f i e d i n i n s t a n c e s and c l a s s v a r i a b l e s can o n l y be m o d i f i e d i n f l a v o r s • Frames i n FRL a r e p a s s i v e d a t a s t r u c t u r e s , w h e r e a s f l a v o r s can be ( r e - ) a c t i v a t e d , c r e a t e d and m o d i f i e d ; t h e y a r e a u t o n o m o u s ;

t h e y are d e c l a r a t i v e and p r o c e d u r a l at the

same time and h e n c e are e n t i t i e s w h i c h are

b e t t e r s u i t e d f o r as formalisms f o r r e p r e s e n t i n g common knowledge ( c f . [ 2 ] ) . Although the f l a v o r system i s a t o o l f o r the development o f l a r g e s o f t w a r e systems and not a knowledge r e p r e s e n t a t i o n language, i t includes the basic concepts f o r the r a p i d design o f s p e c i f i c

k n o w l e d g e r e p r e s e n t a t i o n f o r m a l i s m s . I n c o n t r a s t t o a f u l l - f l e d g e d k n o w l e d g e r e p r e s e n t a t i o n l a n g u a g e t h i s a p p r o a c h r e q u i r e s some a d d i t i o n a l p r o g r a m m i n g i n t h e b e g i n n i n g , b u t i t a v o i d s any p e r m a n e n t o v e r h e a d f o r f e a t u r e s w h i c h a r e s u p e r - f l u o u s f o r t h e t a s k a t hand•

3 . 4 . 2 .

THE MOTION CONCEPT HIERARCHY

The F l a v o r s y s t e m i s used i n HAM-ANS f o r r e p r e s e n t i n g a s p e c i a l i z a t i o n h i e r a r c h y o f m o t i o n c o n c e p t s ( c f . f i g . 4 ) . The r o o t f l a v o r o f t h i s h i e r a r c h y i s t h e m o t i o n c o n c e p t HOVE. D e s c e n d a n t s in the t r e e , e.g. GO_BY, TURN i n h e r i t the

d e c l a r a t i v e and p r o c e d u r a l i n f o r m a t i o n c o n t a i n e d

)

(

)

I TIME

I SPACE

I STOP

IDRIVE-OFF

J

I

VANILLA

I

I .ov,- I

1

I

I TO' N I

)

S U B F L A V O R --0 N S T A N C £ OF

<

HAS A YELLOW CAR TURNED OFF?

HAM-ANS

FLAVOR :TURN SUPERFLAVORS

:

GO_BY

VARS: AGENT, SOURCE...

METHODS :JONLY_ASENT_SLOT_FILLED

J FIND A SOURCE

J

CHECK

DIRECTION CHANGE

I F~O A GOAL NEQ SOURCE

I INSTANCE_OF

APPLICATION_OF

I

TURN120 :

AGENT:

CAR120

SOURCE: HARTUNGST

DIRECTION_CHANGE?:

GOAL: BIBERSTREET

Es.

oNE Y LLOW

FROM

--7

ARTONOSTRE T

ON,O B,BERS ETJ

t I + k 0

e e

1

Fig. 5: Case s l o t f i l l i n g as side e f f e c t o f v i s u a l search

in t h e i r p a r e n t s . I n s t a n c e v a r i a b l e s c o m p r i s e i n f o r m a t i o n a b o u t the d e e p c a s e s a s s o c i a t e d w i t h the m o t i o n c o n c e p t as w e l l as i n f o r m a t i o n n e e d e d

and e x t r a c t e d by m e t h o d s . The m e t h o d s are respon--

sible for c h e c k i n g the r e f e r e n t i a l s e m a n t i c s of

the motion concepts. Instances o f a f l a v o r denote s p e c i f i c events i n the domain o f d i s c o u r s e t h a t could be v e r i f i e d by the a p p l i c a t i o n o f the methods.

The methods of the a d d i t i o n a l l y d e f i n e d f l a v o r s

TIME and SPACE are r e s p o n s i b l e f o r temporal and ~;patial computations. Instances o f these f l a v o r s

d e t e r m i n e the t e m p o r a l and s p a t i a l d e s c r i p t i o n of the a c t u a l scene: the l e n g t h of the s c e n e in time,

the number o f snapshots, the s p a t i a l e x t e n t , e t c . The task o f c h e c k i n g . t h e t r u t h value o f the propo- s i t i o n in ;~ u s e r s q u e s t i o n is a c c o m p l i s h e d t h r o u g h m e s s a q e p a s s i n g . T h e s e m e s s a g e s i n c l u d e :

c r e a t i n g i n ' Lances o f motion concepts, e . g . TURN120, i n s t . , ~ t i a t i n g deep case s l o t s s p e c i f i e d il, the q u e s t i o n , and a c t i v a t i n g a p p r o p r i a t e

( n t ! t h o d S .

L e t ' s now c o n , , z d e r t h e e x a m p l e g i v e n i n f i g . 5 i n more d e t a i l . '.ince o n l y the AGENT was s p e c i f i e d in

the q u e s t i o h , the s e l e c t e d method i s

ONLY AGENT S l ~ ' l !ILLED. A f t e r determinirlg an

interval ~f c~nsideration

f u r t h e r m~.thods, n a m e l y

FIND_A_SOURCE,

DIRECTION_CHAUGE and FIND_A_GOAL NEQ ~;OURCE.

DIRECTION CIIAI;GE i s a s p e c i a l method o f the f l a v o r

TURN. Th~ f i r s t and l a s t methods are i n h e r i t e d ( o f . f i g . 5) from f l a v o r GO_BY because t h e y are

also needed i n t h a t f l a v o r f o r answering q u e s t i o n s l i k e : 'Has the y e l l o w car d r i v e n from B i b e r s t r e e t t o H a r t u n g s t r e e t ~ ' .

FIND A SOURCE i d e n t i f i e s the f i r s t e n t r y o f the agen~'~ t r a j e c t o r y i n the i n t e r v a l o f c o n s i d e r a - t i o n and checks which o f the o b j e c t s o f the s t a t i c background these c o o r d i n a t e s belong t o . For t h i s t e s t o n l y those s t a t i c o b j e c t s are s e l e c t e d t h a t

s a t i s f y the s e l e c t i o n a l r e s t r i c t i o n s for the

s o u r c e slot s p e c i f i e d in the c a s e - f r a m e l e x i c o n .

If the test s u c c e e d s for an o b j e c t , the n a m e o~

this o b j e c t is s t o r e d in the s o u r c e slot,

DIRECTION CHANGE now f o l l o w s the a g e n t ' s t r a j e c -

t o r y look~ng f o r a s i g n i f i c a n t change o f d i r e c - t i o n . If this test is also p o s i t i v e ,

FIND A GOAL NEQ_SOURCE i s t r i e d . This method searches f u r a p o i n t on the t r a j e c t o r y which i s not i n s i d e the ob3ect i d e n t i f i e d i n the source s l o t . I f t h e r e i s such a p o i n t , the same s e l e c - t i o n a l check as f o r the source s l o t i s executed f o r the p o s s i b l e goal o b j e c t . The s u c c e s s f u l a p p l i c a t i o n o f these methods y i e l d s a f u l ] . y i n s t a n t i a t e d f l a v o r i n s t a n c e , e . g . TIJRN120 ( c f . f i g . ? ) .

4.

AN EXAMPLE OF THE PROCESSING OF AN UTTERANCE

The p r o c e s s i n g o f a u s e r ' s u t t e r a n c e may be i l l u s - t r a t e d by an example taken .from the d i a l o g i n f i g . 2.

USER: W h i c h t r u c k s have gone by?

o.,.ov,,

1

TYPE FLAVOR I .SUPERFLAVORS

INSTANCE-VARIABLES

AGENT SOURCE GOAL ... ~XACT.SOURCE EXACT.C..~OAL.-... ~T~RVAL. OF. CON ~DERAI30N CURR~ff.TIHE

METHOOS

INSTANCES I

INANE

GO-BY I

ISUPERFLAVORS~

ITYPE FLAVOR t

INSTANCE -VARIABLES INHERITEI I

I

]

AOOmONAL

]

METHOOS INHERITED

I I '"

AODmONAL C H O ~

~NLY_AOEM T_~.OT _FBJ.ED ~GENT.ANO .SOLI~SP~iED AGENT .AN0 .GOAL_~=ECIF lED AC~ff _AN0 .LOCATW E..cPEO FlED AOENT.SOJ~GDAL .SPECFFn

JTYPE FLAVOR J

TURN ] I P FL VO Jll

t

I

INSTANCE VARIABLES

I ~ _ o ~ N ~

I I

RB~-BNED

ONLY_AOENT~.RLED AII~ONAL

I

ts°

F i g . 6: I n s t a n c e v a r i a b l e s and m e t h o d s i n t h e m o t i o n c o n c e p t h i e r a r c h y

The f o l l o w i n g d i s c u s s i o n of some of the p r o c e s s i n g

p h a s e s can hi:st be u n d e r s t o o d if c o n t i n u a l re~er-

ence i s made t o f i g . B, w h i c h shows a t r a c e d v e r - s i o n o f t h e e x a m p l e .

The processing o f a u s e r ' s NL i n p u t s t a r t s w i t h a r a t h e r e l a b o r a t e l e x i c a l and m o r p h o l o g i c a l

a n a l y s i s - a process which on the one hand reduces s i n g l e words to t h e i r c a n o n i c a l forms w i t h t h e i r morphologi<al and s y n t a c t i c f e a t u r e s ( e . g . gender, person, n u m b e r ) and on the o t h e r hand recognizes syntagmatic groups o f words and d i s c o n t i n u o u s verb

c o n s t i t u e n t s , t r a n s f o r m i n g them a c c o r d i n g to p r e d e f i n e d rules.

The g e n e r a t e d s t r u c t u r e - the p r e t e r m i n a l s t r i n g

(not shown in fi@. 8) - forms the input to the

p a r s e r . The s y n t a c t l c a n a l y s i s c o n s i s t s o f two d i f f e r e n t s t r a t e g i e s , both o f which use the same A T N - d e f i n i t i o n s o f s y n t a c t i c c a t e g o r i e s , e . g . f o r noun phrases and p r e p o s i t i o n a l phrases. One o f

INAME N120 1

INSTANCE_OF

ITYPE

INSTANCEI

INSTANCE VARIABLES

NAM~ VALUE

AGENT CAR 2O

CURRENT_TIME TSD 12B

CURRENT.SPACE SSO 128

INTE~L.0F_CONS~BRATION ( 21 . 5~ )

SOURCE EXACT_SOURCE

OIRETION_CHANGE ? GOAL

EXACT_GOAL

RLLEO.BY.METHOO

MAKE_INSTANCE

OETERM~E_INTERVAL_ OF_CONSIDERATION BIBERSTREET

}

( 50 . 70 ) FINO_A_SOURCE

T CPECR-DIREClqON_CHANGE

HARTUNGS-I'REET } FINO_A_GOAL_NEO_

( 300. I00 ) SOURCE

I'.tg. 7: An i n s t a n c e of TURN

these s t r a t e g i e s - always a p p l i e d for s e n t e n c e s

w i t h c o p u l a v e r b s - uses a s u r f a c e grammar t o cope w i t h w o r d order v a r i a t i o n s . The o t h e r i s a c a s e - d r i v e n a n a l y s i s s t r a t e g y w h i c h i s used f o r s e n - t e n c e s c o n t a i n i n g v e r b s w i t h an a s s o c i a t e d c a s e f r a m e .

Since in the e x a m p l e the verb 'to go by' has a

case frame the second s t r a t e g y is applied. A f t e r

an access t o the case-frame l e x i c o n the case frame i s c o n s t r u c t e d . This case frame i s used t o guide the parsing i n the f o l l o w i n g manner: The al@orithm f i r s t attempts t o recognize those s y n t a c t i c con- s t i t u e n t s t h a t are p o s s i b l e candidates f o r a deep c a s e marked o b l i g a t o r y , and t h e n t o r e c o g n i z e t h o s e c o n s t i t u e n t s t h a t a r e p o s s i b l e c a n d i d a t e s f o r o p t i o n a l deep c a s e s . When t h e i n p u t i s com- p l e t e l y consumed and a l l o b l i g a t o r y deep c a s e s a r e f i l l e d the p r o c e s s e n d s .

The test for d e t e r m i n i n g if a s y n t a c t i c c o n s t i -

tuent i s a p o s s i b l e candidate to f i l l a s p e c i f i c

d e e p case is d i v i d e d into a s y n t a c t i c and a seman-

tic check. The s y n t a c t i c check requires, e.g.,

that in order to fill the agent role a c o n s t i t u e n t

must c o n t a i n the a t t r i b u t e 'nominative' ( s e n t e n c e

i n a c t i v e v o i c e ) and t h a t i t s n u m b e r must correspond t o t h a t o f the v e r b . The semantic check r e q u i r e s t h a t the noun o f the c o n s t i t u e n t f u l f i l l the semantic r e s t r i c t i o n s s p e c i f i e d f o r the s p e c i f i c deep c a s e . T h i s i s a c c o m p l i s h e d t h r o u g h t h e b u i l d i n g o f a SURF e x p r e s s i o n f o r t h e c o n s t i - t u e n t , t h e t r a n s f o r m a t i o n o f t h i s e x p r e s s i o n i n t o a DEEP e x p r e s s i o n , and t h e e v a l u a t i o n o f t h e DEEP e x p r e s s i o n on t h e b a s i s o f t h e c o n c e p t u a l n e t . I n o u r e x a m p l e o n l y t h e a g e n t c a s e is marked as o b l i g a t o r y and t h e noun p h r a s e ' w h i c h t r u c k s ' f u l - f i l l s b o t h t h e s y n t a c t i c and s e m a n t i c r e q u i r e m e n t s t o f i l l t h i s s l o t . S i n c e no o t h e r s y n t a c t i c c o n - s t i t u e n t s a r e e n c o u n t e r e d , t h e complete SURF r e p r e s e n t a t i o n i s c o n s t r u c t e d .

? Which trucks hive gone by# I t Syntactic analysis

;; C a l l frame I r l - i : l g e n t :

( d - l : r O l l - l i t t e r : O r l l t r i c t i o n l

( i s a b e l : I I l i t - i s ISA I I VEHICLE))) o b j e c t i v e :

source:

( e - i t role+marker: F r e s t r i c t i o n s :

I l e l b d l : I | l i t - I t ISA e l THOROUGHFARE))) l o o e t i v l :

( d - l : r o l e - n a r k e ~ : F r l l t r i o t i u n l :

Llimbde: 11 l i E - e : ISA I t THOROUGHFARE))) g o i I :

I d - L : r o l l - m a r k e r : F r e s t r i c t i o n s :

Ileabds: I t l i t - i s ISA =| THOROUGHFAEE])) time:

p i t h : i n l t r u l e u t : ) ;: AGENT p l r l l d

l l l l h d l : IS L i t - i s AGENT

19

I t - s : [q+v: HUICU) I l e l b d l : x$ ( i t - a t XSA x0 TRUCEI)))|

;; SURF representation of i n p u t s e n t e n c e I l l + d : EVENT

I t - s : ( g ' q t : E-ACT) ( l l i b d l : ItO l e E - i s ACT x l 0 G O B Y I l l L d - l : r O l l + h i t :

( r i - e : agent: L l a n b d l : IS

l i t - t : AGENT

=9

( t ' J : (qm+: HHICUI I l e n b d a : aS l e t - x : ISA sO TRUCE))))) o b j e c t i v e :

e o u r c e : l o c l t i v e : goal: t i l e : p i t h : i n l t r u a e n t : ) mud: Id-a: tense:

t ; a l b d l : I t 1 l i t - e : TENSE I I 1 P E R F ) ) v o i c e :

( l a n b d l : I t 2 L e t - e : VOICE 112 ACTIVE)))|I ** i o r m e l i n n t * o n : T r e n l t o r l i n S i n t o DEEP representation

:: 9EEP s t r u c t u r e

I f - d : I t - q : ( f o r : (B-V: NRIEH) e l g ) l i t - R : ISA xt4 TRUCE)) I t - d : ( i - q : ( f o r : ( q - q t : G-ACTI 113) l e t - e : ACT I t S GO BY))

I f - e : r o l e - l i l t : ( r l - d : agent:

l i t - a : AGENT a13 e l 4 ) o b j e c t i v e source: l o c a t i v e : p o l l : t i M : p a t h : i a l t r u n e n t : ) nod:

I t - s : t e n s e : L e t - s : TENSE a t ] PERF) v o i c e : L e t - s : VOICE !13 ACTIVE))I ))

I i g v l l u a l i o n

:; E v a l u a t i o n of i formula u i t h the q u a n t i f i e r (q-w MGICH)

;; E v l l u a t i o . o t i t o r a u l l v i t h t h e q u a n t i f i e r ( q - q t R-ACT)

;; Object TfllICKI his not loved d u r i n g the e n t i r e scene

;: Evaluation of a formula w i t h the q u | n t i t i t r I q - q t : R-ACT)

:; T i l t i n g nf • p a r t i a l l y i n l t a n t i e t l d t i l l frame

I f - e : P o l l - J i g s : I r l - d " agent

[ a t - a : AGENT GG_BY TRUCKi| o b j e c t i v e '

s o u r c e : l o c a t i v e : g o a l : time: path: instrument ) Iod:

I t ' l : t e n s e ( I f ' e : TENSE GO BY PEBF) v o i c e : e l f - i s VOICE GO_BY ACTIVE)J)

;; I n t e r v a l of consideration determined from tense land adverb)

(1 641

:; Thi o b j e c t becomes v i s i b l e b e t l e l n t i l l p o i n t s SG l a d GS

;; The i n t e r v a l et consideration l O d i f i e d i n i c o u r d l n o l v i t h object t i l l i l :

IGG 64)

;; Change determined b e t r o t h t i l l points SG and 57 3; Completed c e i l f r i l l

I f - I S r O l l - l i l t : l + l - d : I g l f l t :

L i t - I S AGENT GO_BY t n g c x i ) o b j e e t i v e t

SOurce: l o c a t i v e :

( I f - i S LOCATZVE G0_BY nON DAOTONGGTGEET) g o l l :

t i n t : path: i n s t r u m e n t : ) nod:

I f - i t annie: Let-s: TENSE GO_BY PERFI voice: ( i f - i s VOICE GO BY ACTIVE)))

: ; + V e r i t i c a t i o n of event win p o l s i b i l

; ; O l s u l t Ot t h e E v a l u a t i o n

I f - d : I t - q : I f o r : (q-s: ITRUCNS)) el4) T)

) f - d : I t - q : ( f o r : Lq-qt: E-ACT) z t + l t i t - a s ACT xlS GO BY)) I t - I s r o l l - l i s t :

l r l - d : agent: ( i t - l : AGENT IT3 ZI4I o b j e c t i v e : source: l o c a t i v e :

l i t = a : LOCATIVE 113 *ON HbRTUNGSTREET) goal:

time: path: instruments) lode

i f - I s t e n l l : ( i t - i s TERSE I t 3 PERF) voice: (at+at VOICE I t 3 ACTIVE}))))

la I n V l r l l n o r l i | i l l t i o n : T F l l l t O F l i n g i n t o SURF r l p r e s e n t l t i o n ; ; EUHF r l p r l l l n t l b i o + ot e l l i o t

l i t + d : EVENT

I t - I S ( q - q k : S-ACT) Llsabdl: xt3 L e t - t : ACT x l 3 GO_DYlll {d-e: r o l e - l i s t :

( r l - I : e v e n t :

I l e l b d l : I t S ( i t - l : AGENT 113 ( t - l ~ (B-a: ITGUCESII T ) I I o b J e e t i v l :

lOUrCl: l o © I t i v a :

( l l l b d | : I t 3 l i t - a s LOCATIVE a l l tON HARTUNGSTREET)) g o a l :

tines p i t h : i n e t r u l e n t : ) ned: I d - a : t i n i e r

( l l a b d l : s t 3 [ i f + i s TENSE s t 3 PERFI] v o i ~ e :

( l l u b d s : =13 L i t - i s VOICE a t 3 ACTIVE))))) ** E l l i p s i s g i n e r a t i o n

; ; E l l i p t i t t e d SURF r e p r e s e n t a t i o n of answer ( r l - e ~ I g e n t :

( 1 e l b a 1 : aS l i t - a s AGENT tO { t - l : ( q - s : (TRUCR2)] T ) ) ) o b j e c t i v e :

lOUrce: l o c a t i v e :

L l | a b d e : sO ( I t - I t LOCATIVE eO *ON UARTUNBSTREETI) g o a l :

t i l l : p i t h : i n l t ? U l l n t i )

I I V l t b l ~ l l l t i O n

t t N P - G e n e r e t i o n f o r TOUCH2 ;; The ggnerited DP f o r TRUCRS i s :

(t-q: ( t o r : l q - q t : A) 1IS) I f - o : AND Let-is ISA l I B TRUED) ( i f - e ~ BEF ItS LIGGT-COLORBDI)I

;; V e r b l I i l l d i t r u c t u r e Of easier

(SENTENCE IAGEDT (HP (HP (H: SOl A LIGHT-COLORED (ELLIPSIS THUCE))I) (LOCATIVE IPP *OH IflP (Ms SOL HARTUNGSTEEETIIII

*l S u r f a c e t r l n s f o r m i t i o n i

A LIGNY+COLOREG ONE ON GARTBNGBTNEET

S i n c e , i n t h e example d i s c u s s e d , t h e q u e s t i o n o p e r a t o r ' w h i c h ' i s s t r o n g e r t h a n t h e e x i s t e n t i a l q u a n t i f i e r f o r v e r b s ' E - A C T ' , t h e s t r u c t u r e i s r e a r r a n g e d .

The t a s k o f e v a l u a t i n g a OEEP f o r m u l a i s g o v e r n e d by a g e n e r a t e and t e s t s t r a t e g y . G e n e r a t e and t e s t p r o c e d u r e s can De v i e w e d as b e i n g a c t i v a t e d by p a t t e r n - d i r e c t e d i n v o c a t i o n and d i f f e r f r o m each o t h e r i n t h a t t h e g e n e r a t e p r o c e d u r e s a s s i g n i n t e r n a l o b j e c t i d e n t i f i e r s t o v a r i a b l e s i n DEEP f o r m u l a s , w h i l e t h e t e s t p r o c e d u r e s y i e l d two v a l u e s , t h e f i r s t o f w h i c h i s e i t h e r a f u l l y i n s t a n t i a t e d f o r m u l a e q u i v a l e n t t o t h e i n p u t f o r - mula o r a m o d i f i e d f o r m u l a , and t h e second o f w h i c h i n d i c a t e s t h e t r u t h v a l u e o f t h e i n p u t f o r - mula i n t h e r a n g e [ 0 , 1 ] . In t h e i n t e r p r e t a t i o n phase t h e s e two p r o c e s s e s i n t e r a c t i n such a way t h a t a t e s t a t t e m p t a c t i v a t e s g e n e r a t e p r o c e d u r e s w h i c h i n t u r n c a l l t e s t p r o c e d u r e s and so on. A c l o s e r l o o k a t o u r example shows t h a t a f t e r t h e f i r s t t e s t a t t e m p t has d i s c o v e r e d a s t r u c t u r e c o n - t a i n i n g a v a r i a b l e i n t h i s case t h e t e r m r e p r e s e n t i n g t h e noun p h r a s e ' w h i c h t r u c k s ' - a package o f g e n e r a t e p r o c e d u r e s i s a c t i v a t e d t o p r o d u c e t h e s e t o f o b j e c t i d e n t i f i e r s d e n o t i n g t h e

r e f e r e n t i a l set of o b j e c t s t h a t are t r u c k s - h e r e

TRUCK1 and TRUCK2. The r e s t o f t h e f o r m u l a i s t h e n r e c u r s i v e l y s e n t t o a t e s t p r o c e s s w i t h t h e v a r i a b l e ' w 1 4 ' r e p l a c e d by e l e m e n t s o f t h e r e f e r - ence s e t f o r t r u c k s one a f t e r t h e o t h e r .

The n e x t f o r m u l a t o be t e s t e d r e q u i r e s t h e g e n e r a - t i o n o f a set o f i n s t a n c e s o f t h e t y p e GO_BY. S i n c e e v e n t s a r e n o t r e p r e s e n t e d i n f u l l y i n s t a n t i a t e d f o r m b u t r a t h e r must be e x t r a c t e d f r o m t h e g e o m e t r i c a l scene d e s c r i p t i o n , a s p e c i a l s e t o f p r o c e d u r e s - t h e methods s p e c i f i e d i n t h e v e r b f l a v o r h i e r a r c h y - i s a c t i v a t e d . (See s e c t i o n 3 . 4 . 2 f o r how t h i s p r o c e s s f u n c t i o n s , )

A v e r i f i c a t i o n o f an e v e n t GO BY i s p o s s i b l e o n l y f o r TRUCK2. The a d d i t i o n a l ~ n f o r m a t i o n e x t r a c t e d d u r i n 9 t h e p r o c e s s o f v i s u a l s e a r c h - t h e s p e c i f i c l o c a t i o n o f t h e e v e n t - i s r e c o r d e d i n t h e l o c a - t i v e s l o t .

D u r i n g t h e f o r m a t i o n o f t h e r e s u l t o f t h e e v a l u a - t i o n , t h e s y s t e m , g u i d e d by g e n e r a l h e u r i s t i c s , d e c i d e s w h e t h e r t h e a d d i t i o n a l d e t a i l w i l l cause t o o ~ r e a t a c o m p l e x i t y i n t h e answer o r n o t [ 1 1 ] . I n t h i s case t h e c o m p l e x i t y i s s u i t a b l e and t h e l o c a t i o n w i l l be m e n t i o n e d i n t h e a n s w e r .

The word ' w h i c h ' i s d e f i n e d as q u a n t i f i e r t h a t causes a d e s c r i p t i o n o f a s e t o f o b j e c t s t o be r e t u r n e d ( i n s t e a d o f a t r u t h v a l u e ) . Thus t h e s e t o f r e f e r e n c e o b j e c t s f o r w h i c h t h e p r o p o s i t i o n i n q u e s t i o n c o u l d be v e r i f i e d , i . e . TRUCK2, i s sub- s t i t u t e d f o r t h e noun p h r a s e ' w h i c h t r u c k s ' . The r e s u l t i n g DEEP e x p r e s s i o n i s t r a n s f o r m e d by t h e i n v e r s e n o r m a l i z a t i o n p r o c e s s i n t o a SURF e x p r e s s i o n . In o r d e r t o v e r b a l i z e e x t e n d e d r e s p o n s e s i n a manner b o t h i n f o r m a t i v e and c o n c i s e as p o s s i b l e , t h e e l l i p s i s g e n e r a t i o n p r o c e s s e l i d e s t h o s e p a r t s o f t h e s e m a n t i c r e p r e s e n t a t i o n o f c o m p l e t e r e s p o n s e s t h a t a r e i d e n t i c a l t o t h e s t o r e d r e p r e s e n t a t i o n o f t h e q u e s t i o n [ ? ] .

The v e r b a l i z a t i o n component p r o d u c e s a s t r i n g o f c a n o n i c a l words and t h e i r g r a m m a t i c a l f e a t u r e s

u s i n g t r a n s l a t i o n r u l e s a t t a c h e d to the v a r i o u s c a t e g o r i e s of S U R F e x p r e s s i o n s , A s p e c i a l s u b c o m -

p o n e n t p r o v i d e s for t h e g e n e r a t i o n of noun p h r a s e s as d e s c r i p t i o n s o f domain i n d i v i d u a l s , i n o u r example TRUCK2. In t h i s case t h e N P - g e n e r a t o r decides not to generate a d e f i n i t e d e s c r i p t i o n

s i n c e n e i t h e r t h e s y s t e m nor t h e u s e r has a l r e a d y r e f e r r e d to TRUCK2 in the p r e v i o u s d i a l o g and the e x i s t e n c e o f TRUCK2 as a moving o b 3 e c t i s n o t i m p l i e d by t h e e x i s t e n t i a l a s s u m p t i o n s s u p p l i e d by t h e a p r i o r i u s e r model ( c f . [ ? ] ) . I n s t e a d , t h e i n d e f i n i t e NP a l i g h t - c o l o r e d t r u c k ' i s gen- e r a t e d , u s i n g t h e p r o p e r t y ' l i g h t - c o l o r e d ' as an i n i t i a l c h a r a c t e r i z a t i o n .

F i n a l l y t h e " s u r f a c e t r a n s f o r m a t i o n ' component [ 1 ] p r o n o m i n a l i z e s t h e noun ' t r u c k ' and y i e l d s a s t a n d a r d word o r d e r o f t h e u t t e r a n c e and t h e c o r r e c t l y i n f l e c t e d f o r m s o f t h e c a n o n i c a l w o r d s .

5.

CONCLUSZON

We have a t t e m p t e d to show t h a t case r o l e f i l l i n g f o r t h e c o n s t r u c t i o n o f an unmarked e x t e n d e d r e s p o n s e can be r e g a r d e d as a s i d e e f f e c t o f t h e v i s u a l s e a r c h n e c e s s a r y t o answer q u e s t i o n s r e f e r - r i n g t o a v i s u a l l y p r e s e n t domain o f d i s c o u r s e . A new method f o r t h e r e p r e s e n t a t i o n o f t h e r e f e r e n - t i a l s e m a n t i c s a s s o c i a t e d w i t h l o c o m o t i o n v e r b s has been p r e s e n t e d i n t h e f r a m e w o r k o f o b j e c t - o r i e n t e d programming based on t h e F l a . v o r s y s t e m . The a p p r o a c h p r e s e n t e d has been u s e f u l i n e x t e n d - i n g t h e c o m m u n i c a t i v e c a p a b i l i t i e s o f t h e d i a l o g s y s t e m HAM-AN$ as an i n t e r f a c e t o a v i s i o n s y s t e m .

REFERENCES

[ 1 ]

[ z ]

[32

[ 4 ]

[ 5 ]

[ s ]

[ 7 ]

[ e ]

[ 9 ]

[ 1 0 ]

( 1 1 ]

[ 1 2 ]

( 1 3 ]

BUSEMANN, S . : Problems i n v o l v i n g t h e a u t o m a t i c g e n e r a t i o n o f u t t e r a n c e s i n German. Hemo ANS-8, Research U n i t f o r I n f o r m a t i o n S c i e n c e and A I , U n i v . o f Hamburg, A p r i l 1082. Ol PRIMIO F., CHRISTALLER, T.: A poor m a n ' s f l a v o r s y s t e m . Working p a p e r No. 47, ISSCO, U n i v . de Geneva, l a B 3 .

FILLHORE, C. 3 . : The case f o r c a s e . I n : Bach, E . , Harms, R. T. ( e d s . ) : U n i v e r s a l s i n l i n g u i s t i c t h e o r y . H o l t , R i n e h a r t & W i n s t o n , 1968, pp. 1 - 8 8 .

HENDRIX, G. G . : S e m a n t i c a s p e c t s o f t r a n s l a - t i o n . I n : W a l k e r , O. E. ( e d . ) : U n d e r s t a n d i n g spoken l a n g u a g e . New Y o r k , N o r t h - H o l l a n d , 1978, pp. 193-228.

HOEPPNER, W.: A T N - S t e u e r u n g d u r c h K a s u s r a h - men. I n : W a h l s t e r , W. ( e d . : GWAI-82. P r o c . Sth German Workshop on A I . B e r l i n : S p r i n g e r , 1982, pp. 2 1 5 - 2 2 6 .

HOEPPNER, W., CHRISTALLER, TH., HARBURGER,

H . , HORIK, K . , NEBEL, B . , O'LEARY, H . , WAHL- STER, W.: Beyond domain i n d e p e n d e n c e : E x p e r i - ence w i t h t h e d e v e l o p m e n t o f a German l a n g u a g e a c c e s s system t o h i g h l y d i v e r s e b a c k g r o u n d s y s t e m s . I n : P r e c . 8 t h I J C A I , K a r l s r u h e 1083, pp. 5 8 8 - 5 9 4 .

3AHESON, A . , WAHLSTER, W.: User m o d e l l i n g i n a n a p h o r a g e n e r a t i o n : E l l i p s i s and d e f i n i t e d e s c r i p t i o n . I n : P r o c . ECAI-82, Orsay 1982. pp. 2 2 2 - 2 2 7 .

HARBURGER, H . , NEBEL, B . : N a t u e r l i - c h s p r a c h l i c h e r Oatenbankzugang m i t HAH-ANS: S y n t a k t i s c h e K o r r e s p o n d e n z , n a t u e r l i c h s p r a - c h l i c h e O u a n t i f i z i e r u n g und s e m a n t i s c h e s H o d e l l des D i s k u r s b e r e i c h s . I n : Kupka, I .

( e d , ) : G I - 1 3 . J a h r e s t a g u n g . (To a p p e a r )

NEUHANN, B . : Towards n a t u r a l l a n g u a g e d e s c r i p t i o n o f r e a l - w o r l d image s e q u e n c e s . I n : Nehmer, J. ( e d . ) : G I - 1 2 . 3 a h r e s t a g u n g . B e r l i n : S p r i n g e r , 1982, pp. 3 4 9 - 3 5 8 .

ROBERTS, R . B . , GOLDSTEIN. I . P . : The FRL m a n u a l . AI Hemo &09, AI L a b . , H I T , C a m b r i d g e , 1977.

WAHLSTER, W., HARBURGER, H . , 3AHESON, A . , BUSEMANN, S . : O v e r - a n s w e r i n g y e s - n o q u e s - t i o n s : Extended r e s p o n s e s i n a NL i n t e r f a c e t o a v i s i o n s y s t e m . I n : P r o c . 8 t h I J C A I , K a r l s r u h e 1983, pp. 6 & ] - B & 6 .

WEBBER, B . , 30SHI, A . , HAYS, E . , HCKEOWN, K . : E x t e n d e d n a t u r a l l a n g u a g e d a t a b a s e i n t e r a c - t i o n . I n : I n t . 3. Computers and M a t h e m a t i c s , S p r i n g 1983.