ON
THE INTERI&$ENDENCE OF LANGUAG AND PE R ~ E P T ION* DavLd L. WaltzC o o r d i n a t e d S c i e n c e L a b o r a t o r y
U n i v e r s i t y o f I l l i n d i s a t Urbana/Champaign
It i s a r g u e d t h a t w i t h o u t a c o n n e c t i o n t o t h e r e a l world v i a p e r c e p t i o n , a l a n g u a g e s y s t e m can- h o t know what i t i s t a l k l n g a b o u t , S i m i l a r l y ,
a
p e r c e p t u a l s y s t e m must have ways o f expressingi t s
o u t p u t s v i a a language (spoken, w r i t t e n , g e s t u r a l o r o t h e r ) . T ~ Q relationship between p e r c e p t k o n and language i s e x p l o r e d , w i t h s p e c i a l a t t e n t ~ s n t o t h e implications of r e s u l t s I n language re- s e a r c h f o r o u r models of v l s l o n s y s t e m s and v l c e - v e r s a . I t r s s u g g e s t e d t h a t e a r l y language l e a r n - l n g i s an e s p e c i a l l y f e r t l l e a r e a f o r this e x p l o - r a t i o n . W l t h i n t h l s a r e a , we a r g u e t h a t p e r c e p t u a l d a t a i s c o n c e p t u a l i z e d p r l o r t o language a c q u i s l t l o n a c c o r d i n g t o l a r g e l y i n n a t e strategies, t h a t t h l s c o n c e p t u a l l z a t ~ o n I s i n terms o f a n ~ n t e r n a l , non- ambiguous " l a n g u a g e , " t h a t l a n g u a g e , p r o d u c t i o n
from i t s b e g r n n i n g s t o a d u l t h o o d i s a p r o j e c t i o n of t h e I n t e r n a l language which s e l e c t s and h r g h -
l i g h t s t h e most r m p o r t a n t p o r t i o n s of l n t e ~ n a l c o n c e p t s , and c h a t schemata produced I n t h e s e n s o r y motor world a r e e v o l v e d j n t o schemata t o d e s c r i b e a b s t r a c t wor I d s . Examples a r e p r ~ v l d e d which s t r e s s t h e i m p o r t a n t of " g e s t a l t " ( f i g u r e - ground) r e l a t i o n s h i p s and p r o j e c t L o n (3-D t o 2-1/2 o r 2-D, c o n c e p t u a l t o l i n g u i s t r c , and
linguistic t o c o n c e p t u a l ) ; f l n a l l y mechanisms f o r a n i n t e g r a t e d v l s i o n - l a n g u a g e s y s t e m a r e p r o p o s e d , qnd some p r e l i m i n a r y r e s u l t s a r e d e s c r i b e d .
I n t r o d u c t i o n
p e r c e p t I o n 1. obs. : CONSCIOUSNESS
2a: a r e s u l t of p e r c e i v l n g : OBSERVATION b : a m e n t a l image. CONCEPT
3 a . a w a r e n e s s of t h e e l e m n t s of environment t h r o u g h p h y s i c a l sensation c o l o r
b : p h y s i c a l s e n s a t i o n i n t e r p r e t e d I n t h e l i g h t of e x p e r i e n c e
4 a : d i r e c t o r i n t u i t l v e cognition. INSLGHT b : a c a p a c l t y f o r c o m p r e h e n s ~ o n
s y n s e e DISCERNMENT
( W e b s t e r ' s S e v e n t h New C o l l e g i a t e O i c t i o n a r y )
q h l s work was s u p p o r t e d by t h e O f f i c e of Naval R e s e a r c h u n d e r C o n t r a c t ONR-N00014-75-C-0612.
TWhlle I i n t e n d p e r c e p t i o n t o r e f e r i n t h e human examples t o all t h e s e n s e s : v f s l o n , h e a r i n g , t o u c h , s m e l l , t a s t e , and motor s e n s e , i n the c a s e o f c o m p u t e r s , o n l y v i s l o n has been e x p l o r e d I n more t h a n a c u r s o r y manner,
Language u n d e r s t a n d i n g i n i t s d e e p e s t s e n s e i s n o t p o s s i b l e w i t h o u t d i r e c t e x p e r i e n c e 8 @
i t s
r e a l w o r l d c o r r e l a t e s . I b e l i e v e t h a t i t i s no a c c i d e n t t h a t t h e s m e word c a n r c f c r bath t o s e n s o r y a w a r e n e s s and t o comprehension. N e a r l y a l l e f f o r t s i n l a n g u a g e p r o c e s s i n g , b o t h i n artificial i n t e l l ~ g e n c e and l i n g u i s t i c s , havec o n c e n t r a t e d on t r a n s f o r m i n g s t r i n g s of words i n t o t r e e s o r o t h e r s t r u c t u r e s o f words (sometimes
s u r f a c e w o r d s , sometimes " p r i r n l t i v e " w o r d s ) o r c o n v e r s e l y , on producing s t r l n g s of words from t h e s e s t r u c t u r e s , Few r e s e a r c h e r s have e v e n r e c o g n i z e d r h e I m p o r t a n c e of i n t e r f a c i n g l a n g u a g e and v i s l o n systerns,t l e t a l o n e u n l t i n g t h e %wo l i n e s o f r e s e a r c h . (Exceptions i n c l u d e [Mlnsky 19751, [Woods 19781, [ M l l l e r & J o h n s o n L a n d [ 1 9 7 6 ] , [Schank & S e l f r i d g e 19771, [ P y l y s h y n 1977 a & b ] , [ C l a r k 19731). At t h l s t i m e I n h l s t o r y , A 1
v i s l o n and n a t u r a l language r e s e a r c h e r s have l l t t l e t o say t o e a c h o t h e r ; most of t h e work which t r e a t s l a n g u a g e and p e r c e p t Ion t o g e t h e r wauld I t h i n k b e c o n s i d e r e d t o l i e I n t h e r e a l m s
of philosophy o r psychology.
Moreover, t h e a r e a s of language processing
which c o u l d h a v e a b e a r l n g on p e r c e p t I o n have
-
b e e n l a r g e l y I g n o r e d . Very l i t t l e work h a s b e e n done on programs t o u n d e r s t a n d language a b u t s p a c e , s p a t l a 1 r e l a t l o n s , o r o b j e c t d e s c r l p t l o n s .(Some e x c e p t l o n s
a r e
[Simmons 19751 and [Novak 19761, [ K u l p e r s 19751, and [Goguen 19733 .)By t h e same t o k e n , c u r r e n t computer v i s i o n s y s t e m s a r e n o t a b l e t o d e s c r l b e w h t t h e y "see" I n n a t u r a l l a n g u a g e , I n f a c t v e f y few programs c a n e v . i d e n t ~ f y o b j e c t s w l t h r n a s c e n e ( e x c e p t f o r programs whlch o p r a t e I n v e r y c o n s t r a i n e d universes). Most v l s l o n s y s t e m s p r o d u c e s c e n e
s e g m e n t a t l o n s , l a b e l l l n g s o r 3 - D i n t e r p r e t a t l o n s of s c e n e portions, e t c . Very l r t t l e p r o g r e s s h a s b e e n made toward t h e g o a l of h a v r n g programs w h ~ c h c o u l d d e s c r i b e a g e e e r a l s c e n c e , l e t a l o n e d e s c r l b e t h e most s a l l e n t f e a t u r e s of a s c e n e .
(Some e x c e p t l o n s a r e [ P r e p a r a t a and Ray 19721, [Yaklmovsky 19731, [ B ~ J C S Y and J o s h 1 19781, [ Z u c k e r , R o s e n f e l d and Davrs 19753 and [Tenen-
The need f o r an i n t e r n a l r e p r e s e n t a t i o n s e p a r a t e n a t u r a l language
It i s now r d a s o n a b l y w e l l - e s t a b l i s h e d t h a t people use l a r g e s t r u c t u r e s l i k e " s c r i p t s " [Schank and # e l s o n 19771 o r llframes-systems" [Minsky
19751 p m v a s i v e l y f o r r e a s o n i n g and language and t h a r a l a r g e s c r i p t can be invoked by r e f e r r i n g t o a s i n g l e s a l i e n t a s p e c t of t h e s c r i p t . Thus we can answer a q u e s t i o n l i k e "How d i d you g e t h e r e ? " by s a y i n g "I borltowed my b r o t h e r ' s c a r
,"
and t h i s answercan
only b e understood i f we a r e a b l e t o u s e i t t o r e l i a b l y r e t ' r i e v e a l a ~ g e r s t r u c t u r e which answers t h e q u e s t i o n more d i r e c t l y . (Examplefrom George Lakoff [1976].) To u n d e r s t a n d l a n - guage a t t h e l e v e l of an a d u l t human w l r l c e r t a i n l y r e q u i r e a huge number of such s c r i p t s , w i t h r i c h * i n t e r c o n n e c t i o n s and powerful, f l e x i b l e matchifig procedures a s i n Bobrow and Winograd [1973]. For s a r i p t s t h a t r e f e r t o t h e p h y s i c a l world d i r e c t l y , what language can be used t o c o n s t r u c t t h e s c r i p t s ? How can we c o n s t r u c t s c r i p t s f o r a b s t r a c t worlds
(e,g, economics, psychology, p o l i t i c s ) ? What language should b e used f o r a b s t r a c t w o r l d s ? Are a l l t h e s e s c r i p t s t o be hand coded?
Consider a l s o s e n t e n c e s l i k e "A man was b i t t e n by a dog". I f we were t o b e asked "Where c o u l d t h e man have
been
b i t t e n , we would probably i n t h e absence a5 more snformatlon g u e s s t h e a n k l e , l e g o r arm. However i f we a r e a l s o t o l d t h a t t h e dog was a doberrnan o r t h a t it was a dashshund o r 6 h a t t h e man was l y i n g down o r t h a t t h e dog was s t a n d i n g a t t h e time, we would g i v e somewhat d i f f e r e n t answers. It seems t o m e t h a t n a t u r a l s t r u c t u r e s f o r representing and answering q u e s t i o n s about such language w i l l be very d i f f e r e n t from t h o s e usedi n
-
a l l prokrams today-
a p r o t o t y p i c a l dog which c a n be s c a l e d , r e p r e - s e n t a t l o n s of a p e r s o n i n v a r i o u s canonicalp o s i t i o n s , s i z e s of mouth openings and limbs, e t c , would be t h e most approp1rlate, economical r e p r e -
s e n t a t i o n s .
There i s a l s o a g r e a t d e a l of prima f a c l e evldence of c l o s e t i e s between p e r c e p t l o n and t h e
language used by a d u l t s t o d e s c r i b e a b s t r a c t p r o c e s s e s such a s thinking, l e a r n i n g , and comnu- n i c a t i n g , and t o d e s c r i b e a b s t r a c t f i e l d s l i k e Qconomlcs, d i ~ lomacy, and psychology. Witness t h e wide u s e of b a s ~ c a l l y p e r c e p t u a l words l i k e : s t a r t , s t o p , a t t r a c t , r e p e l , d ~ v i d e , s e p a r a t e , jorn, connect, s m t t e r , s c r a t c h , smash, t o u c h , l e a n , flow, s u p p o r t , hang, s i n k , s l l d e , s c r a p e , a p p e a r , d i s a p p e a r , emerge, submerge, d e f l e c t , r i s e , f a l l , grow, s h r ~ n k , waver, shake, s p r e a d , congeal, d i s s o l v e , p r e c i p i t a t e , r o l l , bend, w a ~ p , wear, c h l p , break, t e a r , e t c . , e t c . While we obviously do n o t always ( o r even u s u a l l y ) experience per- c e p t u a l images when we u s e o r h e a r such words, I s u g g e s t t h a t much of t h e machinery used d u r i n g p e r c e p t l o n i s used d u r i n g t h e p r o c e s s i n g of
language about space and i s a l s o used d u r i n g t h e p r o c e s s i n g of a b s t r a c t : d e s c r i f l i o n s .
I
do n o t f i n d i t p l a u s i b l e t h a r words l l k e t h e s e have t w b o r more completely d i f f e r e n t meanlngs whlch simply s h a r e t h e same l e x i c a l e n t r y .There a r e s i g n i f i c a n t l i n g u i s t r c g e n e r a l i - z a t i o n s t o be found i n language about p e r c e p t i o n . As an example, C l a r k I19731 d e m o n s t r a t e s b e a u t i -
f u l l y t h e s t r u c t u r a l r e g u l a r l t i e s underlying
p r e p o s i t i o n s which e x p r e s s s p a t i a l r e l a t L o n s and
5 7
t h e m e t a p h o r i c a l t r a n s f e r of s p a b i a l p r e p o s i t i o n st o d e s c r i b e time,
F i n a l l y , language plqys an i m p o r t a n t r o l e i n g u i d i n g o r d i r e c t i n g a t t e n t i o n and i n p r o v i d i n g e x p l a n a t i o n s v i a a n a l o g y o r v i a ~ o m e c t i o n s which a r e n o t d $ r e c t l y a c c e s s i b l e t o s e n s o r y p e r c e p t i o n .
I contend t h a t (1) we should s t r i v e co under- stand and t o l e a r n t o r e p r e s e n t the s e n s o r y / m o t o ~ world;
(2)
we should s t u d y t h e r e l a t i o n s h i p oflanguage t o t h e r e p r e s e n t a t i o n * of t h e world, b e i n g aware t h a t language does n o t i t s e l f r e p r e - s e n t t h e sensory/motor world, but i n s t e a d p o i n t s
-
t o t h e r e p f e s e n t p t i o n s of t h i s world v i a a s e t of word and s t t o c t u r e c o n v e n t i o n s .
The development of p e r c e p t i o n and language
What we l e a r n t o name and d e s c r l b e I n o u r e x p e r i e n c e must
i n
some
s e n s e e x i s t p r i o r t o and s e p a r a t e from t h e words a s s o c i a t e d w i t h t h e e x p e r i - ence. I b e l i e v e t h a t a n i n f a n t d e v e l o p s verye a r l y a kind of "language of p e r c e p t i o n , " i . e . a n a t u r a l , i n n a t e segmentarion of e x p e r i e n c e and o r d e r i n g of t h e importance o r i n t e r e s t of segmented i t e m s , Moreover, b e f o r e an i n f a n t e v e r l e a r n s
(or
can l e a r n ) t h e name of an o b j e c t , t h e i n f a n t m u s t (1) b e a b l e t o p e r c e i v e t h a t o b j e c t a s au n l t a r y c o n c e p t , and (2) must i n f a c t p e r c e i v e t h e u n i t a r y concept of t h e o b j e c t a s t h e most s a l i e n t c h a r a c t e r i s t i c of t h a t o b j e c t . Thus, w e assume t h a t when w e p o i n t t o a t e l e p h o n e and say " t e l e p h o n e , " t h e infant! p r e f e r s t o a t t a c h t h e name t o be e n t i r e o b j e c t and n o t t o some p r o p e r t y
(e.g. c o l o r o r s i z e ) of t h e o b j e c t .
I w i l l u s e t h e word " g e s t a l t " t o r e f e r t o such a u n i t a r y c o n c e p t , because t h e words "con- c e p t " o r "percept" may b e m i s l e a d i n g , and because
f would n o t want t o c o l n an e n t l r e l y new word, By " o b j e c t , " I w i l l mean not o n l y v i s u a l o b j e c t s , b u t a l s o a u d i t o r y " o b j e c t s , " having f i g u r e / g r o u n d r e l a t i o n s h i p s ,
such
as
a c l a p of thunder o r a word spoken I n isolation, and of c o u r s e " o b j e c t s "from
o t h e r s e n s o r y and mQtor domains a s w e l l i l As I w i l l d i s c u s s l a t e r , I b e l i e v e t h a t we can g e t around haviGg t o p o s t u l a t e p e r c e p t u a l primitives by viewing g e s t a l t s a s t h e r e s u l t of information t h e o r e t i c t y p e s of p r o c e s s e s , e . g . p r o c e s s e s which s e l e c t and a t t a c h unportance -to p o i n t s w i t h h i g h l y improbable s u r r o u n d i n g s ( f o rexample, p o l n t s of symmetry).
How much is i n n a t e ?
There h a s r e c e n t l y been a good d e a l of d i s - c u s s i o n about t h e "language" of t h o u g h t o r
'fnentalese" ( [ F o d o ~
19751
),
[Pylyshyn 19781, [Woods 19781, [Johnson-Lgird 19781). The c e n t r a l i s s u e s d i s c u s s e d i n t h e s e a c c o u n t s are: (1) t h e i n n a t e "vocabulary" of such a language ( i n n a t e c o n c e p t s ) , (2) w a y si n
which new c o n c e p t s a r e added t o m e n t a l e s e ; and (3) t h e r e l a t i o n s h i p of mentalese i t e m s t o words.I would l i k e t o focus on4 one a s p e c t of t h e s e d i s c u s s i o n s : i n n a t e c o n c e p t s , To q u o t e Pylyshyn
"There i s no e x p l a n a t i o n , n o t even t h e b e g i n - n i n g s of an approach, f o r d e a l i n g w i t h t h e accommodabion of schemata o r c o n c e p t u a l s t r u c - t u r e s i n t o ones not e x p r e a q i b l e a s d e f i n i t i o n a l composites of e x i s t i n g ones. There i s , i n o t h e r words, no i n k l i n g a s to how a c o m p l e t e l y new n o n ~ e l i m i n a b l e concept can come i n t o b e i n g , "
and l a t e r ,
"The f i r s t approach [ t o t h i s dilemma] i s t o s i m p l e a c c e p t whar seems a n i n e v i t a b l e con- c l u s i o n dnd s e e what i t e n t a i l s . T h i s i s t h e appro&h t a k e n by Fodor 119751 who simply a c c e p t s t h a t mentalese i s i n n a t e . . . "
"121is approach t o t h e i n n a t e n e s s dilemma p l a c e s t h e p u z z l e of c o n c e p t u a l development on a d i f f e r e n t mechanism from t h e u s u a l one of c o n c e p t l e a r n i n g . Now t h e ?rob lem becomes :
g i v e n t h a t most of t h c c o n c e p t s a r e i n n a t e why do t h e y only emerge a s e f f e c t i v e a f t e r c e r t a i n p e r c e p t u a l and c o g n i t i v e e x p e r i e n c e and a t v a r i o u s l e v e l s of m a t u r a t i o n ? "
Pylyshyn goes on t o s k e t c h a n o t h e r s o l u t i o n i n which mentalese i s viewed a s a s o r t of machine language f o r u s e w i t h t h e f i x e d hardware a r c h i - t e c t u r e of t h e nervous system; new c o n c e p t s c o u l d t h e n a r i s e I£ we allow t h e hardwired c o n n e c t i o n s o r architecture t o change. A s h e p o i n t s o u t , t h i s merely b u r i e s t h e problem i n hardware, and does n o t r e a l l y p r o v i d e a s o l u t i o n , b u t a d i f f e r - e n t l o c u s f o r t h e problem; a t l e a s t i t does g e t beyond t h e l i m i t a t i o n s ~ n h e r e n t i f t h e o n l y
formal concept development mechanism a v a i l a b l e i s symbolic composition.
I f i n d t h e n o t i o n of an l n n a t e language t o be u n s a t i s f y i n g , and o f f e r below a d i f f e r e n t s o r t of s o l u t i o n t o t h e problem of t h e s o u r c e of n o b e l c o n c e p t s
.
A s k e t c h of t h e development o f p e r c e p t i o n and language
I n t h i s s e c t a o n I s k e t c h what I f e e l - i s a plausible account of t h e development of p e r c e p t i o n and language. T h i s a c c o u n t i s h e d v i l y based on l n t u l t i o n and on my o b s e r v a t i o n of my t w o r h l l d r e n
(Vanessa, now 5 and Jeremy, now 3 ) , ~t t h u s r e p r e - s e n t s ap extreme c a s e of i n d u c t i v e g e n e r a l i z a t i o n . how eve^, I have a-ttempted t o a l s o c i t e t i e s w i t h and s u p p o r t i v e evidence from o t h e r work of which I am aware
-
1 w i l l be g r a t e f u l t o r e a d e r s of t h i s paper who supply r e l e v a n t s u p p o r t i n g o r c o n f l i c t ~ n g r e f e r e n c e s whlch I do n o t acknowledge.The b a s i c p o s i t i o n s I would l i k e t o a r g u e on t h e s e l s s u e s a r e a s f o l l o w s :
( 1 ) m e n t a l e s e a r i s e s o u t of p e r c e p t u a l e x p e r i e n c e , and i s n o t p e r s e i n n a t e ( i . e . p r e s e n t
a t
b i r t h ) ;( 2 ) t h e development of meritalese depends i n s t e a d bn c e r t a l n i n n a t e a b l l l t i e s and r e f l e x e s , p l d s p e r - c e p t u a l e x p e r i e n c e . The i n n a t e a b i l i t i e s 9 c a r e ( a t l e a s t ) .
a ) t h e a b i l l t y t o form " f l g u r e / g r o u n d f l p e r - c e p t u a l relationships, where f l g u r e s have d i s t i n - g u l s h i n g properties l l k e l o c a l coherence on a homogeneous background ( lob j e c t n e s s " )
,
symmetry, t e p e t i t l o n , l o c a l movement a g a i n s t a f i x e d back- ground, e t c . I w i l l assume t h a t t h e g e s t a l t s eachhave a c e r t a i n s a r i e n c e o r measure o f " i n t e r e s t - . i n g n e s s " t o t h e i n f a n t which i s a f u n c t i o n of t h e i n h e r e n t p e r c e p t u a l c h a r a c t e r i s t i c s of each
g e s t a l t , t h e o r d e r and t i m i n g of a t t e n t i o n t o v a r i o u s g e s t a l t s ( i n t u r n t h e s e a%% e v e n t u a l l y r e l a t e d t o t h e c u r r e n t g o a l b U I I ~ h y p o t h e s e s o f t h e
i n f a n t ) and t h e c u r r e n t d e g r e e of p l e a s u r e o r p a i n b e i n g e x p e r i e n c e d by t h e i n f a n t
-
a t t h e extremes of p l e a s u r e o r p a i n , g e s t a l t s have h i g h s a l i e n c e , and c o u l d become independent g o a l s t o be pursued o r i t e m s t o be avoided.b ) t h e a b i l i t y t o remember q u i t e l i t e r a l l y one o r more f i g u r e s ( " g e s t a l t s " ) from a f i g u r e ground r e l a t i o n s h i p f o r a s h o r t p e r i o d of f h e
(on t h e o r d e r of 10 s e c o n d s ) :
c ) t h e a b i l i t y t o form a s s o c i a t i o n s between g e s t a l t ' s , where by a s s o c i a t i o n I mean t h a t t h e e x p e r i e n c e of one g e s t a l t can l e a d t o t h e e x p e r i - ence of an a s s o c i a t e d g e s t a l t ,
d) i n f a n t s a l s o have r e f l e x e s and c e r t a i n irrilate b e h a v i o r s ,
such
a s c r y i n g when hungry o r I n p a i n . Throughoutt h i s
a r t i c l e , I w i l l assume t h a t t h e s e r e f l e x e s and b e h a v i o r s-
p h y s i c a l ? mental, e m o t i o n a l , e t c .-
a r e a l s o p o r t i o n of an ~ n f a n t ' s p e r c e p t u a l e x p e r i e n c e .( 3 ) The primar? g o a l of a n i n f a n t i s t o maximize i t s p l e a s u r e and minimize ~ t s p a i n , and t h i s goal d r i v e s t h e i n f a n t t o a t t e m p t t o u n d e r s t a n d i t s p e r c e p t u a l e x p e r i e n c e ;
(4) The primary mechanism of u n d e r s t a n d i n g ~ t s e x p e r i e n c e i s t h e organization of g e s t a l t s , t h i s o r g a n i z a t i o n i n v o l v e s :
a ) t h e forrpation o f a taxonomy of t h e g e s t a l t s of e x p e r i e n c e , where t h e taxonomy i s g e n e r a t e d by s u c c e s s i v e l y subdividing e x i s t i n g c a t e g o r i e s i n t o two ( u s u a l l y ) or more new c a t e g o r i e s , and
b ) t h e f o r m a t l o n of a s s o c i a t ~ o n 8 between two o r more g e s t a l t s t o form new ~ e s t a l t s .
~ e o r g a n i z a e i o n o c c u r s when p r e v i o u s taxonomxc d e c i s i o n s appear t o be d e f i c i e n t ( e . g . a r e n o t l e a d l n g t o t h e achievement of p l e a s u r e o r t h e cessation of p a i n ) , and t h e particular form chosen f o r r e o r g a n i z a t i o n depends on which g e s t a l t s a r e c u r r e n t l y a v a i l a b l e , and of t h e s e which a r e most s a l i e n t . The f o r m a t i o n o f g e s t a l t s by a s s o c i a t i o n 1s o n l y p o s s i b l e m i t i a l l y between g e s t a l t s which b o t h f a l l w i t h i n t h e t i m e p e r l o d d u r i n g which
g e s t a l t s can be remembered. Associations i n i t i a l l y a r e ( p r o b a b l y ) merely l i n k s , t h e s e l i n k s a r e them- s e l v e s l a t e r s u b - c a t e g o r i z e d i n t o temporal sequence
(elementary s o u r c e of c a u s e - e f f e c t r e l a t i o n s h i p s a'nd " s c r i p t s " ) , c o n s t a n t Sopresence (elementary s o u r c e of n o t l ~ n s af i d e n t i t y o r i n h e r e n t c o n n e c t - e d n e s s ) , and e v e n t u a l l y s e m a n t i c r e l a t e d n e s s (e.g.
t h e l i n k between t h e g e s t a l t of a p e r c e i v e d v i s u a l o b j e c t and t h e a u d l t o r y g e s t a l t of a word) a s w e l l a s o t h e r connect Ions.
(5)
Once a s s o c i a t i o n s a r e formed, items can become a v a i l a b l e a s g e s t a l t s even i f they a r e n o t a t t h e time d i r e c t l y a v a i l a b l e t o t h e external. senses; t h i s a l l m s escape from t h e narrow bounds of t h e i n i k i a l time window a s s o c i a t e d with e x t e r n a l l ypefceived g e s t a l t s , s i n c e each g e s t a l t can continue t o r e a c t i v a t e o t h e r s a s s o c i a t e d with it f o r inde- f i n i t e periods (though "habituation" and competing e x t e r n a l g e s t a l t s w i l l soon i n t e r f e r e i n g e n e r a l ) .
Taken t o g e t h e r , any g e s t a l t and t h e aseoci- a t e s i t can woke form something l i k e a "frame"
[Minsky 19751
,
every non-isolated concept thus has a frame. Default values f o r s l o t s correspond t o g e s t a l t s evoked i n t h e absence of d e f i n i t e per- c e p t u a l input. Language, then, i s a s o r t of=-
j e c t i o n , where only some of t h e items t o b e com- municated need t o a c t u a l l y be mentioned d i r e c t l y . Syntax can be viewed a s a means of crztastructfing a perspective toward t h e g e s t a l t s s e l e c t e d by words and c o n t e x t ; s p e c i f i c s t r u c t q r e s and words s e l e c t s p e c i f i c connections between g e s t a l t s , a s i n{Fillmore 19771.
Early language i s %n extreme p r o j e c t i o n : a c h i l d beginning t o speak can only output one word per sentence, l a t e r two words ( t h i s i s t h e l i m g t f o r a long time), e t c . Thus ' % a l l " when u t t e r e d
by a one-year o l d might mean "I want t h e b a l l , " "That's a b a l l , " "Where i s my b a l l ? " , "I was j u s t h i t by a b a l l , " e t c . There i s strikllllg recent evidence from t h e study of deaf c h i l d r e n d e l i b e r - a t e l y deprived of language* [Feldman, Goldiq-Meadow and Gle itman 19771 t h a t t h e s e c h i l d r e n develop language independently, and t h a t t h e length and t h e contents of t h e i r "sentences" a r e eiEtretnely s i m i l a r t o "sentences" 0-4 hearing c h i l d r e n , i n which c e r t a i n t y p e s of sentences (e.g. verb f p a t i e n t case) predominate and c e r m i n c a s e r ~ l e s
(e.g. ageot) a r e u s u a l l y omitted. I suggest tlpat t h e i r language development i s s i h i l a r because t h e i r p e r c e p t u a l experiences (and needs t o commu- n i c a t e ) a r e s i m i l a r , and t h a t t h e items chosen t o
appear i n sentences a r e t h e ones with t h e h i g h e s t s a l i e n c e .
In order t o understand p r o j e c t e d language, one must understand t h e context i n which
i t i s occurring.
sr
example, a t age 2 y e a r s 8 montxs Jeremy Waltz h e l d a new toy t r a i n up t o the telephone r e c e i v e r and s a i d "look a t t h e present I got, grammy, ") Later language development can be viewed a s l e a r n i n g t o communicate-in the absence of a shared-physical context.I n t h e d i r e c t i o n of language comprehension, we must then p o s t u l a t e a r e c o n s t r u c t i o n process. Schank [I9731 s u p p l i e s evidence t h a t ' by the age o r one y e a r , c h i l d r e n understand t h e concepts of t h e p r i m i t i v e ACTS of :oncep t u a l dependancy
;t
Schankand S e l f r i d g e [1977] have demonstrated t h a t
c h i l d r e n ' s responses,& sentences a t one year can be mimicked by a program by assuming t h e r h i l d has a s i n g l e word i n p u t b u f f e r , t h a t , ( s ) h e s e l e c t s only one word from a given input s e n t e n c e , and finally picks and executes an ACT which p l a u s i b l y could invglve t h e concept a s s o c i a t e d with the word s e l e c t e d . Thus, when t o l d "Take t h e b a l l t o
Roqger" a c h i l d might simply get t h e b a l l , or take t h e b a l l t o someone e l s e ( i f b a l l were s e l e c t e d ) or run t o Roger ( i f Roger were s e l e c t e d ) .
I would f i n a l l y l i k e t o emphasize t h e i d e a t h a t language a t a l l ages (not j u s t f o r c h i l d r e n ) involves t h e complementary processes o f p r o j e c t i o n from and r e c o n s t r u c t i o n i n t o mentalese. (See a l s o Marcus [I9781 f o r more evidence on i n p u t b u t t e r r e s t r i c t i o n s i n a d u l t s . )
(6) New g e s t a l t s can probably be i n t e g r a t e d into- t h e i n f a n t ' s taxonomy only one a t a time, i . e . , new items must be a s s o c i a t e d w i t h items which a r e
already p a r t of t h e organized taxonomy. Thus words. would u s u a l l y be learned f o r &terns which a r e
a l r e a d y organized c o n c e p t u a l l y , a 1 though novel words could be used So point out t h e need f o r new c a t e g o r i e s (e.g. by po*.tin& out t h a t a dog and sheep a r e d i f f e r e n t ) * The net r e s u l t i s t h e l i k e - lihood of
many
more t o t a l concepts ( i n d i v i d u a l concepts, a s s o c i a t e d i n d i v i d u a l concepts, and a s s o c i a t i o n s of a s s o c i a t i o n s ) than t h e r e a r e con- c e p t s w i t h words a t t a c h e d t o them ([Woods 19781 comes t o a s i m i l a r conclusion).P r o p e r t i e s c a n be s e l e c t i v e l y named by (a) p r e s e n t i n g two or more q u i t e d i f f e r e n t o b j e c t s which s h a r e a s i n g l e p r o p e r t y , say c o l o r , o r @)
c o n t r a s t i n g o b j e c t s which d i f f e r i n only a s i n g l e property (big/small)
,
o r (c) having names rirmly enougb i nplat%
s o t h a t items pointed t o can be understood a s d e t a i l s or p r o p e r t i e s , not t h e name per se. ( ( a ) and (b) a r e lih Winston's I19751 "near-misses"). I would Like t o p o i n t outtm
analogy given above and t h e use of metaphor t o s e l e c t and high-light r e l a t i o n s h i p s f o r which w e do not already have names.Concepts a r e a t l e a s t p o t e n t i a l l y completely unambiguous, with the except i o n of laud i t o r y
g e s t a l t s corresponding t o words.§ C l e a r l y some a u d i t o r y g e s t a l t s corresponding t o words can b e a s s o c i a t e d with two or more d i f f e r e n t g e s t a l t s
(e.g. f a i r ( c a r n i v a l ) , f a i r ( c l e a r o r b e a u t i f u l ) , f a r e ( t r a v e l f e e ) , f a r e (menu i t e m s ) ) ; I suggest t h a t
in
o t d e r t o be understood unambiguously, such words must occur i n a context where one und r-l y i n g concept i s a s s o c i a t e d much more close?) with concepts i n t h e c u r r e n t context ( v e r b a l o r o t h e r p e r c e p t u a l ) .
This
i d e a i s r e l a t e d t o work i nspreading a ~ t i v a t i o n f o r semantic networks [ C o l l i n s and Loftus 19721, a s w e l l a s t o "focussing as i n
[1978].
* 4
B e c a s e the P h i r a d l p h i a school system b e l i e v e s t h a t l i p - r e a d i n g and vocal speech a r e b e s t , .and t h a t Learning s i g n language d e s t r o y s t h e w i l l i q - n e s s of c h i l d r e n t o l e a r n t o l i p read and speak.
t~.g MOVE (a body p a r t ) ,
INGEST,
EXPEL,PTRANS
(trans£ e r a p h y s i c a l ob l e c t ),
ATRANS ( t r a n s f e r an a b s t r a c t r e l a t i o n s h i p , e.g. possession). MTRANS( t r a n s f e r information between or w i t h i n animals), PROPEL (apply force t o ) , GRASP, SPEAK (make a n o i s e , and ATTEND (Focus a sense organ on an o b j e c t [Schank 1975)
.
( 1 )
Jackendoff [I9751 and Gruber[I9651
have p o i n t -ed out evidence that l i n g u i s t i c schemata we develop t o deecrikie GO, BE and STAY e v e n t s i n t h e s e n s o r y / m t o r C"position") world a r e l a t e r - t r a n s f e r r e d v i a a broad metaphor t o d e s c r i b e e v e n t s i n a b s t r a c t worlds (possess i o n , "identification'! and "circum- s t a n t i a l " ) . Thus we l e a r n t o u s e p a r a l l e l s u r f a c e s t r u c t u r e s f o r c o n c e p t u a l l y very d i f f e r e n t s e n t e n c e s Like :
( l a ) The d i s h e s s t a y e d i n t h e s i n k ( p o s i t i o n ) ( l b ) The b u s i n e i s s t a y e d i n t h e family (posses-
s i o n ) .
(2a) H i s puppy went home ( p o s i t i o n ) ,
(2b) H i s f a c e went white ( i d e n t i f i c a t i o n ) .
(3a)
She
g o t i n t o h e r c a r and went t o work( p o s i t i o n ) .
(3b) She s a t down a t h e r desk and went t o work ( c i r c u m s t a n t i a l ) .
Along t h e s e same l i n e s , t h e r e a r e s t r i k i n g p a r a l l e l s i n t h e s t r u c t u r e s of Schank's [I9751 conceptual dependency diagrams f o r PTRANS, ATRANS, and MTRANS (see e a r l i e r f o o t n o t e ) . Reddy [I9771 h a s d e s c r i b e d what
ha
c a l l s t h e "conduit metaphor" f o r l i n g u i s t i c c o m u n i c a t i o n i n which we t y p i c a l l y speak of i d e a s and i n f o r m a t i o n a s though t h e y w i r e o b j e c t s which could be given 6r shipped t o o t h e r s who need only t o look a t t h e " o b j e c t s " t o under-s t a n d them, Thus we
say
"You a r e n ' t g e t t i n g yourmessage a c r o s s , ' ' "She gave me some good ide-as,I1 "He kept h i s thoughts t o himself
,"
"Let me give you a p i e c e of advice," e t c . (Reddy h a s compiled a very long l i s t of examples.)These examples suggest many deep and f a s c i - n a t i n g q u e s t i o n s . It seems c l e a r t h a t t h e same words and s i m i l a r s y n t a c t i c s t r u c t u r e s can be
t r a n s f e r r e d t o d e s c r i b e q u i t e d i f f e r e n t phenomena. What i n t e r n a l s t r u c t u r e s ( i f any) a r e a l s o t r a n s -
f e r r e d i n such c a s e s ? What p e r c e p t u a l c r i t e r i a a r e used t o c l a s s i f y e v e n t s t o b e g i n w i t h ? u l t i - mately? How does a c h i l d t r a n s f e r o b s e r v a t i q n t o i m i t a t i o n ? ,*How a r e memories of s p e c i f i c e v e n t s g e n e r a l i z e 3 to-form-event t y p e s , and how a r e t h e representations of e v e n t t y p e s r e l a t e d
to
memories of s p e c i f i c e v e n t s ?To answer one p o r t i o n of t h e s e q u e s t i o n s , i t scems c l e a r from an economic p o i n t of n e w t h a t i f s y n t a x and words a r e conventional and n o t i n n a t e , we would want t o i n c l u d e only enough d i s t i n c t syn- t a c t i c s t r u c t u r e s and words t o make d i s t i n c t i o n s t h a t a r e necessary and t o unambiguously s e l e c t mentalese r e p r e s e n t a t i o n s , We would t h u s p r e d i c t t h a t words and s y n t a c t i c s t r u c t u r e s would be h e a v i - l y shared (see a l s o [Woods 19781).
I suggest t h a t i n t e r n a l mentalese s t r u c t u r e s a r e
-
not t r a n s f e r r e d , b u t t h a t , j u s t a s s i n g l e words can p o i n t t o m r e t h a n one concept, t h e s e p a r a l l e l s t r u c t u r e s f o r verbs can p o i n t t o more t h a n one mentalese s t r u c t u r e . However, t h e r e are l i m i t a - t i o n s : t h e s t r u c t u r e s pointed t o must $hare some p r o p e r t i e s , e,g, t h e number'of c a s e r o l e s mat bet h e same, ahd s e l e c t i o n r e s t r i c t i o n s on c a s e r o l e s l o t s should be s u f f i c i e n t t o choose t h e a p p r o p r i a t e concept unambiguously.
Another i n t e r e s t i n g q u e s t i o n involves t h e s t a t u s of i n f e r e n t i a l knowledge
-
i s it a t t a c h e d t o mentalese concepts o r t o words? S u r p r i s i n g l y , t h e r e may be some evidence t h a t i n f e r e n t i a l know-ledge i s a t t a c h e d t o w a d s . In t h e p o s i t i o n world
b u
we know t h a t an o b j e c t can o n l y b e i n one p l a c e a ta t h e , t h a t two o b j e c t s cannot occupy t h e same p l a c e a t t h e t h e same time, e t c . Some of t h e s e
same i n f e r e n c e s may be c a r r i e d o v e r i n a p p r o p r i a t e l y t o t h e p o s s e s s i o n world: f o r example my c h i l d r e n appeared t o have some d i f f i c u l t y f u l l y understand- ing concepts l i k e " j o i n t ownership1'. I f we assume t h a t i n t h e c o n c e p t u a l t r a n s f e r a c h i l d c r e a t e s an imaginary "possession b a s k e t " f o r each person, and that t h e i n t e r i o r s
of
twa, such b a s k e t s cannot i n t e r s e c t , t h e n o b j e c t s must be i n one basket o r another,-and s e n t e n c e s Like "Re83 [our Qog] belongs t o a l lOF
u s b u t h e ' s r e a l l y mine" (Vanessa,about 4-112) become more i n t e l l i g i b l e , (There a r e of course o t h e r p l a u s i b l z e x p l a n a t i o n s f o r t h i s s e n t e n c e , ) Reddy [I9771 h a s a l s o p o i n t e d out ways i n whlch t h e "conduit me taphorI1 f o r communication minimizes t h e c o n s t r u c t i v e r o l e of t h e l i s t e n e r , and l e a d s t o t h e n o t i o n t h a t f a i l u r e of communi- c a t i o n i s due p r i m a r i l y t o t h e speaker. Whorf
'
s[1956]
i d e a s and d a t a may be r e l e v a n t h e r e a l s o , The r o l e of a e s t h e t i c sI f e e l t h a t i t i s important t o keep our c e n t r a l a t t e n t i a n on t h e f u n c t i o n a l r o l e s of p e r - c e p t i o n and language f o r t h e s u r v i v a l of t h e i n f a n t , which I
take
t o
be the p r imtirygoal
I n e v o l u t i o n , and She p l a c e where we must looku l t i m a t e l y f o r e x p l a n a t i o n s about i n n a t e a b i l i t i e s and e a r l y development. I a c c e p t Pugh's 119771 s u g g e s t i o n that a l l our v a l u e s ( p l e a s u t e , p a i n , good, bad, happy, unhappy, e t c . ) s e r v e a s "second- a r y v a l u e s , " i . e . a s s u r r o g a t e v a l u e s n f o r t h e primary v a l u e of s u r v i v a l . We have t h e s e second- a r y values because t h e y allow u s t o d i s t i n g u i s h situations which have s i g n i f i c a n t p o % i t i v e o r n e g a t i v e s u r v i v a l value. Woods [ I 9 7 8 3 has p o i n t e d out t h e s a r v i v a l value of language I n a l l o w i n g t h e t r a n s m i s s i o n of knowledge i n t h e absence of
g e n e t i c a l l y % i r e d t ' behavior, (See [Dennett 19741
.
I s u g g e s t t h a t t h e v a l u e s l i k e goodness, economy, a e s t h e t i c s , and i n t e r e s t i n g n e s s a r e p e r - w i v e i n our pe<rceptual s y s t e m and i n t h e
mechanisms which e v a l u a t e hypothesized taxonomies of e x p e r i e n c e , We a t t e n d t o s e n s o r y items which i n t e r e s t u s , s t o r e d e s c r i p t i o n s i n ways t h a t a r e a e s t h e t i c a l l y s a t i ~ f y i n g (e.g. have .good symmetry. p r o p e r t i e s , d i v i d e phenomena iffto b a l a ~ c e d c a t e - g o r i e s , h e l p avoid d a n g l i n g , unexplained phenomena, e t e . ) , i n a d d i t i o n t q e v a l u a t i n g whether our hypo- t h e s e s a r e h e l p i n g us g e t what we wark.
Development of a taxonomy of e x p e r i e n c e
Let us assume t h a t we s t a r t w i t h a u a & t a r y concept of t h e world, and examine a p l a u s i b l e development of d i s t i n c t i o n s i n t h e v i s u a l " w o r l d . * The f i r s t s o r t of d i s t n c t i o n l i k e l y i s moving/not moving, where ''moving" x e f e r s t o a f i g u r e
on
aground. The "movingt> c a t e g o r y i s soon d i v i d e d i n t o c a t e g o r i e s f o r moving items over which the
infant
h a s some c o n t t o l and moving items where (s)he does n o t (random motions). Later, t h i s category i ss e p a r a t e d i n t o items where t h e i n f a n t h a s d i r e c t c o n t r o l (e.g. p a r t s of t h e body), and o t h e r s (e.g.
*
p a r e n t s who sometimes come when t h e c h i l d c r i e s , o b j e c t s nearby which can sometimes be h i t o r touched by body movements, e t c . ) ,
Out of t h i s p r o c e s s e v e n t u a l l y , come b a s i c d i s t i n c t i o n s l i k e s e l f l o t h e r , mindlbody
,
near(reachable) / f a r (unreachable) ; a l s o , c a t e g o r i e s from various sense m o d a l i t i e s can be merged (ob- j e c t s from t h e t a c t i l e and v i s u a l worlds, mother from t h e v i s u a l , a u d i t o r y and t a c t i l e worlds, e t c , ) I have a wealth of o b s e r v a t i o n s on t h e development of t h e s e d i s t i n c t i o n s from watching my c h i l d r e n which cannot be expounded f u r t h e r i n t h e space here. 1 would l i k e t o suggest i n passing t h a t t h e development of t h i s taxonomy can have deep psycho-
l o g i c a l s i g n i f i c a n c e
-
t o p o i n t out one example, consider t h e following c o n t r a s t i n g s i t u a t i o n s :(1) p a r e n t s a r e a t t e n t i v e t o an i n f a n t ' s c r i e s and thus a r e thus i n i t i a l l y w i t h i n t h e c a t e g o r y of moving items c o n t r o l l e d by t h e Ynfant vs.
(2) t h e p a r e n t s a r e i n a t t e n t i v e t o c r i e s , and t h u s
i n i t i a l l y a r e c l a s s i f i e d i n t h e "random motion" category. See Wilber [I9761 f o r an e x t e n s i o n e x p l o r a t i o n of t h e development of fundamental d u a l i t i e s .
A computer model of g e s t a s t format ion
My r e c e n t work i n v i s i o n [Waltz 19781 h a s explored computatiorial methods f o r f i n d i n g p o i n t s i n scenes which have h i g h information c o n t e n t , which I suggest a8 t h e primary b a s i c of t h e de- f i n i t i o n of " i n t e r e s t i n g n e s s , " which i n t u r n
should d r i v e a t t e n t i o n .
Because we (George Hadden and I) have been working with s t a t i c s c e n e s , our programs do n o t
s p e r a t e moving i l g u r e s on grounds (which I t a k e t o be important, a s should be obvious from e a r l i e r d i s c u s s i o n s ) . * We have concentrated i n s t e a d on methods f o r funding symmetry axes, p p i n t s w i t h
high c u r v a t u r e , edges and edge completions, i s o -
l a t e d o b j e c t s on background$, s p a t i a l l y r e p e a t e d p a t t e r n s , and c h a r a c t e r i s t i c s t e x t u r e elements, In each c a s e we a r e assuming t h a t p r o c e s s e s t h a t be bottom-up and task-independent (although I
would b e w i l l i n g t o i n c l u d e some s p e c i a l pre- f e r e n c e s f o r t h i n g s l i k e v e r t i c a l o r h o r i z o n t a l d i r e c t i o n s ) .
T h i s work i s based on t h e n o t i o n t h a t shape
is the b e s t "property" with which t o s o r t items,
i n t o c a t e g o r i e s . Our programs attempt t o l o c a t e unique p o i n t s of h i g h information (e.g. t h e c e n t e r of a c i r c l e ) and t o s t o r e a t t h a t p o i n t s u f f i - c i e n t i n £ o r k t io n t o "unfoldt1 a shape envelope of an o b j e c t ( t h e shape envelope i s t h e same f o r la s d l i d l i n e r e c t a n g l e , d o t t e d l i n e r e c t a n g l e , r e c t a n g l e with a n o t c h removed from t h e s i d e , e t c . ) . The n o t i o n h e r e i s t h a t shape should be represented h i e r a r c h i - c a l l y , with t h e shape envelope t y p i c a l l y a t t h e t o p of t h e h i e r a r c h y , and de- v i a t i o n s from t h e shape envelope l o c a t e d lower, along with o t h e r p r o p e r t i e s l i k e c o l o r , s i z e , e t c .
However, i n t h e long run v i s u a l o b j e c t s should be described in a more f l e x i b l e s t r u c t u r e which draws on a
l i s t
of p p p e r t i e s ; my c u r r e n tf a v o r i t e l i s t of p r o p e r t i e s comes from Pylyshyn [1977b] who i n t u r n g o t t h e l i s t from Basso [196&]. Basso i d e n t i f i e d t h e items through t h e a n a l y s i s
of c l a s s i f i c a t o r y morphemes i n d i v e r s e languages. H e i d e n t i f i e s semantic dimens ions : animal Inon- animal, enclosed/non-enclosed; s o l i d / p l a s t i c / l i q u i d ; one/two/more t h a t two; r i g i d l n o n r i g i d ;
h o r i z o n t a l l e n g t h 2 3 times width o r h e i g h t / tlequidimensional't; p o r t a b l e / n o n p o r t a b l e . These c a n be combined t o form c a t e g o r i e s whgch r e c u r commonly i n o t h e r c u l t u r e s , e.g. " r i g i d and extended i n one dimensionll ( p e n c i l , k n i f e
,
c i g a - r e t t e ) ; " r i g i d end equidimensional" ( p a i l , l i g h t bulb, egg, box, c o i n , book); l l f l e x i b l e and ex- tended i n two dimensions" (paper, b l a n k e t,
s h i r t );
" f l e x i b l e and extended i n one dimens ion" ( r o p e , be1 t
,
c h a i n ) .Of importance i n a l l t h e s e c a s e s i s t h a t t h e d e s c r i p t i o n s be h i e r a r c h i c a l , w i t h meaningful g e n e r a l i z a t i o n s a t t h e t o p of t h e h i e r a r c h y ( s e e P r s p a r a t a and Ray [I9721 f o r o t h e r i d e a s along t h e s e l i n e s t h a t we have adopted), and t h e d e s - c r i p t i o n be capable of being generated bottom up.
V i s u a l imagery
My p o s i t i o n may be a c c e p t a b l e t o both
Pylyshyn [I9731 and Kosslyn [1978]. With Pylyshyn, I b e l i e v e t h a t v i s u a l d e s c r i p t i o n s a r e p r o p o s i - t i o n a l , and t h a t t h e d e s c r i p t i o n s a r e organized h i e r a r c h i c a l l y . However, a s argued i n t h e l a s t s e c t i o n , shape seems t c r be t h e primary d i s t i n - guishing p r o p e r t y of o b j e c t s , and we have reason t o b e l i e v e t h a t shape can i n g e n e r a l be r e p t e - s e n t e d r a t h e r compactly w i t h r e s p e c t t o some
p o i n t (e.g. of symmetry o r a c e n t r o i d ) . I suggest t h a t s h a ~ e r e p r e s e n t a t Ions may a c t u a F l y be capa- b l e of being "run backwards" o r "unfolded, l1 and t h a t t h e r e s u l t may be a c t i v a t i o n of p o r t i o n s of our b r a i n s ( v i s u a l c o r t e x ? ) which a r e a l s o a c t i - vated when an item i s d i r e c t l y perceived.
I n t h i s view, v i s u a l l y imagery could provide u s e f u l c l u e s about t h e n a t u r e of shape r e p r e s e n t a - t i o n . However v i s u a l imagery does n o t seem t o be g e n e r a l l y experiences o r used
-
based on informal q u e s t i o n i n g of my c l a s s e s , fewer than h a l f of e n g i n e e r i n g s t u d e n t s (who might b e expected t o v i s u a l i z e more f r e q u e n t l y t h a n average) r e p o r t o t h e r t h a n o c c a s i o n a l use of imagery. (As a per- son who does u s e v i s u a l imagery e x t e n s i v e l y ,I
found t h i s r e s u l t s u r p r i s i n g . ) Perhaps imagery i s a l a t e n t t a l e n t which c a n be developed; once developed I b e l i e v e it h a s s i g n i f i c a n t value f o r problem s o l v i n g , o r g a n i z a t i o n of m a t e r i a l , and memor i z at ion.*
Moving f i g u r e s a r e however t r i v i a l t a compute by s u b t r a c t i o n o f = s u c c e s s i v e frames of a moving sccsz.
As discussed i n Bajcsy & J o s h i [1978], i n a d u l t speech shape i s d e s c r i b e d v e r b a l l y by r e f e r r i n g t o o t h e r famZliar (or c a n o n i c a l ) o b j e c t s . However, i n o r d e r t o n o t e t h e s i m i l a r i t y of o b j e c t s , we q u s t have n e u t r a l d e s c r i p t i o n s of each, e . g , t h e
I n a r e l a t e d v e i n ,
I am
i n t r i g u e d by (and intend t o follow up f u x t h e r ) t h e i d e s t h a t we may organize memory i n such a way t h a t we can use p e r c e p t u a l s t r a t e g i e s f o r understanding i t s con- t e n t s. Two p a r t i c u l a r l y s u g g e s t i v e phenomena(other t h a n v i s u a l imagery) :
(1)
The s t r i k i n g s h i l a r i t y of some memories t o sensory phenomena: i n order? t o r e t r i e v e t h e punchline of a joke o r c o n t e n t of a s t o r y , I sometimes have t o go through t h e whole joke o r s t o r y ; I can "play back" music; e t c . ,(2) r e c e n t work by Fillmore [I9771 and Grosz [I9781 which suggests t h a t language may guide an anglog of t h e a t t e n t i o n process by suggesting
a
p e r s p e c t i v e from which t o view memory s t r u c t u r e (s) as they a r e r e t r i e v e d .
Can we d i s p e n s e w i t h t h e i d e a of i n n a t e i d e a s ?
In o r d e r t o show t h a t we can account f o r mestalese without r e q u i r i n g i n n a t e i d e a s , I muet shaw (1) t h a t t h e mechanisms proposed a r e c a p a b l e
of
g e n e r a t i n g a l l t h e p r i m i t i v e concepts ofmentalese, and (2) t h a t I have not simply b u r i e d i n q a t e i d e a s somewhere i n t h e mechanisms. Let me say immediately, r e l a t i v e t o p o i n t (2) t h a t t h e r e
are
some i n n a t e i d e a s i n my account; one s e t of i d e a s a r e r e l a t e d t o t h e values (good/bad,symetrical/nonsymmetrical,
e t c . ) d i s c u s s e de a r l ~ e r . There must a l s o be i d e a s r e l a t i n g t o g e n e r a t i n g hypotheses on which t h e values can o p e r a t e , and t h e i d e a qf ob jectness ( i f t h l s c a n be c a l l e d an i d e a ) must b e present. Hypothesis g e n e r a t i o n might seem a candidate for f u r t h e r seazch f o r embedded i d e a s ; howexer, a s I have r e s c r i b e d i t , h y p o t h e s i s g e n e r a t i o n
i s
p r i m a r i l ya
categorizing o p e r a t i o n where it a c t s onthe
"raw m a t e r i a l " of p e r c e p t i o n . On t h e whole f donot belLeve t h a t ~t i s d i f f i c u l t t o accept t h e s o r t s of i h n a t e i d e a s which remain i n my account;
It i s much more d i f f i c u l t t o make a con- vlncing c a s e f o r t h e s u f f i c i e n c y of t h e s e
mechanisms t o e x p l a i n rnentalese. (The s i t u a t i o n i s not a i d e d by the f a c t t h a t t h e r e a r e few
suggest ions corhcerning t h e n a t u r e of n e n t a l e s e , l e t alone g e n e r a l agreement on i t s n a t u r e . )
I
have d e a l t a t l e a s t b r i e f l y h e r e with p h y s i n a l o b j e c t s (from t h e p o i n t s of view of a l l s e n s e s ) , p r o p e r t i e s , a c t i o n s ( t o a s l i g h t d q r e e I do have what I f e e l i s a reasonahae kccount), cause-e f f e c t r e l a t i o n s h b p s , a s p e c t s of t h e mind-body problem, a s w e l l i as a number of o t h e r concepts. h a t i s missing? The two major a r e a s I am awara of a r e (1) q u a n t i f i c a t i o n (I suggest t h i s could be handled by assuming t h a t i t s ~ r i g i q s a r e i n o p e r a t i o n s on f i n i t i e s e t s ) ; and ( 2 ) l o g l c a l
operations
(probably t h e s e a l s o need t o be i n n a t e ) .It has been a long t i m e s f n c e I read Koffka 119351 and ~ i a g q t ' s works ( e . g . [I9671 and
[ P i a g e t & a h a l d e r 1967]), b u t c Learly many of t h e i d e a s i n t h i s paper c a n b e t r a c e d t o &hose two sources. I had n o t r e a d ~ a c e n d o f £"s [I9781 paper i n t h i s volume b e f o r e w r i t i n g t h i ~ paper, "but I wish I had
bdbn
able t o .I would s s p e c i a l l y l i k e t o acknowledge
cne
i d e a s and c r i t i c i s m s I have r e c e i v e d i n conver- s a t i o n s
with
Bill.Woods, P h i l Johnson-Laird, Harry Klopf, Loig Boggess, and J e f f Gibbons.References
Bajcsy, R. and J o s b i , A. (1978), f i e problem of ntyning shapes : v i s i e n - language i n t e r f a c e . I n T
INLAP-2
Basso, K. H. (196@), The w e s t e r n apache c l a s b i f i -
c a t o r y verb system: a formal a n a l y s i s ,
-
South- western J o u r n a l of Anthropology 24, 252-266.Bobrow, D. G. and Winograd, T. 719-77), An overview of KRL, a knowledge r e p r e s e n t a t i o n language. Cognitive Science
1,
1, 1937.Clark,
H.
H. (1973). Space, t i m e , semantics, andt h e c h i l d . In
T.
E. Moore ( e d , ) Cognitive Development and t h e Acqudgition ad Language,Academic P r e s s ,
N.Y.
,
27-63.C o l l i n s , A. and Loftus,
E.
(1975), A s p r e a d i n g a c t i v a t i o n theory of semantic processing. P s y c h o l o ~ i c a l Review 82,(5).
Dennett, D. (1974), Cited by Woods [I9781 a s s o u r c e of many i d e a s ; I could not l o c a t e c i t a t i o n ,
Feldman, H., Goldin-Meadow, S, and Gle$tman, L. (1977,), Beyond Herodotus: The c r e a t i o n of language by l i n g u i s t i c a l l y deprived deaf
c h i l d r e d . I n A. Lock (ed,) Action, Gesture and Symbol: The Emergence o f Language, Academic
Press,
F i l l m o r e , C. (1977). The c a s e f o r c a s e reopened. D r a f t of paper t o appear i n SyntaX and semantics s e r i e s .
Fodor, J. (1975), The language of thought. New York: Crowell,
Goguen N. (1973), A p r o c e d u r a l d e s c r i p t i o n of s p a t i a l p r e p o s i t i o n s . (M.S. t h e s i s ) Urrdversity of Pennsylvania, Dept. of Computer
GroSz, B. J. (1978), FocussYng i n d i a l o g . I n TINLAP-2.
Gruber, J. S. (1965), S t u d i e s i n l e x i c a l " r e l a t i o n s . Unpublished Ph.D. d i s s e r t a t i d n , MIT, Cambridge, MA.
Jackendoff, R. f1975), A system of semantic p r i m i t i v e s . I n R. Schank and
Be
Nash-Webber(eds.) T h e o r e t i c a l I s s u e s i n Natural Languagc Processin&, ACL, Arlington, VA,
Jackendoff, R. (1978), An argument about t h e composition of conceptual s t r u c t u r e . In
J o h n s o n - ~ a i r d (1978), Mental models of meaning. Paper p r e s e n t e d a t t h e Sloan Workshop on
Fof f k a ,
K.
(1935), P r i n c i p l e s of g e s t a l t . psychology, Harcourt Brace,New
York.Kosslyn, S,
M.
(1978),OR
t h e o n t o l o g i c a l s t a t u s of visual' mental images. Tn TINLAP-2.Kuipers, B. J. (1975), A frame f o r frames:, r e p r e s e n t i n g knowledge f o r r e c o g n i t ion@. I n D. Bobrow 6r A. C o l l i n s , R e p r e s e n t a t i o n and t h d e r s t a n d i n g , Academic, New York, 151-184.
Lakoff,
G,
(1978), Comments d u r i n g colloquium a t Dept. of L i n g u i s t i c s , Univ. of I l l i n o i s , A p r i l19Z8.
Marcus, M. (1978), A computational account of some c o n s t r a i n t s on language. I n TINLAP-2.
M i l l e r ,
G,
A. and Johnson-Laird, P. (1976), Language and P e r c e p t h n , Harvard U n i v e r s i t y p r e s s , Cambridge, MA.Minsky, M, L, (1975), A framework f o r r e p r e - s e n t i n g knowledge. TXI Winston (ed,)
The
Psy.chology of Computer Vision, McGraw-Hill,
N.Y.Novak, G. S. (1976), Computer understanding of p h y s i c s probleps s t a t e d i n n a t u r a l language. Tech. Report NL-30, Dept. of'cpmputer Science, Univ. of Texas, Austin.
P i a g e t , J. (1?67), Six Psyqholpg_i_cal s t u d i e s , Vintage, New York.
P i a g e t , J, and I n h e l d e r , B. (19671, The C h i l d ' s Conception of Space. ' N o r t o n , New York.
P r e p a r a t a , F P. and Ray, S. R. (1972), An
apprba_ch t o a r t i f i c i a l nonsymbolic~cognition. ~ n f o r m a t i o n Sciences 4 , 65-86.
Pugh, G. E. (1977), The B i o l d g i c a l O r i g i n of ~ u m a n Values, Basic Books, New ~ o r k .
Pylyshyn, Z.
,W.
(6973), What t h e mind's ege t e l l s t h e mind's b r a i n : a c r i t i q u e of mental imagery, Psychological B u l l e t i n 80, 1, 1-24.Pytyshyn, 2. W. (1977a). What does i t - t a k e t o bookstrap
a
language? In- Language Learnine; and Thought, Academic P r e s s ,N.Y.,-
37-45.Pylyshyn, 2. W. (1977b), c h i l d r e n ' s i n t e r n a l d e s c r i p t i o n s : In Lanmaae
,
Learning, and Thought, Academic P r e s s ,N.Y.,
169-176.Pylyshyn, 2. W. (1978), What h a s language t o do w i t h perception? Some'speculations on t h e Linguarnent is. I n TINLAP-2.
Reddy,,Ji. (1977), k m a r k e d e l i v e r e d a t t h e Con- f e r e n c e on Metaphor and
Thought,
U n i v e r s i t y of I l l i n o i s , Urbana, Sept. 1977.Schank, R. C. ( 1 9 7 9 , The develapment of conceptual s t r u c t u r e s i n c h i l d r e n . Memo AIM-203, Stanford A I Lab., S t a p f o r d , CA.
-ScHaqk, R. C. (1975),
The
p r i m i t i v e ACTSd!
conceptual dependency. In R. Schank & B. Nash- Webber, T h e o r e t i c a l I s s u e s i n ~ a t u r a f LanguageProcessing;'ACL, A r l i n g t o n , VA, 34-7.
Schahk,
R.
C. and Abelson, R. P. (1977), S c r i p t s , P l a n s , Goals; and Understandinq, Lawrence Erlbaum,N s
J.Schank, fE.c C. and s e l f r i d g e ,
H.
(1977), How t o learn/what t o l e a r n . Proceedings of t h e 5 t h I n t ' 1 J o i n t Conf, on A r t i f i c i a l ~ n t e l l i g e n c e , MIT, Cambridge, MA, 8-14.Simmons, R.
F a
(1975), The Clowns Microworld. I n R. Schank and B, Nash-Webber (eds.) T h e o r e t i c a l I s s u e s i n Natural Language P r o c e s s i n g , ACL, A r l i n g t o n , VA.Tenenbaum,
.
and Wayl, S. (1975), A r e g i o n a n a l y s i s subsystem f o r i n t e r a c t i v e scene a n a l y s i s . Advance Papers of t h e 4 t h I n t ' 1. J o i n t Conf. on A r t i f i c i a l I n t e l l i g e n c e , T b i l i s i , USSR, 682-7,Waltz, D. L. (1978), A p o d e l l f o r , Lot; level. v i s i o n . I n Am Hanson & E. Riseman (eds.) Machine Vision, Academic P r e s s , N.Y. ( t o appear).
Whorf
,
B. L. (1956 ),
Language, Thofight and R e a l i t y , MIT P r e s s , Cambridge, MAWilber, K. (1977), The Spectrum of Consciousness, Quest.
Winston, P. H. (1975), Learning s t r u c t u r a l des- cr.iptions from examples. I n Winston (ed.)
The
Psychology of Computer Vision, McGraw-Hill,NY.
Woodsr
W.
A. (1978), Procedural semantics a s a theory of meaning. D r a f t of paper p r e s e n t e d a t Sloan Workshop on Computational Aspects of L i n g u i s t i c S t r u c t u r e and Discourse S e t t i n g , Univ. of Pennsylvania, May 1978,Yakimovsky,
![Investigation of the electronic and thermal rearrangement properties of SPIRO[2,4]HEPTA-4,6-DIENES](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==)