The NOMAD System: Expectation Based Detection and Correction of Errors during Understanding of Syntactically and Semantically Ill Formed Text

(1)

Expectation-Based Detection and

Correction of Errors during Understanding of

Syntactically and Semantically Ill-Formed Text 1

R i c h a r d H . G r a n g e r

A r t i f i c i a l I n t e l l i g e n c e P r o j e c t , C o m p u t e r S c i e n c e D e p a r t m e n t and C o g n i t i v e S c i e n c e s P r o g r a m

U n i v e r s i t y of C a l i f o r n i a Irvine, C A 9 2 7 1 7

Most large text-understanding systems have been designed under the assumption that the input text will be in reasonably " n e a t " form (for example, newspaper stories and other edited texts). However, a great deal of natural language text (for example, memos, messages, rough drafts, conversation transcripts, etc.) have features that differ significantly from " n e a t " texts, posing special problems for readers, such as misspelled words, missing words, poor syntactic construction, unclear or ambiguous interpretation, missing crucial punctuation, etc. Our solution to these problems is to make use of expectations, based both on knowledge of surface English and on world knowledge of the situation being described. These syntactic and semantic expectations can be used to figure out unknown words from context, constrain the possible word senses of words with multiple meanings (ambiguity), fill in missing words (ellipsis), and resolve referents (anaphora). This method of using e x p e c t a - tions to aid the understanding of " s c r u f f y " texts has been incorporated into a working computer program called NOMAD, which understands scruffy texts in the domain of Navy ship-to-shore messages.

1. Introduction

The NOMAD system takes unedited English input in a constrained domain, and works interactively with the user to e n c o d e the message into d a t a b a s e - r e a d a b l e form. The unedited texts in this domain are Naval s h i p - t o - s h o r e messages, written in ' t e l e g r a p h i c ' E n g - lish, often leaving out nouns and verbs, crucial punctuation (such as periods), and making use of ad hoc abbreviations of words. In addition to these problems of surface-text processing, these texts can c o n t a i n problems of interpretation - that is, which of several objects is being referred to, or which possible goal inference is implied. These semantic processing problems are not easily detectable or solvable based on the surface text alone but rather require a data base of

1 This research was supported in part by the Naval Ocean Systems Center under contracts N-00123-81-C-1078 and N66001- 83-C-0255, and by the National Science Foundation under grant IST-81-20685.

knowledge about the domain of discourse, in this case ship movements.

Here are examples of each of these two types of problems. First, one with a n u m b e r of surface-text errors:

(1) ' L o c k e d on open fired d e s t r o y e d '

Example (1) is missing crucial p u n c t u a t i o n (no b o u n d - aries separating the three clauses from each other), is missing subjects and objects for all three verb phrases, and has a tense mismatch in the middle phrase ( ' o p e n fired'). (This is an actual message in the corpus provided to us by the Navy, not a c o n s t r u c t e d example.) NOMAD's output from this example is:

We aimed at an u n k n o w n object. We fired at the object.

The object was destroyed.

A second message has, in addition to some surface problems, a goal-based interpretation problem:

Copyright 1984 by the Association for Computational Linguistics. Permission to copy without fee all or part of this material is granted provided that the copies are not made for direct commercial advantage and the Journal reference and this copyright notice are included on the first page. To copy otherwise, or to republish, requires a fee and/or specific permission.

0 3 6 2 - 6 1 3 X / 8 3 / 0 3 0 1 8 8 - 0 9 5 0 3 . 0 0

(2)

Richard H. Granger The N O M A D System

(2) ' R e t u r n e d bombs to Kashin.'

In addition to the surface problem of a missing subject, this example is a p p a r e n t l y missing m e n t i o n of some previous event, implied by the use of ' r e t u r n e d ' ; and it describes an ambiguous event, that is, either the p e a c e a b l e delivery of b o m b s to the Kashin ship (a type of e n e m y ship) or a battle action of firing bombs in retaliation. Since the input is ambiguous without the previous message, NOMAD returns a n u m b e r of alternative possible outputs to the user, marking one as " p r e f e r r e d " :

( P r e f e r r e d Interpretation):

We fired some b o m b s at a Kashin ship. (Inferred):

The Kashin ship fired at us previously.

(Alternate Interpretation):

We delivered some bombs to a Kashin ship. (Inferred):

The Kashin ship had delivered some bombs to us previously.

NOMAD is interactive: it produces multiple inter- p r e t a t i o n s when necessary, and lets the message- s e n d e r c h o o s e among these alternatives. A typical scenario is:

• the user (message-sender) will enter a 'telegraphic' message;

• NOMAD will produce two different possible interpretations of the message in corrected English, and present them to the user;

• the user will then choose one of the interpretations; and

• a d a t a b a s e - r e a d a b l e version of the c o r r e c t l y - i n t e r p r e t e d message is then f o r w a r d e d f r o m the ship to a central data base.

M a n y of the a p p r o a c h e s to u n d e r s t a n d i n g ill- f o r m e d input focus on syntactic errors separately from semantic errors (for example, H a y e s and M o u r a d i a n 1981 and Kwasny and Sondheimer 1981). Both of these efforts essentially attempt to increase the flexi- bility of an ATN syntactic parser: the first by using 'parse suspension and c o n t i n u a t i o n ' , relaxing constraints on consistency and permitting matches out of their c o r r e c t order, and the s e c o n d by relaxing the constraints required to traverse an ATN arc, and then providing 'deviance n o t e s ' specifying the d i f f e r e n c e s b e t w e e n what was e x p e c t e d and what was actually seen. T h e s e efforts a t t e m p t to c o r r e c t the surface form of the input, that is, to p e r f o r m a transformation from an ill-formed English text to a well-formed Eng- lish text. Their goals are not to produce a meaning representation of the input, and hence c a n n o t be said to ' u n d e r s t a n d ' the input. This also leads to the ina-

bility of these systems to generate alternative interpretations of text; once these systems have guessed at a parse, they c a n n o t back up and re-parse in response to information from a user.

T h e a p p r o a c h t a k e n by H a y e s and C a r b o n e l l (1981) is closer to that described in this paper, in that they do build meaning r e p r e s e n t a t i o n s . H o w e v e r , there are still shortcomings; in particular, their systems c a n n o t u n d e r s t a n d texts in which a missing or unknown word is the one that would have built the main semantic case frame. As will be seen below, NOMAD builds on the FOUL-UP system ( G r a n g e r 1977) to handle such cases (which are f r e q u e n t in our domain). F u r t h e r m o r e , like the systems described above, their systems c a n n o t re-interpret a text when its initial interpretation turns out to be incorrect.

We propose an integrated system of syntactic and semantic processing, in which world knowledge and syntactic k n o w l e d g e are b o t h applied during text proc- cessing to provide a n u m b e r of possible interpretations of a text. Our focus is on interpretations: the goal of the system is to give rise to an unambiguous meaning representation. If surface-text problems occur during processing but an unambiguous interpretation can be provided and confirmed by the user, then the surface- text problems are ignored. It is only when interpretation problems arise that any n o t e d surface-text p r o b - lems will be consulted to see if they might have been the source of the interpretation problem. That is, we are attempting to attack the overall problem of processing text, of which the processing of ill-formed text is a necessary subpart. Our approach implies that the processing of ill-formed text 'falls out' of normal text processing, via the application of generalized e r r o r - c o r r e c t i o n processes that o p e r a t e equally on syntax, semantics, and pragmatics, and are not designed spe- cifically for the processing of ill-formed surface text.

NOMAD builds on previous work on c o n c e p t u a l analysis (Riesbeck and Schank 1976, B i r n b a u m and Selfridge 1979), and on error detection and c o r r e c t i o n during c o n c e p t u a l analysis ( G r a n g e r 1977, 1980, 1982a). Selfridge and E n g e l b e r g ( 1 9 8 4 ) , L e b o w i t z ( 1 9 8 4 ) , and D y e r ( 1 9 8 3 ) have also r e c e n t l y t a k e n approaches that are similar to the one p r o p o s e d here, a t t e m p t i n g to fully exploit the p o w e r of i n t e g r a t e d understanding. NOMAD i n c o r p o r a t e s and integrates e r r o r d e t e c t i o n and c o r r e c t i o n algorithms based on both syntactic and pragmatic error types, and is there- fore capable of correctly processing a wide range of ill-formed texts within the knowledge domain of Navy messages. NOMAD has actually been installed and is being used for message processing by the Naval O c e a n Systems C e n t e r (NOSC) at San Diego.

(3)

2. Background: Tolerant T e x t Processing 2.1. FOUL-UP figured out u n k n o w n w o r d s from

c o n t e x t

The FOUL-UP p r o g r a m (Figuring O u t U n k n o w n L e x - e m e s in the U n d e r s t a n d i n g Process; G r a n g e r 1977) was the first p r o g r a m that could figure out meanings of u n k n o w n w o r d s e n c o u n t e r e d during text u n d e r - standing. FOUL-UP was an a t t e m p t to model the cor- r e s p o n d i n g h u m a n ability c o m m o n l y k n o w n at "figuring out a word f r o m c o n t e x t " . FOUL-UP w o r k e d with the SAM s y s t e m ( C u l l i n g f o r d 1977), using the e x p e c t a t i o n s g e n e r a t e d by scripts (Schank and A b e l s o n 1977) to restrict the possible m e a n i n g s of a word, b a s e d on what o b j e c t or action would have o c c u r r e d in that position according to the script for the story.

F o r instance, consider the following e x c e r p t f r o m a n e w s p a p e r r e p o r t of a car accident:

(1) Friday, a car s w e r v e d o f f R o u t e 69. The vehicle struck an e m b a n k m e n t .

T h e word " e m b a n k m e n t " was u n k n o w n to the SAM system, but it had e n c o d e d predictions a b o u t certain a t t r i b u t e s of the e x p e c t e d c o n c e p t u a l o b j e c t of the PROPEL action (the o b j e c t t h a t the vehicle s t r u c k ) ; namely, that it would be a physical object, and would f u n c t i o n as an " o b s t r u c t i o n " in the v e h i c l e - a c c i d e n t script. (In addition, the c o n c e p t u a l a n a l y z e r (ELI - R i e s b e c k and Schank 1976) had the e x p e c t a t i o n that the word in that s e n t e n c e position would be a noun.)

H e n c e , w h e n the u n k n o w n w o r d was e n c o u n t e r e d , FOUL-UP would m a k e use of those e x p e c t e d attributes to c o n s t r u c t a m e m o r y e n t r y for the w o r d " e m b a n k m e n t " , indicating that it was a noun, a physical o b j e c t , and an " o b s t r u c t i o n " in v e h i c l e - a c c i d e n t situations. It would then create a dictionary definition that the s y s t e m would use f r o m then on w h e n e v e r the word was e n c o u n t e r e d in this context.

2.2. Syntactic (surface) and semantic (interpreation) t e x t errors

But e v e n if the SAM s y s t e m h a d k n o w n the w o r d " e m b a n k m e n t " , it would not have b e e n able to handle a less edited version of the story, such as this ' t e l e g r a p h i c ' message, which might have b e e n sent in by an o n - t h e - s c e n e reporter:

(2) Vehcle ace Rt69; car strck e m b a n k m e n t ; drivr dead one psngr inj; ser d m g to car full rpt frthcmng.

While h u m a n readers would have little difficulty understanding this text, no existing c o m p u t e r p r o g r a m s could do so.

T h e s c o p e of this p r o b l e m is wide; e x a m p l e s of texts that p r e s e n t " s c r u f f y " difficulties to readers are c o m p l e t e l y unedited texts, such as messages c o m p o s e d

in a hurry, with little or no re-writing, rough drafts, m e m o s , transcripts of c o n v e r s a t i o n s , etc. Such texts m a y c o n t a i n these p r o b l e m s , a m o n g others: missing words, ad hoc a b b r e v i a t i o n s of w o r d s , p o o r s y n t a x , confusing order of p r e s e n t a t i o n of ideas, misspellings, lack of punctuation. E v e n edited texts such as news- p a p e r stories o f t e n c o n t a i n misspellings, w o r d s u n - k n o w n to the reader, and ambiguities; and e v e n a p p a r - ently v e r y simple texts m a y c o n t a i n alternative possible i n t e r p r e t a t i o n s , which can cause a r e a d e r to c o n - struct e r r o n e o u s initial i n f e r e n c e s t h a t m u s t later be c o r r e c t e d (see G r a n g e r 1980, 1981a, 1981b).

T h e following sections d e s c r i b e the NOMAD system, which i n c o r p o r a t e s FOUL-UP's abilities as well as significantly e x t e n d e d abilities to use s y n t a c t i c a n d s e m a n t i c e x p e c t a t i o n s to resolve these difficulties, in the d o m a i n of N a v y messages. NOMAD's processing is divided into two m a j o r categories:

(1) blame assignment, that is, the d e t e c t i o n of an error and the a t t r i b u t i o n of t h a t e r r o r to s o m e source; and

(2) error correction, the r e m e d y for the source of the error.

3. H o w N O M A D Recognizes and Corrects Errors

3.1. Introduction

NOMAD i n c o r p o r a t e s ideas from, and builds on, earlier w o r k on c o n c e p t u a l analysis ( f o r e x a m p l e , R e i s b e c k and Schank 1976, B i r n b a u m and Selfridge 1979), situation and intention inference (for example, Cullingford 1977, W i l e n s k y 1978), a n d English g e n e r a t i o n ( f o r example, G o l d m a n 1973, M c G u i r e 1980). W h a t dif- f e r e n t i a t e s NOMAD significantly f r o m its p r e d e c e s s o r s are its error recognition and error c o r r e c t i o n abilities, which enable it to read texts m o r e c o m p l e x t h a n those that can be handled by o t h e r text u n d e r s t a n d i n g systems.

NOMAD o p e r a t e s b y a t t e m p t i n g to p r o c e s s texts left to right, with each w o r d c a p a b l e of suggesting new e x p e c t a t i o n s (for e x a m p l e , a v e r b will follow, the previous noun group should serve as a c t o r of the current act, etc.), and applying those suggested e x p e c t a t i o n s to new inputs. W h e n e x p e c t a t i o n s are met, they result in additions to the ongoing m e a n i n g r e p r e s e n t a t i o n of the text; w h e n they are not met, they result in ' s u r f a c e - text alerts', which are collected f o r p o t e n t i a l later cor- rective processing.

T h e r e are two types of ' e r r o r s ' in NOMAD: s u r f a c e - text errors a n d i n t e r p r e t a t i o n errors. S u r f a c e - t e x t errors are potential p r o b l e m s that c a n be readily de- t e c t e d at s u r f a c e - t e x t p r o c e s s i n g time, including, f o r example, u n k n o w n words and a n y surface e x p e c t a t i o n violations, w h e t h e r s y n t a c t i c or semantic. F o r instance, a syntactic e x p e c t a t i o n failure such as ' n o noun

(4)

group a p p e a r i n g where one was e x p e c t e d ' is a surface alert, but so is a s e m a n t i c / p r a g m a t i c e x p e c t a t i o n failure such as ' t a r g e t noun group was e x p e c t e d to de- scribe an a n i m a t e actor, but d e s c r i b e d an i n a n i m a t e object instead'. E a c h of these is equally an e x p e c t a - tion failure, and no difference need be drawn at this stage of processing b e t w e e n s y n t a c t i c or s e m a n t i c types. It will be seen later that, depending on the type of surface alert, different suggestions will be m a d e as to where to look to 'assign b l a m e ' for the p r o b l e m , and h o w to a t t e m p t to correct it.

I n t e r p r e t a t i o n failures, on the other hand, are de- fined as those that c a n n o t be easily ascribable to the failure of some particular p r e - d e f i n e d surface e x p e c t a - tion; these arise a f t e r s o m e c o n c e p t u a l analysis has b e e n successfully p e r f o r m e d and the resulting representation fails to m a t c h p r a g m a t i c checks such as goal- based or s c r i p t - b a s e d k n o w l e d g e of the situation being described.

Following is a list of nine categories of p r o b l e m s we have identified that occur often in scruffy unedited texts, five s u r f a c e - t e x t p r o b l e m s and four i n t e r p r e t a - tion problems. E a c h p r o b l e m is illustrated by a brief example f r o m the domain of N a v y messages. It will be seen that these errors o f t e n o c c u r in pairs, with s u r f a c e - t e x t p r o b l e m s s o m e t i m e s giving rise to inter- p r e t a t i o n problems. N o t e that while these p r o b l e m s are often referred to in this p a p e r as ' e r r o r s ' in fact some are not actual ' e r r o r s ' , strictly speaking, but are rather p o t e n t i a l p r o b l e m indicators that NOMAD recognizes, which m a y give rise to s u b s e q u e n t i n t e r p r e t a t i o n problems.

Surface-text problems

1. U n k n o w n words.

E n e m y "scudded" b o m b s a t us. - the v e r b is un- k n o w n to the system.

2. Missing subject, object, etc. of sentences.

S i g h t e d e n e m y ship. F i r e d . - the actor who fired is not explicitly stated.

3. Missing sentence and clause boundaries.

L o c k e d on o p e n e d f i r e . - two actions, aiming and firing.

4. A m b i g u o u s word usage.

R e t u r n e d b o m b s to K a s h i n . - " r e t u r n e d " in the sense of retaliation a f t e r a p r e v i o u s a t t a c k , or " r e t u r n e d " in the sense of " p e a c e a b l y delivered t o " ?

5. L a c k of tense agreement.

O p e n f i r e d . - the intended tense of ' o p e n ' is trans- ferred to 'fire'.

Interpretation problems

1. Causality violation.

S h i p s i g h t e d o v e r h e a d . - ships c a n ' t fly; p r o b a b l e m e s s a g e - s e n d i n g error.

2. G o a l violation.

R e t u r n e d b o m b s to K a s h i n . - one of two a m b i g u o u s i n t e r p r e t a t i o n s of ' r e t u r n e d ' ( p e a c e a b l y delivered)

gives rise to a p p a r e n t goal violation (delivering w e a p o n s to e n e m y ) .

3. U s e r c o n f i r m a t i o n failure.

NOMAD's failure is not c o n f i r m e d b y user. ( N o t e that this is considered b y NOMAD to be an inter- p r e t a t i o n p r o b l e m e v e n t h o u g h t it m a y be due to the user's idiosyncrasies, as o p p o s e d to violation of s o m e k n o w n s e m a n t i c rule - the e f f e c t is the same.)

4 . O b j e c t or e v e n t r e f e r e n c e d out of k n o w n e v e n t sequence.

M i d w a y lost c o n t a c t on K a s h i n . - no previous c o n - tact m e n t i o n e d ; this o f t e n arises w h e n typical k n o w n situations are m e n t i o n e d in other t h a n ster- eotypical (scripty) order.

W h e n these p r o b l e m s arise in a m e s s a g e , NOMAD m u s t first r e c o g n i z e w h a t the p r o b l e m ( s ) is(are) (which is o f t e n difficult to do), and t h e n a t t e m p t to correct the error(s). T h e following section outlines the overall p r o c e s s i n g a l g o r i t h m s NOMAD uses to process these errors.

3 . 2 . N O M A D ' s e r r o r - d e t e c t i o n a l g o r i t h m

NOMAD's algorithm for d e t e c t i o n and solution of errors follows a f o u r - s t e p process:

1. Set ' a l e r t ' flags w h e r e v e r p o t e n t i a l s u r f a c e - t e x t p r o b l e m s are detected.

2. Do only partial processing of surface text if necessary due to missing or a m b i g u o u s i n f o r m a t i o n (that is, do as m u c h normal processing as possible in the face of missing i n f o r m a t i o n ) .

3. C h e c k for i n t e r p r e t a t i o n p r o b l e m s (causal, goal, s e q u e n c i n g (script), or user c o n f i r m a t i o n e r r o r s ) after surface sentence processing.

4. T r y solutions b a s e d on surface ' a l e r t ' flag c a t e g o - ries.

To illustrate this p r o c e s s , c o n s i d e r an a m b i g u o u s text, ' c o n t a c t gained on kashin'. During the processing of this text, s o m e s u r f a c e - t e x t alerts arise ( f o r example, ' c o n t a c t ' can be either a n o u n or a v e r b ' if it's a verb, then t h e r e ' s either a missing subject or an e x p e c t e d passive subject coming, etc.), and an inter- p r e t a t i o n ambiguity: the text can be i n t e r p r e t e d as meaning either

(a) We e s t a b l i s h e d visual or r a d a r c o n t a c t with a kashin ship.

(b) O u r c o n t a c t (that is, a ship in c o n t a c t with us) increased its speed in a chase a f t e r a kashin ship. In the case of ' c o n t a c t gained on kashin', NOMAD's blame assignment algorithm m o v e s t h r o u g h the a b o v e steps as follows:

1. (a) Set b o t h ' a m b i g u o u s - w o r d - s e n s e ' and ' a m b i g - u o u s - p a r t - o f - s p e e c h ' alerts for the w o r d ' c o n t a c t ' : it might be either a n o u n (that is, the ship that is currently our c o n t a c t ) or a v e r b (to establish r a d a r or visual contact).

(5)

(b) Set ' a m b i g u o u s - w o r d - s e n s e ' alert for w o r d ' g a i n e d ' : it m i g h t m e a n either ' e s t a b l i s h e d ' as in 'gained (established) radar c o n t a c t ' , or ' a d v a n c e d ' as in ' g a i n e d ( a d v a n c e d ) on e n e m y during chase'. 2. P r o d u c t alternate i n t e r p r e t a t i o n s b a s e d on alternate

a s s u m p t i o n s a b o u t word senses: 'established r a d a r or visual c o n t a c t with kashin', and ' o u r c o n t a c t ship a d v a n c e d on kashin'.

3. (a) L o o k for possible causality or goal violations: n o n e found.

(b) A s k user for confirmation: user c o n f i r m s one i n t e r p r e t a t i o n but not the other.

4. Solution: Select i n t e r p r e t a t i o n c o n f i r m e d b y user. C o n s i d e r a n o t h e r e x a m p l e , ' R e t u r n e d b o m b s to K a s h i n ' . As n o t e d above, one of t w o a m b i g u o u s inter- p r e t a t i o n s of ' r e t u r n e d ' in this text ( t h a t is, the ' ( p e a c e a b l y ) delivered' i n t e r p r e t a t i o n ) gives rise to an a p p a r e n t goal violation (delivering w e a p o n s to e n e m y ) . In the case of ' r e t u r n e d b o m b s to k a s h i n ' , the b l a m e assignment algorithm acts as follows:

1. (a) Set ' a m b i g u o u s - w o r d - s e n s e ' alert for the w o r d ' r e t u r n e d ' : it might h a v e either of t w o categories of meaning, c o r r e s p o n d i n g to ' r e - d o a p r e v i o u s l y - d o n e a c t i o n (as in ' r e t u r n the f a v o r ' , ' r e t u r n a t r a n s m i s s i o n ' ) or ' r e - d e l i v e r a p r e v i o u s l y - d e l i v e r e d o b j e c t (as in ' r e t u r n a ( b o r r o w e d ) b o o k ' ) .

(b) Set ' a m b i g u o u s - w o r d - s e n s e ' alert f o r w o r d ' b o m b s ' : it might m e a n either the v e r b ' t o b o m b ' , p r e s e n t tense, or the plural noun. T h e f o r m e r in- t e r p r e t a t i o n (that b o m b is a v e r b ) also gives rise to a ' m i s s i n g - c l a u s e - b o u n d a r y ' s u r f a c e alert, since t h e n the ' r e t u r n e d ' a n d ' b o m b s ' v e r b s would be next to each other.

2. P r o d u c e alternate i n t e r p r e t a t i o n s b a s e d on alternate a s s u m p t i o n s a b o u t word senses: ' D e l i v e r e d o b j e c t ( b o m b s ) to k a s h i n ' (after they had delivered s o m e to us) or 'fired on k a s h i n ' (after they h a d fired on us). ( T h e e r r o r - r i d d e n a l t e r n a t e i n t e r p r e t a t i o n s that arise f r o m the v e r b sense of ' b o m b s ' are also g e n e r a t e d . )

3. (a) L o o k for possible causality or goal violations: With the ' d e l i v e r y ' i n t e r p r e t a t i o n , a potential violation of o n e of N O M A D ' s k n o w n goals is found: A c t o r s of class (enemies) transferring possession of o b j e c t s of class ( w e a p o n s ) to recipients of class (friends), and vice versa.

(b) O r d e r the i n t e r p r e t a t i o n s in order of p r e f e r - ence, b a s e d on b o t h s u r f a c e - c l a s s and i n t e r p r e t a - tion-class errors; the g o a l - v i o l a t i o n case a b o v e is not preferred, and the ' b o m b s - a s - v e r b ' case is not p r e f e r r e d , while the 'firing b a c k at k a s h i n ' interpretation is preferred.

(e) P r e s e n t p r e f e r r e d i n t e r p r e t a t i o n to user; confirmed. (If this had failed, then u n p r e f e r r e d interpretations would h a v e b e e n p r e s e n t e d . )

4. Solution: Select c o n f i r m e d interpretation.

4. Blame A s s i g n m e n t in N O M A D

As evidenced in the a b o v e examples, there is no simple relationship b e t w e e n types of errors in the i n t e r p r e t a - tion of the input, and possible solutions to those errors. This is primarily b e c a u s e the s o u r c e of an inter- p r e t a t i o n error is difficult to identify. In general, in- t e r p r e t a t i o n p r o b l e m s can arise f r o m any of a n u m b e r of s u r f a c e - t e x t p r o b l e m s , including:

1. words with multiple w o r d senses

R e t u r n e d b o m b s to K a s h i n . - see a b o v e discussion; 2. missing clause b o u n d a r i e s

C h a l l e n g e d s h i p r e f u s e d to h e a v e to. - can be inter- p r e t e d in any of the following ways: (a) W e challenged a ship. T h e y refused to h e a v e to. (b) We challenged a ship. We r e f u s e d to h e a v e to. (c) T h e challenged ship r e f u s e d to h e a v e to.

3. elliptical or telegraphic s e n t e n c e c o n s t r u c t i o n C o n t a c t g a i n e d o n K a s h i n . - can be i n t e r p r e t e d as: (a) We established visual or r a d a r c o n t a c t with a kashin ship. (b) O u r c o n t a c t (that is, a ship in c o n t a c t with us) increased its s p e e d in a chase after a kashin ship).

As m e n t i o n e d earlier, NOMAD's goal is to p r o d u c e correct, u n a m b i g u o u s i n t e r p r e t a t i o n s of input texts. Its ability to handle ill-formed surface text arises f r o m a n e e d to be able to find s u r f a c e - t e x t p r o b l e m s that give rise to i n t e r p r e t a t i o n p r o b l e m s ; it a t t e n d s to s u r f a c e - text p r o b l e m s not b e c a u s e they are useful in their o w n right but only b e c a u s e t h e y m a y be useful l a t e r in solving an i n t e r p r e t a t i o n p r o b l e m . NOMAD collects b o t h s u r f a c e - t e x t p r o b l e m s and i n t e r p r e t a t i o n p r o b - lems as it processes a text, and for e a c h i n t e r p r e t a t i o n p r o b l e m , it a t t e m p t s to find a c o r r e s p o n d i n g surface p r o b l e m that gave rise to it. O n c e it has an i n t e r p r e t a - tion p r o b l e m - surface p r o b l e m pair, it suggests a solution for the overall p r o b l e m b a s e d on the characteris- tics of b o t h the surface p r o b l e m and the i n t e r p r e t a t i o n p r o b l e m . In cases w h e r e only a surface p r o b l e m exists and no i n t e r p r e t a t i o n p r o b l e m has arisen, the surface p r o b l e m is simply ignored as being irrelevant to the true u n d e r s t a n d i n g goal of p r o d u c i n g a correct, u n a m - biguous i n t e r p r e t a t i o n . In cases where an i n t e r p r e t a - tion p r o b l e m exists but no s u r f a c e - t e x t p r o b l e m can be linked to it, NOMAD suggests possible solutions to the i n t e r p r e t a t i o n p r o b l e m that do not d e p e n d on surface problems.

T h e ' b l a m e assignment c h a r t ' b e l o w illustrates s o m e of NOMAD's heuristics for finding s u r f a c e - t e x t alerts that might c o r r e s p o n d to a given i n t e r p r e t a t i o n p r o b - lem.

N O M A D ' s b l a m e a s s i g n m e n t a l g o r i t h m is at the center of its ability to handle syntactically and s e m a n - tically ill-formed text. Blame a s s i g n m e n t in NOMAD is c a p a b l e of dealing with p r o b l e m s at b o t h the s u r f a c e - text level and the i n t e r p r e t a t i o n level, especially where i n t e r p r e t a t i o n p r o b l e m s arise indirectly f r o m s u r f a c e -

(6)

I N T E R P R E T A T I O N P R O B L E M Only partial representation constructed

Causality violation, Goal violation,

User confirmation failure

A c t o r or object reference out of sequence

E v e n t r e f e r e n c e d out of sequence

S U G G E S T E D S U R F A C E - T E X T A L E R T Unknown word

Word with multiple word senses

E x p e c t a t i o n failures: - S y n t a c t i c (word) - S e m a n t i c

- B o u n d a r y (phrase)

(No surface alert)

/

Blame Assignment Chart

S U G G E S T E D P O T E N T I A L S O L U T I O N S FOUL-UP:

Expectations and Act-Preference

T r y alternate word sense

Try inferring clause break

T r y situation-frame inference

level decisions; in general, there is no simple relationship among surface-text problems, interpretation problems, and potential solutions for these problems.

4 , 1 . R e c o g n i z i n g a n d c o r r e c t i n g s u r f a c e e r r o r s F o r each of the five categories of surface problems handled by the system, NOMAD's m e t h o d of recognizing and correcting the p r o b l e m is briefly described here, along with actual English input and output from NOMAD.

1. INPUT:

E N E M Y S C U D D E D BOMBS A T US.

Problem: U n k n o w n word. The u n k n o w n word " s c u d d e d " is trivial to recognize as being unknown, since it is the only word without a dictionary entry. Once it has been recognized, NOMAD checks it to see if it could be (a) a misspelling, (b) an abbrevia- tion, or (c) a regular v e r b - t e n s e of some k n o w n word.

Solution: Use expectations to figure out word meaning from context. When the spelling checkers fail, a FOUL-UP mechanism is called that uses syntactic expectation (and some morphological analysis) to infer that 'scudded' is p r o b a b l y a verb, and then uses pragmatic knowledge of what actions can be done by an ' e n e m y ' ACTOR, to a ' w e a p o n ' OBJECT, direct TO us. At this point, NOMAD uses a mechanism we term ' A C T - p r e f e r e n c e ' ( G r a n g e r 1977), which exploits b o t h pragmatic knowledge of what enemies tend to do with weapons, and word- order knowledge that we have derived of how particular triads of prepositions, noun-categories, and v e r b - c a t e g o r i e s t e n d to c o m b i n e (for example, 'BLAGHED < w e a p o n > AT < s h i p > ' will give rise

to a d i f f e r e n t i n f e r e n c e than 'BLAGHED < w e a p o n > TO < s h i p > ' , or 'BLAGHED < w e a p o n > FOR < s h i p > ' , etc.). This process, detailed in G r a n g e r ( 1 9 7 7 ) , arrives at an i n f e r e n c e that the action is p r o b a b l y a 'PROPEL' (see Schank and Abelson 1977). Again, this is only an e d u c a t e d guess by the system, and may have to be corrected later on the basis of f u r t h e r information (see G r a n - ger 1980, 1981b).

Finally, NOMAD p r o d u c e s an i n t e r p r e t a t i o n of the input, which the user may or may not confirm. In the e v e n t that the user does not c o n f i r m NOMAD's initial interpretation, a n u m b e r of alternative i n t e r p r e t a t i o n s are p r o d u c e d (see G r a n g e r 1981a, 1982c) until one is confirmed, or the process fails. In this and the following examples, NOMAD's ' p r e f e r r e d ' interpretation is confirmed by the user.

N O M A D O U T P U T :

An e n e m y ship fired bombs at our ship.

2. INPUT:

M I D W A Y S I G H T E D E N E M Y . F I R E D .

Problem: Missing subject and objects. 'Fired' builds a PROPEL, and expects a subject and objects to play the conceptual roles of ACTOR (who did the PROPELing), OBJECT (what got PROPELed) and RECIPIENT (who got PROPELed at). H o w e v - er, no surface subjects or o b j e c t s are p r e s e n t e d here.

Solution: Use expectations to fill in conceptual cases. NOMAD uses situational (script-based) ex- p e c t a t i o n s from the k n o w n typical s e q u e n c e of events in an " A T T A C K " - which consists of a m o v e m e n t (PTRANS), a sighting (ATTEND) and

(7)

firing (PROPEL) (as in other script-based understanders; see Cullingford 1978). Those expectations say (among other things) that the actor and recipient of the PROPEL will be the same as the actor and direction of the ATTEND, and that the OBJECT that got PROPELed will be some kind of projectile, which is not further specified here. N O M A D O U T P U T :

We sighted an e n e m y ship. We fired at the ship.

3. I N P U T :

L O C K E D O N O P E N E D F I R E .

Problem: Missing sentence boundaries. NOMAD has no expectations for a new verb ( " o p e n e d " ) to a p p e a r immediately after the c o m p l e t e d clause " l o c k e d o n " . It tries but fails to c o n n e c t " o p e n e d " to the phrase " l o c k e d o n " .

Solution: A s s u m e the syntactic e x p e c t a t i o n s failed because a clause b o u n d a r y was not adequate- ly marked in the message; assume such a b o u n d a r y is there. NOMAD assumes that there m a y have b e e n an i n t e n d e d s e n t e n c e s e p a r a t i o n or clause break before " o p e n e d " , since no expectations can a c c o u n t for the w o r d in this s e n t e n c e position. Hence, NOMAD saves " l o c k e d o n " as one clause, and continues to process the rest of the text as a new sentence.

We aimed at an u n k n o w n object. We fired at the object.

4. I N P U T :

R E T U R N E D B O M B S T O E N E M Y SHIP.

Problem: Multiple word senses of ' r e t u r n e d ' , resulting in a m b i g u o u s i n t e r p r e t a t i o n of action. NOMAD c a n n o t tell w h e t h e r the action here is " r e t u r n i n g " fire to the enemy, that is, firing back at them (after they p r e s u m a b l y had fired at us), or peaceably delivery bombs, with no firing implied.

Solution: Use expectations of probable goals of actors. NOMAD first interprets the s e n t e n c e as " p e a c e a b l y delivering" some b o m b s to the ship. H o w e v e r , NOMAD c o n t a i n s the k n o w l e d g e that enemies do not transfer control of weapons, information, personnel, etc., to each other. H e n c e it attempts to find an alternative interpretation of the sentence, in this case finding the " r e t u r n e d fire" interpretation, which does not violate a n y of NOMAD's knowledge a b o u t goals. It then infers, as in the a b o v e example, that the e n e m y ship must have previously fired on us.

A n u n k n o w n e n e m y ship fired on us. T h e n we fired b o m b s at them.

5. I N P U T : O P E N F I R E D .

Problem: L a c k of tense a g r e e m e n t b e t w e e n

' o p e n ' and 'fired'.

Solution: Use morphological analyzer to correct tense of word. NOMAD identifies the phrase ' o p e n fire', and assumes that past tense was intended (by default); and so constructs a phrase that correctly incorporates the tense into the phrase, to make it ' o p e n e d fire'. NOMAD then adds the inferred missing actor. ( N o t e that were this not a k n o w n phrase to NOMAD then the tense a g r e e m e n t would not have been corrected at the surface level, but rather the semantic c o n t e n t of the two words would have c o n t r i b u t e d to a m e a n i n g r e p r e s e n t a t i o n , which would hae been used to generate a ' c o r r e c t e d ' version of the input.

We fired b o m b s at an unspecified target.

4.2. Recognizing and correcting interpretation errors

The four interpretation error-types given above were: 1. causal violations,

2. goal violations,

3. user c o n f i r m a t i o n failure, and

4. o u t - o f - s e q u e n c e event or object reference.

The process of d e t e c t i n g or c o r r e c t i n g these error types is different in principle f r o m the five surface types, for the simple reason that, as o p p o s e d to surface errors, which can only be a t t r i b u t e d to the message-sender himself, there are m a n y possible different sources of interpretation errors. In particular, some surface errors can give rise to a p p a r e n t interpretation errors. To see this, recall the ' r e t u r n e d b o m b s to k a s h i n ' example above. In this case, NOMAD's default selection of a w o r d sense for an a m b i g u o u s w o r d ( ' r e t u r n e d ' ) can give rise to an a p p a r e n t goal violation error (delivery w e a p o n s to an enemy, as op- p o s e d to firing at an e n e m y ) . H e n c e , the task of blame assignment here is p r o b l e m a t i c : an early surface-processing decision of NOMAD's can give rise to an apparent later interpretation problem.

Similarly, a 'user c o n f i r m a t i o n ' error (that is, the user will not confirm any of the interpretations o f f e r e d by NOMAD) might be due to any of a n u m b e r of things: the user m i s t y p e d the original message, NOMAD made an erroneous surface-text decision, or NOMAD failed to detect a surface or i n t e r p r e t a t i o n problem in the text. And, a 'causal violation' error (that is, 'ship sighted o v e r h e a d ' : ships c a n ' t fly, so the error is apparently a user error) can be due either to user errors or to NOMAD's o w n interpretation errors. Finally, an object or event apparently r e f e r e n c e d out of sequence can be due to either user error or an erroneous inference by NOMAD.

(8)

Richard H. Granger The NOMAD System

5. S u m m a r y and C o n c l u s i o n s

5.1. N O M A D ' s l i m i t a t i o n s and s h o r t c o m i n g s

NOMAD has p r o v e d to be a c a p a b l e a n a l y z e r of ill- f o r m e d text. Some of the standard p r o b l e m s of script- and p l a n - b a s e d u n d e r s t a n d e r s have b e e n satisfactorily addressed in NOMAD, m o s t notably, the handling of u n k n o w n words (via the FOUL-UP m e c h a n i s m ) ; and the s c r i p t - s e l e c t i o n p r o b l e m , that is, k n o w i n g which scripts to apply monitoring w h e n they go wrong (via the m e c h a n i s m s of s u p p l a n t i n g i n c o r r e c t i n f e r e n c e s ( G r a n g e r 1980), and producing a set of alternate in- t e r p r e t a t i o n s of a text ( G r a n g e r 1981a, 198282a,

1982c).

The m o s t i m p o r t a n t d r a w b a c k of NOMAD is its lack of extensibility. Since the s y s t e m ' s k n o w l e d g e is main- ly e m b e d d e d in w o r d - l e v e l routines, adding a n e w word to the s y s t e m requires writing a new routine, possibly duplicating i n f o r m a t i o n elsewhere in NOMAD, and possibly i n t r o d u c i n g new errors into o t h e r w i s e - working NOMAD code. A n y new word routine should ideally take into a c c o u n t i n t e r a c t i o n s with all the w o r d - l e v e l routines a l r e a d y p r e s e n t in the s y s t e m ; s o m e of those routines m a y h a v e to be m o d i f i e d in light of the new entry.

In practice, we do not check every routine w h e n a new word is added. Rather, we test the s y s t e m and m a k e corrections only w h e n a b a d interaction is found. Thus, the s y s t e m is not g u a r a n t e e d to be self- consistent. Since NOMAD has m o r e than a t h o u s a n d - word v o c a b u l a r y , it is impractical to check the entire system when a new word is added.

Encoding g r a m m a t i c a l knowledge at the word level is also c u m b e r s o m e . F o r e x a m p l e , the routine for nearly every v e r b m a k e s its own checks for active or passive usage. A m o r e centralized g r a m m a t i c a l m e c h - anism would eliminate this kind of r e d u n d a n c y . In principle, the k n o w l e d g e c u r r e n t l y e n c o d e d in the word-level routines could be m a d e declarative (that is, stored as data), so as to be m o r e centralized and usa- ble by other parts of the system.

5.2. VOX: A V O c a b u l a r y e X t e n s i o n s y s t e m

To m a k e the NOMAD system m o r e extensible, we are currently building a new system that uses not w o r d - level but

phrasal

analysis. We call this new system VOX (for v o c a b u l a r y extension system). Our goal is to m a k e this s y s t e m extensible by interaction with a user, rather than by adding to the data base p r o g r a m - matically.

Our ideas a b o u t phrasal analysis originate f r o m the w o r k on the PHRAN s y s t e m ( W i l e n s k y and A r e n s 1982). Phrasal analysis consists of matching the input to one or m o r e p h r a s e - l e v e l p a t t e r n s stored in a k n o w l e d g e d a t a base. W h e n the input has b e e n matched, it is said to be understood. Semantic actions can be associated with each phrase, so that w h e n e v e r a

phrase is m a t c h e d to part of the input a c o r r e s p o n d i n g m e a n i n g r e p r e s e n t a t i o n f o r the p h r a s e m a y be c o n - structed.

To extend the k n o w l e d g e b a s e of the system, we simply add new p a t t e r n s to the data base. Ideally, p a t t e r n s are i n d e p e n d e n t entities w h o s e i n t e r a c t i o n introduces no side effects, so that new phrases can be easily added to or r e m o v e d f r o m the data base. A working p r o t o t y p e of VOX is already up and running (see G r a n g e r , M e y e r s , Yoshii, and T a y l o r 1983 and Meyers 1983), i n c o r p o r a t i n g syntactic and g r a m m a t i - cal analyses, semantic analyses and b l a m e assignment, m o r p h o l o g i c a l analysis, and error detection and cate- gorization. VOX's p h r a s e k n o w l e d g e b a s e a l r e a d y consists of hundreds of phrases, and is being exten- sively tested. F u r t h e r m o r e , VOX's data base can be interactively ' e d i t e d ' b y a trained ' t u t o r ' to add new i n f o r m a t i o n , including new v o c a b u l a r y , new syntactic c a t e g o r i e s and c o n s t r u c t i o n s , and n e w m e a n i n g s . H e n c e , we h o p e that VOX m a y be a first step t o w a r d s a ' t r a i n a b l e ' l a n g u a g e - p r o c e s s i n g system. G r a n g e r , Meyers, Yoshii, and T a y l o r (1983) and M e y e r s (1983) p r e s e n t extensive descriptions of the state of VOX and the theories underlying it.

5.3. S u m m a r y : S u r f a c e t e x t and its i n t e r p r e t a t i o n s

T h e ability to u n d e r s t a n d text is d e p e n d e n t on the ability to u n d e r s t a n d w h a t is being d e s c r i b e d in the text. H e n c e , a reader of English must h a v e applicable k n o w l e d g e of b o t h the situations t h a t m a y be described in texts ( f o r e x a m p l e , actions, states, se- quences of events, goals, m e t h o d s of achieving goals, etc.), and the s u r f a c e s t r u c t u r e s that a p p e a r in the language, t h a t is, the relations b e t w e e n the s u r f a c e order of words and phrases, and their c o r r e s p o n d i n g meaning structures. T h e process of text u n d e r s t a n d i n g is the c o m b i n e d a p p l i c a t i o n of these k n o w l e d g e sources as a reader p r o c e e d s through a text. This fact b e c o m e s clearest w h e n we investigate the u n d e r s t a n d - ing of ill-formed texts, texts t h a t p r e s e n t particular p r o b l e m s to a reader. The line b e t w e e n c o r r e c t and i n c o r r e c t English is o f t e n unclear, so a s y s t e m that c a n n o t handle e r r o n e o u s input is of limited use.

H u m a n u n d e r s t a n d i n g is inherently tolerant; people are naturally able to ignore and deal with m a n y types of errors, omissions, p o o r constructions, etc., and get straight to the m e a n i n g of the text. Our theories have tried to take this ability into a c c o u n t b y including knowledge and m e c h a n i s m s of error noticing and correcting as implicit parts of our process models of language understanding. NOMAD and VOX are primarily engineering applications i n c o r p o r a t i n g a series of t h e o - retical results in l a n g u a g e u n d e r s t a n d i n g , including s c r i p t - b a s e d and g o a l - b a s e d understanding, and integrated e r r o r - m o n i t o r i n g and supplanting during u n d e r - standing. The NOMAD and VOX s y s t e m s are the lat-

(9)

est in a line of 'tolerant' language understanders, be- ginning with FOUL-UP, all based on the use of knowl- edge of syntax, semantics, and pragmatics at all stages of the understanding process to cope with errors.

A c k n o w l e d g m e n t s

Rika Yoshii, Chris Staros, Greg Taylor, and Amnon Meyers all contributed to the construction of the NOMAD system; Amnon Meyers has been responsible for the construction of VOX.

References

Birnbaum, L. and Selfridge, M. 1980 Conceptual Analysis of Natural Language. In Schank, R. and Riesbeck, C., Eds., Inside Computer Understanding. Lawrence Erlbaum Associates, Hills- dale, New Jersey.

Cullingford, R. 1977 Controlling Inferences in Story Understand- ing. Proceedings of the Fifth International Joint Conference on Artificial Intelligence (1JCA1). Cambridge, Massachusetts. DeJong, G. 1979 Skimming Stories in Real Time: An Experiment

in Integrated Understanding. Ph.D. Thesis. Computer Science Department, Yale University, New Haven, Connecticut. Goldman, N. 1973 The Generation of English Sentences from a

Deep Conceptual Base. Ph.D. Thesis. Stanford University, Stanford, California.

Granger, R.H. 1977 F O U L - U P : A Program that Figures Out Meanings of Words from Context. Proceedings of the Fifth International Joint Conference on Artificial Intelligence (IJCAI).

Cambridge, Massachusetts.

Granger, R.H. 1980 When Expectation Fails: Toward a Self- Correcting Inference System. Proceedings of the First National Conference on Artificial Intelligence. Stanford University, Stan- ford, California.

Granger, R.H. 1981a Directing and Re-directing Inference Pur- suit: Extra-textual Influences on Text Interpretation. Proceed- ings of the Seventh International Joint Conference on Artificial Intelligence (IJCAI). Vancouver, British Columbia.

Granger, R.H. 1981b Shaping Explanations: Effects of Question- ing on Text Interpretation. Proceedings of the Third Annual Conference of the Cognitive Science Society. Berkeley, California: 193-196.

Granger, R.H. 1982a Judgmental Inference: Inferential Decision- Making during Understanding. Technical Report #182. Com- puter Science Department, University of California, Irvine, California.

Granger, R.H. 1982b Scruffy Text Understanding: Design and Implementation of 'Tolerant' Understanders. Proceedings of the 20th Annual Meeting of the Association for Computational Linguistics. Toronto, Ontario, Canada: 157-160.

Granger, R.H. 1982c Inference Decisions in Text Understanding.

Proceedings of the Fourth Annual Conference of the Cognitive Science Society. Ann Arbor, Michigan.

Granger, R.H.; Meyers, A.; Yoshii, R.; and Taylor, G. 1983 An Extensible Natural Language Understanding System. Proceed- ings of the Artificial Intelligence Conference. Oakland University, Rochester, Michigan.

Hayes, P.J. and Mouradian, G.V. 1981 Flexible Parsing. Ameri- can Journal of Computation Linguistics 7(4): 232-242.

Kwasny, S.C. and Sondheimer, N.K. 1981 Relaxation Techniques for Parsing Grammatically Ill-Formed Input in Natural Lan- guage Understanding Systems. American Journal of Computation Linguistics 7(2): 99-108.

Lebowitz, M. 1981 Generalization and Memory in an Integrated Understanding System. Computer Science Research Report 186. Yale University, New Haven, Connecticut.

Meyers, A. 1983 Conceptual Grammar. Computer Science Tech- nical Report #215. University of California, Irvine, California. McGuire, R. 1980 Political Primaries and Words of Pain. Unpubl-

ished manuscript. D e p a r t m e n t of Computer Science, Yale University, New Haven, Connecticut.

Riesbeck, C. and Schank, R. 1976 Comprehension by Computer: Expectation-based Analysis of Sentences in Context. Computer Science Research Report 78. Yale University, New Haven, Connecticut.

Schank, R.C. and Abelson, R. 1977 Scripts, Plans, Goals, and Understanding. Lawrence Erlbaum Associates, Hillsdale, New Jersey.

Small, S. 1980 Word Expert Parsing: A Theory of Distributed Word-Based Natural Language Understanding. Technical Re- port TR-954. University of Maryland, College Park, Maryland. Wilensky, R. 1978 Understanding Goal-Based Stories. Computer

Science Technical Report 140. Yale University, New Haven, Connecticut.

Wilensky, R. and Arens, Y. 1982 PHRAN: A Phrasal Analyzer. EECS Technical Report. University of California, Berkeley, California.

A P P E N D I X : Some Statistics on NOMAD's Operation

1. Timing: NOMAD uses a b o u t 3 cpu seconds per word when analyzing N a v y messages.

2. Vocabulary size and structure: NOMAD is based on CA (Birnbaum and Selfridge 1979), and incorporates ' w o r d - e x p e r t ' routines (Small 1980). E a c h w o r d - e x p e r t routine can process a whole class of words, not just an individual word. T h e r e are 152 w o r d - e x p e r t routines; there are 440 words, inflect- ed forms, and phrases in NOMAD's dictionary. ( T h e r e are 330 words and phrases, not c o u n t i n g inflections.)

3. Knowledge: T h e r e are 16 situation frames, corre- sponding roughly to: battle, c o m m u n i c a t i o n , l o c a t i o n - c h a n g e , sight, attack, r e p o r t , c o m m a n d , communicate, emanate, detect, project, aim, ptrans, patrol, state-change, causal-result.

4. Benchmarks: NOMAD has successfully p r o c e s s e d a b o u t 4000 N a v y messages of lengths varying f r o m 1 line to 17 lines of text each. No statistics have b e e n compiled on NOMAD's overall success versus failure rate on all N a v y texts.