C o r e f e r e n c e as t h e F o u n d a t i o n s for L i n k A n a l y s i s o v e r F r e e T e x t
D a t a b a s e s
B r e c k B a l d w i n
Institute for Research in Cognitive Science
University of Pennsylvania
3401 Walnut St. 400C
Philadelphia, PA 19104. USA
Phone: (215) 898-0329
Email: breckQlinc.cis.upenn.edu
A m i t
B a g g a
Box 90129
Dept. of Computer Science
Duke University
Durham, NC 27708-0129. USA
Phone: (919) 660-6507
Email: [email protected]
A b s t r a c t
Coreference annotated data has the potential to sub- stantially increase the domain over which link anal- ysis can be applied. We have developed corefer- ence technologies which relate individuals and events within and across text documents. This in turn leverages the first step in mapping the information in those texts into a more data-base like format suit- able for visualization with link driven software.
1
I n t r o d u c t i o n
Coreference is in some sense nature's own hyperlink. For example, the phrase 'Alan Turing', 'the father of modern computer science', or 'he' can refer to the same individual in the world. The communicative function of coreference is the ability to link informa- tion about entities across many sentences and doc- uments. In data base terms, individual sentences provide entry records which are organized around entities, and the method of indicating which entity the record is about is coreference.
Link analysis is well suited to visualizing large structured databases where generalizations emerge from macro observations of relatedness. Unfortu- nately, free text is not sufficiently organized for sim- ilar fidelity observations. Coreference in its simplest form has the potential to organize free text suffi- ciently to greatly expand the domain over which link analysis can be fruitfully applied.
Below we will illustrate the kinds of coreference that we currently annotate in the CAMP software system and give an idea of our system performance. Then we will illustrate what kinds of observations could be pulled via visualization from coreference annotated document collections.
2
C A M P N a t u r a l Language
P r o c e s s i n g Software
The foundation of our system is the CAMP NLP system. This system provides an integrated envi- ronment in which one can access many levels of lin- guistic information as well as world knowledge. Its main components include: named entity recognition,
tokenization, sentence detection, part-of-speech tag- ging, morphological analysis, parsing, argument de- tection, and coreference resolution as described be- low. Many of the techniques used for these tasks per- form at or near the state of the art and are described in more depth in (Wacholder 97), (Collins 96), (Baldwin 95), (Reynar 97), (Baldwin 97), (Bagga, 98b).
3
W i t h i n D o c u m e n t C o r e f e r e n c e
We have been developing the within document coref- erence component of CAMP since 1995 when the system was developed to participate in the Sixth Message Understanding Conference (MUC-6) coref- erence task. Below we will illustrate the classes of coreference that the system annotates.Coreference breaks down into several readily iden- tified areas based on the form of the phrase being resolved and the method of calculating coreference. We will proceed in the approximate ordering of the systems execution of components. A more detailed analysis of the classes of coreference can be found in (Bagga, 98a).
3.1 H i g h l y S y n t a c t i c C o r e f e r e n c e
There are several readily identified syntactic con- structions that reliably indicate coreference. First are appositive relations as holds between 'John Smith' and ~chairman of General Electric' in:
John Smith, chairman of General Electric, resigned yesterday.
Identifying this class of coreference requires some syntactic knowledge of the text and property anal- ysis of the individual phrases to avoid finding coref- erence in examples like:
John Smith, 47, resigned yesterday. Smith, Jones, Woodhouse and Fife an- nounced a new partner.
To avoid these sorts of errors we have a mutual exclu- sion test that applies to such positings of coreference to prevent non-sensical annotations.
between ' J o h n ' a n d ' t h e finest juggler in the world' in:
J o h n is the finest juggler in the world.
Like the appositive case, mutual exclusion tests are required to prevent incorrect resolutions as in:
J o h n is tall. T h e y are blue.
These classes of highly syntactic coreference can play a very i m p o r t a n t role in bridging phrases t h a t we would normally be unable to relate. For example, it is unlikely t h a t our software would be able to relate the s a m e noun phrases in a text like
T h e finest juggler in the world visited Philadelphia this week. J o h n Smith pleased crowds every night in the Annen- berg theater.
This is because we do not have sufficiently sophis- ticated knowledge sources to determine t h a t jug- glers are very likely to be in the business of pleasing crowds. But the recognition of the predicate nomi- nal will allow us to connect a chain of ' J o h n Smith', 'Mr. Smith', 'he' with a chain of 'the finest juggler in the world', ' t h e juggler' and ' a juggling expert'.
3.2 P r o p e r N o u n C o r e f e r e n c e
N a m e s of people, places, products and companies are referred to in m a n y different variations. In jour- nalistic prose there will be a full n a m e of an entity, and t h r o u g h o u t the rest of the article there will be ellided references to the same entity. Some name variations are:
• Mr. J a m e s D a b a h < - J a m e s <- Jim <- D a b a h
• Minnesota Mining and Manufacturing <- 3M Corp. <- 3M
• Washington D.C. < - W A S H I N G T O N < - Wash- ington <- D.C. < - Wash.
• New York < - New York City <- NYC < - N.Y.C.
This class of coreference forms a solid foundation over which we resolve the remaining coreference in the document. One reason for this is t h a t we learn i m p o r t a n t properties a b o u t the phrases in virtue of the coreference resolution. For example, we m a y not know whether ' D a b a h ' is a person name, male name, female name, c o m p a n y or place, but upon resolution with 'Mr. James D a b a h ' we then know t h a t it refers to a male person.
We resolve such coreferences with partial string m a t c h i n g subroutines coupled with lists of hon- orifics, corporate designators and acronyms. A sub- stantial problem in resolving these names is avoiding overgeneration like relating 'Washington' the place with the name 'Consuela Washington'. We control
the string matching with a range of salience func- tions and restrictions of the kinds of partial string matches we are willing to tolerate.
3.3 C o m m o n N o u n C o r e f e r e n c e
A very challenging area of coreference a n n o t a t i o n involves coreference between c o m m o n n o u n s like ' a shady stock deal' and ' t h e deal'. F u n d a m e n t a l l y the problem is t h a t very conservative a p p r o a c h e s to ex- act and partial string m a t c h e s overgenerate badly. Some examples of actual chains are:
• his dad's trophies <- those trophies
• those words < - the last words
• the risk < - the potential risk
• its accident investigation < - the investigation
We have a d o p t e d a range of matching heuristics and salience strategies to t r y and recognize a small, b u t accurate, subset of these coreferences.
3.4 P r o n o u n C o r e f e r e n c e
T h e pronominal resolution component of the system is perhaps the m o s t advanced of all the components. It features a sophisticated salience model designed to produce high accuracy coreference in highly am- biguous texts. I t is capable of noticing a m b i g u i t y in text, and will fail to resolve pronouns in such cir- cumstances. For example the system will n o t resolve 'he' in the following example:
Earl and Ted were working together w h e n suddenly he fell into the threshing machine.
We resolve pronouns like ' t h e y ' , 'it', ' h e ' , 'hers', 'themselves' to p r o p e r nouns, common nouns and other pronouns. Depending on the genre of d a t a being processed, this c o m p o n e n t can resolve 60-90% of the pronouns in a text with very high accuracy.
3.5 T h e O v e r a l l N e x u s o f C o r e f e r e n c e in a D o c u m e n t
Once all the coreference in a document h a s been computed, we have a good approximation of which sentences are strongly related to other sentences in the document b y counting the number of corefer- ence links between the sentences. We know which entities are mentioned m o s t often, and w h a t other entities are involved in the same sentences or para- graphs. This s o r t of information has been used to generate very effective summaries of d o c u m e n t s and as a foundation for a simple visualization interface to texts.
4 C r o s s D o c u m e n t C o r e f e r e n c e
Coreference Chains for doc.O I
Cross-Document Coreference Chains
i I
J I VSM- I Disambiguate
~ summary.O1 1 J
" ~ J summary.nn I ~
Figure 1: Architecture of the Cross-Document Coreference System
John Perry, of lVeston Golf Club, an- nounced his resignation yesterday. He was the President of the Massachusetts Golf Association. During his two years in of- rice, Perry guided the MGA into a closer relationship with the Women's Golf Asso- ciation of Massachusetts.
Figure 2: Extract from doc.36
,
'
!
@
,
I
Figure 3: Coreference Chains for doc.36
This module takes as input the coreference chains produced by CAMP's within document coreference module. Details about each of the main steps of the cross-document coreference algorithm are given below.
• First, for each article, the within document coreference module of CAMP is run on that article. It produces coreference chains for all
the entities mentioned in the article. For exam- ple, consider the two extracts in Figures 2 and 4. The coreference chains output by CAMP for the two extracts are shown in Figures 3 and 5.
• Next, for the coreference chain of interest within each article (for example, the coreference chain that contains "John Perry"), the Sentence Ex- tractor module extracts all the sentences that contain the noun phrases which form the coref- erence chain. In other words, the SentenceEx- tractor module produces a "summary" of the ar- ticle with respect to the entity of interest. These sun~maries are a special case of the query sensi- tive techniques being developed at Penn using CAMP. Therefore, for doc.36 (Figure 2), since at least one of the three noun phrases ("John Perry," "he," and "Perry") in the coreference chain of interest appears in each of the three sentences in the extract, the summary produced by SentenceExtractor is the extract itself. On the other hand, the summary produced by Sen- tenceExtractor for the coreference chain of in- terest in doc.38 is only the first sentence of the extract because the only element of the corefer- ence chain appears in this sentence.
[image:3.612.103.504.57.335.2] [image:3.612.83.280.481.578.2]01iver "Biff" Kelly of Weymouth suc- ceeds John P e r r y as president of the Mas- sachusetts Golf Association. "We win have continued growth in the future," said Kelly, who will serve for two years. "There's been a lot of changes and there win be continued changes as we head into the year 2000."
Figure 4: E x t r a c t from doc.38
I I
I I
I
Figure 5: Coreference Chains for doc.38
100 90 80 70 g 6o ~ 5o
~.
40{
3 0
Precision/Recall vs Threshold
~ u r A I g : Precision o
\~,n _ Our AIg:" Recall -~--
Atg: F-Measure -~--
]
" \ ~.
" / ; ~ ~ * ~ ~ ' ~ " O' "~" O - Q--O- G--O-..E]-- O--O-.[~. _
i:-e/
",.,,.
c~20'
10
0 I I I I I l 1 1 I
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Threshold
Figure 7: Precision, Recall, and F-Measure Using Our Algorithm for the John Smith D a t a Set
the summaries extracted from each of the other articles. The VSM-Disambiguate module uses a standard vector space model (used widely in in- formation retrieval) (Salton, 89) to compute the similarities between the summaries. Summaries having similarity above a certain threshold are considered to be regarding the same entity.
4.1 E x p e r i m e n t s a n d R e s u l t s
We tested our cross-document system on two highly ambiguous test sets. T h e first set contained 197 articles from the 1996 and 1997 editions of the New York Times, while the second set contained 219 articles from the 1997 edition of the New York Times. T h e sole criteria for including an article in the two sets was the presence of a string matching the " / J o h n . * ? S m i t h / " , and the " / r e s i g n / " regular expressions respectively.
T h e goal for the first set was to identify cross- document coreference chains about the same John Smith, and the goal for the second set was to identify cross-document coreference chains about the same "resign" event. The answer keys were manually cre- ated, but the scoring was completely automated.
T h e r e were 35 different John Smiths in the first set. Of these, 24 were involved in chains of size 1. T h e other 173 articles were regarding the 11 remain- ing John Smiths. Descriptions of a few of the John Smiths are: Chairman and CEO of General Motors, assistant track coach at UCLA, the legendary ex- plorer, and the main character in Disney's Pocahon- tas, former president of the Labor P a r t y of Britain. In the second set, there were 97 different "resign" events. Of these, 60 were involved in chains of size 1. T h e articles were regarding resignations of several different people including Ted Hobart of ABC Corp., Dick Morris, Speaker Jim Wright, and the possible resignation of Newt Gingrich.
4.2 S c o r i n g a n d R e s u l t s
In order to score the cross-document coreference chains o u t p u t by the system, we had to map the cross-document coreference scoring problem to a within-document coreference scoring problem. This was done by creating a m e t a document consisting of the file names of each of the documents t h a t the system was run on. Assuming t h a t each of the doc- uments in the two data sets was a b o u t a single John Smith, or a b o u t a single "resign" event, the cross- document coreference chains produced by the system could now be evaluated by scoring the correspond- ing within-document coreference chains in the m e t a document.
Precision and recall are the measures used to eval- uate the chains o u t p u t by the system. For an entity, i, we define the precision and recall with respect to that entity in Figure 6.
The final precision and recall numbers are com- puted by the following two formulae:
N
Final Precision = Z wi * Precision~
i = l
N
Final Recall = ~ wl * Recall~
i = l
where N is the number of entities in the document, and wi is the weight assigned to entity i in the docu- ment. For the results discussed in this paper, equal weights were assigned to each entity in the m e t a doc- ument. In other words, wi = -~ for all i. Full details about the scoring algorithm can be found in (Bagga, 98).
[image:4.612.322.544.55.221.2] [image:4.612.354.512.511.578.2]number of correct elements in the output chain containing entityi Precisioni =
Recalli =
number of elements in the output chain containing entityi
number of correct elements in the output chain containing entityi number of elements in the truth chain containing entityi
Figure 6: Definitions for Precision and Recall for an Entity i
E CD
D - 100
90
80
70
60
50
40
30
20
10
0 I I I I I I I I I
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Threshold
Precision/Recall vs Threshold
"'~, ~ Our Alg: Precision
"~-,'O--E~. ~ Our AIg: Recall ---~---
• ~' \,% ~ - G . . Our AIg: F-Measure -o---
.,]/ ' - ~ . 13"" O . . D . (3.. Q
,,'~ ~-..,% "E3"" O'-O-.E3.. 13.. O
.+,,. 4.. ,4,,. ÷
Figure 8: Precision, Recall, and F-Measure Using Our Algorithm for the "resign" D a t a Set
the s y s t e m on this d a t a set was 93% and 77% re- spectively (when the threshold for the vector space model was set to 0.15). Similarly, Figure 8 shows the s a m e three statistics for the "resign" d a t a set. T h e best precision and recall achieved by the sys- t e m on this d a t a set was 94% and 81% respectively. This occurs when the threshold for the vector space model was set to 0.2. The results show t h a t the sys- t e m was very successful in resolving cross-document coreference.
5 P o s s i b l e G e n e r a l i z a t i o n s A b o u t L a r g e D a t a C o l l e c t i o n s D e r i v e d F r o m C o r e f e r e n c e A n n o t a t i o n s
Crucial to the entire process of visualizing large doc- u m e n t collections is relating the same individual or event across multiple documents. This single aspect of our s y s t e m establishes its viability for large col- lection analysis. It allows the drops of information held in each d o c u m e n t to be merged into a larger pool t h a t is well organized.
5.1 T h e P r i m a r y D i s p l a y o f Information
T w o display techniques immediately suggest them- selves for accessing the coreference annotations in a d o c u m e n t collection, the first is to take the identi- fied entities as atomic and link t h e m to other entities which co-occur in the same document. This might reveal a relation between individuals and events, or
individuals a n d other individuals. For example, such a linking m i g h t indicate t h a t no n e w s p a p e r article ever mentioned b o t h Clark K e n t a n d S u p e r m a n in the same article, b u t t h a t m o s t all other famous in- dividuals t e n d e d to overlap in some article or an- other. On the positive case, individuals, over time, m a y tend to congregate in m e d i a stories or events m a y tend to be more tightly linked t h a n otherwise expected.
The second technique would be to take as a t o m i c the documents and relate via links other d o c u m e n t s t h a t contain m e n t i o n of the s a m e entity. W i t h a t e m - poral dimension, the role of individuals a n d events could be assessed as time m o v e d forward.
5.2 F i n e r G r a i n e d A n a l y s i s o f t h e D o c u m e n t s
T h e fact t h a t two entities coexisted in the s a m e sen- tence in a d o c u m e n t is n o t e w o r t h y for correlational analysis. Links could be restricted to those between entities t h a t co-existed in the s a m e sentence or para- graph. Additional filterings are possible with con- straints on t h e sorts of verbs t h a t exist in the sen- tence.
A more sophisticated version of the above is to access the a r g u m e n t s t r u c t u r e of the document. C A M P software provides a limited predicate argu- ment structure t h a t allows s u b j e c t s / v e r b s / o b j e c t s to be identified. This ability moves our a n n o t a t i o n closer to the fixed record d a t a s t r u c t u r e of a tra- ditional d a t a base. One could select an event and its object, for instance 'X sold a r m s to I r a q ' a n d see what the fillers for X were in a link analysis. There are limitations to predicate a r g u m e n t struc- ture m a t c h i n g - f o r instance getting the correct p a t - tern for all the selling of a r m s variations is quite difficult.
In any case, there a p p e a r to be a m y r i a d of appli- cations for link analysis in the domain of large t e x t d a t a bases.
6 C o n c l u s i o n s
[image:5.612.54.292.152.329.2]space of information provided by coreference analy- sis.
formation by Computer, 1989, Reading, MA: Addison-Wesley.
7
Acknowledgments
The second author was supported in part by a Fel- lowship from IBM Corporation, and in part by the University of Pennsylvania. Part of this work was done when the second author was visiting the Insti- tute for Research in Cognitive Science at the Uni- versity of Pennsylvania.
References
Bagga, Amit, and Breck Baldwin. Algorithms for
Scoring Coreference Chains. Proceedings of the
the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation, May 1998.
Bagga, Amit. Evaluation of Coreferences and Coref-
erence Resolution Systems. Proceedings of the
First Language Resource and Evaluation Confer- ence, pp. 563-566, May 1998.
Bagga, Amit, and Breck Baldwin. Entity-Based Cross-Document Coreferencing Using the Vec-
tor Space Model. To appear at the 17th Inter-
national Conference on Computational Linguis- tics and the 36th Annual Meeting of the Asso- ciation for Computational Linguistics (COLING- ACL'98), August 1998.
Baldwin, Breck. CogNIAC: A Discourse Processing Engine. University of Pennsylvania Department of Computer and Information Sciences Ph.D. Thesis, 1995.
Baldwin, B., C. Doran, J. Reynar, M. Niv, and M. Wasson. EAGLE: An Extensible Architecture
for General Linguistic Engineering. Proceedings
RIA O, Computer-Assisted Information Searching on Internet, Montreal, Canada, 1997.
Collins, Michael. A New Statistical Parser Based on
Bigram Lexical Dependencies. Proceedings of the
34 th Meeting of the Association for Computational Linguistics, 1996.
Ratnaparkhi, Adwait. A Maximum Entropy Model
for Part-Of-Speech Tagging. Proceedings of the
Conference on Empirical Methods in Natural Lan- guage Processing, pp. 133-142, May 1996. Wacholder, Nina, Yael Ravin, and Misook Choi. Dis-
ambiguation of Proper Names in Text. Proceedings
of the Fifth Conference on Applied Natural Lan- guage Processing, pp. 202-208, 1997.
Reynar, Jeffrey, and Adwait Ratnaparkhi. A Max- imum Entropy Approach to Identifying Sentence
Boundaries. Proceedings of the Fifth Conference
on Applied Natural Language Processing, pp. 16- 19, 1997.
Salton, Gerard. Automatic Text Processing: The
Transformation, Analysis, and Retrieval of In-
