The MASC Word Sense Sentence Corpus
Rebecca J. Passonneau
∗, Collin Baker
†, Christiane Fellbaum
‡, Nancy Ide
¶∗Columbia University
New York, NY, USA [email protected]
†ICSI
Berkeley, CA
‡Princeton University
Princeton, NJ, USA [email protected]
¶Vassar College
Poughkeepsie,NY, USA [email protected]
Abstract
The MASC project has produced a multi-genre corpus with multiple layers of linguistic annotation, together with a sentence corpus containing WordNet 3.1 sense tags for 1000 occurrences of each of 100 words produced by multiple annotators, accompanied by in-depth inter-annotator agreement data. Here we give an overview of the contents of MASC and then focus on the word sense sentence corpus, describing the characteristics that differentiate it from other word sense corpora and detailing the inter-annotator agreement studies that have been performed on the annotations. Finally, we discuss the potential to grow the word sense sentence corpus through crowdsourcing and the plan to enhance the content and annotations of MASC through a community-based collaborative effort.
Keywords:word sense annotation, open data, multi-layer annotation
1.
Introduction
Annotated corpora play an increasingly significant role in computational linguistics to address lexical, propositional, and discourse semantics. Each of these dimensions is im-portant in its own right, yet they also interact in the de-termination of meaning. As a result, there is an increased demand for high quality linguistic annotation to represent a wide range of phenomena, especially semantic and prag-matic ones, to support machine learning and computational linguistics research. At the same time, there is a demand for annotated corpora representing a broad range of genres, due to the increasingly apparent impact of domain on both syntactic and semantic characteristics. Finally, there is a keen awareness of the need for annotated corpora that are both easily accessible and available for use by anyone. To address these needs, construction of the Manually An-notated Sub-Corpus (MASC) (Ide et al., 2010) was un-dertaken in response to a community mandate at a 2006 workshop funded by the US National Science Foundation. MASC is a half million word corpus of American English language data drawn from the 15 million word Open Amer-ican National Corpus (OANC).1 It is annotated for an in-creasing number of linguistic phenomena, all of which are either manually produced or validated, and all of the data and annotations are freely available from the MASC web-site and the Linguistic Data Consortium (LDC).2
Here we give an overview of the contents of MASC and
1http://www.anc.org/OANC
.
2urlhttp://www.ldc.upenn.edu.
then focus on the word sense sentence corpus, describ-ing the characteristics that differentiate it from other word sense corpora. To date, MASC has been developed and cre-ated through a combination of the efforts of the MASC col-laborators and contributions from other annotation projects. However, for MASC to grow in size and in number of an-notation types, we aim to encourage a variety of collective efforts. We briefly discuss a range of such efforts, including the use of Amazon Mechanical Turkers to perform word sense annotation, and ways in which the community can help grow the corpus and its annotations.
2.
MASC Contents
Rnd Word Pos Senses α(95% conf. inter) -1 Ann αmasi Pairwise
9 late adj 9 (0.77) 0.83 (0.90) 0.87 0.84 0.89
10 high adj 9 (0.73) 0.82 (0.90) 0.85 0.82 0.90
7 poor adj 13 (0.50) 0.54 (0.63) 0.67 0.59 0.66
10 common adj 13 (0.32) 0.40 (0.46) 0.53 0.40 0.52
10 particular adj 9 (0.09) 0.21 (0.26) 0.30 0.21 0.45
9 normal adj 3 (-0.09) -0.02 (0.04) 0.08 -0.02 0.38
7 strike noun 7 (0.83) 0.88 (0.94) 0.93 0.88 0.93
8 state noun 11 (0.65) 0.72 (0.79) 0.78 0.73 0.79
8 number noun 8 (0.45) 0.68 (0.86) 0.79 0.68 0.95
10 control noun 15 (0.27) 0.35 (0.42) 0.44 0.35 0.46
9 level noun 12 (0.16) 0.22 (0.28) 0.30 0.26 0.43
8 family noun 16 (0.18) 0.14 (0.36) 0.26 0.35 0.51
8 live verb 13 (0.62) 0.70 (0.76) 0.73 0.70 0.79
9 appear verb 8 (0.53) 0.61 (0.68) 0.70 0.61 0.72
10 book verb 10 (0.50) 0.63 (0.75) 0.68 0.65 0.79
10 help verb 10 (0.01) 0.31 (0.35) 0.41 0.31 0.68
9 fold verb 10 (0.08) 0.18 (0.26) 0.29 0.18 0.40
8 ask verb 8 (0.00) 0.05 (0.09) 0.48 0.05 0.35
Table 3: Agreement results for words with the three highest and three lowest agreement scores, for each part of speech
among annotators on such sentences, which might suggest additional features for WSD classifiers.
4.
Growing MASC
To grow the MASC word sense sentence corpus, we are ex-ploring the potential to use crowdsourcing to collect high quality word sense annotations. In a pilot study reported in (Passonneau et al., To Appear), we compared the qual-ity of annotations from a half dozen trained annotators on a single round of ten words with annotations from over a dozen Amazon Mechanical Turkers (AMT). While results were inconsistent, they were sufficiently promising that we are currently collecting AMT word sense annotations for a much larger set of words. We also continue to add manual annotations, and should produce tags from multiple annota-tors for approximately 50 new words within the next several months.
MASC as a whole is intended to serve as a base for a sustained collaborative resource development effort by the community. In our view, a community-wide, collaborative effort to produce open, high quality annotated corpora is one of the very few possible ways to address the high costs of resource production and ensure that the entire commu-nity, including large teams as well as individual researchers, has access and means to use these resources in their work. The ultimate goal is to build on the MASC base to pro-vide an ever-expanding, open linguistic infrastructurefor the field, in which the community engages in a distributed effort to provide, enhance, and evaluate data, annotations, and other linguistic resources that are easily accessible and free for community use.
MASC is at present a half million words, substantially smaller than some other multiply-annotated corpora and therefore less ideal for training language models. The cor-pus will be soon increased in size to a million words, al-though there are currently no resources for further in-house validation; we will depend on the community to contribute annotations to fill in the gap. At present, researchers and developers are invited to contribute annotations of MASC
and/or OANC data of any type, in any format, to be in-corporated into MASC in a common format that makes all MASC annotations usable together. Others can con-tribute their annotations of MASC and/or OANC data by sending them in email [email protected]. The MASC/OANC team transduces contributed annotations to GrAF so that all MASC annotations are usable together and can be input to services such as ANC2Go.9
In addition to growing MASC and the sense-annotated sen-tence corpus, an effort to create ”Multi-MASC”, consist-ing of open corpora in other languages that are built to be comparable in genre distribution to MASC, is just get-ting underway (Ide, 2012). The goal is to produce a mas-sive multi-lingual corpus including language-specific data with comparable genre distribution and annotations, ulti-mately with linkages among annotations across languages. Like the continued development of the American English MASC, creation of comparable Multi-MASC corpora will rely on community effort.
5.
Conclusion
The MASC project has produced a multi-genre corpus with multiple layers of linguistic annotation, together with a “sentence” corpus containing WordNet 3.1 sense tags for 1000 occurrences of each of 100 words produced by mul-tiple annotators, accompanied by in-depth inter-annotator agreement data. All data and annotations are completely open and free for community use. The intent is that the community will use these resources as a base upon which to build by contributing additional data and annotations of all kinds. Hopefully, a substantial community-based collabo-rative effort to develop the MASC resources will ultimately serve to avoid redundant work, enable substantive evalua-tion and replicaevalua-tion of results, and empower all members of the community with access to high-quality resources in the years to come.
9We have developed and will soon make available software to
Acknowledgments
This work was supported by National Science Foundation grants CRI-0708952 and CRI-1059312.
References
Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596.
Collin F. Baker and Hiroaki Sato. 2003. The FrameNet data and software. InCompanion Volume to the Proceed-ings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 161–164.
Vikas Bhardwaj, Rebecca J. Passonneau, Ansaf Salleb-Aouissi, and Nancy Ide. 2010. Anveshan: A frame-work for analysis of multiple annotators’ labeling behav-ior. InProceedings of the Fourth Linguistic Annotation Workshop (LAW IV), pages 47–55.
Alioscha Burchardt and Marco Pennacchiotti. 2008. FATE: a FrameNet-annotated corpus for textual entail-ment. InProceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008), Marrakech, Morocco. European Language Resources As-sociation (ELRA).
Christiane Fellbaum. 1998.WordNet: An Electronic Lex-ical Database. MIT Press, Cambridge, MA.
Eduard Hovy, Mitchel Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. InProceedings of HLT-NAACL 2006, pages 57–60.
Nancy Ide and Keith Suderman. Submitted. The Lin-guistic Annotation Framework: A Standard for Annota-tion Interchange and Merging. Language Resources and Evaluation.
Nancy Ide, Collin Baker, Christiane Fellbaum, and Re-becca J. Passonneau. 2010. The manually annotated sub-corpus: a community resource for and by the people. In
Proceedings of the Association for Computational Lin-guistics, pages 68–73.
Nancy Ide. 2012. MultiMASC: An open linguistic in-frastructure for language research. InProceedings of the Workshop on Building and Using Comparable Corpora. Rebecca J. Passonneau, Nizar Habash, and Owen Ram-bow. 2006. Inter-annotator agreement on a multilingual semantic annotation task. InProceedings of the 5th Inter-national Conference on Language Resources and Evalua-tion (LREC), pages 1951–1956, Genoa, Italy.
Rebecca J. Passonneau, Ansaf Salleb-Aouissi, and Nancy Ide. 2009. Making sense of word sense variation. InSEW ’09: Proceedings of the Workshop on Semantic Evalua-tions: Recent Achievements and Future Directions, pages 2–9, Morristown, NJ, USA. Association for Computa-tional Linguistics.
Rebecca J. Passonneau, Ansaf Salleb-Aouissi, Vikas Bhardwaj, and Nancy Ide. 2010. Word sense annotation of polysemous words by multiple annotators. In Proceed-ings of the 7th International Conference on Language Re-sources and Evaluation (LREC), pages 3244–3249. Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. To Appear. Multiplicity and word
sense: evaluating and learning from multiply labeled word sense annotations.Language Resources and Evaluation. Rebecca J. Passonneau. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic an-notation. InProceedings of the 5th International Con-ference on Language Resources and Evaluation (LREC), pages 831–836, Genoa, Italy, May.
