Towards a Universal Grammar for
Natural Language Processing
Joakim Nivre
Uppsala University
Department of Linguistics and Philology
Based on collaborative work with Filip Ginter, Yoav Goldberg, Jan Hajič,
Chris Manning, Ryan McDonald, Natalia Silveira, Marie de Marneffe, Slav Petrov, Sampo Pyysalo, Reut Tsarfaty, Daniel Zeman and many others
“In its substance, grammar is one
and the same in all languages,
“In its substance, grammar is one
and the same in all languages,
“In its substance, grammar is one
and the same in all languages,
Universal Grammar
Universal Grammar
• All human languages are species of a common genus • Language structure is constrained by a universal cause
Universal Grammar
• All human languages are species of a common genus • Language structure is constrained by a universal cause • There is order in the chaos of linguistic variation
Natural Language Processing
• Linguistic diversity makes our life harder
• Why 90% parsing accuracy for English but only 80% for Finnish? • Can we even compare the numbers?
Natural Language Processing
• Linguistic diversity makes our life harder
• Why 90% parsing accuracy for English but only 80% for Finnish? • Can we even compare the numbers?
• Current NLP relies heavily on linguistic annotation: • “In its substance, grammar is the same in all languages, even if
Natural Language Processing
• Linguistic diversity makes our life harder
• Why 90% parsing accuracy for English but only 80% for Finnish? • Can we even compare the numbers?
• Current NLP relies heavily on linguistic annotation: • “In its substance, grammar is the same in all languages, even if
the annotation accidentally varies.”
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language X
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language X
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Y
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language X
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Y
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Z
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language X
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Y
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Z
Which languages are most closely related?
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language X
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Y
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Z
Which languages are most closely related?
1/5
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language X
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Y
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Z
Which languages are most closely related?
1/5
2/5
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language X
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Y
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager r˚ader og møs
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
1
Language Z
Which languages are most closely related?
1/5 2/5 2/5 Swedish Danish English
En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
En katt jagar r˚attor och m¨oss det nsubj conj
dobj
conj
En kat jager rotter og mus nsubj
? dobj cc conj
A cat chases rats and mice det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc
Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
Why is this a problem?
Why is this a problem?
• Hard to compare empirical results across languages • Hard to evaluate cross-lingual learning
Why is this a problem?
• Hard to compare empirical results across languages • Hard to evaluate cross-lingual learning
Why is this a problem?
• Hard to compare empirical results across languages • Hard to evaluate cross-lingual learning
• Hard to build and maintain multilingual systems
Universal Dependencies
http://universaldependencies.org
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
Universal Dependencies
http://universaldependencies.org
Part-of-speech tags
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
Universal Dependencies
http://universaldependencies.org
Part-of-speech tags
Morphological features
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
Universal Dependencies
http://universaldependencies.org
Part-of-speech tags
Morphological features Dependency relations
En katt jagar r˚attor och m¨oss
det nsubj conj
dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
Universal Dependencies
Universal Dependencies
http://universaldependencies.org
Stanford Dependencies
Universal Dependencies
http://universaldependencies.org
Stanford Dependencies
Universal Dependencies
http://universaldependencies.org Stanford Dependencies Google UD Stanford UDUniversal Dependencies
http://universaldependencies.org Stanford Dependencies Google UD Stanford UD HamleDTUniversal Dependencies
http://universaldependencies.org Stanford Dependencies Google UD Stanford UD HamleDT IntersetUniversal Dependencies
http://universaldependencies.org Stanford Dependencies Google UD Stanford UD HamleDT Interset Google universal tagsUniversal Dependencies
Universal Dependencies
Universal Dependencies
• Milestones:
• Kick-off meeting at EACL in Gothenburg, April 2014
• Release of annotation guidelines, Version 1, October 2014 • Release of treebanks for 10 languages, January 2015
• Release of treebanks for 18 languages, May 2015
• Release of treebanks for 33 languages, November 2015
• Open community effort – anyone can contribute!
Universal Dependencies
Goals and Requirements
Goals and Requirements
• Cross-linguistically consistent grammatical annotation • Support multilingual research and development in NLP
Goals and Requirements
• Cross-linguistically consistent grammatical annotation • Support multilingual research and development in NLP
Design Principles
• Dependency
• Widely used in practical NLP systems
Design Principles
• Dependency
• Widely used in practical NLP systems
• Available in treebanks for many languages • Lexicalism
• Basic annotation units are words – syntactic words • Words have morphological properties
Design Principles
• Dependency
• Widely used in practical NLP systems
• Available in treebanks for many languages • Lexicalism
• Basic annotation units are words – syntactic words • Words have morphological properties
• Words enter into syntactic relations • Recoverability
Golden Rules
• Maximize parallelism
• Don’t annotate the same thing in different ways • Don’t make different things look the same
Golden Rules
• Maximize parallelism
• Don’t annotate the same thing in different ways • Don’t make different things look the same
• But don’t overdo it
• Don’t annotate things that are not there
• Languages select from a universal pool of categories • Allow language-specific extensions
Morphology
En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
• Lemma representing the semantic content of the word
En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , le fille adorer le dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct
Toutefois , les filles adorent les desserts . toutefois , le fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc
Number=Plur Number=Plur Person=3 Number=Plur Number=Plur Tense=Pres advmod punct det nsubj root det dobj punct 1
Morphology
En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
• Lemma representing the semantic content of the word • Part-of-speech tag representing the abstract lexical
category associated with the word
En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , le fille adorer le dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct
Toutefois , les filles adorent les desserts . toutefois , le fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc
Number=Plur Number=Plur Person=3 Number=Plur Number=Plur Tense=Pres advmod punct det nsubj root det dobj punct 1
Morphology
En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1 En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
• Lemma representing the semantic content of the word • Part-of-speech tag representing the abstract lexical
category associated with the word
• Features representing lexical and grammatical properties
associated with the lemma or the particular word form
En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , le fille adorer le dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct
Toutefois , les filles adorent les desserts . toutefois , le fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc
Number=Plur Number=Plur Person=3 Number=Plur Number=Plur Tense=Pres advmod punct det nsubj root det dobj punct 1
Morphology
En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1 En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1 En katt jagar r˚attor och m¨oss
det nsubj conj dobj
conj
En kat jager rotter og mus
nsubj
? dobj cc conj
A cat chases rats and mice
det nsubj dobj cc
conj
Toutefois , les filles adorent les desserts .
toutefois , les fille adorer les dessert .
ADV PUNCT DET NOUN VERB DET NOUN PUNCT
Definite=Def Gender=Fem Number=Plur Definite=Def Gender=Masc Number=Plur Number=Plur Person=3 Number=Plur Number=Plur
Tense=Pres advmod punct det nsubj root det dobj punct 1
• Taxonomy of 17 universal part-of-speech tags, based on
the Google Universal Tagset (Petrov et al., 2012)
• All languages use the same inventory, but not all tags
have to be used by all languages
Part-of-Speech Tags
Open Closed Other
ADJ ADP PUNCT
ADV AUX SYM
INTJ CONJ X NOUN DET PROPN NUM VERB PART PRON SCONJ
Features
• Standardized inventory of morphological features, based
on the Interset system (Zeman, 2008)
• Languages select relevant features and can add
language-specific features or values with documentation
Lexical Inflectional Nominal Inflectional Verbal
PronType Gender VerbForm
NumType Animacy Mood
Poss Number Tense
Reflex Case Aspect
Definite Voice
Degree Person
Syntax
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT det nsubj aux aux root det det dobj case det nmod punct
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT det nsubj aux aux root det det dobj case det nmod
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT nsubj
root
dobj
nmod
Syntax
• Content words are related by dependency relations
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT
det nsubj aux aux root det det dobj case det nmod punct
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT
det nsubj aux aux root det det dobj case det nmod
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT
nsubj
root
dobj
nmod
Syntax
• Content words are related by dependency relations
• Function words attach to the content word they modify
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT det nsubj aux aux root det det dobj case det nmod punct
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT det nsubj aux aux root det det dobj case det nmod
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT nsubj
root
dobj
nmod
Syntax
• Content words are related by dependency relations
• Function words attach to the content word they modify • Punctuation attach to head of phrase or clause
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT det nsubj aux aux root det det dobj case det nmod punct
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT det nsubj aux aux root det det dobj case det nmod
The cat could have chased all the dogs down the street .
DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT nsubj
root
dobj
nmod
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT
det nsubjpass auxpass root case det nmod punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def
nsubjpass
root
case nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT
nsubjpass
root
nmod
punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def
nsubjpass
root
nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT
det nsubjpass root det nmod punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def
nsubjpass
root
nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT det nsubjpass auxpass root case det nmod punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def
nsubjpass
root
case nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT nsubjpass
root
nmod
punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def
nsubjpass
root
nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT det nsubjpass root det nmod punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def
nsubjpass
root
nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT
det nsubjpass auxpass root det nmod punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def
nsubjpass
root
nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT
det nsubjpass auxpass root case det nmod punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def nsubjpass
root
case nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT
nsubjpass
root
nmod
punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def nsubjpass
root
nmod
punct
The dog was chased by the cat .
DET NOUN AUX VERB ADP DET NOUN PUNCT
det nsubjpass root det nmod punct
Hunden jagades av katten .
NOUN VERB ADP NOUN PUNCT
Definite=Def Voice=Pass Definite=Def nsubjpass
root
nmod
punct
Dependency Relations
• Taxonomy of 40 universal grammatical relations, broadly
attested in language typology (de Marneffe et al., 2014)
Dependency Relations
• Taxonomy of 40 universal grammatical relations, broadly
attested in language typology (de Marneffe et al., 2014)
• Language-specific subtypes may be added • Organizing principles
• Three types of structures: nominals, clauses, modifiers
Dependents of Clausal Predicates
Nominal Clausal Other
Core nsubj nsubjpass dobj iobj csubj csubjpass ccomp xcomp Non-Core nmod vocative discourse expl advcl advmod neg aux auxpass cop mark punct
Mary was quietly reading a book in the garden .
PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT
nsubj aux advmod root det dobj case det nmod punct
If you are sick , you should not exercise .
SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT
mark nsubj cop advcl punct nsubj aux neg root punct
Peter thought that he should stop smoking .
PROPN VERB SCONJ PRON AUX VERB VERB PUNCT
nsubj root mark nsubj aux ccomp xcomp punct
Cairo , the lovely capital of Egypt PROPN PUNCT DET ADJ NOUN ADP PROPN
mark det amod appos case nmod
Hewey , Dewey and Louie PROPN PUNCT PROPN CONJ PROPN
punct
conj
cc
conj
Mary was quietly reading a book in the garden .
PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT
nsubj aux advmod root det dobj case det nmod punct
If you are sick , you should not exercise .
SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT
mark nsubj cop advcl punct nsubj aux neg root punct
Peter thought that he should stop smoking .
PROPN VERB SCONJ PRON AUX VERB VERB PUNCT
nsubj root mark nsubj aux ccomp xcomp punct
Cairo , the lovely capital of Egypt PROPN PUNCT DET ADJ NOUN ADP PROPN
mark det amod appos case nmod
Hewey , Dewey and Louie PROPN PUNCT PROPN CONJ PROPN
punct
conj
cc
conj
5
Mary was quietly reading a book in the garden .
PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT
nsubj aux advmod root det dobj case det nmod punct
If you are sick , you should not exercise .
SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT
mark nsubj cop advcl punct nsubj aux neg root punct
Peter thought that he should stop smoking .
PROPN VERB SCONJ PRON AUX VERB VERB PUNCT
nsubj root mark nsubj aux ccomp xcomp punct
Cairo , the lovely capital of Egypt PROPN PUNCT DET ADJ NOUN ADP PROPN
mark det amod appos case nmod
Hewey , Dewey and Louie PROPN PUNCT PROPN CONJ PROPN
punct
conj
cc
conj
Mary was quietly reading a book in the garden .
PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT
nsubj aux advmod root det dobj case det nmod punct
If you are sick , you should not exercise .
SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT
mark nsubj cop advcl punct nsubj aux neg root punct
Peter thought that he should stop smoking .
PROPN VERB SCONJ PRON AUX VERB VERB PUNCT
nsubj root mark nsubj aux ccomp xcomp punct
Cairo , the lovely capital of Egypt PROPN PUNCT DET ADJ NOUN ADP PROPN
mark det amod appos case nmod
Hewey , Dewey and Louie PROPN PUNCT PROPN CONJ PROPN
punct
conj
cc
conj
5
Mary was quietly reading a book in the garden .
PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT
nsubj aux advmod root det dobj case det nmod punct
If you are sick , you should not exercise .
SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT
mark nsubj cop advcl punct nsubj aux neg root punct
Peter thought that he should stop smoking .
PROPN VERB SCONJ PRON AUX VERB VERB PUNCT
nsubj root mark nsubj aux ccomp xcomp punct
Cairo , the lovely capital of Egypt PROPN PUNCT DET ADJ NOUN ADP PROPN
mark det amod appos case nmod
Hewey , Dewey and Louie PROPN PUNCT PROPN CONJ PROPN
punct
conj
cc
conj
5
Mary was quietly reading a book in the garden .
PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT
nsubj aux advmod root det dobj case det nmod punct
If you are sick , you should not exercise .
SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT
mark nsubj cop advcl punct nsubj aux neg root punct
Peter thought that he should stop smoking .
PROPN VERB SCONJ PRON AUX VERB VERB PUNCT
nsubj root mark nsubj aux ccomp xcomp punct
Cairo , the lovely capital of Egypt PROPN PUNCT DET ADJ NOUN ADP PROPN
mark det amod appos case nmod
Hewey , Dewey and Louie PROPN PUNCT PROPN CONJ PROPN
punct
conj
cc
conj
Dependents of Nominals
Nominal Clausal Other
nummod appos nmod acl amod det case
Mary was quietly reading a book in the garden .
PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT
nsubj aux advmod root det dobj case det nmod punct
If you are sick , you should not exercise .
SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT
mark nsubj cop advcl punct nsubj aux neg root punct
Peter thought that he should stop smoking .
PROPN VERB SCONJ PRON AUX VERB VERB PUNCT
nsubj root mark nsubj punct ccomp xcomp punct
Cairo , the lovely capital of Egypt
PROPN PUNCT DET ADJ NOUN ADP PROPN
mark det amod appos case nmod 5
• Coordinate structures are headed by the first conjunct • Subsequent conjuncts depend on it via the conj relation
• Conjunctions depend on it via the cc relation
• Punctuation marks depend on it via the punct relation
Mary was quietly reading a book in the garden .
PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT
nsubj aux advmod root det dobj case det nmod punct
If you are sick , you should not exercise .
SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT
mark nsubj cop advcl punct nsubj aux neg root punct
Peter thought that he should stop smoking .
PROPN VERB SCONJ PRON AUX VERB VERB PUNCT
nsubj root mark nsubj aux ccomp xcomp punct
Cairo , the lovely capital of Egypt
PROPN PUNCT DET ADJ NOUN ADP PROPN
mark det amod appos case nmod
Huey , Dewey and Louie
PROPN PUNCT PROPN CONJ PROPN
punct conj cc conj 5
Coordination
Coordination conj cc (punct)Multiword Expressions
• UD annotation does not permit “words with spaces” • Multiword expressions are analysed using special relations
• The mwe, name and goeswith relations are always head-initial • The compound relation reflects the internal structure
Relation Examples
mwe in spite of, as well as, ad hoc
name Roger Bacon, Carl XVI Gustaf, New York
compound phone book, four thousand, dress up
Other Relations
Relation Explanation
parataxis Loosely linked clauses of same rank list Lists without syntactic structure
remnant Orphans in ellipsis linked to parallel elements reparandum Disfluency linked to (speech) repair
foreign Elements within opaque stretches of code switching dep Unspecified dependency
Language-Specific Relations
• Language-specific relations are subtypes of universal
relations added to capture important phenomena
• Subtyping permits us to “back off” to universal relations
Relation Explanation
acl:relcl Relative clause
compound:prt Verb particle (dress up)
nmod:poss Genitive nominal (Mary’s book) nmod:agent Agent in passive (saved by the bell)
cc:preconj Preconjunction (both … and) det:predet Predeterminer (all those …)
Word Segmentation
• How do we segment sentences into words?
• Dependent on language and writing system, often non-trivial • Segmentation must be reproducible on new data
Word Segmentation
• How do we segment sentences into words?
• Dependent on language and writing system, often non-trivial • Segmentation must be reproducible on new data
• Two options provided:
• Only include words in treebank, but document segmentation
Word Segmentation
• How do we segment sentences into words?
• Dependent on language and writing system, often non-trivial • Segmentation must be reproducible on new data
• Two options provided:
• Only include words in treebank, but document segmentation
• Include mapping from low-level tokenisation to words in treebank
Vamos nos a el mar .
VERB PRON ADP DET NOUN PUNCT
V´amonos al mar .
? ? NOUN PUNCT
• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar .• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar . LEMMA _ ir nosotros _ a el mar .• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar . LEMMA _ ir nosotros _ a el mar . UPOSTAG _ VERB PRON _ ADP DET NOUN .• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar . LEMMA _ ir nosotros _ a el mar . UPOSTAG _ VERB PRON _ ADP DET NOUN . XPOSTAG _ _ _ _ _ _ _ _ _• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar . LEMMA _ ir nosotros _ a el mar . UPOSTAG _ VERB PRON _ ADP DET NOUN . XPOSTAG _ _ _ _ _ _ _ _ _ FEATS _ Mood=Imp|Number=Plur|Person=1 PronType=Per|Number=Plur|Person=1 _ _ Definite=Def|Number=Sing| Gender=Masc Number=Sing|Gender=Masc _• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar . LEMMA _ ir nosotros _ a el mar . UPOSTAG _ VERB PRON _ ADP DET NOUN . XPOSTAG _ _ _ _ _ _ _ _ _ FEATS _ Mood=Imp|Number=Plur|Person=1 PronType=Per|Number=Plur|Person=1 _ _ Definite=Def|Number=Sing| Gender=Masc Number=Sing|Gender=Masc _ HEAD _ 0 1 _ 5 5 1 1• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar . LEMMA _ ir nosotros _ a el mar . UPOSTAG _ VERB PRON _ ADP DET NOUN . XPOSTAG _ _ _ _ _ _ _ _ _ FEATS _ Mood=Imp|Number=Plur|Person=1 PronType=Per|Number=Plur|Person=1 _ _ Definite=Def|Number=Sing| Gender=Masc Number=Sing|Gender=Masc _ HEAD _ 0 1 _ 5 5 1 1 DEPREL _ root expl _ case det nmod punct• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar . LEMMA _ ir nosotros _ a el mar . UPOSTAG _ VERB PRON _ ADP DET NOUN . XPOSTAG _ _ _ _ _ _ _ _ _ FEATS _ Mood=Imp|Number=Plur|Person=1 PronType=Per|Number=Plur|Person=1 _ _ Definite=Def|Number=Sing| Gender=Masc Number=Sing|Gender=Masc _ HEAD _ 0 1 _ 5 5 1 1 DEPREL _ root expl _ case det nmod punct DEPS _ _ _ _ _ _ _ _• Revised version of the CoNLL-X format
• Two-level segmentation and secondary dependencies
CoNLL-U Format
ID 1-2 1 2 3-4 3 4 5 6 FORM Vámonos Vamos nos al a el mar . LEMMA _ ir nosotros _ a el mar . UPOSTAG _ VERB PRON _ ADP DET NOUN . XPOSTAG _ _ _ _ _ _ _ _ _ FEATS _ Mood=Imp|Number=Plur|Person=1 PronType=Per|Number=Plur|Person=1 _ _ Definite=Def|Number=Sing| Gender=Masc Number=Sing|Gender=Masc _ HEAD _ 0 1 _ 5 5 1 1 DEPREL _ root expl _ case det nmod punct DEPS _ _ _ _ _ _ _ _ MISC _ _ _ _ _ _ _ _Where are we now?
• Universal Dependencies, Version 1 • Guidelines released October 2014
• Latest treebank release November 2015 (v1.2):
Ancient Greek, Arabic, Basque, Bulgarian, Croatian, Czech, Danish, Dutch, English,
Estonian, Finnish, French, German, Gothic, Greek, Hebrew, Hindi, Hungarian,
Indonesian, Irish, Italian, Japanese, Latin, Norwegian, Old Church Slavonic, Persian, Polish, Portuguese, Romanian, Slovenian, Spanish, Swedish, Tamil
Where are we now?
• Universal Dependencies, Version 1 • Guidelines released October 2014
• Latest treebank release November 2015 (v1.2):
Ancient Greek, Arabic, Basque, Bulgarian, Croatian, Czech, Danish, Dutch, English,
Estonian, Finnish, French, German, Gothic, Greek, Hebrew, Hindi, Hungarian,
Indonesian, Irish, Italian, Japanese, Latin, Norwegian, Old Church Slavonic, Persian, Polish, Portuguese, Romanian, Slovenian, Spanish, Swedish, Tamil
• Future plans:
• New releases every six months (May, November) • Revision of guidelines as needed
Where are we now?
• Universal Dependencies, Version 1 • Guidelines released October 2014
• Latest treebank release November 2015 (v1.2):
Ancient Greek, Arabic, Basque, Bulgarian, Croatian, Czech, Danish, Dutch, English,
Estonian, Finnish, French, German, Gothic, Greek, Hebrew, Hindi, Hungarian,
Indonesian, Irish, Italian, Japanese, Latin, Norwegian, Old Church Slavonic, Persian, Polish, Portuguese, Romanian, Slovenian, Spanish, Swedish, Tamil
• Future plans:
• New releases every six months (May, November) • Revision of guidelines as needed
So what exactly is UD?
• A new linguistic theory?
• Not at all, but we like to think it is informed by linguistic theory and potentially useful also for linguistic studies
So what exactly is UD?
• A new linguistic theory?
• Not at all, but we like to think it is informed by linguistic theory and potentially useful also for linguistic studies
• A better parsing framework?
• Probably not, since parsers seem to prefer function words as heads so we may have to tweak the representations for parsing
So what exactly is UD?
• A new linguistic theory?
• Not at all, but we like to think it is informed by linguistic theory and potentially useful also for linguistic studies
• A better parsing framework?
• Probably not, since parsers seem to prefer function words as heads so we may have to tweak the representations for parsing
• The ultimate annotation scheme?
• Not quite, more like a lingua franca for treebank developers and definitely useful for some annotation projects
So what exactly is UD?
• A new linguistic theory?
• Not at all, but we like to think it is informed by linguistic theory and potentially useful also for linguistic studies
• A better parsing framework?
• Probably not, since parsers seem to prefer function words as heads so we may have to tweak the representations for parsing
• The ultimate annotation scheme?
• Not quite, more like a lingua franca for treebank developers and definitely useful for some annotation projects
• A universal grammar?
• Not in the Chomskyan sense, but hopefully in the more practical sense of facilitating multilingual NLP by bringing a little order into the chaos
So what exactly is UD?
• A new linguistic theory?
• Not at all, but we like to think it is informed by linguistic theory and potentially useful also for linguistic studies
• A better parsing framework?
• Probably not, since parsers seem to prefer function words as heads so we may have to tweak the representations for parsing
• The ultimate annotation scheme?
• Not quite, more like a lingua franca for treebank developers and definitely useful for some annotation projects
• A universal grammar?
• Not in the Chomskyan sense, but hopefully in the more practical sense of facilitating multilingual NLP by bringing a little order into the chaos
Acknowledgments
• Core UD group: Filip Ginter, Yoav Goldberg, Jan Hajič, Chris Manning, Ryan
McDonald, Natalia Silveira, Marie de Marneffe, Slav Petrov, Sampo Pyysalo, Reut Tsarfaty, Dan Zeman
• UD contributors: Željko Agić, Riyaz Ahmad, Maria Jesus Aranzabe, Masayuki
Asahara, Aitziber Atutxa, Cristina Bosco, Giuseppe G. A. Celano, Jinho Choi, Çağrı Çöltekin, Kaja Dobrovoljc, Timothy Dozat, Binyam Ephrem, Tomaž Erjavec, Richárd Farkas, Jennifer Foster, Koldo Gojenola, Iakes Goenaga, Bruno Guillaume, Nizar
Habash, Dag Haug, Anders Trærup Johannsen, Hiroshi Kanayama, Jenna Kanerva, Simon Krek, Juha Kuokkala, Veronika Laippala, Alessandro Lenci, Krister Lindén, Nikola Ljubešić, Olga Lyashevskaya, Teresa Lynn, Aibek Makazhanov, Catalina
Maranduc, Héctor Martínez Alonso, Anna Missilä, Verginica Mititelu, Yusuke Miyao, Simonetta Montemagni, Shinsuke Mori, Hanna Nurmi, Petya Osenova, Lilja Øvrelid, Elena Pascual, Marco Passarotti, Jussi Piitulainen, Barbara Plank, Prokopis
Prokopidis, Loganathan Ramasamy, Wolfgang Seeker, Mojgan Seraji, Maria Simi, Kiril Simov, Arne Skjæerholt, Aaron Smith, Jan Štěpánek,Takaaki Tanaka, Francis Tyers, Sumire Uematsu, Veronika Vincze, Rob Voigt, Jonathan Washington
