Resources and Evaluation
Takehito Utsuro 3
, Manabu Sassano 33
3
DepartmentofInformationandComputerSciences,ToyohashiUniversityofTechnology
Tenpaku-cho,Toyohashi,441-8580,Japan
33
FujitsuLaboratories,Ltd.
4-4-1,Kamikodanaka,Nakahara-ku,Kawasaki211-8588,Japan
Abstract
Approaches to named entity recognition that rely on hand-crafted rules and/or supervised learning techniques have
limitations interms of their portability into new domains as wellas in the robustness over time. For the purposeof
overcomingthoselimitations,thispaperevaluatesnamedentitychunkingandclassicationtechniquesinJapanesenamed
entityrecognition inthe context ofminimallysupervisedlearning. Thisexperimentalevaluationdemonstrates thatthe
minimally supervisedlearning methodproposed hereimprovedthe performanceof theseedknowledgeonnamedentity
chunkingandclassication.Wealsoinvestigatedthecorrelationbetweenperformanceoftheminimallysupervisedlearning
andthesizesofthetrainingresourcessuchastheseedsetaswellastheunlabeledtrainingdata.
1. Introduction
It is widely agreed that named entity recognition
is animportant stepforvariousapplications of
natu-ral language processingsuch as information retrieval,
machine translation, information extraction and
nat-ural language understanding. In the English
lan-guage, the task of named entity recognition was one
ofthetasksoftheMessageUnderstandingConference
(MUC)(e.g.,MUC-7(MUC,1998))andhasbeen
stud-ied intensively. Inthe Japanese language, several
re-cent conferences, such as MET (Multilingual Entity
Task, MET-1 (Maiorano, 1996) and MET-2 (MUC,
1998))and IREX (InformationRetrievaland
Extrac-tionExercise)Workshop(IREXCommittee,1999),
fo-cusedonnamedentityrecognitionasoneoftheir
con-testtasks,thuspromotingresearchonJapanesenamed
entityrecognition.
In Japanese named entity recognition, it is quite
commonto applymorphological analysis asa
prepro-cessing stage and to segmentthe sentence string into
asequenceofmorphemes. Then,hand-craftedpattern
matching rulesand/or statisticalnamed entity
recog-nizer are applied to recognize named entities.
How-ever, in named entityrecognition, itis oftenthe case
that newentitiesareintro ducedas thedomainof the
texts changes, and even in the same domain, as for
newspap er articlesfor example,where itis quite
nat-ural to assume that entities which have never been
encountered in past articles are newly intro duced in
futurearticles. Therefore,approachestonamedentity
recognitionthat relyonhand-craftedrulesand/or
su-p ervisedlearningtechniqueshavelimitationsinterms
oftheirportabilityinto newdomainsas wellasin the
robustness over time.
For the purpose of overcoming those limitations,
recognition proposed by (Collins and Singer, 1999;
CucerzanandYarowsky,1999)ismorepromising. The
centralideaof theminimallysupervised approach
re-lates to bootstrapping utilizing redundancy in
unla-beleddata, withthehelp ofa minimal number of
la-beleddataasinitialseeds. Theideaofutilizing
redun-dancyintheunlabeleddataforentitytaggingwas
pro-posedby(Yarowsky,1995)inthecontextofwordsense
disambiguation, in which cross-model redundancy in
contextual features of word sense disambiguation is
exploited. This idea was then theoretically
formal-ized in the context of computational learning theory
by(BlumandMitchell,1998)andhasbeentermedas
\co-training". (CollinsandSinger,1999;Cucerzanand
Yarowsky,1999)independentlyappliedco-training
al-gorithms to named entity classication/ recognition,
both of which utilize the redundancy of
morphologi-calandcontextualevidenceofnamedentity
classica-tion/recognition.
Following this prior research which employed the
minimallysupervisedapproachtonamedentity
classi-cation/recognition, in thispaper wefocus ona
min-imally supervisedapproachto Japanesenamed entity
recognition. In Japanese named entity recognition,
named entities to be recognized may have dierent
segmentationboundariesfromthoseofmorphemes
ob-tainedbythemorphological analysis. For example,in
our analysis of the IREX workshop's training corpus
ofnamedentities,44%ofthenamedentitieshave
seg-mentation boundaries that dierfromboundaries
ob-tained throughmorphological analysis by a Japanese
morphological analyzer breakfast (Sassano et al.,
1997) (section 2.). Thus, in Japanese named entity
recognition,themostdicultproblemsincludethis
more,in almost 90% ofcasesof segmentation
bound-ary mismatches, named entities to be recognized can
b e decomposed into several morphemes as their
con-stituents. Thismeansthattheproblemofrecognizing
named entities in those cases may be solved by
in-corp orating techniques of base noun phrase chunking
(b oundary detection) (Ramshaw and Marcus, 1995;
Mu ~noz etal.,1999).
Inthispaper,wefocusonboththeissuesofnamed
entity chunking and classication in Japanese named
entity recognition, and evaluate named entity
chunk-ingtechniques in thecontextof minimallysupervised
learning. First, as a supervised learning method, we
employ the supervised decision list learning method
of (Yarowsky, 1994), into which we incorporate
sev-eralnounphrasechunkingtechniques(sections3. and
4.). We chose the decision list learning method as
the supervised learning technique because it is easy
to implement and quite straightforward to extend
a supervised learning version to a minimally
super-visedversion(Yarowsky,1994;Yarowsky,1995;Collins
and Singer, 1999) (section 5.). Then, we applied a
minimally supervised learning algorithm to Japanese
named entity recognition, where a list of frequent
named entities are extracted from unlabeleddata by
a human and fed to the learning algorithm as seeds.
The minimally supervised learning method improves
the performance ofthe seed knowledgeon named
en-tity recognition by iteratively applying it to unseen
dataunlabeledwithnamedentitytagsandeectively
utilizingredundancyin theunlabeleddata.
The minimally supervised learning method
em-ployed in thispaper isa variantof theones basedon
the supervised decision listlearning (Yarowsky,1995;
Collins and Singer, 1999), which, as discussed above,
havebeenapplied toclassication taskssuch assense
disambiguation and named entity classication, but
not to the full named entity chunking and
classica-tion task. Theresultsof the experimental evaluation
showsthat ourminimallysupervised learningmethod
improves the performance of the seed knowledge on
namedentityrecognition. Wealsoinvestigatethe
cor-relationbetweenperformanceoftheminimally
super-vised learning and the sizes of the resources such as
the seed as well as theunlabeledtraining data.
(sec-tion6.).
2. Japanese Named Entity
Recognition
2.1. Task ofthe IREXWorkshop
Thetask ofnamed entityrecognitionof theIREX
workshop is to recognize eight named entity typ es in
Table 1 (IREX Committee, 1999). The organizer of
theIREXworkshopprovided1,174newspaperarticles
which include 18,677 named entities as the training
data. Intheformalrun(generaldomain)ofthe
work-shop,theparticipatingsystemswererequestedto
rec-ognize 1,510 named entities included in the held-out
Table1: StatisticsofNE Typ esofIREX
frequency(%)
NEType Training Test
ORGANIZATION 3676(19.7) 361(23.9)
PERSON 3840(20.6) 338(22.4)
LOCATION 5463(29.2) 413(27.4)
ARTIFACT 747(4.0) 48(3.2)
DATE 3567(19.1) 260(17.2)
TIME 502(2.7) 54(3.5)
MONEY 390(2.1) 15(1.0)
PERCENT 492(2.6) 21(1.4)
Total 18677 1510
Table 2: Statistics of Boundary Match/Mismatch of
Morphemes(M) andNamedEntities (NE)
Match/Mismatch freq. ofNETags(%)
1Mto1NE 10480(56.1)
n(2)Ms n=2 4557(24.4)
to n=3 1658(8.9) 7175
1NE n4 960(5.1) (38.4)
otherboundarymismatch 1022(5.5)
Total 18677
2.2. SegmentationBoundaries of Morphemes
and Named Entities
In the work presented here, we compare the
seg-mentation boundaries of named entities in the IREX
workshop's training corpus with those of morphemes
which were obtained through morphological
analy-sis by a Japanese morphological analyzer
break-fast(Sassanoetal.,1997). 1
Detailedstatisticsofthe
comparison are provided in Table 2. Nearly half of
thenamedentitieshaveboundarymismatchesagainst
themorphemesandalsoalmost90%ofthenamed
en-tities with boundary mismatches can be decomposed
into more thanone morpheme. Figure 1 showssome
examples ofthosecases. 2
3. Chunking and Tagging Named
Entities
Inthissection,weformalizetheproblemofnamed
entitychunkinginJapanesenamedentityrecognition.
3.1. Task Denition
First,wewill provide ourdenition of the task of
Japanese namedentitychunking. Supposethat a
se-1
Thesetofpart-of-speechtagsofbreakfastconsistsof
about300tags. breakfastachieves99.6%part-of-speech
accuracyagainstnewspaperarticles.
2
Inmostcasesofthe\otherboundarymismatch"in
Ta-ble2,oneormorenamedentitieshavetoberecognizedas
a part ofa correctlyanalyzedmorpheme andthose cases
arenotcausedbyerrorsofmorphologicalanalysis. One
fre-quentexampleofthistyp eisaJapaneseverbalnoun
\hou-bei(visitingUnitedStates)"whichconsists oftwo
charac-ters\hou(visiting)"and\bei(UnitedStates)",where\bei
(UnitedStates)"havetoberecognizedas<LOCATION>. We
believethatboundarymismatchofthistyp ecanbeeasily
NamedEntityTag <ORG> <LOC> <LOC>
MorphemeSequence 111 M M M M M M M M 111
Inside/OutsideEncoding O ORGI O LOCI LOCI LOCI LOC B O
Open/CloseEncoding O ORGU O LOCS LOCC LOCE LOC U O
2Morphemesto 1NamedEntity
<ORGANIZATION>
111 Roshia gun 111
(Russian) (army)
<PERSON>
111 Murayama Tomiichi shushou 111
(lastname) (rstname) ( prime
minister )
3Morphemesto 1NamedEntity
<TIME>
111 gozen ku ji 111
(AM) (nine) (o'clock)
<ARTIFACT>
111 hokubei jiyuu-boueki kyoutei 111
( North
America
) (freetrade) (treaty)
Figure 1: Examples of Boundary Mismatch of
Mor-phemesand NamedEntities
quence ofmorphemesaregivenasbelow:
( Left
Context
) (NamedEntity) ( Right
Context )
111M L
0k 111M
L
01 M
NE
1 111M
NE
i 111M
NE
m M
R
1 111M
R
l 111
"
(CurrentPosition)
Then, giventhat the current position is at the
mor-pheme M NE
i
,thetaskofnamedentitychunkingisto
assignachunkingstate(tobedescribedinsection3.2.)
aswellasanamedentitytyp etothemorphemeM NE
i
at the current position, considering the patterns of
surrounding morphemes. Note that in thesupervised
learning phase we can use the chunking information
on which morphemes constitute a named entity, and
whichmorphemesarein theleft/rightcontextsofthe
namedentity.
3.2. Encoding Schemes ofNamedEntity
ChunkingStates
Inthispaper,weusethefollowingtwomethodsfor
enco dingchunkingstatesofnamedentities.Examples
ofthoseencodingschemesareshownin Table3.
3.2.1. Inside/Outside Encoding
The Inside/Outside scheme of encoding chunking
statesofbasenounphraseswasstudiedby(Ramshaw
andMarcus, 1995;Mu~noz etal.,1999). Theirscheme
the currentposition is outsideanybase noun phrase.
I { the word at the current position is inside some
base noun phrase. B { the wordat the current
posi-tion marksthe beginningof a base noun phrase that
immediatelyfollowsanotherbasenounphrase. We
ex-tendthisschemeto namedentitychunkingbyfurther
distinguishing each of the states I and B into eight
named entitytyp es. 3
Thus, this schemedistinguishes
228+1=17states.
3.2.2. Open/Close Encoding
The Open/Close scheme of encoding chunking
statesofnamedentitieswasemployedin(Sekineetal.,
1998;Borthwicketal.,1998). Asimilarschemeisalso
studied in (Mu~nozet al.,1999)in thecontextofbase
nounphraseschunking.Thisschemedistinguishesthe
followingfour states foreach named entitytyp e: S {
the morpheme at the current position marksthe
be-ginningofanamedentityconsistingofmorethanone
morpheme. C{themorphemeat thecurrentposition
marksthemiddleofanamedentityconsistingofmore
thanonemorpheme. E{themorphemeatthecurrent
positionmarkstheendingofanamedentityconsisting
ofmorethanonemorpheme. U{themorphemeatthe
current position is a named entity consisting of only
one morpheme. The scheme also considers one
addi-tionalstateforthepositionoutsideanynamedentity:
O { the morpheme at the current position is outside
any named entity. Thus, in this setting, ourscheme
distinguishes 428+1=33states.
3.3. Preceding/Subsequent Morphemesas
Contextual Clues
Inthispaper,weemploythemodelusedin(Sekine
et al.,1998; Borthwicket al.,1998)as that of
consid-ering preceding/subsequent morphemesas contextual
cluestonamedentitychunking/tagging. 4
Herewe
pro-videabasicoutlineofthemodel,andthedetailsofhow
to incorporate itinto thedecisionlistlearning
frame-workwillbedescribedinsection4.2.2..
Suppose that the current position is at the
mor-phemeM
0
,asillustratedbelow. Then,whenassigning
a chunking state as well as a named entity typ e to
themorphemeM
0
,themodelconsidersthepreceding
single morpheme M
01
as well as the subsequent one
3
WeallowthestatexBforanamedentitytyp exonly
whenthemorphemeatthecurrentpositionmarksthe
be-ginningofanamedentityofthe typ exthatimmediately
followsanamedentityofthesametyp ex.
4
In(SassanoandUtsuro,2000),weproposedand
eval-uated a novel model that incorporates richer contextual
111 Murayama Tomiichi shushou ha Kooru shushou to 111
(lastname) (rstname) ( prime
minister
) (Topic) (lastname) ( prime
minister
) (with)
Figure2: TheBasicIdeaofUtilizingRedundancyin theMinimallySupervised Learning
1998; Borthwick et al., 1998), the most appropriate
sequenceof thedecisionsthat areconsistent
through-out the whole sequence is searched for. By
consis-tency of thedecisions,wemeanrequirementssuchas
that the decision representing the beginning of some
named entity typ e has to be followed by that
repre-sentingthemiddleofthesameentitytyp e(inthecase
of the Open/Close encoding). Also, in our case, the
appropriatenessofthesequenceofthedecisionsis
mea-sured by thesumof thelogoflikeliho od ratiosofthe
corresp ondingdecisionrules.
5. Minimally Supervised Learning
5.1. The Basic Idea ofUtilizing Redundancy
Thefollowingexampledescribeshowto utilize
re-dundancy in theunlabeled data in theminimally
su-p ervisedlearning.
Supp ose that the sequence of morphemes in
Fig-ure 2 is includedin an unlabeled dataset, where the
names of theprime ministers of Germanyand Japan
need toberecognizedas <PERSON>s. Both ofthetwo
names are followedby a noun representing thesocial
p osition,i.e.,inthiscase,\shushou"(primeminister).
In theJapanese language, the name of a person at a
certainsocialpositionisfollowedbyanoun
represent-ingtheposition,as inthecaseof\PresidentClinton"
inEnglish. Alsosupposethat,asaseed,thelearneris
giventhenameofJapaneseprimeminister\Murayama
Tomiichi"as a <PERSON>named entity. Then, the
al-gorithmcanlearnthefollowingdecisionrulefromthis
seed andunlabeleddata inFigure2.
IfX isfollowedby\shushou"(prime minister),
thenX is<PERSON>.
By applying this decision rule to \Kooru", the
al-gorithm can easily annotate \Kooru" as a <PERSON>
named entity. In this case, the redundancy of the
seed knowledge and a reliable contextual clue at the
namedentityposition\MurayamaTomiichi"isthekey
tobootstrappinginminimally supervised learning.
5.2. Learning Algorithm
Infollowing section we provide our minimally
su-p ervisedlearningalgorithm,whichusesalistofnamed
entitiesasseedsandalargeamountofunlabeleddata,
but notuseanylabeleddata.
1. Initialization
Seed Selection
First,alistoffrequentnamedentitiesareextracted
from theresultof themorphological analysis of
unla-b eleddata. Thisisdonebyahumanwhoconsultsthe
denition ofthenamedentitytagsifnecessary.
Then, the list of frequent named entities is
con-verted into a decision list which is to be used as an
initial decisionlist in theminimallysupervised
learn-ing.
2. Minimally Supervised Learning
Thefollowing applicationand trainingiterationis
iterated and the changes in precision, recall, and
f-measureagainstheld-outdatalabeledwithnamed
en-titytagsareobserved.
Applying Decision Listto Unlabeled Data
Usually,inthecaseofsupervisedlearning,thebest
performance against test data is obtained when only
those rules witha logof likeliho od ratioabovea
cer-tainthresholdareconsidered. However,inour
experi-mental results,thebestperformancein theminimally
supervisedlearningwasobtainedwhenalltherulesin
the decision list were considered at each step of the
trainingiteration. Theresultsofminimallysupervised
learning reported in this paper, then, are those
ob-tainedwithoutathresholdinthedecisionlists.
Learning Decision List
A new decision list is learned from the data with
named entity tags annotated by the decision list
learned in the previous step. The data may contain
errorsofnamedentityrecognition.
6. Experimental Evaluation
Wenext experimentally evaluate the performance
of the supervised and minimally supervised learning
for Japanese named entity recognition on the IREX
workshop'strainingandtest data. Inthis evaluation,
weexclude the named entities with \otherboundary
mismatch"inTable2.
6.1. Comparison of NamedEntity Chunking
Methods
First,wecomparethe performanceof the two
en-coding schemes of named entity chunking states(the
Inside/OutsideandtheOpen/Closeencodingschemes)
in fullsupervisedandminimallysupervised learning.
6.1.1. SupervisedLearning
Foreachof thoseencoding schemes,a decisionlist
is learned by full supervised learning from the IREX
workshop's training data and evaluated against the
IREX workshop's test data. We searched for an
op-timalthresholdofthelogoflikelihoodratiointhe
de-cisionlist. The performance ofeachencodingscheme
measured by f-measure ( = 1)/precision/recall is
givenin Table4. Weclassied thesystemoutput
ac-cording to the number of constituent morphemes of
nMorphemesto 1NamedEntity
n1 n=1 n=2 n=3 n4
F-measure(=1) 72.9 75.4 79.7 51.4 29.2
Inside/Outside (Precision) (72.7) (68.0) (78.2) (74.7) (87.5)
(Recall) (73.1) (84.7) (81.1) (39.2) (17.5)
F-measure(=1) 72.7 76.1 79.5 43.7 29.6
Open/Close (Precision) (76.7) (71.8) (82.2) (79.6) (95.5)
(Recall) (69.0) (81.0) (77.0) (30.1) (17.5)
the whole set, we show the higher performance in
f-measurewith bold-facedfont.
The Inside/Outside encoding scheme achieves
slightlybetterperformanceintotalf-measureand
sig-nicantly better in f-measure of the \n = 3"
sub-set, while the Open/Close encoding scheme achieves
slightly better performance in f-measure of \n= 1"
and \n 4" subsets. The Inside/Outside encoding
schemeachievesbetterperformanceinrecall,whilethe
Op en/Close encoding scheme achieves better
perfor-mancein precision. Thisdierencein precision/recall
is consistent with the degree of generalization of the
two encodingschemes.
6.1.2. Minimally Supervised Learning
Next,wesimulate theminimallysupervised
learn-ing algorithm of section 5.2. by taking a small
por-tionof highly rankeddecision ruleslearned by
super-vised learning as seeds. First, from the rst half of
the IREX workshop's training data, a decisionlist is
learned by supervised learning. Then, top 100 and
500 rulesare extractedfrom thedecisionlistas seeds
of minimally supervised learning, and the minimally
sup ervised learning algorithm is run with the other
halfoftheIREXworkshop'strainingdata(containing
ab out 9,300 named entities) as the unlabeled
train-ingdatabyignoringtheannotatednamedentitytags.
The whole decision list learned by supervised
learn-ing has approximately 60,000 rules for both the
In-side/Outside and the Open/Close encoding schemes
and the top 500 rules are those without ambiguities
of decisions (i.e., the unsmoothed conditional
proba-bilityP(D=xj E=1) equals to 1). Figure 3 shows
theresults ofcomparing theperformancechanges(in
f-measure(=1))ofthetwoencodingschemesinthe
minimallysupervised learning.
This results clearly shows that the initial
perfor-mance of theInside/Outside encoding schemeis
bet-ter than that of the Open/Close encoding scheme
with the same numb er of decision rules. Also with
resp ect to the maximum f-measure throughout the
trainingiteration,theInside/Outsideencodingscheme
achievesbetterperformancethantheOpen/Close
en-co dingscheme. Oneofthe obviousadvantagesof the
Inside/Outsideencodingschemeover theOpen/Close
enco ding schemeis thattheformercangeneralize
de-cision rules among named entities consisting of
dif-ferent number of morphemes, while the latter can
0
10
20
30
40
50
60
0
2
4
6
8
10
F-measure
Learning Iteration
Inside/Outside, Seed Size = 500
Open/Close, Seed Size = 500
Inside/Outside, Seed Size = 100
Open/Close, Seed Size = 100
Figure 3: Comparison of Inside/Outside and
Open/Close Encoding Schemes in Minimally
Super-visedLearning
ing of one morpheme and those consisting of more
thanone morpheme. Thegeneralizationabilityofthe
Inside/Outside encoding scheme is particularly
eec-tivein minimally supervised learning where thedata
sparsenessproblemeasilyoccurs. Fromthisresult,we
conclude that the Inside/Outside encoding scheme is
more suitable for minimally supervised learning and
the rest of our evaluation is carried out with the
In-side/Outsideencodingscheme.
6.2. Minimally SupervisedLearning with
Inside/Outside Encoding Scheme
Japanesenamed entitychunking andtaggingwith
the Inside/Outsideencoding scheme was evaluatedin
the minimally supervised learning of section 5.2.. In
this evaluation, a list of frequentnamed entities
con-sistingofonemorphemeisextractedasseedsbya
hu-manfrom thersthalfoftheIREXworkshop's
train-ing data without referring to the named entity tags
annotated to the original training data. Then, the
minimally supervised learning algorithm is run with
the other half of the IREX workshop's trainingdata
as the unlabeled training data. The performance of
20
25
30
35
40
45
50
55
60
65
70
0
2
4
6
8
10
F-measure/Precision/Recall
Learning Iteration
F-measure
Precision
Recall
(b)F-measureClassiedbytheNumb erof
ConstituentMorphemes
0
10
20
30
40
50
60
70
80
90
0
2
4
6
8
10
F-measure
Learning Iteration
Total
1 NE to 1 Morpheme
1 NE to 2 Morphemes
1 NE to 3 Morphemes
1 NE to n(>3) Morphemes
Figure 4: PerformanceChangesacross Minimally
Su-p ervisedTrainingIteration
6.2.1. Changes of
Precision/Recall/F-measurein
TrainingIteration
First,forthecasewherethemostfrequent200seed
named entities are used, Figure 4 showsthe changes
of precision/recall/f-measure in the iteration of
mini-mallysupervisedlearning(Figure4(a))aswellasthe
changesoff-measureclassiedaccordingtothenumber
of constituentmorphemes of each named entity
(Fig-ure 4 (b)). These performance are measured against
theunlabeledtrainingdata. ThecurveofFigure4(a)
represents a typicalperformancechanges in the
min-imally supervised learning, in that the recall and the
f-measure dramatically increases, while the precision
decreases. One of the remarkable results of Figure 4
(b) which has to be pointed out is that the
perfor-manceforthe\1namedentityto2morphemes"and\1
named entityto3 morphemes" increases eventhough
each named entity in the seed list consists of exactly
one morpheme. Thisresultclearlysupportstheclaim
that theInside/Outsideencodingschemecan
general-ize decision rulesamong named entities consisting of
0
10
20
30
40
50
60
50
100
200
500
1000
Max F-measure / Precision / Recall
Seed Size
Max F-measure
Precision
Recall
(b)SeedsSelectedinDecreasing FrequencyOrder
0
10
20
30
40
50
60
50
100
200
500
1000
Max F-measure / Precision / Recall
Seed Size
Max F-measure
Precision
Recall
Figure5: PerformanceatDierentSeedSizes
6.2.2. Evaluationat DierentResource Sizes
Next,weevaluateminimallysupervisedlearningat
dierentsizesof seednamed entitiesas wellas of
un-labeledtrainingdata.
First,wechangethenumberofseednamedentities
as 50,100,200,500,and1000,andrun theminimally
supervised learning algorithm andplotthe maximum
f-measure value throughout the training iteration for
each seed size. Figure 5 (a) shows the result when
the seed namedentities areselectedso that their
fre-quencycountsarebalancedamongdierentseedsizes,
whileFigure5(b)showsthatwhenthemostfrequent
named entities areselected as seeds. InFigure 5 (a),
themaximumf-measurevalueincreasesroughly
loglin-earlywith thenumb ertheseednamed entities,which
isconsistentwiththeresultsreportedin(Cucerzanand
Yarowsky, 1999). InFigure5 (b), onthe otherhand,
the maximumf-measurevaluestops to increasewhen
the numb er of theseed named entities is 500, simply
becausethelargerseedlistsareconstructedbyadding
lessfrequentnamedentities tothesmallerseed lists.
Second,wechangethesizeoftheunlabeledtraining
data andrun the minimallysupervised learning
algo-rithmwith200 mostfrequentnamedentities asseeds
andplotthemaximumf-measurevaluethroughoutthe
46
48
50
52
54
56
58
100k
150k
750k
1500k
Max F-measure / Precision / Recall
Unlabeled Training Data Size (# of Morphemes)
Max F-measure
Precision
Recall
(b)UnlabeledTrainingData
55
60
65
70
75
100k
150k
750k
1500k
Max F-measure / Precision / Recall
Unlabeled Training Data Size (# of Morphemes)
Max F-measure
Precision
Recall
Figure 6: Performance at Dierent Unlabeled Data
Sizes
measurevalueismeasuredagainsttheheld-outIREX
workshop's test data, while Figure 6 (b) shows that
when the the maximum f-measure value is measured
against thetraining unlabeled data. In Figure 6 (a),
the maximum f-measure value slightly increases due
to increases in precision,as the size of the unlabeled
trainingdataincreases,whichisconsistentwiththe
re-sultsreportedin(CucerzanandYarowsky,1999). This
pap erclaimsthatifmoreunlabeleddataareavailable,
more accurate rulesfor named entityrecognition will
b e learned byminimally supervised learning. In
Fig-ure6(b),ontheotherhand,themaximumf-measure
value against the unlabeled training data decreases
with the size of the unlabeled training data. This is
b ecause as the unlabeled training data set becomes
larger, the unlabeled training data tends to include
more infrequentnamedentitiesthat aremore dicult
torecognizethanfrequentnamedentities.
7. Conclusion
This paper evaluated named entity chunking and
classication techniques in Japanese named entity
recognition in the context of minimally supervised
learning. Theexperimentalevaluationshowedthatour
minimally supervised learning method improves the
recognition. We also investigated the correlation
be-tweenperformanceof theminimallysupervised
learn-ing and the sizes of the resources such as the seed
as well as the unlabeled training data. We are now
working on incorporating the model of (Sassano and
Utsuro,2000)intominimallysupervisedlearning
envi-ronments,whererichercontextualinformationaswell
aspatternsofconstituentmorphemeswithinanamed
entityareconsidered. Theresultsofthisexperimental
evaluationwillbereportedinthenearfuture.
8. Acknowledgments
This research was carried out while the authors
werevisitingscholarsatDepartmentofComputer
Sci-ence, Johns Hopkins University. The authors would
liketothankProf. DavidYarowsky ofJohnsHopkins
Universityforvaluable supportstothisresearch.
9. References
Blum, A. and T.Mitchell, 1998. Combining labeledand
unlabeleddatawithco-training. InProc.11thCOLT.
Borthwick,A.,J.Sterling,E.Agichtein,andR.Grishman,
1998. Exploiting diverse knowledge sources via
maxi-mumentropyinnamedentityrecognition. InProc.6th
WVLC.
Collins, M.and Y. Singer,1999. Unsupervised modelsof
namedentityclassication.InProc.1999JointSIGDAT
Conference onEMNLPandVLC.
Cucerzan, S.and D. Yarowsky, 1999. Language
indepen-dentnamedentityrecognitioncombiningmorphological
and contextualevidence. InProc.1999 Joint SIGDAT
Conference onEMNLPandVLC.
IREXCommittee(ed.),1999. Proc.the IREXWorkshop.
(inJapanese).
Maiorano, S., 1996. The multilingual entitytask (MET):
Japanese results. In Proc. TIPSTER PROGRAM
PHASEII.
MUC,1998. Proc.7thMessageUnderstandingConference
(MUC-7).MUC.
Mu~noz,M.,V.Punyakanok,D.Roth,andD.Zimak,1999.
A learning approachtoshallowparsing. InProc.1999
JointSIGDATConferenceonEMNLPandVLC.
Ramshaw,L. and M.Marcus,1995. Textchunkingusing
transformation-basedlearning. InProc.3rdWVLC.
Rivest, R. L., 1987. Learning decision lists. Machine
Learning,2:229{246.
Sassano,M.,Y.Saito,andK.Matsui,1997. Japanese
mor-phological analyzerfor NLP applications. InProc.3rd
AnnualMeetingoftheAssociationforNaturalLanguage
Processing. (inJapanese).
Sassano,M.and T.Utsuro,2000. Namedentitychunking
techniques in supervised learning for Japanese named
entityrecognition. InProc.18thCOLING.(toappear).
Sekine, S., R. Grishman, and H. Shinnou, 1998. A
deci-sion tree method for nding and classifying names in
Japanesetexts. InProc.6thWVLC.
Yarowsky, D., 1994. Decision lists for lexical ambiguity
resolution: ApplicationtoaccentrestorationinSpanish
andFrench. InProc.32ndACL.
Yarowsky,D.,1995. Unsupervisedwordsense
