by Similarity Template
and Retrieval Query Generation
Takehito Utsuro
Graduate Scho ol of Information Science,
Nara Institute of Science and Technology
8916-5, Takayama-cho, Ikoma-shi, Nara, 630-01, JAPAN
phone : +81-7437-2-5242 fax : +81-7437-2-5249
email: [email protected]
Abstract
The nearest neighbor algorithm is the most basic class of techniques in the sub- elds
of machine learning such as case-based reasoning (CBR), memory-based reasoning (MBR),
and instance-based learning (IBL). In the nearest neighb or algorithm, the computational
cost of example retrieval is one of themost important issues. This paper proposes a novel
technique foroptimizing the nearest neighb or algorithm. Its basicidea is based on the use
of similarity template,whichis adatastructure thatenumeratesallthepossiblepatternsof
calculating similarityb etween two examples. In themethod, the nearest neighb or retrieval
processisoptimizedbygeneratingretrievalqueriesfromaninputandsimilaritytemplatesin
a certain order. Itsmajor advantages areasfollows: 1)unlike thehierarchicalorganization
of memory, it is guaranteed that examples with the greatest similarity value according to
the givensimilaritymeasureare retrieved,2)itis easytoadd new examplestotheexample
database, 3)without any expensive hardware such as massively parallel computers, nearly
constant time nearest neighb or retrieval can be achieved, independently of the numb er of
examples inthedatabase.
Inthe eld ofmachinelearning(ML),mosttraditionallearning methodsformsomeabstrac-
tion from the experience and they store this structure in their memory. However,in recent
years there has been growing interest in metho ds that store instances (examples or cases)
in the memory and apply this sp eci c knowledge directly to new situations. In the eld of
ML,case-basedreasoning (CBR) [
Kolodner,1993 ]
,memory-basedreasoning (MBR) [
Stan ll
and Waltz,1986 ]
,and instance-basedlearning (IBL) [
Ahaet al., 1991 ]
are inthis approach.
In CBR, MBR, and IBL, the most basic class of techniques is called the nearest neighbor
algorithm. It originated in the eld of pattern recognition [
Cover and Hart, 1967 ]
and has
been applied to many classi cation tasks. In the nearestneighbor algorithm, each training
example issimply storedin the memory. Givena newtest example,the algorithm nds the
stored training example that is nearest according to some similarity measure, records the
class of the retrieved example,and predicts that the new example will havethe same class.
In the nearest neighbor algorithm, the computational cost of example retrieval is one
of the most important issues, especially when the numb er of examples in the database b e-
comes larger. The way of coping with this issue di ers from one approach to another,
and there existat leasttwo major approaches, namely,hierarchical organization of memory
and parallel search of at memory. Most practical CBR systems on serial computers (e.g.,
MEDIATOR [
Kolodner and Simpson, 1989 ]
and REMIND CBR Shell [
Cognitive Systems,
1992 ]
) adopted hierarchical organization of memory, while MBR [
Stan ll and Waltz, 1986 ]
andthemassivelyparallelexample-basednaturallanguageprocessing(NLP)approach(e.g.,
[
Sumitaetal.,1993 ]
)adoptedparallelsearchof atmemoryonmassivelyparallelcomputers.
Inapproachesthatadoptedhierarchicalorganizationofmemory,suchasadiscrimination
network and a decision tree, the case library is hierarchically partitioned using several fea-
tures asindices, and according tothe hierarchy,the searchin the case library isguided and
the search space islimited. When retrieving similar cases fromthe case library, cases satis-
fying the constraints of several indices are eciently retrieved. Then, the nearest neighb or
algorithmisusedforrankingretrievedcasesaccordingtosomesimilaritymeasure. Although
this approachis adopted inmany practical CBRsystems onserial computers, ithas several
disadvantages. Oneof the mostsigni cantdisadvantagesisthatthe memorysearchand the
ranking of casesare separate. Similar cases are searchedwithin the hierarchically organized
memory by checking several features as indices. During the search, however, the ranking
of cases according to some similarity measure is not considered at all, but it is considered
afterwards. Since the cases are not searched by checking all the features but only a few,
there is no guarantee that a good case will not be missed. Another disadvantage is that
the adding of cases is a complex operation and it is hard to keep the hierarchical memory
optimalas cases are added.
MBRandmassivelyparallelexample-basedNLP,thesimilarityfunctionisappliedinparallel
toalltheexamplesinthe exampledatabase, andallthoseexamplesare rankedaccordingto
the similarity values and the b estmatches are returned. In this approach,the whole library
is searchedand thusaccuracy of retrieval isdependent onlyon the similarity measure. It is
easytoaddnew examplestothedatabase,sinceexamplesarestoredinthedatabasewithout
anyhierarchy. However,expensivehardwareisneededandthisisoneofmajordisadvantages
in this approach.
This paperproposes anovel techniquefor overcoming the problem of the computational
cost of example retrievalin the nearest neighb oralgorithm. The basic idea of the proposed
method isbased onthe use ofsimilarity template. A similarity template isa data structure
which enumerates all the possible patterns of calculating similarity between two examples
anditisindependentofeachexample. Thesystemcontainsasetofallthepossiblesimilarity
templates and itgenerates a retrievalquery fromaninput and a similaritytemplate. Using
the generated retrievalqueries, the example space can be structuredaround the input asin
Figure 1. Inthe structured examplespace,the nearestneighb orexamples arelocated atthe
positions closest tothe input. In this framework,the nearestneighb or retrievalprocess can
be optimized by generating retrieval queries in a certain order. This optimization problem
is a problem of nding a retrieval query which retrieves the nearest neighb or of the input
as quickly as possible. In this paper, we describ e two distinct strategies for optimizing the
retrieval query generation process: linear search along similarity value ordering and binary
search along subsumption ordering of retrieval queries and similarity value ordering. Com-
paredwiththehierarchicalmemoryorganizationtechniques,inwhichthememorysearchand
the ranking of cases (or examples) are separate, the proposed method integrates these two
in an optimalway, by directly searching the nearest neighb or(s) according to the structure
of the example space.
The prop osedmethodcombines the advantagesof both of the previous two approaches,
i.e., hierarchical organization of memory and parallel search of at memory. The major
advantagesareasfollows: 1)itisguaranteedthatexampleswiththegreatestsimilarityvalue
according to the given similarity measure are retrieved, 2) it is easy to add new examples
to the example database, 3) without any expensive hardware such as massively parallel
computers, nearly constant time nearest neighb or retrieval can be achieved, independently
of the numb erof examples inthe database.
In the previous paper [
Utsuro et al., 1994 ]
, we presented the method for optimizing the
nearest neighb or retrieval only speci cally in the case of retrieving Japanese surface case
structures. In this paper,we providemore generalmotivations toour methodby extending
itinthe contextofML,CBR,andMBR. Theremainderofthepaperisorganized asfollows.
The Input
: Example
Example Space
Similarity
The Nearest Neighbor Example
Figure 1: Structured Example Space
First, in section 2, we describe the basic idea of the proposed method in its most general
form,without giving any speci c de nitions tothe data structure of an exampleand to the
similarity measure. Then, in section 3, we apply the optimization method to the standard
data structure and the similarity measure which are most commonly used in the nearest
neighb orretrievalinCBRandexample-basedNLP,namely,thesetoffeature-valuepairsand
theweightedsumofsimilaritiesforindividualfeatures. Wedescribeatechniqueforapplying
our method tothe generalcase of the data structure and the similarity measure, and then,
as an example, we describe the retrieval of Japanese surface case structures with speci c
de nitions of the data structure and the similarity measure. We also evaluate our metho d
by showingthat nearlyconstant time nearest neighb orretrievalis achieved, indep endent of
the numb erof examples inthe database.
2 The Basic Idea
This sectionpresentsthe basic idea ofoptimizing nearestneighb orretrievalusing similarity
templatesandretrievalquerygenerationinitsmostgeneralform,withoutgivinganyspeci c
de nitions tothe data structure of anexample and to the similaritymeasure.
2.1 Similarity Measure and Nearest Neighb or Retrieval
First,supposingthatE bethesetofalltheexamples,weintro ducethesimilaritycalculation
function sim from E 2E to R (the set of real numb ers) which gives the similarity value
between twoexamples:
sim: E 2E 7!R
Next, let Eg be the set of examples in the example database. Retrieval of the nearest
neighb or(s) of the input e
in
can then be regarded as the retrieval of the set Eg
max (e
in ) of
Eg
max (e
in ) =
n
ej e2Eg;sim(e;e
in
)=max
e
sim(e;e
in )
o
2.2 Similarity Template and Similarity Calculation
The key idea of optimizing the nearest neighb or retrieval is the use of similarity template.
First, we assume that a similarity template is de nable only when the similarity measure
satis es the followingtworequirements:
Requirements on the Similarity Measure
1. Similarity can be regarded as a function of factors inthe examples. Each example
containsonly asmall numb erof factors.
2. Eachfactorvariesoverafewdiscretevalues. Ifafactorvariesovercontinuousvalues,
these values can be transformed intoa few discretevalues.
A similarity template is now de ned as a data structure which enumerates all the pos-
sible patterns of calculating similaritybetween twoexamples, and it is independentof each
example. The left half of Figure 2 shows the idea of similarity calculation using similar-
ity templates. Given any element e
1
from the example set 1 and any element e
2
from the
example set 2, the result of their similarity calculation b ecomes the similarity template 1.
Each similarity template has a similarity value whichcorresponds to the similarity between
the twoexamples, and the similaritytemplates 1 3 inthe gure havethe same similarity
value.
More formally, supposing that T is the set of all similarity templates, the similarity
calculation function simcan be de nedas a comp osite function sim
t
tmpl , where tmpl is
a function fromE2E to T which gives a similarity template of twoexamples, and sim
t is
a functionfrom T toR which givesthe similarity value of a similaritytemplate:
tmpl: E2E 7!T
sim
t
: T 7!R
sim=sim
t
tmpl : E2E 7!R
2.3 Retrieval Query Generation
A retrieval query can be considered as a set of constraints on the examples to b e retrieved
by the query. A retrievalqueryis generated fromaninput and asimilaritytemplate bythe
#1 #2 #3 Similarity Template
Similarity
Example Set
#1 #2 #3
!_
Example Similarity Template Constraints on the Example Set
= Retrieval Query
SimilarityCalculation RetrievalQuery Generation
Figure 2: The BasicIdea of Similarity Template
retrieval query generation function qgen. qgen is a function from E 2T to Q, where Q is
the set of all retrieval queries. The right half of Figure 2 showsthe idea of retrieval query
generation from an input and a similarity template. Given an input e
in
and a similarity
template t, qgenreturns aretrievalquery q:
1
qgen: E2T 7!Q
q = qgen (e
in
;t)
2.4 Overall Framework
Figure 3 showsthe overall frameworkof the optimized nearestneighb orretrievalusing sim-
ilarity templates and retrieval queries. The system contains only the setof similarity tem-
plates, the indexed retrieval module, and the example database. It do es not contain any
retrieval queries in advance. A similaritytemplate isa data structure which isindependent
ofeachexample,whilearetrievalqueryisgeneratedfromtheinputandasimilaritytemplate,
and itisthusdependentontheinput. Foragiveninput,retrievalqueriesare generatedina
certain orderusing similaritytemplates. Then, exampleswhich satisfya retrievalqueryare
quickly retrievedfrom the example database using the indexed retrieval module. The opti-
mization problem of the nearestneighbor retrieval isconsidered as a problem of optimizing
the retrieval query generation process. This is a problem of nding a retrieval querywhich
retrievesthe nearest neighbor of the input as quickly asp ossible, and to dothis, itmustb e
guaranteed that the example database doesnot containanyexamples whichhave agreater
similarity value.
1
IfEg(q)isthesetofexamples which satisfytheconstraintsoftheretrievalqueryq,itisnotnecessary
thecasethatthesimilaritytemplateofeande
in
,i.e. tmpl (e
in
;e),becomestforalltheexampleseinEg(q).
This is so b ecause a retrieval queryq represents constraintson the examples and some example e which
satis esq couldalsosatisfy much strongerconstraints. Inthose cases,thesimilaritytemplate ofeand e
in
would b ecome much stronger than the similarity template t. Infact, this is truefor the de nition of the
retrievalqueryin section3.1.
YES NO
....
....
Input
Retrieval Query
Example Database Retrieval
Set of Similarity Templates ( Linear Sequence or Binary Search Tree )
Indexed Retrieval Module
Found?
Figure 3: The Framework of the Optimized Nearest Neighb orRetrieval
2.5 Optimizing the Nearest Neighb or Retrieval
This section describ es two distinct strategies for optimizing the retrieval query generation
process: linear search along similarity value ordering and binary search along subsumption
ordering of retrieval queries and similarity value ordering.
2.5.1 Linear Search along Similarity Value Ordering
This is a simple linear search strategy which starts from a retrieval query for the greatest
similarity value and continues to those for lower similarity value, gradually decreasing the
similarity value, until it nds at least one example. In Figure 1, linear search corresponds
to the searchfromthe inputoutward untilat least one nearest neighb or isfound.
For the linear search, the set of all similarity templates is rst divided into a sequence
T
1
;...;T
n
which istotally ordered by the similarityvalue:
8t2T
i
;8t 0
2 T
j
; i<j ) sim
t
(t)>sim
t (t
0
)
i=j )sim
t
(t)=sim
t (t
0
)
From the sequence T
1
;...;T
n
of the sets of similarity templates, a sequence Q
1
;...;Q
n of
the setsof retrievalqueriesisconstructedbycollectingretrievalqueriesgenerated fromeach
similarity template as below:
Q
i
= n
qj9t2T
i
;q=qgen (e
in
;t) o
(i=1;...;n) (1)
1 1 n
retrievesexamples using retrievalqueries ineachsetuntil atleast one example is retrieved.
If atleast one example is retrievedusing one of retrievalqueries inthe set Q
i
, then the set
Eg(Q
i
)of examples retrievedbyallthe retrievalqueries inQ
i
isthe setof nearest neighbor
examples.
2
2.5.2 BinarySearchalongSubsumptionOrderingofRetrieval Queriesand Sim-
ilarity Value Ordering
It can happen that the nearest neighb or of an input has a small similarity value and it is
inecienttosearchthe nearestneighbor linearly alongsimilarityvalueordering. Compared
with the linear search strategy, the binary search strategy makesecient retrieval possible
independently of whether the nearest neighbor has a high similarity valueor not.
The basic idea of the binary search strategy is to partition the example space using
retrievalqueriesaccording tothe subsumptionrelationofretrievalqueries andthe similarity
value. Subsumption relation of retrieval queries corresponds to the subsumption relation
of the example sets retrieved by the queries. This means that if constraints in a retrieval
query q
1
subsume those in another retrieval query q
2
, then the set Eg(q
1
) of the examples
retrieved by q
1
subsumes the set Eg(q
2
) of the examples retrieved by q
2
. The example
spaceispartitionedand structuredusing subsumptionrelationof theexamplesets, and also
using similarity value ordering. With the structured example space, it b ecomes p ossible to
binary-searchthe nearestneighb or(s) of the input.
Totally Ordered Example Sets
In the same way as for the linear search, also for the binary search, the set of all similarity
templates is rst divided into a sequence T
1
;...;T
n
, and from this sequence, a sequence
Q
1
;...;Q
n
of the sets of retrieval queries is constructed by Formula 1 in section 2.5.1.
However,nowthesequenceEg(Q
1
);...;Eg(Q
n
)oftheexamplesetsretrievedbytheretrieval
queries in each set Q
i
(i=1;...;n) has to satisfy the following two requirements of total
ordering:
1. Subsumption Relation of Example Sets
The sequence Eg(Q
1
);...;Eg(Q
n
) is totally ordered by the subsumption relation as in
Figure 4:
Eg(Q
1
)111Eg(Q
n )
2
ItisnotnecessarytoconstructallthesetsQ
1
;...;Q
n
inadvancebeforestartingexampleretrievalusing
retrievalqueries ineachset. Itissucientto constructa set Q
i
ofretrievalqueries whennoexamples are
retrievedusingtheprecedingsets Q
1
;...;Q
i01
. Thisalsoholds forthebinarysearchin thenextsection.
Eg(Q1) Eg(Q2)
Eg(Qn) Example Space
The Input
Figure 4: SubsumptionRelation of Example Sets
2. Similarity Value of Each Example
The examples in the inner sets have higher similarity values than the examples in the
outer sets:
Eg(Q
i
)Eg(Q
j ); 8e
i
2Eg(Q
i );8e
j
2Eg(Q
j );
( e
j
62Eg(Q
i
) ) sim(e
in
;e
i
)>sim(e
in
;e
j ) )
If itis possible todivide the setof all similarity templates intoasequence T
1
;...;T
n
which
satis esthetworequirementsabove,thenthefollowingbinarysearchstrategyforthenearest
neighb orretrievalbecomes applicable.
Binary Search of the Nearest Neighb or
Withthetotally orderedsequence ofexamplesets, thenearestneighbor(s) withthegreatest
similarityvalueexistintheinnermostnon-emptysetofexamples. Theinnermostnon-empty
set can be eciently found by the binary search procedure using the binary search tree in
Figure5. Thebinary searchtreeinFigure5isconstructedfromthetotallyorderedsequence
Eg(Q
1
);...;Eg(Q
n
) of example sets, in which each nodeis a set of examples Eg(Q
i
). The
following is the binary searchstrategy for nding the nearestneighb or(s) of the input.
3
1. At any non-leaf no de Eg(Q
i
), if Eg(Q
i
) contains at least one example then the search
process goes down tothe left child Eg(Q
j
). Otherwise, the search process goes down to
the rightchildEg(Q
k ).
3
Inthis section,wedescrib ethebinarysearchstrategyusing thebinary searchtree ofthe example sets
justforexplanation. Intheframeworkdescrib edin section2.4,thesystemcontainsonlythebinary search
tree of the sets of similarity templates but not the binary search tree of the example sets like the one in
Figure5. Aswementionedintheprevious section,itissucienttoconstructthesetQ
i
ofretrievalqueries
onlywhenthesearchprocesshaspro ceededtoT
i .
YES NO
Eg(Qi)
Eg(Qj) Eg(Qk)
YES NO
Eg(Qm)
Eg(Ql) YES
NO NO
Figure 5: Binary SearchTree of the Nearest Neighb orRetrieval
2. At any leaf no de Eg(Q
l
), if Eg(Q
l
) contains at least one example then Eg(Q
l
) is the
innermost non-empty set. Otherwise, the innermost non-empty set is the lowest non-
empty ancestor Eg(Q
m
) of Eg(Q
l ).
3. Construct the set Eg
max (e
in
) of the nearest neighb or(s) from the innermost non-empty
set.
3 An Example
This section applys the optimization method to the standard data structure and the sim-
ilarity measure which are most commonly used in the nearest neighb or retrieval in CBR
and example-based NLP, namely, the set of feature-value pairs and the weighted sum of
similarities for individual features. We then de ne similarity template for the de nition of
the similarity measure, and also showhow to optimize the nearest neighb or retrieval based
onthe binary searchstrategy given in section 2.5.2. After that, we describe the retrieval of
Japanesesurfacecasestructures asanexample. Finally,weevaluatethesystem sizeand the
computational cost of the overall framework.
3.1 De nitions
Data Structure of Example
Werepresent anexample e as ann-tuple of feature-value pairs:
e = D
hf
1
;v
1
i;...;hf
n
;v
n i
E
Let w
i
b e the weight for the i-th feature and s(v
1i
;v
2i
) be the similarity of the i-th values
of the examples e
1
and e
2
. Then, the similarity sim(e
1
;e
2 ) of e
1
and e
2
is de ned as the
weighted sum of similarities for individualfeatures:
sim(e
1
;e
2 ) =
n
X
i=1 w
i 2s(v
1i
;v
2i )
. n
X
i=1 w
i
(2)
We assume that the weights w
i
(i=1;...;n) and the similarity function s(v
1i
;v
2i
) for indi-
vidual features satisfy the tworequirements onthe similaritymeasure stated insection 2.2.
Similarity Template
For the similarity measure de ned in Formula 2, we de ne a similarity template as an n-
tuple of the similarities for individual features. The similarity function sim
t
for similarity
template is also de ned fromthe similarity function sim of examplesin Formula 2:
t = hs
1
;...;s
n
i (3)
sim
t (t) =
n
X
i=1 w
i 2s
i
. n
X
i=1 w
i
Retrieval Query
A retrievalqueryq isde ned as ann-tuple of the constraints onindividualfeature values:
q = D
hf
1
;V(v
1
;s
1
)i;...;hf
n
;V(v
n
;s
n )i
E
(4)
Inthis de nition,eachconstraintisdenoted asV(v
i
;s
i
),whichmeansthatthe i-th valuev
r i
of theexample e
r
toberetrievedhas tosatisfythe similarity constraintthatv
r i andv
i have
the similarityvalue greaterthan or equal tos
i :
s(v
r i
;v
i
) s
i
Asdescrib edinsection2.3,aretrievalqueryisgeneratedfromtheinpute
in
andthesimilarity
template t by the functionqgen . Ife
in
is represented ashhf
1
;v
1
i;...;hf
n
;v
n
ii and t isgiven
as Formula 3,the retrievalquery q generated by qgen (e
in
;t) is given asFormula 4.
3.2 Total Ordering of the Sets of Similarity Templates
Forthe binarysearchstrategygiveninsection2.5.2,we rsthavetodividethesetofallsim-
ilarity templates intoa sequence T
1
;...;T
n
which satis esthe requirementsof subsumption
relation of example sets and similarity value of each example. In the case of the similarity
measureand the similarity template inthis section,these tworequirementscan berestated
using the subsumption relationand the similarity value of similarity templates.
First,wede nethesubsumptionrelationofsimilaritytemplatesandderivethesubsumption
relation of example sets from that. The subsumption relation of similarity templates can
be regarded as the subsumption relation of the similarity constraints on individual feature
values. Let t and t 0
be similarity templates hs
1
;...;s
n
i and hs 0
1
;...;s 0
n
i respectively. Then
the subsumptionrelation
t
ofsimilaritytemplatesis de nedasfollows: t
t t
0
(t subsumes
t 0
) is true if and only if each similarity s
i
is smaller than or equal to s 0
i
for all the features
and t is not identical tot 0
, i.e.
t
t t
0 def
() s
i
s 0
i
(i=1;...;n) and t isnot identical to t 0
Then,assumethattsubsumest 0
andlete
in
b eaninputandq=qgen (e
in
;t)(=hhf
1
;V(v
1
;s
1 )i;
...; hf
n
;V(v
n
;s
n
)ii) and q 0
=qgen (e
in
;t 0
) (=hhf
1
;V(v
1
;s 0
1
)i; ...; hf
n
;V(v
n
;s 0
n
)ii) be the
retrieval queries generated by qgen . Then, with the de nition ab ove, it is obvious that q
representsa weakerconstraintonthe examplestoberetrievedthan q 0
, sinceeachsimilarity
constraint on its feature values is weaker than or equal to the similarity constraint on q 0
's
feature values. Thus, letting Eg(q) and Eg(q 0
) be the example sets retrieved by q and q 0
respectively,we can conclude that Eg(q)subsumes Eg(q 0
):
t
t t
0
) Eg(q)Eg(q 0
) (5)
Total Ordering of the Sets of Similarity Templates
Next, we show how to divide the set of all similarity templates into a sequence T
1
;...;T
n
whichsatis estherequirementsofsubsumption relationof examplesets and similarityvalue
of each example in section 2.5.2. We give the division into a sequence T
1
;...;T
n
using the
subsumption relation of similarity templates and the similarity value of similarity template.
Then, we showthis division satis es the tworequirements insection 2.5.2.
1. Subsumption Relation of Similarity Templates
Weassumethat thesequenceT
1
;...;T
n
satis esthe conditionthatforeacht
i inT
i ,there
exists at
j in T
j
(i<j) suchthat t
j
subsumes (
t )t
i :
i<j ) 8t
i 2T
i
; 9t
j 2T
j
; t
i
t t
j
(6)
As described in section 2.5.2, a sequence Q
1
;...;Q
n
of the sets of retrieval queries is
constructedfromthesequenceT
1
;...;T
n
usingFormula1. FromFormulas5and 6above,
it can be concluded that the sequence Q
1
;...;Q
n
satis esthe following:
i<j ) 8q
i 2Q
i
; 9q
j 2 Q
j
; Eg(q
i
)Eg(q
j )
j i
Q
1
;...;Q
n
satis es the requirement of the subsumption relationof examples setsin sec-
tion 2.5.2:
Eg(Q
1
)111Eg(Q
n )
2. Similarity Value of Similarity Template
We assume that the sequence T
1
;...;T
n
satis es the condition that for eacht
i inT
i and
t
j in T
j
(i<j), sim
t (t
i
) isgreater than sim
t (t
j ):
i<j ) 8t
i 2T
i
; 8t
j 2T
j
; sim
t (t
i
)>sim
t (t
j )
In order to prove that the sequence T
1
;...;T
n
satis es the requirementof the similarity
value of each example in section 2.5.2, it is sucient to prove the requirement for the
case of j =i+1. For each e
i
in Eg(Q
i
) and for each e
i+1
in Eg(Q
i+1 ), if e
i+1
is not
in Eg(Q
i
), then there exist a t
i in T
i
and a t
i+1 in T
i+1
such that e
i
2 Eg(qgen (e
in
;t
i ))
and e
i+1
2Eg(qgen (e
in
;t
i+1
)),and for each t
i in T
i , e
i+1
62Eg(qgen (e
in
;t
i
)). From this,
the following relation of the similarity values of examples and the similarity values of
similarity templates holds 4
:
sim(e
in
;e
i
) min
t
i 2T
i sim
t (t
i
) > sim(e
in
;e
i+1
) min
t
i+1 2T
i+1 sim
t (t
i+1 )
Thussim(e
in
;e
i
) is greaterthan sim(e
in
;e
i+1 ).
Then, as we presented in section 2.5.2, it becomes possible to binary-search the nearest
neighb or(s) of the input using the sequence T
1
;...;T
n
of setsof similarity templates.
3.3 Retrieval of Japanese Surface Case Structures
As an example of optimizing the nearest neighbor retrieval, this section describes retrieval
of Japanese surface case structures.
5
We represent an example of a Japanese surface case
structure as a3-tuple of feature-valuepairs 6
:
e = D
hverb;Sem
v
i;hsubj;Sem
s
i;hobj ;Sem
o i
E
4
Note that if e 2 Eg(qgen(e
in
;t)) is true, then sim(e
in
;e) sim
t
(t) is true. And also note that if
e
i+1
62Eg(qgen(e
in
;t
i
))istrueforallthet
i inT
i
,then min
t
i 2T
i sim
t (t
i
)>sim(e
in
;e
i+1
)istrue.
5
Inexample-basedNLP,retrievalofsimilarsurfacecasestructuresisa basictechniquein thetaskssuch
as example-based target wordselection in machinetranslation [Sato
and Nagao, 1990]
and example-based
casestructureanalysis [
Kurohashi andNagao, 1993 ]
.
6
In [
Utsuro et al., 1994 ]
, we adopted much more practical data structure of examples and similarity
measure for retrievalof Japanese surface case structures, and also discussed how to optimize the nearest
neighb orretrieval.
(accusative case or wo case) of the surface case structure, respectively. The values Sem
v ,
Sem
s
, and Sem
o
represent the semantic categories of the verb, subject noun, and object
noun, resp ectively. For representing the semantic categories, we use an on-line Japanese
thesaurus called Bunrui Goi Hyou(BGH)
[NLRI, 1964].
BGH has asix-layered abstraction
hierarchy and morethan 60,000Japanese wordsare assigned tothe leaves.
The similarity measuresimfor this data structureof Japanese surface casestructures is
de nedonthebasis ofthe similaritymeasureinsection3.1. Theweightsw
1 ,w
2
,and w
3 are
set as 1 for simplicity. In the following de nition, s(Sem
1v
;Sem
2v
), s(Sem
1s
;Sem
2s ), and
s(Sem
1o
;Sem
2o
) representthe similarities of correspondingsemanticcategories:
sim(e
1
;e
2 ) =
s(Sem
1v
;Sem
2v
)+s(Sem
1s
;Sem
2s
)+s(Sem
1o
;Sem
2o )
.
3
We de ne the similarity s(Sem
1
;Sem
2
) of two semantic categories Sem
1
and Sem
2 as a
monotonically increasing function of the most speci c common layer mscl (Sem
1
;Sem
2 ) of
Sem
1
and Sem
2
inthe thesaurus asbelow:
mscl(Sem
1
;Sem
2
) 1 2 3 4 5 6 exact match
s(Sem
1
;Sem
2
) 0 5 7 8 9 10 11
Forexample,the similarity of the surface case structures e
1 ande
2
of Examples1 and 2
is calculatedas follows. First,the surface case structures e
1 and e
2
are givenas below:
Example 1 Example 2
kare- ga hon - wo kau kanojo - ga no oto - wo kau
he - NOM book - ACC buy she - NOM notebook - ACC buy
(He buysa book.) (She buys a notebook.)
e
1
= D
hverb;Sem
buy
i;hsubj;Sem
he
i;hobj;Sem
book i
E
e
2
= D
hverb;Sem
buy
i;hsubj;Sem
she
i;hobj;Sem
nb i
E
(nb=notebook)
The similarity values forthe semanticcategories inthe thesaurus are asbelow:
s(Sem
buy
;Sem
buy
)=11; s(Sem
he
;Sem
she
)=10; s(Sem
book
;Sem
nb )=9
Finally,the similarity ofe
1 and e
2
is calculatedas below:
sim(e
1
;e
2
) = (11+10+9) =3 = 10
Forthe similaritymeasure de nedab ove, wede ne a similarity template as a3-tuple of
the similarityfor individual features:
t = hs
v
;s
s
;s
o i
s
v
;s
s
;s
o
2f0;5;7;8;9;10;11g
10
Y
T
5
Y
T
3
Y
T
2
Y
T
1
N
T
4
N
T
8
Y
T
7
Y
T
6
N
T
9
N
T
15
Y
T
13
Y
T
12
Y
T
11
N
T
14
N
T
18
Y
T
17
Y
T
16
N
T
19
Figure 6: An Example of Binary Search Treeof the Sets of Similarity Templates
The similarity function sim
t
for the similarity template and the retrieval query are de ned
onthe basis of the de ntionsinsection 3.1. In thesimilaritycalculation of e
1 ande
2
above,
the similaritytemplate is h11;10;9i.
Next,wedividethesetofallsimilaritytemplatesintoasequenceT
1
;...;T
n
whichsatis es
the requirements of subsumption relation of similarity templates and similarity value of
similarity template insection3.2. The numb erof p ossiblesimilaritytemplates is343 (=7 3
),
and the set of all similarity templates is divided into a sequence of 19 subsets T
1
;...;T
19 .
7
Figure6showsthebinarysearchtreeconstructedfromthesequenceT
1
;...;T
19
. Asdescrib ed
in section 2.5.2, the arc labels \Y" and \N" corresp ond to the two p ossibilities whether or
notthereexistsasimilaritytemplateintheparentnodesuchthataretrievalquerygenerated
from this retrievesat least one example.
Supp ose that the input is the surface case structure e
1
of Example 1 and the nearest
neighb or in the example database is e
2
of Example 2. The pro cess of binary search in the
binary search tree of Figure 6 is then as follows. The similarity template of e
1
and e
2 is
t
12
=h11;10;9i and this is contained in T
4
. The binary search process starts from the ro ot
node T
10
and proceeds in the order of T
10
!T
5
!T
3
! T
4
. Finally, aretrievalquery q
12 is
generated frome
1 and t
12
as below, and e
2
isretrieved by q
12 :
q
12
= D
hverb;V(Sem
buy
;11)i;hsubj;V(Sem
he
;10)i;hobj;V(Sem
book
;9)i E
In the process of the binary search, subsumption relation of similarity templates can be
used for eliminating similarity templates whichare useless for retrievingexamples from the
7
Thelengthof thesequenceofthesubsets ofsimilaritytemplates di ersdep endingonthede nition of
thesimilaritymeasure: thelongerthesequence,themoreecientthebinarysearchisforthesame numb er
ofsimilaritytemplates. Sometimesitcanhappenthatthereisnouniquedivisionofthesetofallsimilarity
templates.
0 1000 2000 3000 4000 5000 6000
0 200 400 600 800 1000 1200
Retrieval Time (msec)
Number of Examples Full Retrieval Optimized Retrieval
Figure 7: RetrievalTimeperNumber of Examples
example database.
8
3.4 Evaluation
Thissectionevaluatesthesystemsizeandthecomputationalcostoftheoverallframework
of the nearest neighb or retrieval proposed in section 2.4, in the case of the retrieval of
Japanese surface case structures presented in section 3.3. As an indexed retrieval module
of section2.4, weuse atree-structured index module constructed fromthe on-lineJapanese
thesaurus, which we call sub-thesaurus
[Utsuro
et al.,
1994].
For each feature, examples
whichsatisfythe constraintV(Sem;s) are collectedfor alltheleaf semanticcategories Sem
in the thesaurus and all the similarity values s. Then, a sub-thesaurus for that feature is
constructed as a sub-structure of the whole thesaurus in which each node corresponds to a
setofexamplesthat satisfytheconstraintofthe semanticcategoryofthe no de. Usingthose
sub-thesauri as the index module, examples which satisfy a retrieval query are obtained as
examples which satisfyconstraints onallthe features. This retrieval can b edone quicklyin
constant time indep endetly of the numb erof examples inthe example database.
Next, letN b ethe size of the exampledatabase. The number ofall similaritytemplates
is independent of N and that of the indexed retrieval module, i.e., the set of sub-thesauri,
is O(N). Thus,the total order of the system size isO(N). Finally, inorder to evaluate the
computational cost, weplot the computation time(in CPU time),increasing the numb er of
examples N, and compare the result with a full retrieval program. The example database
contains example Japanese surface case structures and b oth programs retrieve the most
similarexamplesfromthe exampledatabase, givenaninputsurface case structure. Thefull
8
Formula5insection3.2representstherelationb etweenthesimilaritytemplatesubsumption
t
andthe
examplesetsubsumption. Fromthis,wecanconcludethatifaretrievalquerygeneratedfroma similarity
templatetretrievesnoexample,thenanysimilaritytemplatet 0
subsumedbyt generatesnoretrievalquery
that would retrieve at least one example. Thus, we can eliminate all the similarity templates that are
subsumedbyt.
examples in the example database, and retrieves the examples with the greatest similarity.
Both programs are implemented in SICStus Prolog 2.1 on a SPARC station-10. Figure 7
illustrates the results. The computation time of the full retrieval program is prop ortional
toN, while that of the optimized retrieval program is nearly constant. Thus,our optimized
retrievalprogram achieveddrasticimprovementindecreasing computational cost compared
with the full retrieval program.
4 Concluding Remarks
This paper prop osedanoveltechniquefor optimizingthe nearestneighb oralgorithm, based
on the use of similarity template, which is a data structure to enumerate all the possible
patterns of calculating similarity b etween two examples. The nearest neighb or retrieval
process is optimized by generating retrieval queries from aninput and similarity templates
in acertain order.
AmongpreviousworksonCBR,
[Shimazu
etal., 1993]
tookanapproachrelativelysimilar
toours. In
[Shimazu
et al., 1993],
givenaninput,retrievalqueries are generatedusingStan-
dard Query Language (SQL) and then nearest neighb orsare retrieved froma commercially
available relational data-base system (RDBMS). Their search strategy corresponds to our
linearsearchin section2.5.1. Compared with the formalizationof this paper, theyprovided
onlyaspeci ctechniqueofgeneratingSQLqueriesfromaninputandthuslackedanyformal
accounts of optimizing the nearest neighb or algorithm.
Thesetoffeature-valuepairsandtheweightedsumofsimilaritiesforindividualfeatures,
towhichweappliedour method insection3,are most commonlyused asthe standard data
structure and the similarity measure in CBR. Sometimes, however, it is not sucient to
assign only one set of weights, especially when the case library is used for several di erent
purposes. If it is necessary to assign several di erent sets of weights dep endening on the
purpose of the reasoning, then we have to prepare several di erent sequences T
1
;...;T
n of
the setsof similarity templates according tothe setsof weights.
References
[
Aha et al.,1991 ]
D. W. Aha, D. Kibler, and M. K. Albert. Instance-based learning algo-
rithms. Machine Learning, 6(1):37{66, 1991.
[
Cognitive Systems, 1992 ]
CognitiveSystems. ReMindDeveloper'sReferenceManual. 1992.
and Hart, T.M.CoverandP.E.Hart. Nearestneighb orpatternclassi cation.
IEEE Transactions on InformationTheory,13(1):21{27,1967.
[Kolodner
and Simpson, 1989]
J. Kolodner and R. Simpson. The MEDIATOR: Analysis of
anearlycase-based problem solver. Cognitive Science,13(4):507{549, 1989.
[Kolodner, 1993]
J. Kolodner. Case-Based Reasoning. Morgan Kaufmann, 1993.
[
Kurohashi and Nagao, 1993 ]
S. Kurohashi and M. Nagao. Structural disambiguation in
Japanese by evaluating case structures based on examples in case frame dictionary. In
Proceedings of the 3rd IWPT,pages 111{122, 1993.
[
NLRI, 1964 ]
NLRI, (NationalLanguage ResearchInstitute). Word List by Semantic Prin-
ciples. Syuei Syuppan,1964. (in Japanese).
[
Sato and Nagao, 1990 ]
S.Sato and M. Nagao. Toward memory-based translation. In Pro-
ceedings of the 13th COLING, volume3, pages 247{252, 1990.
[
Shimazu et al., 1993 ]
H. Shimazu, H. Kitano, and A. Shibata. Retrieving cases from rela-
tional data-bases: Another stride towards corp orate-wide case-base systems. In Proceed-
ings of the 13th IJCAI, pages 909{914, 1993.
[
Stan lland Waltz, 1986 ]
C.Stan llandD. Waltz. Towardmemory-basedreasoning. Com-
munications of the ACM,29(12):1213{1228, 1986.
[
Sumitaet al.,1993 ]
E. Sumita, K. Oi, O. Furuse, H. Iida, T. Higuchi, N. Takahashi, and
H. Kitano. Example-based machine translation onmassivelyparallel processors. In Pro-
ceedings of the 13th IJCAI,pages 1283{1288, 1993.
[
Utsuro et al., 1994 ]
T. Utsuro, K. Uchimoto, M. Matsumoto, and M. Nagao. Thesaurus-
based ecientexample retrievalby generating retrievalqueries fromsimilarities. In Pro-
ceedings of the 15th COLING, pages 1044{1048,August 1994.