V
alidated
Retriev
al
in
Case{Based
Reasoning
Ev
angelos
Simoudis
James
Miller
Digital
Equipmen
tCorp
oration
Cam
bridge
Researc
hLab
CRL
90/2
Decem
ber
12,
1990
Abstract
W
ecom
bine
simple
retriev
al
with
domain-sp
ecic
validation
of
retriev
ed
cases
to
pro
duce
au
seful
practical
tool
for
case-based
reasoning.
Based
on
200
real-w
orld
cases,
we
retriev
eb
etw
een
three
and
six
cases
over
aw
ide
range
of
new
problems.
This
represen
ts
aselectivit
yranging
from
1.5%
to
3%,
com-pared
to
an
average
selectivit
yof
only
11%
from
simple
retriev
al
alone.
cDigital
Equipmen
tC
orp
oration
1990.
All
righ
ts
reserv
ase-2
2
RETRIEV
AL
IN
CBR
man
tic
net
work8
].
In
order
to
build
our
validation
mo
del
we
are
faced
with
aclassic
kno
wledge
acquisition
task.
By
perusing
existing
data
bases
used
by
sp
ecialists
we
are
able
to
acquire
this
kno
wledge
with
areasonable
amoun
to
f
eort
|
and
with
only
asmall
investmen
tof
specialists'
time.
2
Retriev
al
in
CBR
CBR
systems
rst
retriev
ea
set
of
cases
from
acase
base
and
then
reason
from
them
to
nd
asolution
to
an
ewly
posed
problem.
Existing
systems
(1],
2],
3],
4],
9]
and
10])
mak
et
wo
assumptions
ab
out
the
initial
retriev
al
of
cases
from
the
case
base:
1.
Very
few
cases
will
be
retriev
ed
from
the
case
library
.
2.
The
retriev
ed
cases
are
relev
an
tto
the
problem
being
solv
ed.
In
man
yp
ractical
applications,
retriev
al
alone
issucien
tt
oso
lve
the
dif-cult
part
of
atask.
For
example,
in
our
domain
of
diagnosis
of
computer
soft
ware
failures,
sp
ecialists
can
easily
resp
ond
to
customer
problems
if
they
can
quic
kly
locate
afew
similar
cases
from
their
collectiv
epast
exp
erience.
For
this
reason,
weh
ave
concen
trated
on
the
retriev
al
asp
ect
of
case-based
reason-ing.
In
MBR
Talk
10
],
also,
the
essen
tial
task
is
retriev
al
the
\reasoning"
comp
onen
tconsists
of
merely
passing
the
retriev
ed
information
directly
to
an
output
unit.
2.1
Related
W
ork
Closest
to
our
own
work
is
the
work
of
Koton
on
casey
5],
aCBR
system
whic
h
has
been
applied
in
the
domain
of
medical
diagnosis.
casey
has
a
justication
comp
onen
twhose
goal
is
to
determine
whether
the
causal
expla-nation
of
aretriev
ed
case
applies
to
anew
problem.
This
frequen
tly
allo
ws
casey
to
avoid
invoking
its
causal
mo
del
when
creating
an
explanation
for
a
new
case.
casey
's
justication
phase
is
similar
to
our
validation
phase.
But
there
isa
n
imp
ortan
tdierence
betw
een
these
two
systems
arising
from
dieren
tassumptions
abo
ut
tests.
casey
relies
on
precisely
two
tests
(EK
G
and
X-ra
ys),
both
of
whic
h
are
inexp
ensiv
eand
non-in
vasiv
e.
Both
of
these
tests
are
performed
prior
to
the
retriev
al
phase
and
the
results
are
used
to
pro
vide
surface
features
for
the
retriev
al
algorithm.
By
con
trast,
there
2.2
Tw
oPhases:
Retriev
al
and
Validation
3
literally
hundreds
of
tests
to
be
performed
in
our
domains
and
it
is
far
too
exp
ensiv
eto
perform
all
of
them
in
adv
ance
of
initial
case
retriev
al.
As
a
result,
our
systems
dev
ote
atten
tion
to
minimi
zing
the
num
ber
of
tests
that
are
performed.
W
enot
only
perform
tests
incremen
tally
and
cac
he
the
results,
but
also
emplo
yk
nowledge
ab
out
the
tests
themselv
es
to
reduce
the
num
be
r
of
tests
performed.
The
CBR
system
chef
2],
whose
domain
is
Chinese
co
oking,
also
has
a
justication
comp
onen
t.
In
order
to
justify
eac
h
retriev
ed
case,
chef
uses
bac
kw
ard
chaining
rules.
While
our
validation
mo
del
is
not
appropriate
to
this
domain,
weh
ave
examined
and
rejected
the
use
of
rules
for
our
valida-tion
mo
dels.
This
decision
isb
ased
on
the
dicult
yof
acquiring
the
exp
ert
kno
wledge
needed
to
create
large
rule
sets,
esp
ecially
in
comparison
to
the
simplicit
yof
constructing
our
validation
mo
dels.
These
mo
dels
are
captured
by
aseman
tic
net
work
that
represen
ts
groups
of
tests
and
information
about
the
relationships
betw
een
the
groups.
Furthermore,
as
with
casey
,
chef
do
es
not
use
the
results
of
comparisons
with
earlier
cases
to
prune
its
searc
hf
or
relev
an
tcases.
The
sw
ale
6]
system
concen
trates
primarily
on
mo
difying
the
explanation
(con
tained
in
an
explanation
pack
et
or
XP)
ofa
retriev
ed
case
to
matc
ha
new
problem.
Nonetheless,
it
has
asub
comp
onen
t,
xp
accepter
,that
justies
the
application
of
a
retriev
ed
XP
to
a
curren
tsituation.
The
accepter
veries
an
XP
by
determining
ifit
can
believ
et
he
applicabilit
y
checks
that
are
pac
kaged
with
eac
hX
P.
Eac
hs
uch
test
is
similar
to
the
tests
asso
ciated
with
our
validation
mo
del.
Because
of
the
small
num
ber
of
cases
(eigh
t,
by
our
coun
t)
xp
accepter
nev
er
addressed
the
issues
of
scale
whic
h
are
our
ma
jor
concern.
Th
us,
sw
ale
nev
er
dev
elop
ed
ajustication
mo
del
to
relate
the
various
applicabilit
yc
hec
ks
to
one
another.
2.2
Tw
oPhases:
Retriev
al
and
Validation
Ou
rg
oali
stot
ake
asizable
pre-existing
case
base
along
with
an
ew
problem
and
pro
duce
asmall
num
ber
ofrelev
an
tcases.
Lik
eour
human
sp
ecialists,
our
systems
perform
diagnosis
in
two
phases:
Retriev
al:
it
poses
aq
uery
to
the
case
base
using
asubset
of
the
features
that
describ
ethe
new
3.1
What
is
avalidation
mo
del?
5
validation
mo
dels
:structures
that
capture
muc
hof
an
exp
ert's
kno
wledge
in
a
wayt
hatm
akes
it
easy
for
the
validation
phase
to
pro
cess
the
tests
itrequires.
3.1
What
is
avalidation
mo
del?
Rather
than
require
acomplete
and
accurate
description
ofe
acht
est
used
by
the
sp
ecialist,
we
capture
the
overall
structure
of
the
test
space
itself.
The
resulting
structure,
our
validation
mo
del
,consists
of
related
groups
oft
ests
and
information
ab
out
the
relationship
betw
een
the
groups.
For
example,
if
we
wan
tt
ok
noww
hy
ahouse
is
hot
(the
problem),
wem
ay
rst
wan
tto
see
ifthe
air
conditioner
isw
orking
but
this
requires
us
to
nd
out
ifthe
house
has
an
air
conditioner.
In
this
example,
the
desired
test
(is
the
air
conditioner
working?)
is
related
to
another
test
(is
there
an
air
conditioner?).
This
kno
wledge,
as
well
as
kno
wledge
ab
out
the
imp
ortan
toutcomes
and
implications
ofa
test,
is
captured
by
the
validation
mo
del.
W
eh
avec
hosen
to
represen
tthis
kno
wledge
in
the
form
of
aseman
tic
net
work
whose
no
des
corresp
ond
to
sets
of
tests
and
whose
arcs
indicate
relationships
betw
een
these
sets.
3.2
Creating
aV
alidation
M
odel
W
ebuild
our
validation
mo
dels
by
rst
examining
existing
data
bases
that
are
used
byh
uman
sp
ecialists.
These
data
bases
ma
yb
eeither
formalized
(as
in
the
case
of
our
W
PS-PLUS
1
system)
or
merely
informal
notes
prepared
by
the
sp
ecialists
for
their
own
perusal
(as
in
the
case
of
our
VAX/VMS
system).
In
our
two
case-based
systems,
the
existing
data
con
tains
atextual
description
of
the
steps
that
the
specialists
used
to
verify
ah
yp
othetical
explanation
of
the
problem.
In
constructing
the
validation
mo
del,
itis
our
goal
to
capture
the
interrelationships
betw
een
the
validation
tests.
As
aresult,
weh
aveb
uilt
validation
mo
dels
that
corresp
ond
to
ap
articular
case
base
by:
1.
Reading
the
validation
pro
cedures
of
eac
h
case
and
building
a
list
of
all
the
validation
steps
used
in
the
en
tire
data
base.
In
the
pro
cess
of
reading
the
data
base
and
preparing
this
list,
the
implem
en
tor
dev
elops
asense
of
the
underlying
(but
unstated)
relationships
betw
een
tests
that
are
men
tioned
in
the
data
base.
1
DEC,
VAX,
VMS
and
WPS-PLUS
are
registered
trademarks
of
Digital
Equipmen
t
Corp
6
4
AN
EXTENDED
EXAMPLE
2.
Examining
the
resulting
list,
looking
for
groups
oftests
that
app
ear
to
form
related
sets.
Organizing
the
list
pro
vides
ab
asis
for
discussion
with
domain
exp
erts,
who
help
\debug"
the
prop
osed
organization.
3.
Rening
the
structure
of
the
list
through
kno
wledge
acquisition
sessions
with
domain
exp
erts.
During
these
sessions,
signican
tranges
of
test
results
are
iden
tied,
as
are
inferences
from
these
results
that
eliminate
the
need
to
perform
other
tests.
That
is,
ad
ep
endency
graph
based
on
test
results
is
dev
elop
ed.
4.
Iterating
the
ab
ovet
wo
steps
after
consulting
additional
information
suc
ha
sm
anuals
and
cod
ed
ocumen
tation.
The
structure
of
the
domain
becomes
clearer
at
eac
hiteration.
(W
eh
ave
found
that
three
iterations
are
sucien
tto
pro
duce
auseful
structuring.)
The
nal
validation
mo
del
consists
primarily
of
en
tries
corresp
onding
directly
to
information
that
app
ears
in
the
original
data
base.
5.
Integrating
the
test
sets
into
the
structure
deriv
ed
in
the
previous
step.
This
integration
mak
es
explicit
the
prerequisites
of
eac
htest,
as
well
as
pro
viding
alternativ
ew
ays
of
obtaining
information
ordinarily
pro
vided
by
aparticular
critical
test
in
cases
where
that
test
cannot
be
performed.
4
An
Extended
Example
In
order
to
understand
the
validated
retriev
al
pro
cess,
consider
the
follo
wing
example.
Our
domain
is
automobile
diagnosis
and
repair,
and
we
assume
an
existing
case
base
with
its
asso
ciated
validation
mo
del.
W
eare
giv
en
the
follo
wing
case:
NEW
CASE
mak
e
:
MAZD
A
mo
del
:
626
mo
del
year
:
1985
engine
typ
e
:
2.0L
EFI
miles
:
50,000
problem
:
engine
do
es
not
start.
The
retriev
al
phase
uses
the
mak
e,
mo
del,
problem,
and
appro
ximate
year
of
man
ufacture
to
searc
hthrough
acase
base
of
previous
automobile
problems.
Based
on
these
surface
features,
we
retriev
ethree
cases
to
be
validated
before
presen
tation
to
areasoning
comp
onen
7
CASE
1
mak
e
:
MAZD
A
mo
del
:
626
mo
del
year
:
1988
engine
typ
e
:
2.0L
EFI
miles
:
10,000
problem
:
engine
do
es
not
start.
validation
:
The
fuel
injector
was
clogged.
Fuel
was
not
deliv
ered
to
the
com
bustion
cham
ber
for
the
engine
to
ignite.
For
this
reason
the
engine
could
not
start.
solution
:
cleaned
the
fuel
injector.
CASE
2
mak
e
:
MAZD
A
mo
del
:
626
mo
del
year
:
1984
engine
typ
e
:
2.0L
miles
:
60,000
problem
:
engine
do
es
not
start.
validation
:
The
car
had
afault
y
gas
pump.
Fuel
could
not
be
de-liv
ered
to
the
com
bustion
cham
ber.
For
this
reason
the
engine
could
not
start.
solution
:
Replaced
the
gas
pump.
CASE
3
mak
e
:
MAZD
A
mo
del
:
626
mo
del
year
:
1987
engine
typ
e
:
1.8L
miles
:
20,000
problem
:
engine
do
es
not
start.
validation
:
A
leak
existed
in
the
gas
line.
Fuel
could
not
be
deliv
ered
through
the
fuel
line.
For
this
reason
the
engine
could
not
start.
solution
:
Fixed
the
leak.
The
validation
mo
del
con
tains
(at
least)
the
three
tests
that
are
referenced
by
these
cases:
\chec
kif
afuel
injector
isc
logged",
\chec
kif
the
gas
pump
is
working",
and
\chec
kif
there
is
aleak
in
the
fuel
line".
The
rst
of
these
is
actually
comp
osed
of
two
simpler
tests:
atest
for
fuel
presen
tin
the
reserv
oir
of
the
injector
and
atest
for
fuel
exiting
the
injector's
nozzle.
Ifthere
is
no
fuel
in
the
injector
then
we
can
deduce
that
the
injector
is
not
at
fault.
Rather,
the
problem
lies
earlier
in
the
fuel
system
|
either
in
the
pump
or
the
fuel
line.
The
system
rst
attempts
to
validate
Case
1b
yrep
eating
the
validation
steps
from
that
case.
That
is,
we
wish
to
test
if
the
fuel
injector
is
clogged.
In
the
pro
cess
ofp
erforming
this
two-step
test
we
actually
acquire
kno
wledge
that
is
relev
an
tto
Cases
2a
nd
3:
if
the
fuel
reserv
oir
is
not
empt
yw
ec
an
eliminate
both
cases
ifit
is
empt
y,w
ecan
eliminate
Case
1.
This
8
5
RECENT
RESUL
TS
is
enco
ded
in
the
seman
tic
net
work
that
represen
ts
our
validation
mo
del
and
is
used
in
the
validation
phase.
In
the
best
case,
this
validation
mo
del
allo
ws
us
to
reduce
the
work
required
to
validate
cases
from
four
tests
to
two
tests
and
sim
ultaneously
reduces
the
num
ber
ofcases
to
be
considered
by
the
reasoner
from
three
to
one
(selectivit
y
of
33.3%).
The
rst
test
is
for
an
empt
yfuel
reserv
oir
if
the
reserv
oir
isfull
then
Cases
2and
3are
eliminated.
W
ethen
test
the
nozzle
for
fuel
exiting.
If
no
fuel
lea
ves
the
nozzle,
then
Case
1is
presen
ted
to
the
reasoner
but
if
fuel
is
lea
ving
the
nozzle
we,
unfortunately
,eliminate
Case
1as
well
and
lea
ve
the
reasoning
comp
onen
tto
its
own
resources.
The
worst
case
requires
all
four
tests
and
pro
vides
either
zero
or
one
case
to
the
reasoner.
5
Recen
tR
esults
5.1
An
Op
erating
System:
VMS
The
rst
system
we
dev
elop
ed
isused
for
the
diagnosis
ofd
evice
driv
er
induced
crashes
ofDigital's
VMS
op
erating
system.
The
kno
wledge
ab
out
surface
fea-tures
was
obtained
primarily
from
DEC
internal
publications
and
was
com-plemen
ted
by
an
exp
ert
from
the
VMS
supp
ort
team
during
three
kno
wledge
acquisition
sessions.
It
took
atotal
of
84
hours
to
acquire
the
domain
sp
e-cic
kno
wledge
about
surface
features.
Based
on
this
information,
the
domain
kno
wledge
used
by
unimem
in
order
to
organize
the
cases
into
ageneralization
hierarc
hyw
as
impleme
nted
in
v
ed
ays.
It
took
an
additional
four
days
of
reading
validation
pro
cedures
in
the
data
base
to
dev
elop
avalidation
mo
del
for
device
driv
ers.
In
addition,
four
more
kno
wledge
acquisition
sessions,
lasting
40
hours,
were
needed
to
rene
and
impro
vet
he
validation
mo
del.
Enco
ding
the
actual
validation
mo
del
too
k
ab
out
80
additional
days.
The
total
num
be
rof
days
sp
en
to
n
kno
wledge
acquisition
and
dev
elopmen
ti
ssh
own
belo
w:
Activity
Person
Days
Kno
wledge
acquisition
20
Dev
elopmen
t
85
Since
this
was
our
rst
attempt
to
build
acase
base
and
validation
mo
del,
these
num
bers
are
muc
hlarger
than
we
exp
ect
for
subsequen
tsystems.
5.2
AW
ord
Pro
cessing
System:
WPS-PLUS
9
work
to
date
on
the
system
describ
ed
in
Section
5.2
app
ears
to
conrm
this
exp
ectation.
The
system
was
evaluated
using
acase
base
of
200
cases
that
were
obtained
from
notes
written
by
specialists.
The
surface
feature
retriev
al
phase
of
the
system
was
evaluated
by
presen
ting
eac
ho
fthe
200
cases
to
the
retriev
er
(as
new
problems)
and
preparing
ah
istogram
of
the
num
ber
of
cases
retriev
ed.
unimem
pro
vides
am
echanism,
kno
wn
as
retrieval
weights
,for
tuning
its
re-triev
al
capabilities.
After
some
exp
erimen
tation,
wed
isco
vered
that
the
use
of
larger
retriev
al
weigh
ts
(i.e.
more
stringen
tm
atching
criteria)
caused
the
re-triev
er
to
miss
man
yrelev
an
tcases
and,
in
man
yoccasions,
to
fail
to
retriev
e
an
ycases
at
all.
With
less
stringen
tcriteria
this
problem
was
rectied.
Ho
w-ev
er,
man
yof
the
retriev
ed
cases
were
not
relev
an
tto
the
problem.
W
ith
the
optimal
weigh
ting,
wew
ere
able
to
retriev
eo
n
average
22
cases
per
retriev
al
(11%).
The
validation
phase,
ho
wev
er,
was
able
to
reduce
this
num
ber
ofc
ases
to
an
average
of
4.5
cases
out
of
200
(2.25%).
In
addition,
we
presen
ted
three
new
cases
to
the
system.
Based
on
surface
features
alone,
we
retriev
ed
20,
25,
and
16
cases
(10,
12.5,
and
8%
selectivit
y).
The
validation
phase
reduced
this
to
3,
5a
nd
3cases,
resp
ectiv
ely
(1.5,
2.5,
and
1.5%
selectivit
y).
Our
exp
erts
conrm
that
these
validated
cases
are
the
only
ones
relev
an
tto
the
problems
presen
ted.
5.2
AW
ord
Pro
cessing
System:
W
PS-PLUS
The
second
system
performs
diagnosis
of
customer
problems
with
the
word
pro
cessing
comp
onen
tof
an
oce
automation
pro
duct.
During
15
hours
of
kno
wledge
acquisition
sessions,
the
kno
wledge
ab
out
surface
features
was
ob-tained
from
asupp
ort
engineer
for
the
pro
duct.
It
then
too
k
an
additional
v
ed
ays
to
enco
de
the
domain
kno
wledge
for
use
by
unimem
.
The
validation
mo
del
was
obtained
from
the
validation
pro
cedures
of
the
cases
in
the
data
base,
an
internal
publication,
and
10
hours
of
kno
wledge
acquisition
with
the
same
engineer.
While
the
work
is
not
yet
complete
(only
50
out
of
340
cases
ha
ve
been
enco
ded),
ith
as
tak
en
only
10
da
ys
to
implem
ent
the
validation
mo
del.
This
system
is
still
under
dev
elopmen
t.
Ho
wev
er,
the
time
wes
pen
to
n
kno
wledge
acquisition
and
dev
elopmen
tis
sho
wn
belo
10
6
SCOPE
OF
W
ORK
Activity
Person
Days
Kno
wledge
acquisition
5
Dev
elopmen
t
10
This
system
was
evaluated
using
acase
base
of
340
cases.
Rep
eating
the
same
exp
erimen
tperformed
with
the
VMS
case
base
led
to
an
average
of
26
cases
per
retriev
al,
or
7.6%
selectivit
y.T
he
validation
phase
reduced
this
to
two
cases,
or
0.58%
selectivit
y.
Since
the
validation
mo
del
for
this
case
base
is
not
yet
fully
enco
ded,
weh
ave
not
presen
ted
new
problems
to
the
system.
6
Scop
eof
W
ork
Our
validated
retriev
al
metho
dcan
be
applied
in
man
yt
yp
es
of
tasks.
The
basic
requiremen
ts
are:
an
existing
data
base
of
previous
practical
exp
eri-ence
aset
of
quic
ktests
that
serv
eto
reduce
the
searc
hspace
at
low
cost
a
set
of
more
exp
ensiv
etests
that
can
further
reduce
the
searc
hspace
and
an
understanding
of
the
relationships
bet
ween
the
exp
ensiv
etests.
W
eh
ave
iden
tied
four
areas
of
poten
tial
interest,
but
weh
ave
limited
our
implem
en
tation
work
to
the
rst
of
these:
Diagnostic
tasks
.A
ssh
own
in
the
example,
we
use
the
symptoms
of
ap
roblem
as
the
surface
features
for
the
retriev
al
phase.
The
validation
pro
cedure
describ
es
whic
htests
to
perform
in
order
to
determine
ifthe
case
is
relev
an
tto
the
new
problem.
Design
tasks
.The
surface
features
are
sp
ecications
that
adesign
must
satisfy
.The
validation
pro
cedures
verify
that
ap
rop
osed
design
meets
the
specication.
Sales
tasks
.The
tec
hnique
can
be
used
to
help
iden
tify
sales
prosp
ects
for
a
new
pro
duct.
The
surface
features
are
the
characteristics
of
a
customer
suc
has:
size
of
business,
typ
eof
business,
location
of
business.
Eac
hv
alidation
pro
cedure
describ
es
the
customer's
requiremen
ts
that
were
satised
in
a
previous
sale.
The
validation
mo
del
includes
tests
that
determine
whether
or
not
a
customer
needs
a
particular
typ
eo
f
pro
11
Managemen
ttasks
.T
hese
tasks
include
accoun
ting,
credit
analysis,
investmen
tdecisions,
and
insurance
underwriting.
In
eac
ho
fthese
ar-eas,
sp
ecialists
can
iden
tify
easily
recognized
features
in
their
problem
domain
(typ
eof
compan
y,size,
etc.)
that
allo
w
rapid
retriev
al
of
similar
situations
encoun
tered
in
the
past.
They
then
ha
ve
more
detailed
tests
that
can
be
applied
(debt/equit
yr
atio
,pa
ymen
thistory
,t
ypeo
fc
lien
t,
balance
sheets,
etc.).
7
Conclusions
Our
work
has
concen
trated
exclusiv
ely
on
the
issue
of
case
retriev
al.
A
care-ful
study
of
two
applications
in
whic
h
people
consciously
use
case
retriev
al
has
sho
wn
that
retriev
al
based
solely
on
surface
features
is
not
sucien
tly
discriminating
for
use
with
large
case
bases.
It
results
in
large
num
bers
of
cases
returned
to
the
reasoner,
eac
ho
fwhic
hm
ust
then
be
further
examined
at
great
exp
ense.
Adding
avalidation
phase
that
uses
kno
wledge
of
domain-sp
ecic
tests
to
prune
the
retriev
ed
cases
dramatically
reduces
the
num
be
rof
cases
that
must
be
examined
by
the
reasoner.
W
eh
ave
found
that
acquiring
kno
wledge
ab
out
domain-sp
ecic
tests
is
aided
by
an
initial
perusal
ofthe
existing
data
base
used
by
spe
cialists.
With
areasonable
amoun
tof
eort,
and
with
only
asmall
investmen
tof
special-ists'
time,
this
information
can
be
captured
in
avalidation
mo
del
represen
ted
as
aseman
tic
net
work.
W
eh
ave
used
this
metho
dology
to
pro
duce
twos
ys-tems.
One
of
these
systems
has
been
successful
in
practice,
and
the
other
(incomplete)
system
is
lik
ely
to
be
equally
useful.
While
the
burden
of
kno
wledge
acquisition
in
our
metho
dology
is
small
compared
with
other
metho
ds,
itis
not
negligible.
Automating
this
work
by
com
bining
anatural
language
system
to
analyze
existing
data
bases
with
AI-assisted
statistical
comparison
of
surface
features
pro
vides
a
fertile
area
for
further
investigation.
Furthermore,
we
susp
ect
that
acareful
study
of
suc
h
asystem
in
practice
will
rev
eal
validation
tests
that
are
sucien
tly
common
that
it
ma
yb
ereasonable
to
promote
them
to
surface
features,
leading
to
a
system
with
better
retriev
al
capabilities.
This
analysis,
whic
hm
ust
rst
be
performed
man
ually
to
validate
our
assumption,
isitself
an
excellen
tarea
for
the
application
AI
metho
ds.
Ac
kno
wledgemen
12
7
CONCLUSIONS
The
authors
wish
to
thank
Prof.
David
W
altz
for
his
help
with
this
researc
h.
In
addition,
weh
ave
receiv
ed
helpful
commen
ts
from
Mark
Adler,
Andrew
Blac
k,
Da
ve
Hanssen,
Rose
Horner,
and
Candy
Sidner.
W
eare
also
indebted
to
the
engineers
who
ha
veg
iven
their
time
and
kno
wledge
to
help
us
understand
the
two
domains
weh
ave
studied:
the
Digital
Supp
ort
Engineers
at
Colorado
Springs,
Colorado
and
Spitbro
ok,
New
REFERENCES
13
References
1]
Ra
yB
areiss,
Karl
Bran
ting,
and
Bruce
Porter.
The
role
ofe
xplanation
in
exemplar-based
classication
and
learning.
In
Pr
ocee
dings
of
Case-Base
d
Reasoning
Workshop
,1
988
.
2]
Kristian
Hammond.
Case-Base
dP
lanning:
AnI
ntegr
ate
dThe
ory
ofP
lan-ning,
Learning,
and
Memory
.P
hD
thesis,
Yale
Univ
ersit
y,1
986
.
3]
Janet
L.Kolo
dner.
Reconstructiv
em
emory:
A
computer
mo
del.
Co
gnitive
Scienc
eJournal
,7:281{328,
1983.
4]
Janet
L.K
olo
dner,
Jr.
Rob
ert
L.
Simpson,
and
Katia
Sycara-Cyranski.
A
pro
cess
mo
del
ofcase-based
reasoning
in
problem
solving.
In
Pr
ocee
dings
of
the
International
Joint
Confer
enc
eo
nA
rticial
Intel
ligenc
e
,1
985
.
5]
Ph
yllis
Koton.
Using
exp
erienc
ei
n
learning
and
pr
oblem
solving
.
PhD
thesis,
Massac
hussetts
Institute
of
Tec
hnology
,1988.
6]
Da
vid
Leak
e.
Ev
aluating
explanations.
In
Pr
ocee
dings
of
the
Seventh
National
Confer
enc
eo
nA
rticial
Intel
ligenc
e
,1
988
.
7]
Mic
hael
Leb
owitz.
Exp
erimen
ts
with
incremen
tal
concept
formation:
Unimem.
Machine
Learning
,2:103{138,
1987.
8]
M.R.
Quillian.
Seman
tic
memory
.In
Marvin
Minsky
,editor,
Semantic
Information
Pro
cessing
,pages
227{270.
MIT
Press,
1968.
9]
Edwina
Rissland
and
Kenneth
Ashley
.H
yp
otheticals
as
heuristic
device.
In
Pr
ocee
dings
of
the
Fifth
National
Confer
enc
eo
nA
rticial
Intel
ligenc
e
,
1986.
10]
Craig
Stanll
and
Da
vid
W
altz.
Tow
ard
memory-based
reasoning.
CA
CM
,
29:1213{122
14
A
CREA
TING
THE
VMS
VALID
ATION
MODEL
A
Creating
the
VMS
Validation
Mo
del
This
app
endix
describ
es,
in
detail,
the
metho
dused
to
construct
the
valida-tion
mo
del
for
the
VAX/VMS
device
driv
er
diagnosis
system.
Eac
hsubsection
corresp
onds
to
one
of
the
steps
describ
ed
in
Section
3.2.
For
exp
ository
pur-poses,
of
course,
weh
ave
remo
ved
an
um
be
rof
intermediate
steps
and
sho
w
only
selected
results.
A.1
Reading
the
Initial
Data
Base
W
eb
egin
with
the
data
base
in
curren
tuse
by
Customer
Supp
ort
Sp
ecialists.
This
data
base
(actually
,an
informal
textual
\notes
le")
consists
of
ab
out
150
en
tries.
A
typical
en
try
is
sho
wn
in
Figure
1.
At
this
stage
we
not
only
read
these
en
tries,
but
we
tidy
them
up
a
bit.
For
example,
some
of
these
en
tries
were
incomplete
and
were
man
ually
expanded
into
multiple
cases
for
our
case
base,
resulting
in
200
entries
to
be
studied.
The
primary
purp
ose
of
this
initial
reading
isto
create
alist
ofall
of
the
tests
that
are
explicitly
men
tioned
by
the
sp
ecialists
in
their
explanations
of
the
resolv
ed
cases.
From
the
case
presen
ted
in
Figure
1w
eextract
three
tests:
1.
Retriev
eP
rocess
PCB
2.
Chec
kJ
IB
Address
3.
Chec
kCoun
t
For
this
particular
data
base,
we
deriv
ed
an
initial
set
of
ab
out
100
tests.
A.2
Grouping
the
Tests
Reading
through
this
data
base
led
to
anatural
decomp
osition
of
the
tests
by
grouping
them
according
to
data
structures
men
tioned
in
the
cases.
Th
us,
we
group
ed
tests
related
to
the
JIB
(job
information
blo
ck)
together,
joining
the
tests
\Chec
k
JIB
Address"
and
\Chec
kC
oun
t"
from
the
case
sho
wn
in
Figure
1.
In
this
particular
data
base,
we
group
ed
the
tests
into
roughly
sev
en
sets
of
tests.
By
con
trast,
ino
ur
work
with
the
WPS-PLUS
data
base,
the
natural
group-ing
was
along
functional
lines.
While
weh
ave
no
rm
evidence,
it
seems
lik
ely
that
suc
h\natural"
groupings
will
occur
in
most
sizable
data
A.2
Grouping
the
Tests
15
Version
ofs
ystem:
VAX/VMS
V4.2
Reason
for
bugc
hec
kexception:
ACCESS
VIOLA
TION
Pro
cess
curren
tly
executing:
SP=>
8018E7AC
00000004
8018E7B0
7FFE7DE4
8018E7B4
FFFFFFFD
8018E7B8
0000026A
8018E7BC
00000000
!THIS
S
HOULD
H
AVE
B
EEN
O
N
PREVIOUS
LINE
!AND
VISA
VERSA
8018E7C0
00000001
8018E7C4
00000005
!BEGINNIN
GO
F
S
I
G
N
A
LA
R
R
A
Y
8018E7C8
0000000C
!ACCES
VIOLATION
8018E7CC
00000004
!WRITE
ACCESS
PROTECTIO
N
8018E7D0
00000020
!VA
8018E7D4
80004A4E
IOC$BUFPO
ST+
017
8018E7D8
04040000
8018E7DC
80004A6D
IOC$BUFPO
ST+
036
8018E7E0
800CD108
DZDRIVER+
118
the
reason
for
this
bugc
hec
kis
because
the
pid
eld
ofthe
IRP
has
been
zero
ed.
IN
post
pro
cessing
the
system
tries
to
giv
ethe
buered
byte
coun
tb
ackt
o
the
pro
cess
that
did
the
io.
HE
doe
sthis
by
using
the
PID
eld
oft
he
irp
to
index
into
the
pid
table
then
gets
the
PCB(n
ull
pro
cess
in
this
case)
and
gets
the
JIB
address.
THE
JIB
address
for
the
null
pro
cess
is0
.T
HE
system
then
go
es
to
the
JIB$L
BYTCNT
oset
(20)
from
the
base
of
the
JIB
(0
in
this
case)
and
tries
to
add
bac
ki
nt
he
byte
coun
t.
IN
this
case
we
get
an
access
violation.
ha
ve
resp
onse
from
VMS
dev
elop
emen
tthat
there
is/w
as
ab
ug
xed
in
4.2
it
has
been
men
tioned
to
try
and
low
er
the
baud
rate
on
terminals
as
a
work
around.
Figure
1:
An
Initial
Data
Base
En
16
A
CREA
TING
THE
VMS
VALID
ATION
MODEL
A.3
Rening
the
Test-Group
Structure
Armed
with
this
group
oftests,
we
returned
tot
he
sp
ecialists
who
had
prepared
the
data
base.
With
their
help,
we
learned
the
meaning
ofeac
hof
the
tests
we
had
iden
tied.
The
sp
ecialists
poin
ted
out
that
our
natural
groupings
didn't
corresp
ond
to
groupings
that
they
would
ha
ve
made
|
primarily
because
they
had
a(non-articulated)
set
of
abstractions
that
our
groupings
did
not
mak
eclear.
By
reviewing
our
groupings
they
were
forced
to
verbalize
the
abstractions
that
they
took
for
gran
ted,
and
wew
ere
able
to
sort
our
tests
into
15
groupings
that
reected
the
specialists'
notion
of
the
correct
abstractions
for
their
domain.
In
the
case
of
the
VAX/VMS
data
base,
the
specialists
typically
sub
divided
our
groupings
into
three
distinct
categories:
those
that
test
for
the
existence
of
the
exp
ected
location
in
memory
,those
that
tested
the
internal
consistency
of
the
data
structure,
and
those
that
prob
ethe
data
structures
for
spe
cic
value.
In
the
case
ofthe
tests
extracted
from
Figure
1,
\Chec
kJIB
Address"
is
in
the
rst
category
,the
sp
ecialists
poin
ted
out
for
the
need
for
an
additional
test
we
had
not
iden
tied
that
performed
the
consistency
test,
and
\Chec
kC
oun
t"
is
in
the
third
category
.
Th
us,
our
initial
mo
del
of
sev
en
structure-based
test
groups
was
rened,
through
interaction
with
specialists,
into
15
groups
with
ala
yered
structure.
Testing
alw
ays
pro
ceeds
from
one
layer
to
another,
starting
with
tests
for
exis-tence
ofthe
data
structure,
pro
ceeding
to
tests
on
the
consistency
ofthe
data
structure,
and
nally
on
to
probing
specic
values
within
the
data
structure.
A.4
Acquiring
Additional
Kno
wledge
This
step
is
primarily
an
iteration
of
the
earlier
one,
but
twou
nusual
items
arose
during
this
phase
for
the
VAX/VMS
system.
The
rst
was
the
realiza-tion
that
certain
terms
used
in
the
descriptions
were
not
reected
in
anyo
f
the
tests
our
groupings
we
had
already
deriv
ed.
In
this
particular
case,
there
was
con
tin
ual
use
of
phrases
lik
e\during
AST
deliv
ery
,"
and
\during
post-pro
cessing."
Examination
of
additional
material
(material
from
an
internal
DEC
course
suggested
by
the
sp
ecialists
as
a
goo
d
starting
poin
t)
rev
ealed
that
these
reected
the
pro
cessing
stages
of
the
system
wew
ere
diagnosing.
Our
initial
breakdo
wn
had
been
along
data
structure
boundaries,
but
an
or-thogonal,
equally
imp
ortan
tstructuring
is
possible
along
temp
oral
lines
based
on
these
stages.
It
is
these
two
alternativ
edecomp
ositions
that
we
referred
A.5
Integrating
Tests
into
the
Mo
del
17
in
Section
3.2
as
the
\structure
oft
he
domain."
Further
iteration
with
the
course
materials
and
the
specialists
rev
eals
a
muc
hric
her
structure,
consisting
of
multiple
lev
els
of
abstraction.
With
the
temp
oral
decomp
osition
tak
en
as
primary
,F
igures
2and
3sho
wt
wo
dieren
t
lev
els
of
abstraction.
Figure
2is
tak
en
at
avery
abstract
lev
el,
sho
wing
the
four
ma
jor
pro
cessing
stages
and
the
data
structures
used
at
eac
hstage.
Figure
3
is
a\close-up"
of
the
single
pro
cessing
stage
kno
wn
as
\AST
deliv
ery
."
A.5
Integrating
Tests
into
the
M
odel
Armed
with
am
uc
h-im
pro
ved
mo
del
of
the
domain,
we
built
aseman
tic
net-work
capturing
as
muc
hof
this
mo
del
as
bears
directly
on
the
abilit
yto
select
and
apply
the
tests
and
test
groups.
In
the
VAX/VMS
system,
our
validation
mo
del
reects
both
decomp
ositions
along
data
structures
and
along
pro
cess-ing
stages.
The
seman
tic
net
work
has
15
nod
es,
for
eac
htest
group,
77
no
des
that
represen
tk
nowledge
ab
out
the
pro
cessing
stages
of
device
driv
ers,
and
22
no
des
that
represen
tk
nowledge
ab
out
data
18
A
CREA
TING
THE
VMS
VALID
ATION
MODEL
LEGEND:
data structure
test
test groups
IRP
UCB
PCB
valid
PCB?
processing
device
driver
postprocessing
AST Delivery
preprocessing
starting
address
starting
address
valid
IRP?
starting
address
valid
UCB?
Figure
2:
Device
Driv
er
Validation
Mo
del:
Abstract
A.5
Integrating
Tests
into
the
Mo
del
19
AST Delivery
Break Internal PID
retrieve process PCB
null process PCB?
PCB$PID PCB Vector
first ACB ACB$ASTQBL JIB
PCB$ASTQBL PCB$JIB JIB$BYTCNT Check ACB
type
get ASTQBL Check JIB Valid
Return byte count Get JIB
address
JSB to AST routine REMQ AST
Enqueue ACB in AST queue
Table
Check
address JIB? byte count
Figure
3:
Device
Driv
er
Validation
Mo
del:
AST
Deliv
