PERFORMANCE EVALUATION OF THE INFERENCE STRUCTURE IN EXPERT SYSTEM Yang Tao He Zh A r t i f i c i a l I n t e l 1 Department of Computer Zhejlang U n i v e r s i t y , ABSTRACT
There are a v a r i e t y of f a c t o r s that a f f e c t the performance of an expert system. This paper presents an e v a l u a t i n g method f o c u s i n g on the c o m p l e x i t y , costs and e f f i c i e n c y of
i n f e r e n c e s t r u c t u r e in a r u l e / Knowledge-source based expert system. The performance measuring f a c t o r s proposed here, when fedback to Knowledge engineers, can help them gain a q u a n t i t a t i v e understanding of reasoning performance of Knowledge acquired at the ( e a r l y ) stages of
the development l i f e c y c l e , and enable them to make a numeric comparison and a choice among several s t r u c t u r e s , or have some r e f i n e m e n t s . This paper also presents some a p p l i c a t i o n examples of t h i s e v a l u a t i n g method and describes a n e v a l u a t i n g f a c i l i t y based on i t .
I INTRODUCTION
In the l a s t decade, expert systems and t o o l s f o r b u i l d i n g them have been being developed at a s u r p r i s i n g l y r a p i d r a t e [Waterman, 1985]. This makes e v a l u a t i o n of expert system more and more important in i t s development process. As p o i n t e d out in [Gaschnig, et a l . , 1983], e v a l u a t i o n s pervade the expert system b u i l d i n g process and are c r u c i a l f o r improving system design and performance, but the e x i s t i n g techniques f o r e v a l u a t i n g these systems are few and p r i m i t i v e . C e r t a i n l y many e v a l u a t i n g methods used in other software systems [Shneiderman, 1980] could be a p p l i e d to expert systems, but an expert system is unique in the sense that it is a Knowledge-based system c o n t a i n i n g human e x p e r t i s e , and has many d i s t i n g u i s h i n g f e a t u r e s such as h e u r i s t i c s e a r c h i n g , r e a s o n i n g , high l e v e l r e p r e s e n t a t i o n , e t c . . T h e r e f o r e , expert system should have i t s own e v a l u a t i n g techniques.
Gaschnig and others in t h e i r paper
i j u n Yu Ruizhao igence Laboratory Science and Engineering
Hangzhou, P.R. China
have presented many general e v a l u a t i n g c r i t e r i a concerning on what, how, and when the e v a l u a t i o n should be .performed. Our approach , on the other
hand, emphasizes the performance of i n f e r e n c e s t r u c t u r e which means c o m p l e x i t y , costs or e f f i c i e n c y in the context of t h i s paper, i . e , the degree o f d i f f i c u l t y , e f f o r t , o r the reasonable time f o r a system w i t h such an i n f e r e n c e s t r u c t u r e to a t t a i n c e r t a i n g o a l s . As was proposed in [ N e w e l l , 1982], there does e x i s t a d i s t i n c t computer system l e v e l , the Knowledge l e v e l , l y i n g immediately above the program l e v e l . A c c o r d i n g l y , our p o i n t i s that t h i s e v a l u a t i o n is performed at the Knowledge
l e v e l whereas the others f o r common software systems are performed at the program l e v e l .
J. McDermott, et al.CMcDermott, et a l . , 1978] have f i r s t made an approach and presented several e x c i t i n g r e s u l t s on the e f f i c i e n c y of c e r t a i n p r o d u c t i o n system implementations f o c u s i n g on impacts of c o n f l i c t set f i n d i n g s t r a t e g i e s . The e v a l u a t i n g method presented here is towards the
i n f e r e n c e s t r u c t u r e i t s e l f , as the reasoning network c o n s t i t u t e d by the i n t r i n s i c r e l a t i o n s h i p s of Knowledge has a great i n f l u e n c e on the performance of expert system.
The a p p l i c a t i o n s i g n i f i c a n c e of t h i s e v a l u a t i n g method has been shown in the sense that it helps Knowledge engineers
(1) o b t a i n a q u a n t i t a t i v e measure of inference performance of acquired Knowledge. Example 1, f o r i n s t a n c e , gives the e v a l u a t i n g r e s u l t of an expert system.
(2) make a numeric comparison of performance among several s t r u c t u r e s
( n o r m a l l y w i t h the same f u n c t i o n in problem s o l v i n g ) . For i n s t a n c e , Example 2 analyzes the impacts of a g e n e r a l i z a t i o n s t r a t e g y on the i n f e r e n c e performance, and Example 3 compares two mechanisms f o r p a t t e r n c o n s i s t e n c y c h e c k i n g .
i n f e r e n c e s t r u c t u r e w i t h o u t the d e s t r u c t i o n of the f u n c t i o n to enhance
the performance of the system, and speed up the r e a s o n i n g . For i n s t a n c e , Example 2 i n d i c a t e s the e f f e c t i v e n e s s of t h i s g e n e r a l i z a t i o n s t r a t e g y f o r knowledge r e f i n e m e n t , and Example 3 shows that a mechanism f o r p a t t e r n - c o n s i s t e n c y checking can be improved f u r t h e r .
The development of t h i s model is motivated by our work on expert system development environment [He, 1986], as knowledge a c q u i s i t i o n may be performed a f t e r or before the s e l e c t i o n of supported system b u i l d i n g t o o l s . Also in KMIX [Yang, e t a l . , 1986], i t i s p o s s i b l e to modify the c o n t r o l s t r a t e g y s t a t i c a l l y o r d y n a m i c a l l y . That i s why we need to make an e s t i m a t i o n on i n f e r e n c e s t r u c t u r e i t s e l f , and have a p r i o r p r o b a b i l i s t i c p e r s p e c t i v e o f performance i n c o n t r o l s t r a t e g y a p p l i e d . This w i l l enable us to have an approximate preview of performance at e a r l y stages i n the development l i f e c y c l e .
II THE NETWORK MODEL OP INFERENCE STRUCTURE
IV SCOPE AND EXAMPLES OF THE APPLICATION where q i s t h e mean p r o b a b i l i t y o f s u c c e s s f u l a c t i v a t i o n o f a t r a n s i t i o n , w h i c h i s a p p r o x i m a t e d a s 1 / b . I n t h e above s t a t e m e n t , t h e f i r s t i t e m d e n o t e s t h a t t { i s s e l e c t e d f i r s t , and t h e o t h e r i t e m s d e n o t e t i s s e l e c t e d a f t e r t h e u n s u c c e s s f u l a c t i v a t i o n s o f o t h e r s i n I ( x ) . Thus t h e p r o p a g a t i n g f a c t o r used, i n t h e r u l e l i s w e i g h t e d a s 1 ( 1 -1 / b ) . The f i n a l e v a l u a t i n g s t a g e i s t o compute p e r f o r m a n c e m e a s u r i n g f a c t o r s o f i n f e r e n c e s t r u c t u r e R . L e t t h e r e s u l t o f t h e above c o m p u t a t i o n b e I n t h i s e v a l u a t i n g mod c o r r e l a t e d w i t h t h e e f f o r t o f r e a s o n i n g h o r i z o n t a l l y i n m a t c h i n g , s e l e c t i o n , c o n f l i c t i o n r e s o l u t i o n , s w i t c h i n g , e t c . , w h i c h i s t h e most i m p o r t a n t f a c t o r a f f e c t i n g t h e r e a s o n i n g speed o f s y s t e m . The g r e a t e r BF is , t h e more e f f o r t is n e e d e d . DF and MDF, on t h e o t h e r h a n d , a r e measure o f t h e v e r t i c a l l y - a c c u m u l a t e d e f f o r t i n c h e c k i n g , m a t c h i n g , a c t i v a t i o n and a c t i o n e t c . . ! T ! and ! S i a r e o f t e n a s s o c i a t e d w i t h t h e s i z e o f s t o r a g e . 1FI i n v o l v e s t h e number o f p r i m i t i v e d a t a w h i c h need t o b e a c q u i r e d . Commonly, i t i s e a s i l y f o u n d i n most o f t h e k n o w l e d g e - b a s e d e x p e r t s y s t e m s t h e t r a c k o f r e a s o n i n g n e t w o r k o f t e n c o n s t i t u t e d v i a a s e t o f r u l e s o r k n o w l e d g e s o u r c e s . I t i s t h e r e f o r e t r a c t a b l e t o a b s t r a c t i n f e r e n c e s t r u c t u r e f r o m t h e m . P r o p e r t y 3 o f a n i n f e r e n c e s t r u c t u r e s t a t e d i n S e c t i o n I I seems to be a m a j o r l i m i t a t i o n of t h e a p p l i c a t i o n w h i c h e x c l u d e s t h e s e l f -r e f e -r e n c i n g i n f e -r e n c e t -r a n s i t i o n s i n t h e s y s t e m . When s u c h a s y s t e m l i k e MYCIN i s c o n f r o n t e d , t h e a d d i t i o n a l mechanism i s needed t o a v o i d r e a s o n i n g l o o p s , s o i s t h e p r o p a g a t i o n p r o c e s s s t a t e d i n S e c t i o n I I I t o p r e v e n t t h e c o m p u t i n g l o o p s . The s i m i l a r p r o b l e m has a l s o been e x p l o r e d i n [He&Yu, 1 9 8 6 ] , and t h e s o l u t i o n can b e t h e m a r k i n g method d i s c u s s e d i n [ B u c h a n a n & S h o r t l i f e , 1 9 8 4 ] o r o t h e r s . A n o t h e r p r o b l e m a r i s e s i n d e t e r m i n i n g t h e p e r f o r m a n c e f a c t o r o f t r a n s i t i o n s and t h e s i z e o f s t a t e , w h i c h r e q u i r e s t h e a b s t r a c t i o n i s p e r f o r m e d u n d e r a f i x e d s t a n d a r d . One o f t h e f u r t h e r d i r e c t i o n o f t h i s a p p r o a c h i s t o e x p l o r e t h e volume f a c t o r o f i n f e r e n c e s t r u c t u r e , w h i c h has been i n v e s t i g a t e d a s f ' ( l ) i n o u r w o r k , b u t i t s a p p l i c a t i o n s i g n i f i c a n c e r e m a i n s t o b e c l a r i f i e d . I t s h o u l d b e n o t e d t h a t a l t h o u g h t h i s method i s m o t i v a t e d f o r r u l e o r k n o w l e d g e - s o u r c e based r e a s o n i n g n e t w o r k , Example 3 b e l o w , h o w e v e r , i s t o e v a l u a t e two p a t t e r n - c o n s i s t e n c y c h e c k e r s , w h i c h i n d i c a t e s t h a t t h e i n f e r e n c e s t r u c t u r e i s a k i n d o f t h e c o g n i t i v e m o d e l i n g o f human r e a s o n i n g . The a p p l i c a t i o n scope o f t h i s method i s t h e r e f o r e e x t e n d e d f u r t h e r .
The f o l l o w i n g examples show t h e a p p l i c a t i o n s i g n i f i c a n c e o f t h i s m e t h o d . The p e r f o r m a n c e f a c t o r o f each
t r a n s i t i o n i s a s s i g n e d a s 1 , X f u n c t i o n
u n i f o r m l y s p e c i f i e s t h e domain and c o n t r o l k n o w l e d g e . A s e v e r y k n o w l e d g e e n t i t y i n KMIX i s s p e c i f i e d a s a s t a t e m a c h i n e w i t h c o n s t r a i n t s , i t can b e d i v i d e d i n t o a c o l l e c t i o n o f s u b s t a t e s , t h e n k n o w l e d g e s o u r c e s c o n s t i t u t e t h e i n f e r e n c e r e l a t i o n s h i p s b e t w e e n t h o s e s t a t e s . EVAL a c c e p t s a c o l l e c t i o n o f k n o w l e d g e e n t i t i e s , and a b s t r a c t s a n i n f e r e n c e s t r u c t u r e f r o m t h e m , t h e n c a l c u l a t e s i t s p e r f o r m a n c e measure f a c t o r s . ACKNOWLEDGMENTS
We t h a n k Wang S h e n k a n g , Gao J i , Cao X u e j u n , Z h a n g w e i , Wang W e i n i n g , Tong X u e j u n , Zhang L e i , and Tongming a t t h e A r t i f i c i a l I n t e l l i g e n c e L a b o r a t o r y o f Z h e j i a n g U n i v e r s i t y , and Hang J i a n g c h a o a t t h e Academic S i n i c a , f o r c o m m e n t i n g e a r l y d r a f t s and p r o v i d i n g v a l u a b l e d i s c u s s i o n s . REFERENCES [ 1 ] D.A. W a t e r m a n , A G u i d e t o E x p e r t S y s t e m , ( A d d i s o n - W e s l e y , M a s s . , 1985) [ 2 ] J . G a s c h n i g , P . K l a h r , H . P o p l e , E . S h o r t l i f f e , and A . T e r r y , E v a l u a t i o n o f e x p e r t s y s t e m s i s s u e s and case s t u d i e s , i n F . H a y e s - R o t h , D.A. Waterman and D . B . L e n a t ( E d s . ) , B u i l d i n g E x p e r t Systems ( A d d i s i o n - W e s l e y , M a s s . , 1 9 8 3 ) . , 2 4 1 - 2 8 0 [ 3 ] E . S h n e i d e r m a n , S o f t w a r e P s y c h o l o g y : Human F a c t o r s in Computer and I n f o r m a t i o n S c i e n c e , ( W i n t h r o p , M a s s . , 1 9 8 0 ) . [ 4 ] J . M c D e r m o t t , A . N e w e l l and J . M o o r e , The e f f i c i e n c y o f c e r t a i n p r o d u c t i o n s y s t e m i m p l e m e n t a t i o n s , i n D. A. Waterman and F. H a y e s - R o t h ( E d s . ) , P a t t e r n - D i r e c t e d I n f e r e n c e Systems ( A c a d e m i c P r e s s , NY, 1 9 7 8 ) , 1 5 5 - 1 7 6 . [ 5 ] A . N e w e l l , The k n o w l e d g e l e v e l , A r t i f i c i a l I n t e l l i g e n c e 1 8 ( 1 9 8 2 ) 8 7 -1 2 7 . [ 6 ] B . G . B u c h a n a n , E . H . S h o r t l i f f e ( E d s . ) , R u l e - B a s e d E x p e r t Systems The MYCIN E x p e r i m e n t s o f t h e S t a n f o r d H e u r i s t i c P r o g r a m m i n g P r o j e c t , ( A d d i s o n - W e s l e y , M a s s . , 1 9 8 4 ) . [ 7 ] C . L . F o r g y , R e t e : a f a s t a l g o r i t h m s f o r t h e many p a t t e r n s / many o b j e c t s p a t t e r n m a t c h p r o b l e m , A r t i f i c i a l I n t e l l i g e n c e 1 9 ( 1 9 8 2 ) 1 7 - 3 2 . [ 8 ] H e Z h i j u n and Y u R u i z h a o , ( E d s . ) , S e l e c t e d P a p e r s o n A r t i f i c i a l I n t e l l i g e n c e , Computer V i s i o n , and I n t e l l i g e n t CAD, ( A I L a b / Z h e j i a n g U n i v e r s i t y , C h i n a , 1 9 8 6 ) . [ 9 ] H e Z h i j u n , E x p e r t s y s t e m d e v e l o p m e n t e n v i r o n m e n t , i n [ 8 ] 1 - 2 1 .
CIO] Yang T a o , He Z h i j u n , and Yu R u i z h a o , Knowledge m o d e l i n g and t h e KMIX g e n e r a l - p u r p o s e k n o w l e d g e base s y s t e m , i n C8] 4 4 - 6 0 . [ 1 1 ] H e Z h i j u n , Gao J i , L u Z h i q i n g , and Ru Z h i m i n g , GPE: a g e o l o g i c a l p r o s p e c t i n g e x p e r t s y s t e m , i n t h e 3 r d N a t i o n a l C o n f . o n A p p l i c a t i o n o f m i c r o c o m p u t e r , N a n j i n g , C h i n a , O c t . , 1 9 8 5 . [ 1 2 ] J . L . P e t e r s o n , P e t r i Net T h e o r y and t h e M o d e l i n g o f S y s t e m s , ( P r e n t i c e - H a l l , N J , 1981) [ 1 3 ] W. Van M e l l e , A d o m a i n - i n d e p e n d e n t s y s t e m t h a t a i d s i n c o n s t r u c t i n g k n o w l e d g e - b a s e d c o n s u l a t i o n p r o g r a m s , STAN-CS-820, S t a n f o r d U n i v e r s i t y , 1980. [ 1 4 ] Zhang W e i , Yu R u i z h a o , and He Z h i j u n , Optimum n e t w o r k i n g i n many p a t t e r n s / many o b j e c t s m a t c h i n g , T e c h n i c a l R e p o r t , D e p t . o f Computer S c i & E n g r . , Z h e j i a n g U n i v e r s i t y , O c t . , 1 9 8 6 . [ 1 5 ] R . S . M i c h a l s k i , J . G . C a r b o n e l l and T . M . M i t c h e l l ( E d s . ) , M a c h i n e L e a r n i n g , ( T i o g a , CA, 1984) [ 1 6 ] K a t s u h i k o O g a t a , Modern C o n t r o l E n g i n e e r i n g , ( P r e n t i c e - H a l l , N J , 1 9 7 0 ) . 950 REASONING