CiteSeerX — Knowledge acquisition; Knowledge

(1)

The internet-based knowledge acquisition and management method to construct large-scale distributed medical expert systems

Hongmei Yan

^a

, Yingtao Jiang

^b,

*, Jun Zheng

^b

, Bingmei Fu

^c

, Shouzhong Xiao

^a

, Chenglin Peng

^a

aBioengineering Institute, Chongqing University, Chongqing 400044, China

bDepartment of Electrical and Computer Engineering, University of Nevada, Las Vegas, NV 89154, USA

cDepartment of Mechanical Engineering and Cancer Institute, University of Nevada, Las Vegas, NV 89154, USA

Received 27 June 2002 ; received in revised form 22 September 2002; accepted 21 October 2002

KEYWORDS Internet;

Knowledge acquisition;

Knowledge management;

Medical expert system;

Distributed client/server

Summary The Internet offers an unprecedented opportunity to construct powerful large-scale medical expert systems (MES). In these systems, a cost-effective medical knowledge acquisition (KA) and management scheme is highly desirable to handle the large quantities of, often conflicting, medical information collected from medical experts in different medical fields and from different geographical regions. In this paper, we demonstrate that a medical KA/management system can be built upon a three-tier distributed client/server architecture. The knowledge in the system is stored/managed in three knowledge bases. The maturity of the medical know-how controls the knowledge flow through these knowledge bases. In addition, to facilitate the knowledge representation and application in these knowledge bases as well as information retrieval across the Internet, an 8-digit numeric coding scheme with a weight value system is proposed. At present, a medical KA and management system based on the proposed method is being tested in clinics. Current results have showed that the method is a viable solution to construct, modify, and expand a distributed MES through the Internet.

1. Introduction

Clinical decision support systems (CDSSs) have great usefulness to medical practitioners, and design of such systems has been an active research area for

*Corresponding author. Tel.: +1-702-895-2533;

fax: +1-702-895-4075.

E-mail addresses: [email protected] (H. Yan), [email protected] (Y. Jiang), [email protected] (J. Zheng), [email protected] (B. Fu).

years. Since the ﬁrst publicly available medical expert system (MES)-MYCIN in 1970s[1,2], doctors and knowledge engineers have resorted to many CDSSs to generate clinical alerts, interpretations or diagnosis [3—5]. In the development of these knowledge-based systems (KBS), very often knowledge acquisition (KA) becomes a leading problem to efﬁciently build and expand knowledge bases with high accuracy. There are two KA methods that have been applied: automatic KA and manual KA.

doi:10.1016/S0169-2607(03)00076-2

(2)

Automatic KA method (also referred as Knowl- edge Discovering and Data Mining) is a relatively new technique. The most important step of an automatic KA is to induce abstract rules from a large number of cases. Neural networks, nearest neighbor methods, discriminant analysis, cluster analysis, linear programming, rough set and evolu- tionary algorithms are the most common automatic KA methods [6—11]. However, up-to-date the ef- ﬁciency of the automatic KA is still unsatisfactory due to over-complex algorithms and immature methodology [12]. For instance, data mining re- quires a large database to be used as a source of useful rules; however, these rules may come along with large sets of irrelevant or incorrect ones. A great deal of time and effort thus may be wasted as experts may end up with selecting a few trivial rules instead.

At present, most medical knowledge bases still rely on manual KA to acquire knowledge, although the knowledge bases constructed using this method are usually small, which deal with very specific and thus relatively narrow medical fields. Manual KA is usually accomplished through close cooperation between medical experts and knowledge engineers [13,14]. It has been long recognized that medical diagnosis is a complex cognitive process. Sometimes it is difficult, if not impossible, for medical experts to even formalize their knowledge and experiences. The accuracy of the acquired knowledge using manual KA tends to be questionable as the information is collected from a very limited number of medical experts with their expertise refrained in very specific and narrow fields. In addition, the speed of manual KA is quite low because significant amount of time and effort is required by both medical experts and knowledge engineers to create and maintain these knowledge bases. Furthermore, the knowledge bases constructed using manual KA method are not widely accessible. Therefore, it is extremely difficult for users outside the originating institutes to access and reuse the knowledge.

In contrast to traditional platforms, the Internet provides a more effective base for expert system delivery[15]: (i) The Internet is widely accessible;

(ii) Web-browsers provide a common multimedia interface; (iii) Several Internet-compatible tools for expert system development are available; and (iv) Emerging protocols support cooperation among expert system. In a word, it is more of a reality that by organizing medical experts worldwide to communi- cate/exchange/share their knowledge and experiences, globally available large-scale medical knowledge bases can be built in a timely manner. Not only can such knowledge bases contribute to the stan- dardization effort of medical information process-

ing, they can also, to a great extent, help avoid un- necessary duplication of work, which may lead to tremendous social and economic savings.

The goal of our work, therefore, is 3-fold: (i) to design a medical KA and management system to facilitate the creation and maintenance of medical knowledge bases, (ii) to maximize the sharing and reuse of the information among medical institutions and practitioners, and (iii) to ease the medical decision making of MES[4,5,16].

In this paper, we present an Internet-based KA/management method to construct large-scale medical knowledge bases. The medical knowledge collected from experts of different domains and from different regions through the Internet is stored/managed in three knowledge bases in ac- cordance with the accuracy levels of the acquired knowledge. The disease hierarchy is categorized using an 8-digit numeric coding scheme. Our test- ing system has been designed using Delphi 5.0 and Microsoft SQL server 2000 tools, and it has been put into test on the Internet for less than 1 year.

Current results show that the proposed method is capable of acquiring and managing large quantities of knowledge reliably and easily.

The rest of the paper is organized as follows. We will ﬁrst describe the coding scheme for knowledge representation inSection 2, followed by the organization details of the medical knowledge bases in Section 3. InSection 4, we present the architecture of the medical KA system. The implementation details of the proposed medical KA and management method are provided inSection 5. Conclusions are summarized inSection 6with information on future work.

2. Representation of medical knowledge

It has been shown that expert physicians organize knowledge on the basis of similarities between disease categories, forming numerous small worlds, which consist of small subsets of logically related diseases and their distinguishing features [17]. In the same token, medical knowledge can be organized in a similar way to facilitate medical doctors in making prompt and sound diagnostic decisions.

Knowledge bases constructed by simulating medical experts’ cognitive process, in the end, will consid- erably beneﬁt MES as well.

Recently, a three-character alphanumeric coding scheme (a single letter followed by two num- bers spanning from ‘‘A00’’ to ‘‘Z99’’) is adopted in revised International Classiﬁcation of Diseases, Tenth Revision (ICD-10) standard [18] to include many new disease categories. Compared with the

(3)

numeric codes (001—999) used in ICD-9 (ninth revision), the number of categories in ICD-10 has been signiﬁcantly expanded. If the disease categories continue to expand, one can expect that the future ICD standards may need to adopt a new coding scheme. As a result, instead of going to ICD-10, we adopt an augmented 8-digit numeric coding scheme as shown inFig. 1. This scheme, with sig- niﬁcant redundancy, shall accommodate enough disease categories with no concern to exhaust its capacity in a very long period of time.

2.1. 8-Digit numeric coding scheme

This 8-bit coding scheme (Fig. 1) rests on a tree-like structure to classify diseases in knowledge bases, and as such, it can efficiently represent the hierarchical nature of the disease classification. The first two digits (AA inFig. 1) describe different medical subjects. Take a few diseases caused by the blood and blood-forming organs as an example. 01 repre- sents internal medicine. Each medical subject in- cludes several disease categories, represented by the next two digits (BB in Fig. 1). For instance, 0101 is the code name for blood internal medicine.

Each disease category consists of several subsets (or diagnostic entities), which are represented by the following two digits (CC inFig. 1). Purpura with hemorrhagic conditions, for instance, is coded as 010106. The diseases in each subset are marked by the very last two digits (DD in Fig. 1). 01010601, for instance, is the digital representation of allergic purpura.

Note that this coding scheme can handle maximum 100 medical subjects; each subject can have as many as 100 disease categories. Each category is further divided into 100 subsets with each subset includes maximum 100 speciﬁc diseases. Together, theoretically speaking, we can cover as many as 10⁸ diseases. Compared with ICD-10, the proposed scheme should be sufﬁcient in dealing with today’s

Fig. 1 Coding scheme for knowledge representation.

knowledge level and future expansions. Note that the mapping between the alphanumeric code of IDC-10 and the proposed 8-digit numeric code can be easily performed through simple look-up tables.

2.2. Weight value system

For each speciﬁc disease, there are several symptoms or manifestations linked to diagnostic decision-making process. A weight system for the diagnostic importance of the disease manifestations can be applied to quantitatively and also collectively describe the associated symptoms or manifestations so that a sound diagnostic decision can be made for MES.

For a certain disease with a total of n associated symptoms or manifestations, the diagnostic decision can be made using the simple algebraic sum of the weight values:

_n

t−1W_i> T |W_i≤ 1|

W_i> 0 the ith symptom or manifestation supports the diagnosis

W_i< 0 the ith symptom or manifestation denies the diagnosis

(1) where T is the diagnostic threshold.

There are two types of methods to determine the values of Wi and T: (i) empirical methods, and (ii) adaptive methods such as Bayes’ method or neural networks, etc. One can predict that using the adaptive approaches, more meaningful and accurate weight values shall be obtained. However, this prediction is subject to further intensive study as assigning appropriate weight value to a symptom has been always a challenging problem. In [19], it has been indicated that the diagnostic accuracy rates of using empirical method are superior to those of using Bayesian algorithm, and it is thus the approach adopted in this work.

(4)

Following the scheme suggested in [20], for the symptom or manifestation supporting the diagnosis of a certain disease, four importance levels are used: low, medium, high, and critical, and their re- spective diagnostic weight values are suggested to be 0.2, 0.4, 0.6 and 1.0. On the other hand, for a symptom or manifestation denies the diagnosis of the disease, four importance levels are considered:

low, medium, high and critical, and their corresponding diagnostic weight values are −0.2, −0.4,

−0.6 and −1.0, respectively. The ﬁnal weight for each symptom or manifestation in one disease is determined by the authorized experts.

2.3. Structure of the knowledge representation

By using the described simple coding scheme and the diagnostic weight value system, disease diagnostic criteria subsystem can be assembled auto- matically and efﬁciently whenever needed. That is, this coding scheme can make the knowledge be eas-

Fig. 2 General outline of the medical hierarchy representation: I: The hierarchical classiﬁcation of diseases; II: The collection of symptoms and weights of manifestations to corresponding disease; W is the diagnostic weight value of corresponding symptom or manifestation for certain disease.

ily expanded, shared and reused by other institutes to satisfy their particular needs. A sketch of medical knowledge representation used in this study is provided inFig. 2.

3. Organization of medical knowledge bases

To efﬁciently manage the acquired medical knowledge, three knowledge bases (Fig. 3) have been established according to the accuracy level of the knowledge: (i) the medical core knowledge base (MCKB), (ii) the expert temporary knowledge base (ETKB), and (iii) the expert optimized knowledge base (EOKB). The knowledge in these knowledge bases is represented following the coding scheme described inSection 2.

In our system, medical experts are classiﬁed into three professional levels: (i) residents, (ii) attending, and (iii) advanced physicians. Depend- ing on the professional level of the medical ex-

(5)

Fig. 3 Organization of knowledge bases.

perts, these databases can be accessed through three corresponding modules: (i) experts’ KA module, (ii) experts’ knowledge optimization module, and (iii) medical core knowledge base renewal module.

3.1. Organization of MCKB

MCKB, characterized for its high knowledge accuracy, includes the information regarding disease classiﬁcation and diagnostic criteria/features, etc.

It serves as the foundation and core component in both the KA system and the overall MES. MCKB is made up of ﬁve logically relevant tables (Fig. 4):

(i) Tb subject, (ii) Tb group, (iii) Tb subset, (iv) Tb disease, and (v) Tb symptom.

Tb subject is created to store all information related to the medical subject at the top of the hierarchy. Each data entry of this table consists of three ﬁelds: (i) Subject id, (ii) Subject name, and (iii) Subject memo. Subject id, corresponding to AA in the coding scheme (Fig. 1), identiﬁes each med-

Fig. 4 Logical connections among Tb subject, Tb-group, Tb subset, Tb disease and Tb symptom tables in MCKB.

ical subject. Each medical subject has a unique Subject id. Subject name denotes the name of the medical subject, whereas Subject memo provides the general introduction/description of each associated medical subject.

Tables Tb group, Tb subset, Tb disease, Tb symptom in MCKB are constructed in the way similar to Tb subject. All these four tables are designed by ex- ploring the so-called Master/Detail relationship, as illustrated inFig. 4. For example, Tb group is linked to Tb subject by a pointer (Subject id) and this link- age is chained across all the tables. This method can clearly represent the hierarchical characteris- tics of medical ﬁelds, and facilitate the utilization of the acquired knowledge.

As there may exist n symptoms for a disease in Tb disease, n data entries may appear in Tb symptom with their corresponding Disease id ﬁelds all pointing to the same disease in Tb disease.

The ﬁeld weight in Tb symptom records the weight value of a symptom in supporting or denying a diagnosis, as detailed inSection 2.

(6)

Fig. 5 Logical connections among Tb group, Tb subset, Tb disease and Tb symptom tables in ETKB.

3.2. Organization of ETKB

ETKB consists of four tables (Fig. 5): (i) Tb group, (ii) Tb subset, (iii) Tb disease, and (iv) Tb symptom.

Note that Tb group, Tb subset, Tb disease and Tb symptom are also constructed following Mas- ter/Detail relationship (Fig. 5), with similar data ﬁelds as those shown inFig. 4.

For a certain disease or diagnostic criterion, several suggestions may exist in the ETKB. As a result, the medical knowledge in ETKB has to be repeat- edly checked, reviewed, and/or modified before it is accurate or matured enough to be moved up to EOKB. That is, unlike MCKB, the knowledge, and experts’ experiences and opinions stored in ETKB are frequently added, updated or even deleted by medical experts wired through the Internet. It is for this purpose that one additional field, remark, has been included into Tb group, Tb subset, Tb disease and Tb symptom, to flag any modification made by the medical expert(s). That is, if the medical record in ETKB is a newly-added item, then remark is set to 1; if the medical record has recently been updated, remark is set to 2. The remark is set to 3 once an expert has deleted the record. One additional field, exp name, traces the names of those experts who contributed to the modification process.

3.3. Organization of EOKB

EOKB shares a very similar structure as the one em- ployed in ETKB. The knowledge in EOKB can be further added into MCKB if it passes the clinical trials.

Otherwise, its accuracy will continue to be exam- ined. Consequently, the knowledge stored in EOKB and MCKB must be managed and maintained by a limited number of advanced experts with various expertises.

4. Architecture of the medical knowledge acquisition system

The medical KA system is built upon a distributed client/server architecture consisting of three tiers (Fig. 6): (i) server tier, (ii) middle tier, and (iii) client tier. The three-tier design supplies a tier of dis- tribution separate from the front-end presentation and the back-end data access. Such arrangement has many advantages over traditional two-tier or single-tier designs in terms of ease of load balancing, performance, ﬂexibility, robustness, and scal- ability.

The remote server (Server Tier) is a server for the application, which carries out all kinds of request of the application, and provides the following three key features to application on the network: (i) security management, which includes the management of access rights and operation rights of medical experts as well as the management of data security, (ii) ﬂow control, which can control the number of users and make a smooth data ﬂow between the server and the clients, and (iii) data retrieval, which handles the input and output of data.

The middle tier is introduced for the application logic, and it acts like an ‘‘agent’’ between the client and the server. The middle tier maintains three business logics, which can handle the problems of load balancing and database connection.

In addition, the three business logics also regulate different access rights to the knowledge bases for experts with different professional levels as well as the security, integrity and rightness of the acquired medical knowledge. The network trafﬁc between the clients and the middle tiers is drastically re- duced as all queries are generated by the business logics in the middle tier.

Client terminal (Client Tier) provides the interface (e.g. GUI and application-speciﬁc entry

(7)

Fig. 6 The structure of distributed Client/Server medical KA system.

forms/interactive windows) for medical experts to access to the remote database server so that the stored medical knowledge (data) can be manipulated, updated, and validated. Proper authoriza- tion/authentication procedures restrict medical expert to access to the speciﬁc part of the knowledge bases only related to his/her own expertise.

5. Implementation

A medical KA system has been implemented following the proposed method. We used Delphi 5.0 and Microsoft SQL server 2000[21]as the main de-

Fig. 7 The process of medical KA, addition and update in the proposed MES.

sign tools. We also chose popular TCP/IP protocols [22]for data communications. The interface at the client terminal has been specially designed to make it easily manipulated by the medical experts. The process of medical KA, addition and update is shown in Fig. 7. This system is currently operating very well as we continue to test and improve it with the assistance of clinic experts. Current results have shown that the completeness, integrity and consistency of acquired medical knowledge have been improved signiﬁcantly, and all three knowledge bases are growing in a healthy pace. In the following, three scenarios will be provided to reveal many design details of this work.

(8)

5.1. Scenario 1: how a junior medical practitioner contributes to the system

When a less experienced/knowledgeable medical practitioner, say a resident specializing in blood internal medicine, tries to login into the system from a client terminal, after been identiﬁed by the remote server about his identity, the remote server will send data related to blood internal medicine stored in MCKB to the client and establish a local temporary base. After reviewing the detailed clas- siﬁcation and the diagnostic criteria of blood internal medicine stored in MCKB, this junior doctor is entitled to perform the following operations:

• He/She can add, modify and delete the diagnostic indications according to his/her opinions and experiences through the GUI-based interactive window shown inFig. 8. Special caution needs to be exercised through a double conﬁrmation proce- dure when deleting a piece of information.

• He/She can create new disease groups, subsets and diseases. He/She then can add detailed information around these new nodes in the disease hierarchy.

• He/She can relocate certain groups, subsets and diseases in the current disease hierarchy to new positions.

Fig. 8 The GUI of medical KA module at the client terminal.

Fig. 9 The Login Monitoring window on the server side.

• He/She can obtain on-line help through the GUI-supported help function. There he/she can ﬁnd information regarding the system operations and the rules of assigning diagnostic weight values, etc.

On the other hand, an expert’s name and his/her expertise can be monitored on the server side when he/she logins into the system (Fig. 9). After he/she finishes the work, only the modified or newly added knowledge will be sent back to ETKB. In this way, the server will not experience the huge data traffic otherwise encountered, and the rightness and integrity of the medical knowledge in MCKB remain untouched.

(9)

5.2. Scenario 2: how a highly regarded expert contributes to the system

Continue from the previous example. After the first KA module, ETKB has stored significant amount of coarse knowledge and experiences in terms of blood disease classification as well as the diagnostic information. All the information, however, may be accurate or inaccurate. If one highly regarded medical expert in the same field, who has the authority to check, assess and modify these knowledge and experiences, logins into the knowledge optimization module, all the knowledge related to blood internal medicine stored in ETKB and EOKB are sent to the client terminal. The GUI interface this expert physi- cian may see is similar to the one shown inFig. 10.

This expert now is ready to perform following operations:

• He/She can move the knowledge in ETKB that is in consensus with his/her experiences and opinions to the EOKB by using Add function.

• On the other hand, if the knowledge is immature or he/she completely objects to, he/she can add his/her own knowledge by using the Add-myself function.

• If he/she wants to check the related knowledge in MCKB, he/she can use the Brow-MCKB function.

Fig. 10 The GUI of experts’ knowledge optimization module at the client terminal.

The related information in MCKB will be shown in a pop-up form (right bottom window shown in Fig. 10).

All the knowledge in temporary EOKB at the client terminal will be batch-updated back to EOKB in the server after the expert ﬁnishes his/her work.

5.3. Scenario 3: updating/expanding MCKB

Each particular medical ﬁeld in EOKB and MCKB is strictly managed and maintained by pre-authorized advanced medical experts pertain to his/her own expertise. If the diagnostic records for a certain disease in EOKB have passed the clinical trials, they can be moved into the MCKB. At the same time, the knowledge already in MCKB will be updated if its accuracy is challenged during the clinical trials. The interface of knowledge renewal module of MCKB is similar to the one used in expert’s knowledge optimization module (Fig. 10).

5.4. System test

This medical knowledge system has been tested with the assistance from the doctors afﬁliated with the Southwest Hospital located in Chongqing, China. In particular, eight residents and two expert

(10)

physicians, wired through the Internet connection, have actively participated in the test. In less than 1 year, we have observed signiﬁcant improvement of the medial databases in terms of the completeness, integrity and consistency of the acquired medical knowledge. At present, ETKB has included over 1000 diseases and the corresponding diagnosis manifestations, and the database is still growing.

We expect that this system will grow at an even faster pace if it is fully available in the Internet to more clinical experts worldwide.

Our test also shows that a friendly and uniform user interface at the client terminal is essential for an Internet-based KA system to succeed. This is par- ticularly true when this KA system involves a large number of doctors with very different skill levels in language and computer. Our system user interface has been improved signiﬁcantly by taking account of many suggestions and comments made by the par- ticipating medical doctors. The Help function, for instance, has been included to answer many FAQ’s a doctor may have when using the system. This function, however, is currently available in English only and support of other major working languages may soon become a necessity if the system is to be used in the Internet.

6. Conclusions

In this paper, we have presented both architec- tural model and the implementation details of an Internet-based KA method capable of acquiring and managing knowledge in a cost-effective manner.

The system is built upon a three-tier distributed client/server architecture. The knowledge in this system is organized/managed using three knowledge bases, with 8-digit encoding scheme and weight value system for knowledge representation. Further intensive test as well as the system improvement is still on the way to build a truly large-scale MES.

Acknowledgements

This work was supported partially by UNLV, Chongqing University, and the Southwest Hospital (Chongqing, China).

References

[1] E.H. Shortliffe, S.G. Axline, B.G. Buchanan, T.C. Meri- dan, S.N. Cohen, An artiﬁcial intelligence program to ad-

vise of physicians regarding antimicrobial therapy, Com- put. Bimed. Res. 6 (1973) 544—560.

[2] E.H. Shortiffe, B.G. Buchanan, A model of inexact reasoning in medicine, Math. Biosci. 23 (1975) 351—379.

[3] N.I. Birndorf, J.O. Pentecost, J.R. Coakley, K.A. Spackman, An expert system to diagnose anemia and report results directly on hematology forms, Comput. Biomed. Res. 29 (1) (1996) 16—26.

[4] L.M. Brasil, F.M. Azevedo, J.M. Barreto, Hybrid expert system for decision supporting in the medical area: complex- ity and cognitive computing, Int. J. Med. Inform. 63 (2001) 19—30.

[5] M. Suojanen, S. Andeassen, K.G. Olesen, A method for diagnosing multiple diseases in MUNIN, IEEE Trans. Biomed.

Eng. 48 (5) (2001) 522—532.

[6] B. Kovalerchuk, E. Vityaev, J.F. Ruiz, Consistent knowledge discovering in medical diagnosis, IEEE EMB Mag. 19 (2000) 26—37.

[7] J. Stempczynska, E. Kacki, Problems of knowledge acquisition automation in medical expert systems, Medinfo 8 (Pt.

1) (1995) 857—860.

[8] S. Tsumoto, Automated knowledge acquisition from clinical databases based on rough sets and attribute-oriented generalization, Proc. AMIA Symp. 1 (1998) 548—552.

[9] M.L. Wong, W. Lam, K.S. Leung, P.S. Ngan, C.Y. Cheng, Discovering knowledge from medical databases using evo- lutionary algorithms, IEEE EMB Mag. 19 (2000) 45—52.

[10] G.G. Yen, P. Meesad, Constructing a fuzzy rule-based system using the ILFN network and genetic algorithm, Int. J.

Neural Syst. 11 (5) (2001) 427—443.

[11] Y. Shahar, M.A. Musen, Knowledge-based temporal abstrac- tion in clinical domains, Artif. Intell. Med. 8 (3) (1996) 267—298.

[12] K.J. Cios, A. Teresinska, S. Konieczna, J. Potocka, S.

Sharma, A knowledge discovering approach to diagnosing myocardial perfusion, IEEE EMB Mag. 19 (2000) 17—

[13] S.L. Achour, M. Dojat, C. Rieux, P. Bierling, E. Lepage,25.

A UMLS-based knowledge acquisition tool for rule-based clinical decision support system development, J. Am. Med.

Inform. Assoc. 8 (4) (2001) 351—360.

[14] M.B. Amaral, Y. Satomura, M. Honda, T. Sato, A psychiatric diagnostic system integrating probabilistic and categorical reasoning, Methods Inf. Med. 34 (3) (1995) 232—243.

[15] R. Grove, Internet-based expert systems, Expert Syst.

17 (3) (2000) 129—135.

[16] R. Baud, C. Lovis, A.M. Rassinoux, P.A. Michel, J.R. Scher- rer, Automatic extraction of linguistic knowledge from an international classiﬁcation, Medinfo 9 (Pt. 1) (1998) 581—

[17] A.W. Kushniruk, V.L. Patel, A.J. Marley, Small worlds and585.

medical expertise: implications for medical cognition and knowledge engineering, Int. J. Med. Inf. 49 (1998) 255—

[18] ICD-10,271. http://www.who.int/whosis/icd10/index.html.

[19] S.Z. Xiao, C.L. Peng, Z.S. Wang, F.L. Wang, Using the algebraic sum method in medical expert systems, IEEE EMB Mag. 15 (1996) 80—82.

[20] S.Z. Xiao, Knowledge-Based Systems 2 (2000) America Aca- demic Press, chapter 21.

[21] F.G. Guerrero, C.E. Rojas, C. Rojas, SQL Server 2000 Pro- gramming by Example, Que, First Ed., 2001.

[22] B.A. Forouzan, S.C. Fegan, TCP/IP Protocol Suite, McGraw- Hill, 1999.