Lung Function Computational Studies

As with most major medical fields, lung function has been the subject for expert system development and knowledge discovery. The most well known of these is the

PUFF expert system for interpreting lung function data, but this has been followed by other studies and developments.

Expert Systems

While PUFF is the most well known expert system for lung function, at least in research literature, there have been more recent systems developed. One example is Pulmonary Consult, a commercial product from the Medical Graphics Corporation (MedGraphics, 2011). As it is a commercial product little detail is available on its development and content; however it is known to have been built upon the knowledge base from PUFF and so largely covers the same area (Thomson, 2009). PUFF

PUFF was developed in the early 1980s as a test of the Essential MYCIN (EMYCIN) framework, which was a generalisation of the MYCIN expert system such that it could be applied to different domains. PUFF was deployed in the Pacific Medical Centre in San Francisco to assist pulmonary physiologists in interpreting the results of patient lung function tests, by taking in spirometry, lung volume, and diffusing capacity test results and returning interpretations based on the rules in its knowledge base.

The reasons for the expert system being developed in the lung function field were many: the interpretation of lung function tests is a daily problem, and so fills a need; the interpretation task was complex enough to be challenging; the lung function data was mostly self-contained, not requiring large amounts of data apart from that gathered in the lung function tests; there was available data; expert interpretations tended to be phrased similarly; and there was significant tedious work involved for the experts in generating reports.

It used classification rules, an inference engine, a knowledge acquisition module, and an explanation module. The system would function by asking the user, a lung function expert, a series of questions about the current case, thereby building the data about the current case. Once received, it would infer from that data and the rules in its knowledge base interpretations, which it would respond with. Over 4 years, the system interpreted over 4000 cases, providing interpretations for approximately 10 patients each day in use. Figure 2-8: shows a sample of the output

from PUFF’s interpretation, following a standard lung function report format where possible.

Figure 2-8: PUFF sample report output (Aikins, et al., 1983)

It was concluded that PUFF was a ―practical assistant to the pulmonary physiologist‖ (Aikins, et al., 1983), as it had the support of hospital staff and administration and was in daily use. However, areas for improvement were noted. The system lacked the ability to identify prototypical patterns; there was difficulty involved in adding new knowledge to the system, as the addition of a new rule may affect the behaviour of existing rules in unexpected ways; there were problems with the order that data was requested; and it lacked the ability to adequately explain the results that were reached (Aikins, et al., 1983).

Pulmonary Consult

Pulmonary Consult is another expert system for assisting in the interpretation of lung function test results. It is a commercial product from the Medical Graphics Corporation (MedGraphics, 2011), and as such little detail is available on its development and content; however it is known to have been built upon the knowledge base from PUFF and so largely covers the same area (Thomson, 2009). It has been available since the 1980s and is used in many clinical settings.

Knowledge Discovery

There have been surprisingly few applications of data mining and knowledge discovery to the field of lung function. Numerous studies have been performed in highly specialised areas of lung function, such as analysing a thoracic lung cancer database (J. Goldman, Chu, Parker, & Goldman, 2008), an association study between gene variations and bronchopulmonary dysplasia attempting to find the causes of that one lung disease (Rova, et al., 2004), and another data mining study into a dataset of a specific lung abnormality (solitary pulmonary nodules) (Kusiak, Kern, Kernstine, & Tseng, 2002). Other studies have also been performed on data related to lung function, such as a case based reasoning approach to automatically building a classifier for molecular biology, which also tested over a lung microarray dataset (Arshadi & Jurisica, 2005). However, there has been very little work into broader attempts to analyse lung function test data, and almost no exploratory data mining: all data mining studies in lung function seem to be explanatory in nature, trying to find detailed reasons for specific events or phenomenon.

Exposed MCRDR

An approach which combined MCRDR knowledge acquisition, data mining and expert-driven analysis was developed in 2006 (Ling, 2006). The method, given the name Exposed Multiple Classification Ripple-Down Rules (EMCRDR), was based on the premise that the MCRDR methodology would allow the acquisition of a strong knowledge base. From that base, experimental hypotheses could be added as new ―knowledge‖, which would then be validated (or not) through the MCRDR validation process: allowing an exploratory approach to knowledge discovery. The validation process would use a dataset to provide evidence for the hypothetical knowledge, point out the inconsistencies, and assist in developing the hypothesis

until it was compatible with existing knowledge and data. It also suggested that extra validation mechanisms might be added to allow the expert to further verify that their hypotheses were supported by the data, and a rudimentary data mining feature was added that could either assist in defining rule conditions to match a group of cases, or could identify the cases that matched conditions defined by the expert.

Modifications to the MCRDR process

The study contained a few small but significant modifications to the basic MCRDR approach to facilitate the knowledge discovery application. The most significant of these modifications was to allow the expert free access to view and modify the knowledge base, to the extent of being able to edit or delete existing rules. This is in direct contrast to the traditionally accepted wisdom in RDR development that the knowledge base only ever be added to, never edited or deleted from (Compton & Edwards, 1994). Exception rules and stopping rules provide all the functionality of editing and removing without invalidating the context of any existing knowledge (Kang, 1996).

The second significant departure from a normal MCRDR implementation was the addition of a dataset, which caused the cornerstone case model to be much different: rather than the cornerstone cases being any previously seen case which matched a rule when the rule was made, the EMCRDR system maintained a list of all classifications for all cases in the dataset. When the expert was defining a rule, all cases matching the rule would be displayed, and this was used to provide validation for the rule.

Impact of EMCRDR modifications

The EMCRDR study tested a small dataset of approximately 400 cases, with one expert, in the domain of lung function. While it found evidence to suggest that the EMCRDR approach worked, the study was far from conclusive (Ling, 2006). It also made no conclusions as to how well the method worked. It did however highlight many areas for potential improvement; in particular, the study demonstrated the effects of the modifications to the base MCRDR approach and how they might be better adapted and applied.

It was found that while allowing the expert to view the knowledge base provided a relatively effortless way of expressing the existing knowledge, it caused a shift of focus from a case-based expression of knowledge to a rule-based one; and unfortunately this shift invalidates many of the advantages of the MCRDR approach. It requires the expert to understand precisely how the knowledge base works in order to add rules correctly, which is an unrealistic expectation. Given the inevitable restrictions on expert time mentioned previously, it is in most instances impractical, if not impossible, to take the time to teach the expert exactly how the rules they define inter-relate. Depending on how the knowledge base was built, understanding exactly how the rules combine and what applies in any given instance can be a very difficult task regardless of how familiar the person is with the MCRDR method. Supporting this, the study reported that the expert struggled with determining exactly what rules should be applied where to achieve the desired results (Ling, 2006). This was attributed to the contrast between the traditional MCRDR implementation style which hides the structure of the knowledge base, and the more explicit style used in places to show the structure of the knowledge base. The combination of these conflicting styles and the inherent problems with the expert understanding how the knowledge base works resulted in confusion from the expert on how to add rules, which type of rule to use, and how to solve errors in the knowledge base.

As a direct consequence of this confusion, the ability to edit and delete rules was very rarely utilised, and mostly to little effect. It was noted that the expert liked having the ability to edit rules and used it commonly to correct small mistakes; it is suggested however that had the expert had a better understanding of how to define rules to begin with, less errors would have required correction. Rule deletions were very rarely used, and seemed to offer no real benefit over the normal stopping rules, and in fact may have hindered progress as at least stopping rules could have provided an indication of the problems the expert was encountering.

A related issue, not discussed in the study but possibly the underlying cause of much of the confusion, is that when the knowledge base is viewed as a single entity it is missing the integral components of context and evidence. Although a parent rule gives an outline of the context for its exception rules, without the context of the cases themselves there can be significant missing information. It has been noted by

many that the knowledge added to a knowledge base, in any form, is not concrete: it may (and is even likely to) change over time, and it may change when presented with a different context of application (Compton & Jansen, 1989). This means that when a rule is considered outside of the context it was made in, and without a framework of data showing how it is applied, there is a stronger chance that the expert will misunderstand the intention and application of the rule. Similarly, while the method in the study provided a list of all the cases that the new version of a rule covered, it gave no indication of which cases were no longer covered by the rule. As such, a case which the expert had previously decided was complete, and hence would be very unlikely to look at again, could now have different results. This again reinforced the rule-centred mode of thinking which was determined to be detrimental to the knowledge acquisition and discovery process.

A further problem raised in the study is that, because the expert is no longer considering individual cases until they are completed, it would be expected that the ability to derive tacit knowledge is reduced, or even removed completely (Ling, 2006). Traditionally an expert will consider one case at a time and continue working with that case until they are satisfied that it is completely correctly classified. However under a rule-focused approach, the expert will define rules for their most commonly used knowledge first, in the order that the knowledge occurs to them, without completing their current case. Unless they revert to a case-focused approach later, they will likely miss some of the rarer conclusions as the expert is unlikely to recall them from memory without prompting. Tacit knowledge – that is, knowledge which is difficult to define and express – will likely be missed completely as the expert is not given a situation requiring such knowledge.

This also has serious implications for the validation process. If there are no cases completely reviewed then indications of cornerstone case conflicts will be less likely and less meaningful. This problem was addressed in the EMCRDR study by the removal of traditional cornerstone case validation, and instead while a rule was being created, showing all the cases in the dataset that match the rule. The expert would then look through each of those cases to determine if the rule is correct. This is clearly an impractical solution for any sizable dataset, requiring the expert to look through and classify potentially hundreds, even thousands, of cases to check that each rule is correct. Also, for this validation strategy to be effective at all it requires

that the dataset be representative of the frequency and range of cases in the domain – and the typical way to ensure an unclassified dataset has these attributes is to use as large a dataset as possible. While a size balance might well be found between the two, it is a fundamental issue that any method which does not take advantage of all the data available will not be as effective as it could be. Also, the lack of cornerstone validation meant that the expert was required to examine and evaluate every case in the dataset which matched their new rule, in order to find those that invalidated the rule or provided additional information, if any existed.

However, the approach was found to achieve the desired goal, with the expert discovering new knowledge and being apparently satisfied with the method and the result. It did however highlight many areas for potential improvement, particularly in resolving the issues with rule-based thinking and errors in knowledge acquisition, misunderstanding the knowledge base, and the potential for providing data mining assistance to the user.