EXPERIMENTAL RESULTS
5.3 Testing the Validation Model
This section presents the empirical evaluation of the algorithm presented in Chapter 3 by applying it to the application domain described above and by using the test environment just explained. First, section 5.3.1 classies the kind of er- rors that can occur when using the system. Then, it describes the experimental method utilized in the evaluation. Second, section 5.3.2 presents the results of evaluating the accuracy of the model and the eect of the criteria for detecting faulty sensors. This is done by evaluating the probabilistic phase of the system, i.e., the accuracy of the system for detecting faults. Then, section 5.3.3 presents the results of evaluating the two phases of the model: the detection and isolation of real faults.
5.3.1 Experimental method
The test environment receives as its input, a data set from the process. Then, the
o line module partitions the data set in two subsets: one partition for training the network, and the other partition for testing. The training/testing partition used was 70-30 % of the original data set, i.e., 610 instances for training the model (calculating the prior and conditional probabilities), and 260 instances for testing.
Theoretically, the system should always detect and isolate single faults cor- rectly. However, as mentioned earlier, in reality, some errors may occur since in practice it is unlikely that the dependency model will be perfect. Consequently, two types of errors could occur: a correct reading might be considered faulty, and a real fault might not be detected. These two possible errors are called type I and type II errors in the literature, and dened as follows [Cohen 1995]:
type II:
acceptance of the null hypothesiswhen it is actually false.The null hypothesis used (dened in Chapter 3) refers to the hypothesis that a sensor is working properly. Thus, in other words, type I errors occur when a correct sensor is reported as faulty while type II errors occur when faulty sensors are not detected. Table 5.2 presents the four possible cases.
Table 5.2: Dierent cases of the status of the hypothesis and decision taken. Choice hypothesis true hypothesis false
acceptance correct type II error rejection type I error correct
As described in the introduction of this chapter, the criteria for deciding if a reading is faulty or not can result in a trade o between these two types of errors. At the end of Chapter 3, the following two criteria were mentioned:
1. Calculate the distance of the real value from the expected value, and map it to faulty if it is beyond a specied threshold and to correct if it is less than a specied threshold.
2. Assume that the sensor is working properly and establish a condence level at which this hypothesis can be rejected, in which case it can be considered faulty. This condence level is known as the p value.
The accuracy of the model, i.e., the proportion of type I and II errors, is evaluated by varying the possible thresholds for each of these criteria.
A testing session includes the following steps: 1. Obtain a random partition of the data set. 2. Run the o line module.
4. Run the in line module utilizing the instantiation data set. This test cor- responds to a simulation with no errors.
5. Modify the instantiation data set to insert a single failure in one sensor, in every one of the 260 lines of the le.
6. Run the in line module again.
7. Compare the results obtained with the expected results, and generate a results table.
Step 5 introduces the simulated failures and requires further explanation. A single line of the testing le includes the readings of all the sensors considered. In every line, one sensor is modied in order to represent an erroneous reading. The rst line modies the rst sensor. The second modies the second, and so on, until all the sensors have been modied. This operation is repeated, until all the lines, starting with the rst sensor, have been edited. Two dierent faults were simulated:
Severe.
The value modied is the most distant extreme value, i.e., if (maximum value - real value) is greater than (real value - minimumvalue) then the real value is substituted by the maximumvalue, and by the minimumotherwise.Mild.
The real value is replaced by one which diers by 25 %. This test procedure was used to evaluate:the accuracy of the validation phase, and the accuracy of the isolation phase.
The validation phase is an intermediate phase of the model that determines if a sensor is potentially faulty. It is therefore important to test its accuracy as well as the accuracy of the isolation phase. The following two subsections present the results of these two evaluations.
5.3.2 Accuracy of the probabilistic validation phase
To evaluate the accuracy of the validation phase, two experiments were carried out. First, a process with no faulty sensors was simulated and the number of faulty sensors incorrectly found (i.e., type I errors) was determined. Second, single failures were simulated as described in the test procedure in section 5.3.1, and the number of type I and II errors determined.