1.1 A FRAMEWORK FOR AUTOM ATED VISUAL INSPECTION
6.2.3 The normal ruleset as a benchmark
T he m e a s u re m e n ts u sed to a ssess the q u a lity o f a ru leset can b e d e fin e d as o b je c tiv e and su b jectiv e. T h e fo rm e r m e a su re is ta k e n by m e a s u rin g th e en d c lassificatio n . T h e latter is an a sse ssm e n t o f th e ru le se t a c c o rd in g to th e five c rite ria o f sectio n 1.3.
T o m ak e an asse ssm e n t o f th e n o rm a l ru le se t it is n ecessary to c o m p a re th e resu lts o b ta in e d u sin g th e tra in in g d ata (fig u re 6 .2 0 ) an d the test d ata (fig u re 6 .19).
Correct
classificatbns Incorrectclassificatbns
Missed defects (escapes) D em p ster S h afer Sm ets W esley Defect type 12 3 4 5 6 7 Defect type 1 2 3 4 5 Defect type 20 No. 20 No. 1 2 3 4 5 6 7 Defect type
a
20 No.n
12 3 4 5 6 7 Defect type 1 2 3 4 5 • Defect typeDefect type Defect type Defect type
Non-defects classifbd as defects (Overkill) \ / 2 0 2 0 2 0 2 0
No. No. No. No.
0 0 O s 0 V 0 BV ^ Defect type 20 h No. 0 Defect type 20 20 20 20
No. No. No. No.
1 —
0 1 1 0 --- 1—1— ^ 0 □ . 0
67
Defect type
Key to Defect Types; l-> Fish-Eye 2 White-Spot 3 - > Gel 4 - > Oval 5 - > Line 6 ^ Dust 7 Fibre
Fig. 6.20 Classification data using norm al ruleset with training data.
Chapter 6 - The classification system
In general the ruleset classified the training data correctly. Dust was classified as overkill in all three decision criteria due to its similar appearance to white spots. All the results for this defect type had borderline results and the conflict in the evidence was high.
Using Smets’ pignistic transformation for the classification of fibres highlighted a problem in applying the insufficient reasoning principle. The principle assumes that everything can be described as singletons at an equal hierarchical level. This is not the case in this problem.
In figure 6.21 the leaf nodes have their respective pignistic values in brackets. Selection of the most likely using Smets’ algorithm means dust would be selected. An expert, however, would probably choose fish since the total evidence sums to 0.3. Smets doesn’t consider that decisions may be made where subjects are at different hierarchical levels.
Results from sample 1804 (training data)
I
BetP = 0.15 {red fish} BetP = 0.15 {blue fish} BetP = 0.22 {dust}
Defect Type
fish dust
(0.22)
red fish blue fish
(0.15) (0.15)
Fig. 6.21 Results showing problem encountered with insufficient reason principle
The several instances of ‘escape’ classifications using Wesley’s decision criteria were caused by the difficulty in selecting singletons from the evidence list. This was particularly evident for white spots which were generally difficult to differentiate from other defects.
On applying the ruleset to the test data the number of incorrect classifications for ovals, lines and fibres increased. In particular 45 of the 50 lines were classified incorrectly. This was influenced by two factors. The lines in the test data were wider than those in the sample data, which affected the primary rule for lines (width / length ratio low), and the lighting arrangement was not particularly robust for capturing line data. The images had an occasional pixel to the side of long objects making them appear ‘square’.
Ovals had several incorrect classifications due to their likeness with fish eyes. Their shape information is similar however ovals tend to be an order of magnitude bigger than fish eyes. Such a property enables a
Chapter 6 - The classification system
fairly easy distinction to the naked eye. The ruleset, however, did not take into account the physical size o f the defect. Originally the focus of the rules had been to use size invariant features since a concept of defect size did not exist.
The final area of incorrect classification was for defect type fibre which was mistakenly classified as dust. This was the result o f a bias towards dust in the ruleset under certain conditions (i.e. when fibres were positioned diagonally creating a square rather than thin outline - width and length are measured as maximum horizontal and maximum width).
Whilst the normal ruleset did not classify the training or test data perfectly, it provided a good test bed for performing further experimentation. The following section highlights the proposed modifications to the normal ruleset.
6.3 Experimental Design
This section specifies the tests under which the experimental data was observed. Each experiment was designed to investigate the properties outlined in chapter 1 defined as the ‘five criteria for an inspection system’.
All the experiments were based upon the normal ruleset of the previous section. The remainder of this section declares the experiment number and name, and provides a description o f the experiment along with the modifications made and the experiments objective. The results are summarised in chapter 7.
1. Increased Rules
In an industrial environment new sensor data or product data often becomes available. This experiment simulates the introduction of new data by defining a new data input - a ratio between dark field and bright field. Three new rules were added to implement the new source o f information. The input was modified
( BF - D F \
to allow for a new measurement: BFiDF ratio. --- .
\b f+d f)
Using the area values in the A threshold, if the output was 1 then the bias was to bright field, if to -1 then the bias was to dark field. The three new rules were:
90% confident that if ratio is low {fish, oval} 90% confident that if ratio is high (fibre, dust, white} 90% confident that if ratio is even {white}
Chapter 6 - The classification system
2. Increased Rules
Similar to experiment 1 however no new inputs used. The existing data available was used to add two new rules. The two new rules were:
75% confident that if BF A %AREA is low {fibre, oval} 75% confident that if BF_A W/L is low {line}
Objective: analyse effect o f adding new information as experience of ruleset increases.
3. Deleted Rules
Deleting rules enables the analysis of their effect on the ruleset. Both general and specific rules were tested. In experiment 3, rules 14 and 15 were deleted which both held specific classification data regarding fibres. By removing these rules the overall knowledge regarding fibres was reduced dramatically. The rules removed were:
Rule 14: if BF A W/L Square & BF_A %Area is low then {fibre} Rule 15: if BF A W/L is NOT Square then {fibre}
Objective: analyse effect of classifying contamination types.
4. Deleted Rules
As per experiment 3.
Rule 8 deleted. This was seen to be a general rule since the consequent had seven possible outcomes. It was necessary to analyse the effect of the rule deletion on all end classifications. The rule removed was:
Rule 8: if DF A W/L Square then { dust large white oval small fish large fish small gel large g e l }
Objective: test to see if removal has impact on any one class.
5. Deleted Rules
As per experiment 3.
Rule 1 deleted. The consequent here was quite specific. The possible outcomes were line or gel. Gels had always been classified accurately when this rule fired. This experiment therefore tested if it was a key rule. The rule removed was:
Rule 1: if No BF then { line small_gel large g e l}
Chapter 6 - The classification system
Objective: analyse if rule 1 is key for classifying gels and if so test to see if ruleset in general is affected by this key rule.
6. Norm al with changes
This experiment analyses how modifying information in a ruleset (rather than just adding new rules) affects the end classification. The existing ruleset remains the same but the consequent of one of the rules was modified. In general, this experiment seeks to make changes to the rules without adversely affecting the ruleset globally. Rule 14 was modified so the consequent could also be oval. The modified rule 14 was:
Rule 14: if BF A W/L Square & BF A %A LOW then { oval small fibre large fibre }
Objective: test to see how this rule and the key oval rule (10) interact.
7. Crisp not Fuzzy
This experiment tests to see if fuzzy numbers allow for a more general ruleset to be constructed. The fuzzy numbers in rules 5, 8, 9, 10, 13, 14 & 15 were reduced to crisp numbers. Since the fuzzy numbers had quite simple distributions the defuzzifying process replaced the ramps with a step change from 0 to 1
(false to true) at the midpoint o f the distribution. The values set in f u z z y A rg [ i ] were simply modified to the crisp format.
The graphs in figure 6.22 show the fuzzy region on the left and the resultant crisp number on the right.
1 Degree o f Membership 0 1 Degree o f Membership Possible Inputs 0 0.5 1 Possible Inputs
Fig. 6.22 Graphs to show change from fuzzy to crisp number at mid-point
Chapter 6 - The classification system
The normal code:
BF_A_WL_SQUARE DF_A_WL_LONG DF_A_WL_SQUARE DF_A_PA_LOW BF_A_PA_HIGH BF A PA LOW
The experiment 7 code:
BF_A_WL_SQUARE DF_A_WL_LONG DF_A_WL_SQUARE DF_A_PA_LOW BF_A_PA_HIGH BF A PA LOW C F u z z y N u m b e r (0.3, 0.7, 2, 3); CF u z z y N u m b e r (0, 0,01, 0,3, 0,6); CFuzz y N u m b e r : : r a m p L o H i (0,3, 1,0); C F u z z y N u m b e r (0, 0,01, 0,2, 0,5); CFuzzyN u m b e r : :r a m p L o H i (0,2, 0,8); CF u z z y N u m b e r (0, 0,01, 0,2, 0,8); CF u z z y N u m b e r (0,5, 0,5, 2,5, 2,5); CF u z z y N u m b e r (0, 0,1, 0,4, 0,4); CF u z z yNumber::r a m p L o H i (0,6, 0,6) CF u z z y N u m b e r (0, 0,1, 0,35, 0,35) CF u z z yNumber::r a m p L o H i (0,5, 0,5) CF u z z y N u m b e r (0, 0,1, 0,5, 0,5);
Objective: analyse the end classification after crispening of the antecedents.
8. Crisp not Fuzzy
As per experiment 7.
In this experiment the step change was implemented between the mid point used above and the true statement. The graphs in figure 6.23 show the fuzzy region on the left and the resultant crisp number on the right (note: on the y axis 1 is true and 0 is false).
The code below highlights the changes made to the f u z z y A rg .
The normal code:
BF_A_WL_SQUARE DF_A_WL_LONG DF_A_WL_SQUARE DF_A_PA_LOW BF_A_PA_HIGH BF A PA LOW
The experiment 8 code:
BF_A_WL_SQUARE DF_A_WL_LONG DF_A_WL_SQUARE DF_A_PA_LOW BF_A_PA_HIGH BF A PA LOW CF u z z y N u m b e r (0,3, 0,7, 2, 3); C F u z z y N u m b e r (0, 0,01, 0,3, 0.6); CF u z z yNumber::r a m p L o H i (0,3, 1,0); CF u z z y N u m b e r (0, 0,01, 0,2, 0,5); CF u z z yNumber::r a m p L o H i (0,2, 0,8); CF u z z y N u m b e r (0, 0,01, 0,2, 0,8); C F u z z y N u m b e r (0,6, 0,6, 2,25, 2,25); C F u z z y N u m b e r (0, 0,1, 0,4, 0,4); CFuzzy N u m b e r : :r a m p L o H i (0,8, 0,8); C F u z z y N u m b e r (0, 0,1, 0,3, 0,3); CFuzzy N u m b e r : :r a m p L o H i (0,65, 0,65); C F u z z y N u m b e r (0, 0,1, 0,35, 0,35); 142
Chapter 6 - The classification system Fuzzy Crisp 1 Degree o f Membership 0 1 Degree o f Membership Possible Inputs 1 0 Possible Inputs Degree o f Membership 0 Degree o f Membership Possible Inputs 0 .2 5 0 0 .5 Possible Inputs
Fig. 6.23 G raphs to show change from fuzzy to crisp num ber near true
Objective: analyse how changing the point at which the number is defuzzified changes the end classification.
9. Crisp not Fuzzy
As per experiment 7.
In this experiment the step change was implemented between the mid point used in experiment 7 and the false statement. The graphs in figure 6.24 show the fuzzy region on the left and the resultant crisp number on the right. The code below highlights the changes made to the f u z z y A rg .
The normal code:
BF_A_WL_SQUARE DF_A_WL_LONG DF_A_WL_SQUARE DF_A_PA_LOW BF_A_PA_HIGH BF A PA LOW
The experiment 9 code:
BF_A_WL_SQUARE DF_A_WL_LONG DF_A_WL_SQUARE DF_A_PA_LOW BF_A_PA_HIGH BF A PA LOW C F u z z y N u m b e r (0.3, 0.7, 2, 3); C F u z z y N u m b e r (0, 0.01, 0.3, 0.6); C F u z z y N u m b e r ::r a m p L o H i (0.3, 1.0); C F u z z y N u m b e r (0, 0.01, 0.2, 0.5); C F u z z y N u m b e r : :r a m p L o H i (0.2, 0.8); C F u z z y N u m b e r (0, 0.01, 0.2, 0.8); C F u z z y N u m b e r (0.4, 0.4, 2.75, 2.75); C F u z z y N u m b e r (0, 0.1, 0.5, 0.5); C F u z z y N u m b e r : :r a m p L o H i (0.5, 0.5); C F u z z y N u m b e r (0, 0.1, 0.4, 0,4); CF u z z y N u m b e r : :r a m p L o H i (0.35, 0.35); C F u z z y N u m b e r (0, 0.1, 0.65, 0.65);
Chapter 6 - The classification system Fuzzy Crisp 1 Degree o f Membership 0 Possible Inputs 1- Degree o f Membership 0 ° 0.5 Possible Inputs Degree o f Membership 0 1 Possible Inputs 1- Degree o f Membership o'.5 1 Possible Inputs
Fig. 6.24 Graphs to show change from fuzzy to crisp number near false
Objective: analyse how changing the point at which the number is defuzzified changes the end classification.
10. Flat hierarchy
This experiment has a similar purpose to those in experiment 7, 8 & 9. The ease o f use o f the ruleset shall be tested by restricting the language which can be used to describe the domain. Concepts such as large, small, non-defect and contamination can no longer be used. The frame of discernment here is described as parent types only (i.e. fish eye, white spot, gel, oval, line, dust, fibre and no defect). All the consequents o f the rules were modified to allow for the parent types only. For example the code below shows the consequent of the rule 8 as it was in the normal ruleset and how it was modified for this experiment:
Before:
Rule 8: if DF A W/L Square then { dust large white oval small fish large fish small gel large g e l } After:
Rule 8: if DF_A W/L Square then { dust white oval fish g e l }
Objective: analyse how the end classification is restricted by this limited terminology.
Chapter 6 - The classification system
11. Flat hierarchy
As per experiment 10.
The frame of discernment here is described at the leaf node level (i.e. gel is described as small gel and large gel). The code below shows the consequent of the rule 2 as it was in the normal ruleset and how it was modified for this experiment:
Before:
Rule 2: if No DF then { small white contam } After:
Rule 2: if No DF then { small white dust large fibre small fibre }
Objective: analyse how the end classification is restricted by this limited terminology.
12. Mr. Optimistic
The impact o f a rule can be controlled by the value set as the ruleConfidence. This experiment analyses the effect of making the belief in the rules more committed. This represents the information which might be obtained from an expert confident about their knowledge. Figure 6.25 shows the function used to represent this optimistic viewpoint. The existing ruleConfidence value was replaced by its respective ‘new value’. .0.5 ■y=x' 1 — N ew Value Original RuleConfidence 0 1
Fig. 6.25 Graph to show function used for Mr. Optimistic viewpoint
The new values for the ruleConfidence are shown below:
Rule: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Original: .90 .50 .90 .90 .75 .75 .75 .75 .90 .90 .90 1.0 .75 .50 .50 New: .95 .71 .95 .95 .87 .87 .87 .87 .95 .95 .95 1.0 .87 .71 .71 Objective: test the effect of the change in ruleConfidence on the outcome o f the rules.
Chapter 6 - The classification system
13. Mr. Pessimistic
As per experiment 12 except the rules become less committed. This represents the information from a less confident expert. Figure 6.26 shows the function used to represent this pessimistic viewpoint.
,y=x' 1 — N ew Value Original RuleConfidence 0 1
Fig. 6.26 G raph to show function used for M r. Pessimistic viewpoint
The new values for the ruleConfidence are shown below:
Rule: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Original; .90 .50 .90 .90 .75 .75 .75 .75 .90 .90 .90 1.0 .75 .50 .50 New: .81 .25 .81 .81 .56 .56 .56 .56 .81 .81 .81 1.0 .56 .25 .25 Objective: test the effect of the change in ruleConfidence on the outcome of the rules.
14. Multiple experts
This experiment looks at how different views of the world, Mr. Optimistic and Mr. Pessimistic, might be combined in one ruleset. The ruleset was split into dark field and bright field rules. It was assumed that Mr. Optimistic had knowledge o f dark field and that Mr. Pessimistic had knowledge of bright field. This assumption enabled the division o f their belief. The existing ruleConfidence values were updated to the respective ‘new values’ accordingly.
The new values for the ruleConfidence are shown below:
Rule: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Original: .90 .50 .90 .90 .75 .75 .75 .75 .90 .90 .90 1.0 .75 .50 .50 New: .81 .71 .81 .95 .56 .87 .87 .87 .95 .95 .88 1.0 .56 .71 .25 Objective: test the effect of this multiple knowledge source by analysing the end classification.
Chapter 6 - The classification system
15. Multiple experts
As per experiment 14, except Mr. Optimistic had knowledge o f bright field and that Mr. Pessimistic had knowledge o f dark field.
The new values for the ruleConfidence are shown below:
Rule: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Original: .90 .50 .90 .90 .75 .75 .75 .75 .90 .90 .90 1.0 .75 .50 .50 New: .95 .25 .95 .81 .87 .56 .56 .56 .81 .81 .88 1.0 .87 .25 .71 Objective: test the effect of this multiple knowledge source by analysing the end classification.
6.4 Sum m ary
This chapter described the classification scheme used with the experimental machine vision system. The following points can be summarised:
1. A classification scheme was written in C++ which enabled uncertainty techniques to be applied to this inspection problem. Classes to work with fuzzy expressions and calculate beliefs using Dempster Shafer were included in the classification scheme. Methods proposed in the literature by Smets and Wesley for selecting the most likely defect type using the evidence collected using Dempster Shafer were also included to enable an assessment o f classification results. The design o f the classification