Rule Confirmation by Weight Analysis

The first part ot the rule extraction procedure has involved checking the sensitivity of the trained models in order to identity the significant input parameters that will have some effect on the predicted network outputs i.e. the spectral line size.

During network training, a variable selection technique implements a genetic algorithm that is able to select the most relevant input data fields and will reject the not so relevant ones. I he rejected inputs will show a lower frequency of occurrence (less than 50%) on the input data fields which could indicate to the user that ignoring that data field could produce a more accurate predictive model with a smaller architecture. The genetic variable selection technique is essentially a form of feature selection that will be very useful for pruning very large input data sets. Thus, making it easier to adopt the rule extraction procedure presented in this thesis for problem domains with larger input data sets.

As discussed before, since the final trained ANN species models were able to fit the complex nonlinear relationship between the controllable process variables and spectral line sizes with a minimum of three hidden units, an assessment of the weighted connections within the network architecture was a feasible consideration.

Extracting rules from single weight connections between units in an ANN can be dangerous due to the fact that network training involves arbitrary weight initialisation and modification within the context of the distributed learning that occurs within ANN's. Ultimately, unless the extraction procedure is based on discrete data with a single binary output (examples in Sestito 1993 and Shavlik 1997), then obtaining rules directly from single weight connections between units in the network architecture can be deceptive.

The weight assessment of the units in each trained ANN species model here employs a summed weight to individual network units backtracking approach rather than assessing only the individual weight connections.

By assessing the summed weights to individual hidden units in both the hidden layer and output layer, the network multi-outputs (i.e. spectral line sizes) were related directly to their input data. Testing the summed weights for each output unit by backtracking to the those inputs that contribute towards the weighted sum of the hidden units identifies a direct inputs-outputs relationship. The bias weights are particulaily significant as they act as the activation threshold which the summed weights to hidden units should exceed. It this condition is satisfied then contributory or inhibitory input effects on the network outputs can be determined.

The summed weight to a single hidden unit represents all the contributions from each input variable to that unit. I he summed weights at each hidden unit in the hidden layer are transformed by an appropriate transfer function. Each hidden unit represents a basis function whereby the weights to that unit correspond to the steepness of the basis function and the bias weight is the shift along the x-axis direction (of a two- dimensional x-y plot) ol the basis function. If the summed weight of the hidden units exceed their bias threshold via the nonlinear activation function, then the positive (and negative) weight contributions from inputs to that hidden unit are significant to relating specific input variables to the spectral line size.

4.8.1 Extraction Procedure by Backtracking

All seven species models have three hidden units in the hidden layer. The transfer (activation) function for the hidden layer is tanh. whilst the linear activation function acts on the output layer. The summed weights to individual units in the hidden and output layers are listed in Table 4.13 . The bias weights are listed for direct comparison with the summed weights to individual units in the hidden layer and output layer. The summed weights to hidden units that exceed the bias threshold, shown in bold, identify the hidden unit(s) to assess in terms of positive and negative weights (i.e. contributory or inhibitory) for determining the most significant inputs.

Once the hidden units whose summed weights exceed their bias threshold have been identified, then their contribution to the linear activation of the weighted sum outputs should relate the sizes of the spectral lines to the most significant inputs. This method of relating the outputs to the inputs via the hidden units is termed here as

backtracking. (Individual units (Hidden I Hidden 2 Hidden 3 Ar(420) Ar(750) | Ar(763) 1 | f * | « f

Activation sum wts. Bias (Individual units Activation sum wts. Bias ( 0.29 -0.03 0 57 0.46 0.26 0.251 tanh tanh tanh linear linear linear • (U6 -0.15 -0.31 0.07 -1 .0 4 0.03 ' -0.08 Hidden 1 -0.09 Hidden 2 -0.15(Hidden 3 0.29 CH(314) 0.26 CH(387) 0.231 CH(431) tanh tanh tanh linear linear linear -1.40 0.02 0.44 -0.13 0.46 -1 .6 2

(Hidden 1 tanh -0.23 -0.03(Hidden 1 tanh -0.03 0.27

iHidden 2 tanh -1.48 _{-0.20|Hidden 2 tanh -0.20 -0 13}

(Hidden 3 tanh -0.37 -0.08 Hidden 3 tanh -0.89 0.73

H(434) linear -0.88 0.23 CH+(395) linear 0.65 0.20

H(486) linear -0.01 0.20 CH+(422) linear -1 .4 8 0 37

I H(656) linear -1 .4 7 0.26|Hidden 1 tanh oTT 0.68

(Hidden 1 tanh 029 -0.28 Hidden 2 tanh 0.63 -0.34

(Hidden 2 tanh -1.62 0.53 Hidden 3 tanh -1.77 -0.23

(Hidden 3 tanh 0.05 -0.05 N2(337) linear 3 .0 5 0 33

H2(406)

I f /it * 7 \

linear

1 •

0 .2 9 0 .16| N2(389) _{linear -1.37 Q.30|}

H2(4 17) linear -0.45 0.17 Hidden 1 tanh -0.40 -0.49

1 H2(420) linear -1.63 0.19 Hidden 2 tanh 0.31 0.07

(Hidden 3 tanh -1.24 -0.21

I N2+(391) linear -0 .3 2 0.29

| N2+(427) linear -1.36 0 33

fable 4.13 Weighted sum to hidden units and output units

I he tanh activation function acting on the hidden layer is expressed as follows:

hidden output activation = tanh (X w ikXj + 0k) (2)

wik: value of weight connection from input to hidden unit (from i = 1 to n , n = no. of inputs)

Xj : input value (from i=l to n, n = no.of inputs)

0k : bias to hidden unit (from k=l to z, z = no. of hidden units)

I his method is effectively dealing with the summed weight, X w jkXj, to the hidden

units as the premise for rule activation. The summed weight to at least one unit in the hidden layer must exceed the threshold bias weight for this rule activation. This condition is satisfied for Ar, H2, CM, N2 and N2+. Bold highlights in Table 4.13, show

this for Hidden 1 tor Ar; Hidden3 for H2; Hidden2 for CH; Hidden2 for N2; and Hidden2 tor N2 . I he most positive and negative input contributions to these

significant hidden units are as follows:

Ar : Power - positive ; Methane flow rate - negative

Hydrogen flow rate. Power - positive ; Methane flow rate - negative : Methane flow rate and Power - positive ; Hydrogen flow rate - negative

Hydiogen flow rate , Pressure - positive ; Nitrogen flow rate - negative . Flow rates ot Methane, Hydrogen . Pressure - positive ;

Nitrogen flow rate - negative H2 :

CH N, : N

These contributory (positive) and inhibitory (negative) input parameters were determined by backtracking to the input layer from the summed weights of the most significant hidden units, identified in I able 4.13, for each trained ANN species model.

Hidden 1 | _{Hidden 2 Hidden 2}

1 Ar(420) 0.63 N2(337) _{1.22 CH(314) -0.42} Ar(750) 0.22| N2(389) _{-0.60 CH(387) 0.27} | Ar(763) 0.12| _{Hidden 2 CH(431) -1.07} Hidden 3 N2+(391) 0.59 H2(406) 0.8 91 N2+(427) -0.32 H2(4 17) 0.63 | H2(420) -0.781

Table 4.14 Output weights from Significant Hidden Units

The output weights from the significant hidden units are shown in Table 4.14 . The • •

activation of these output weights have contributed to the summed weight of the

multi-output units which determines the relative spectral line sizes.

Alter the backtracking process of determining the positive and negative input parameters directly from weighted values of the significant hidden unit connections to the summed weight outputs, the following rules were determined from the relation between output spectral line sizes and input process parameters.

I- IF (Increase Power) AND (Decrease Methane How rate) THEN (Increase Ar420, Ar750, Ar763 spectral 1 ine sizes)