T-DTS self-tuning procedure validation using classification benchmarks

Fig. IV.26 : Validation T-DTS self-tuning threshold procedure, Average learning rate (including its corridor of the standard deviations) as a function of θ - threshold: 4 Spiral benchmark, 2 classes, generalization database size 500 prototypes, learning

database size 500 prototypes, DU – CNN, PU – PNN, Fisher measure based complexity estimator

tel-00481367, version 1 - 6 May 2010

Fig. IV.27 : Validation T-DTS self-tuning threshold procedure, Average

generalization rate (including its corridor of the standard deviations) as a function of θ - threshold: 4 Spiral benchmark, 2 classes, generalization database size 500 prototypes, learning database size 500 prototypes, DU – CNN, PU – PNN, Fisher

measure based complexity estimator

A very good illustration of how the self-tuning procedure works is done on the benchmark of Spiral classification benchmark. The Fig. V.26 – Fig. V.29 provide the details of the possible quasi optimum search for this specific classification problem.

Fig. IV.28 : Validation T-DTS self-tuning threshold procedure, Average clusters’

number as a function of θ - threshold: 4 Spiral benchmark, 2 classes, generalization database size 500 prototypes, learning database size 500 prototypes, DU – CNN, PU

– PNN, Fisher measure based complexity estimator

tel-00481367, version 1 - 6 May 2010

Intuitively and heuristically analyzing the trends, it’s expected to find quasi-optimal threshold in the subinterval [0.7 ; 0.8], where the generalization rate reaches it maximum and learning rate continue to grow. Now, when we come to the question of what is an optimal solution for us, it is time to determine its meaning in a term of performance estimating function P(θ) (equation III.12).

For this purpose I have set b1=3, b2=2, but b3=0. It was done in order to simplify the number of parameters that defines quasi optimum ignoring T-DTS executing time which proportional to the number of prototypes Fig. V.28.

Fig. IV.29 : Validation T-DTS self-tuning threshold procedure, Performance estimating function P(θ): 4 Spiral benchmark, 2 classes, generalization database size 500 prototypes, learning database size 500 prototypes, DU – CNN, PU – PNN, Fisher

measure based complexity estimator

On Fig. IV.29 described P(θ) – function evolution in the θ-threshold interval [0.1;0.8]

for 4 Spiral academic benchmark and fixed range of parameter. In is shown that for θ close to 0.7 T-DTS reaches its performance maximum (minimum of P(θ)=0.42) in a term of combination generalization and learning rates, where the high priority is set for overall maximizing of the generalization rate. More princely, using implemented into T-DTS self-tuning θ-threshold procedure, the minimum of P(θ) has been found for θ = 0.7217.

For different selected complexity estimators, DU and PU, the different combination of the satisfactory results is possible. These results are given in Table V.5. “Gr” stands for

tel-00481367, version 1 - 6 May 2010

generalization rate and “Lr” – learning rate. “Std” is the standard deviation of this parameters.

Table IV.5 : Classification results: 4 Spiral benchmark, 2 classes, generalization database size 500 prototypes, learning database size 500 prototypes

DU Complexity estimator PU Gr±Std/2 (%) Lr±Std/2 (%) Avr. leaf No. ±Std/2 Θ

CNN Collective entropy based Elman_BN 79.1583±0.4960 96.4870±0.1813 104.00±5.95 0.2798 SOM Collective entropy based Elman_BN 77.7956±1.6256 97.6846±0.2961 144.20±6.21 0.3353 CNN Fisher measure based PNN 80.4008±0.4216 95.8882±0.2505 176.2±1.98 0.7217

Based on given results, one may have selected one that which satisfies the given constrains better such as low standard deviation of the generalization rate, maximal possible generalization rate or satisfactory generalization rate, but maximal learning rate.

Let us stress that in a framework of T-DTS there is no a priory “the best”, the most

“optimal” solution. There is the bunch of possible quasi-optimal solution.

Fig. IV.30 : Validation T-DTS self-tuning threshold procedure, Clusters’ number distribution: 4 Spiral benchmark, 2 classes, learning database size 500 prototypes,

DU – CNN, Collective entropy based complexity estimator

Minimizing P(θ) - performance function inducts the configurations of the results where generalization and learning rates satisfies the user expectation. To obtain this minimum, different types of PU must be applied. The selection of appropriate PU is heuristic procedure that requires a user-experience. The given in advanced characteristics

tel-00481367, version 1 - 6 May 2010

of the problem and its type, and knowledge of PUs’ features are helpful for particular PU selection. The possibility to reaches the quasi optimum has been predefined by the form of histogram given on Fig IV.30.

Therefore, the form of histogram Fig IV.30, defines the divisibility of the initial database and if the complexity estimator defines the classification complexity in proper manner we have the appropriate histogram.

However, another key factor that determines the form of the histogram and that is present invisibly, each time when I build it, is the used decomposition. If the decomposition produces the sub-clustering in such manner that it doesn’t reduce complexity (regardless the problem), whatever prefect or not complexity estimator one has used, the histogram by its form exhibits this case. Concluding, the pair of DU and complexity estimator determines divisibility of the initial database.

Fig. IV.31 : Validation T-DTS self-tuning threshold procedure: 10 Stripe benchmark, 2 classes, generalization database size 1000 prototypes, learning database size 1000

prototypes, DU – CNN, PU – LVQ1, 4 complexity estimators

The Fig. IV.31 describes the results obtained for the classification benchmarks and then for real world problems (ANN based complexity estimator is marked under name ZISC). Hither, we start from two-class, 10 stripe benchmark problem Fig. V.31. The x-axis represents the decision threshold, the y-x-axis – the percentage of learning and generalization rate.

For a two-class benchmark problem, the quasi optimal thresholds for four complexity estimators were found in the range of [0.8591;0.9992]. Because of the benchmark artificiality, these optimal thresholds lay close to each other regardless of the complexity

tel-00481367, version 1 - 6 May 2010

estimator. For this sub-interval, the four complexity estimators achieve their maximums (98-99% of generalization for the proposed θ – thresholds). These results correspond to the result obtained in the work (Bouyousef, 2007).

The aim of this experiment was not to find the best complexity estimator in T-DTS framework or determine the best performance, but rather to test the self-tuning threshold procedure for the same range of fixed parameters as for the given classification benchmark including bi – priorities of P(θ).

The most interesting in these results is that defining h = 9 allows T-DTS in the user-free mode to find the quasi optimal θ - threshold with the high accuracy, particularly because of the continuous behaviour of predefined P(θ). However, this does not mean that a human operator cannot heuristically achieve this accuracy.

The next section is dedicated to a validation of the self-tuning threshold T-DTS procedure for two real-world classification problems.

IV.2.2.2 T-DTS self-tuning procedure validation using real-world