Error Rate in Predictions

In order to test the reliability of the framework, the error rate in predictions made is tested using different parameters and scaling factors to compute RVs. The acceptable error rate of D3-FRT will depend on the criticality of the application domain, and the risk appetite in the domain. With a penalty of 0.4 and 0.5 for intoxication and collusive behaviour respectively, and a reward of 0.5 for normal behaviour, the following default parameters are used: µh = 0.3, µo = 0.5, µf = 0.17. Initially, several iterations of the model were

done and the observation was that these default values produced the optimal results. For every 100-tick cycle of the simulation, N = 50 with 12% and 10% of the agents exhibiting collusive and intoxicating behaviours and for the duration of 9000 ticks.

Figure 6.3: θr and θf, µo = 0.5 Figure 6.4: Estimation error, µo = 0.5

simulation. In these experiments, there is a 5.1% prediction error rate at an error threshold of ±0.6.

In order to reduce the possibility for intoxication attacks in the experiments, µo has

to be greater than µh; reducing the effect of historical behaviour on predictions and

all reputation computations. This is because choosing a larger value for µh biases the

reputation value of an agent a to the observations currently received about a. A larger value of µo gives heavier weight to the performance of the agent in the past. With µo set

to 0.6 and retaining the error threshold value from the previous experiments, D3-FRT made some inaccurate predictions about agent reputation values at the rate of 15%; this is much higher than error rate of when µo = 0.5. The graph in Figure 6.5 shows the

overall RVs of the peers compared with the predicted values. From the figure, as the simulation progresses the actual and predicted values seem to deviate. In Figure 6.6, from approximately 5500 ticks in the simulation, the error rate reduced considerably and remained above the lower bound −0.6 of the threshold.

Figure 6.5: θr and θf, µo = 0.6 Figure 6.6: Estimation error, µo = 0.6

We tested the behaviour of D3-FRT when µo is the same value as µh which is 0.3 and

the results are given in the Figures 6.7 and 6.8 below. The error rate in these experiments is 21% and Figure 6.8 shows that the errors in prediction initially fluctuated above and

6.2 Experimental Setup 127

within the threshold boundary but later remained within the threshold boundary from around 3800 and for the rest of the simulation.

Figure 6.7: θr and θf, µh = µo = 0.3 Figure 6.8: Estimation error, µh = µo = 0.3

From these, one might remark that the best value for the scaling factors are µh =

0.3, µo = 0.5 as the best results were obtained with these values and in addition reducing

the possibility of intoxication attacks.

Figure 6.9: Comparison of ratings in different values of the scaling factors for collusive agents

Furthermore, when using quantitative data, it is possible that small differences in individual values produce relatively large differences in the overall ratings [Mar94]. Since

the mathematical model is defined by a series of equations, parameters, and variables, it is subject to many sources of uncertainty including errors of measurement, absence of information and poor or partial understanding of the driving forces and mechanisms. This uncertainty imposes a limit on our confidence in the response or output of the model [Cen12]. In our case, we try to gain insight on the whether our model is sensitive to the scaling factors used in computing agent ratings. The aim of the analysis is to determine the effect of the scaling factors on computed values.

The sensitivity analysis is performed by running different simulations of the algorithm with modification of parameters in order to evaluate the behaviour of the algorithm. When conducting the experiments, we run 10 simulations of the algorithm for each parameter, to make statistically correct conclusions. We analysed 3 parameters: µh, µo, µf and 3 different

values were used for each parameter and for each iteration, µh is always greater than

µo. These parameters are important to our model because they influence the accuracy

of the model output. The simulations were carried out using the default values for the parameters chosen (µh = 0.3, µo = 0.5 and µf = 0.17). The results from using default

values are used as a benchmark for the other values tested, because with default values our model is optimal, as shown in the the preceding experiments of this subsection.

In Figures 6.9-6.12 we show the sensitivity of D3-FRT with respect to changes in its input parameters based on the type of behaviour exhibited. Constant µh in the graph legend is the trend line obtained from keeping the factor constant at its default value whilst varying the values of the other factors and this is the case for the trend lines of constant µo and constant µf as well (using the values in Table 6.1).

6.2 Experimental Setup 129

Figure 6.10: Comparison of ratings in different values of the scaling factors for intoxicating agents

Parameter values for sensitivity test Parameter 1 Parameter 2 Parameter 3 µo = 0.5 µf = 0.1 µh = 0.4 µf = 0.17 µh = 0.3 µf = 0.2 µh = 0.2 µf = 0.17 µh = 0.2 µo = 0.3 µh = 0.3 µo = 0.4 µh = 0.4 µo = 0.5 µh = 0.3 µf = 0.1 µo = 0.4 µf = 0.17 µo = 0.5 µf = 0.2 µo = 0.6

Table 6.1: Parameter list for sensitivity test

ratings. Although, the graphs show similar trend lines for the different cases, slight changes to the input values have resulted in noticeable variations in the output. In each of the graphs, one of the variables is kept constant at its default value while the other factors change.

The experiments done have shown that there is a noticeable change on the behaviour of the model depending on the values of the parameters analysed. The motivation behind the sensitivity analysis of the D3-FRT algorithm was to identify the influence of the variability

Figure 6.11: Comparison of ratings in different values of the scaling factors for collusive and intoxicating agents

of the parameters of the algorithm in order to know which at which point the parameters can improve the performance of the algorithm. The ratings obtained from running the experiments using default values are significantly different from the other ratings. In particular, we have proven that the use of adaptive factors in computing ratings potentially influences the behaviour of our model.