Simulations

The results of some simulation runs of the STOKLAIMspecification, per- formed by using SAM, are reported in Figures 6, 7, 8 and 9 (see Ap- pendix A for further results). The results are averaged across 1500 sim- ulation runs1_{. The charts present the trend of the reputation of a given}

party in the system and the error committed by the reputation model in the computation of reputation scores. In the charts an horizontal line de- notes the true paty’s behaviour. The x-axis reports the numbers of rating values used to compute reputation scores. The y-axis reports both the reputation scores and the error committed. As measure of the error we

1_{On an Apple MacBook Pro computer (2.4 GHz Intel Core 2 Duo and 4 GB of memory)}

Figure 6:Reputation and error trends for parties with behaviour θ = 0 and no initial ratings assigned

used the norm-1 distance between probability distributions (we already used it in Chapter 3 as a possible definition for loss functions). In our analysis we compare the Beta model and the ML model in case of binary ratings by studying:

• the trend of the reputation score computed by the models, reported in the charts as ’BETA’ and ’ML’ lines, respectively;

• the trend of the error committed by the models, reported in the charts as ’Norm-1 BETA’ and ’Norm-1 ML’ lines, respectively. We compare the two models both in presence and in absence of initials ratings. Such initials ratings are used to determine the initials reputation scores for parties, before any interaction in the system.

From simulation data we observe that the ML model performs better than the Beta model if we consider the computed reputation score. In-

Figure 7: Reputation and error trends for parties with behaviour θ = 0.20 and no initial ratings assigned

deed the reputation score computed by the ML model is always closer to the true party’s behaviour than the reputation score computed by the Beta model. Such evaluation does not give us a complete information about models performances. Indeed looking at the error committed by the models, we observe that the ML model does not perform always bet- ter than the Beta model. Roughly speaking, for behaviour values lower than θ = 0.9 and higher than θ = 0.1 the error committed by the Beta model is always lower than the one committed by the ML model. It hap- pens the opposite for behaviour values lower than θ = 0.1 and higher than θ = 0.9. Finally, we observe that the Beta model tend to over- or under- estimate party’s behaviour. Specifically, the Beta model underes- timates behaviours higher than θ = 0.5, and overestimates behaviours lower than θ = 0.5. We believe that this is an unwanted behaviour for trust and reputation models. Indeed the Beta model gives an advantage

Figure 8: Reputation and error trends for parties with behaviour θ = 0.10 and 4 initial ratings fixing initial reputation score to 0.5

to parties with a bad behavior, computing for them a reputation score (in average) higher than their true behaviour.

Figures 8 and 9 report some simulation results in the case of initial rating values. The charts report analysis results in presence of four ini- tial rating values for each party that fix parties’ initial reputation to 0.5. We observe that also in this case the ML model performs better than the Beta model in computing reputation scores. Moreover, we notice that the ML model benefits from initial ratings if compered with the Beta model. Specifically, we observe that for behaviour values lower than θ = 0.3 and higher than θ = 0.7 the error committed by the ML model is always lower than the one committed by the Beta model. The Beta model ben- efits from initial ratings only for behaviour values lower than θ = 0.6 and higher than θ = 0.4. That happens because the true behaviour of the party is close to the initial reputation score, i.e. party’s behaviour is close

Figure 9: Reputation and error trends for parties with behaviour θ = 0.20 and 4 initial ratings fixing initial reputation score to 0.5

to θ = 0.5. Indeed, the Beta model has a bias towards reputation values close to 0.5.

To sum up the results of our analysis allow us to argue that initial ratings can improve the models’ performances, but the number and the value of such ratings has to be choosen carefully in order to get the de- sired result. Notice that initial ratings play a different role for different behaviours, i.e. behaviours close2 _{to the initial reputation value are in-}

ferred with a lower error than in the case of no initial ratings. It happens the opposite for others behaviours, for that the error committed by the models is higher compared to a system with no initial ratings. Initial rat- ings could be used as an incentive for good behaviour, i.e. fixing a low initial reputation score should push parties to behave well if they want

2_{The term close denoting the distance between probability distributions may not have a}

clear meaning. Anyway, in this case, where Bernoulli distributions are used and a single parameter identifies a distribution, we believe that this term makes sense.

“the reputation of sparty iconverges to its actual behaviour θiwithin time t”

Initials Ratings Beta Model ML Model

0 1 1

2 0.804444828548 1

4 0.00745483834375 0.802581118962

Table 2:Satisfaction probability for party’s behaviour θ = 1 and time t = 50

to achieve an high reputation score.