Changing the Service Provider Population Composition

It is critical to evaluate the performance of the model in environments where the population, from which it is obtaining opinions, is changed with respect to the number of ASPs and BSPs it contains. It is feasible to assume that in a real life application it is unlikely that the entire popu- lation of the agent system will consist completely of agents that are either accurate or biased in their opinion provision. For this reason, evaluating the mechanisms in a range of preconfigured populations establishes when it is beneficial to use TRAVOS-R instead of TRAVOS. TRAVOS requires a number of opinion provision instances to learn which agents are providing good opin- ions and which agents are providing bad opinions. On the other hand, TRAVOS-R does not rely on past opinion provision episodes and is able to adjust opinions based on heuristics. For this reason, when the number of opinion experience interactions is low, TRAVOS is likely to be out- performed by TRAVOS-R, and viceversa when the number of opinion experience interactions observed by the TRAVOS agent increases. More formally, we state the hypothesis for this part of the evaluation as follows.

Hypothesis 2

Increasing the percentage of BSPs in the population will result in TRAVOS-R out- performing both TRAVOS and the Zero Intelligence consumer agents. For a given number of experience interactions,nei, and a given number of opinion experience

interactions, noi, as the percentage of ASPs increases, TRAVOS will outperform

the other two.

Results have been obtained to test this hypothesis by running experiments where the service provider population composition is varied from 100% ASPs to 0% ASPs in decrements of 25%, as shown in Table 6.1. The results clearly show one trend, which is that as the percentage of BSPs increases, the performance of both TRAVOS and TRAVOS-R consumer agents falls. This is largely due to the increase in the number of false opinions that the consumer agents receive.

FIGURE6.4: Plots showing the results obtained from environments containing a 100% Biased population.

However, it should be noted that, in the majority of cases, when the population consists of only BSPs, TRAVOS-R outperforms TRAVOS as expected. Results obtained for a subset of such environments are shown in Figure 6.4, which shows TRAVOS-R outperforming TRAVOS in environments containing 100% BSPs. The reason for the superior performance by TRAVOS-R in such environments is simply that it learns the relationships between agents, and so selects the appropriate relationship-based heuristics to combat the biased opinions. On the other hand, TRAVOS has no such method of combatting biased opinions, and as a result is misled by the opinions and therefore produces a larger mean error in its estimations. The process by which TRAVOS is sufficiently misled by the biased opinions is described in more detail below. The strength of the TRAVOS mechanism for adjusting opinions relies on the assumption that an opinion provider agent employs a fixed strategy in providing opinions. That is, it always provides the same kind of opinion such as always false positives, or always false negatives, for all opinion requests. Simply, this means that as long as an opinion provider uses the same strategy (regardless of the agent requesting the opinion and the agent to whom the opinion applies) to provide all of its opinions, TRAVOS is able to perform relatively well. This is because it is able to learn a fixed strategy and, as shown in Chapter 4, by doing so it performs well.

0 0.5 1 0.0 − 0.2 Bin 0 0.5 1 0.2 − 0.4 Bin 0 0.5 1 0.4 − 0.6 Bin 0 0.5 1 0.6 − 0.8 Bin 0 0.5 1 0.8 − 1.0 Bin

Inaccurate opinion provider with a static strategy for providing opinions Accurate opinion provider

FIGURE 6.5: A TRAVOS consumer agent’s opinion provision history bins for an opinion provider that provides inaccurate opinions using a static strategy.

More specifically, its learning strategy associates the opinion with the corresponding outcome observed in a set of bins that group together similar opinions (as discussed in Section 3.4.2 ). Such fixed behaviour leads to the bins recording useful information, which can be used to adjust opinions appropriately. For example, Figure 6.5 shows a set of bins belonging to a TRAVOS consumer agent that has encountered a particular opinion provider employing a fixed strategy in providing opinions. In this case, the opinion provider overestimates the trust level represented by the opinion before providing this false positive opinion. It does this for all opinion requests. As Figure 6.5 shows, the TRAVOS consumer agent’s bins for this opinion provider are skewed to the right.6

Now, if the opinion provider employs a range of strategies to provide opinions, for example by providing a number of false positives, false negatives and honest opinions, it is able to confuse the TRAVOS mechanism. As a result, the information held in the bins tends to become more uniform as shown in Figure 6.6. Here, all the bins contain a similar plot, indicating that the TRAVOS consumer is unable to accurately interpret the report of the opinion provider when it provides a particular opinion, and fails to adjust it accordingly.

Then, as the number of biased service agents increases, the number of agents dynamically gener- ating opinions (based on shared relationships) increases too, creating the dynamic environment in which TRAVOS is misled. In such cases, however, the TRAVOS-R mechanism is able to continue to adjust the opinions using the appropriate relationship-based heuristic to counteract the dynamic strategies. The relationship-based heuristics (described in Section 5.5) allow the TRAVOS-R agent to modify the biased opinions by reducing or removing the bias that may have been introduced into them. The TRAVOS-R agent knows which heuristic to apply as it learns the relationships (that cause the bias) between the service provider population, through observing signals generated by interacting service providers.

6

The plots are skewed to the right of plots compared to what would have been obtained had the opinion provider provided accurate opinions.

0 0.5 1 0.0 − 0.2 Bin 0 0.5 1 0.2 − 0.4 Bin 0 0.5 1 0.4 − 0.6 Bin 0 0.5 1 0.6 − 0.8 Bin 0 0.5 1 0.8 − 1.0 Bin

Inaccurate Opinion Provider with a dynamic strategy for providing opinions Accurate Opinion Provider

FIGURE 6.6: A TRAVOS consumer agent’s opinion provision history bins for an opinion provider that provides inaccurate opinions using a dynamic strategy.

However, this ability of TRAVOS-R to adjust opinions using relationship-based heuristics has its limitations. As the percentage of accurate agents increases in the service provider population, the relative performance of TRAVOS improves, and in environments where there are sufficient opinion experience interactions (here this isnoi > 5), it is able to outperform TRAVOS-R (the

impact of varying experience interactions on the performance of TRAVOS and TRAVOS-R is discussed further in the discussion of Hypothesis 5). The poorer performance by TRAVOS-R is a result of its inability to distinguish accurately between ASPs and BSPs. In particular, the model assumes all opinions are biased, and therefore applies the appropriate relationship-based heuristic to the adjustment of the opinion. This adds noise to an otherwise honest and accurate opinion, causing it to calculate a level of trust with increased error. On the other hand, the TRAVOS mechanism prevents this from happening, keeping the information contained within the honest opinion intact, and thus allowing the agent to calculate a level of trust with less error.