The TRAVOS-R model needs to be bootstrapped with a certain amount of relationship informa- tion before it can be used in a system. This prior information allows the agents to effectively learn the relationships that are present, and thereby choose the correct relationship-based heuris- tic to apply to the opinions they encounter. Here, we describe the evaluation of the performance of TRAVOS-R consumer agents under different forms of prior information.
4
This form of interaction provides experience that is utilised by the TRAVOS mechanisms.
MISLED CONSUMER AGENT 1 2 3 4 5 6 7 8 9 10 11 12 0 0.5 1 1 2 3 4 5 6 7 8 9 10 11 12 0 0.5 1 1 2 3 4 5 6 7 8 9 10 11 12 0 0.5 1
KNOWLEDGEABLE CONSUMER AGENT
IGNORANT CONSUMER AGENT
p(S ignal|R ela tionship) Signal Signal Signal p(S ignal|R ela tionship) p(S ignal|R ela tionship) Depended Upon Depends On Cooperates With Competes With KEY
FIGURE 6.2: Relationship given signal distributions used in three different configurations of TRAVOS-R.
6.2.1 Types of TRAVOS-R Agents
In detail, the prior information about relationships that a TRAVOS-R consumer agent has is held as conditional probability tables (CPTs, see Section 5.4.2), which represent the probability of observing particular signals given the presence of particular relationships. We create types of the TRAVOS-R consumer agent that have different CPTs, representing different forms of prior information. The three subtypes are described below, and the their associated prior distributions are shown in Figure 6.2:
1. Knowledgeable Consumer Agent (KCA) — This agent is configured with an accurate dis- tribution, which is very similar to the actual distribution used to generate the signals in the simulation. This type of configuration is representative of a system where the designer is aware of the possible actions and signals that can be produced by the presence of a particular relationship.
so that it is unable to make any sense of the signals that it observes. The ICA is representa- tive of systems where the designer does not know the connection between the presence of a particular relationship type and the signals and actions produced by interacting agents. 3. Misled Consumer Agent (MCA) — This agent is configured with an inaccurate distribu-
tion. More specifically, it is configured with CPTs that either give it completely incorrect information about what relationship a particular signal represents, or uniform prior infor- mation (indicating that it has no knowledge). This agent represents the scenario where the system designer may, mistakenly, assign incorrect CPTs due to lack of (or incorrect) knowledge about the relationships and signals in a system.
6.2.2 Experimental Process
The three types of TRAVOS-R consumer agents, as described above, are tested in two envi- ronments: (i) where the entire service provider population consists of ASPs, and (ii) where the entire population is made up of BSPs. These configurations are selected as they represent envi- ronments in which the TRAVOS-R mechanism is required and one in which it is not required. In each case, we vary the number of experience interactions,nei ∈ {0,5,10,15,20}, and for
each setting of experience interaction we vary the number of signal experience interactions,
nsi∈ {0,5,10,15,20}.
In this set of experiments the ACA is expected to be able to identify the correct relationships that are present, and should therefore be able to apply the correct relationship-based heuristics. Ultimately, this should lead it to outperform the other two. By outperform we mean that it produces a lower mean estimation error, meaning that its estimates for the trustworthiness of others are more accurate. More formally, we state the aim of this experiment in a hypothesis as follows.
Hypothesis 1
When varying the prior information about relationships that a TRAVOS-R agent has, the KCA will outperform both the MCA and the ICA. The MCA’s performance will degrade as the number of signals increase, and it will be poorer than the ICA.
The results clearly show that more informative prior information, such as the knowledgeable distribution, enables a consumer agent to perform better (see Figure 6.3). In both environments (100% BSP and 100% ASP populations) the KCA is able to outperform the other two, a result that validates Hypothesis 1. In the best case it is able to outperform the ICA by a mean error of 0.2, and the MCA by a mean error of 0.05. The results also show a notable change in the performance of the models. As can be seen in Figure 6.3, the models produce a larger mean error in environments where there is a 100% BSP population, because each agent adds a lot of biased noise to the opinions before supplying them to the consumer. This causes the consumer to be misled.
KCA ICA MCA KCA ICA MCA
FIGURE 6.3: Plots showing how different configurations of TRAVO-R performed in environ- ments with 100% ASP population (on the left) and a 100% BSP population (on the right).
However, contrary to our expectations, even though the performance of the MCA is worse than the KCA, the performance of each does not differ significantly (as can clearly be seen in the full set of results shown in Appendix A). Although not an obvious result, the MCA is able to outperform the ICA (by a mean error of 0.15). It also closely follows the performance of the KCA because much of the signal given relationship distribution of the MCA (see Figure 6.2) is uniform, which results in a reduction of confidence in the relationships it perceives are present. More specifically, the aspects of the distribution that are not uniform (for example Signal 3) cause TRAVOS-R to use incorrect relationship-based heuristics. However, signals such as Sig- nal 1 in the MCA’s prior information have a uniform CPT, leading the MCA to believe that every relationship is equally likely, and hence it lowers its confidence in the relationship type it believes is present. Such a low confidence implies a minor impact on the adjustment the relationship-based heuristic makes on the opinion. Therefore, after the adjustment, the opinion contains much more of its original information. In environments with 100% ASPs this clearly presents a benefit, which ultimately results in the MCA’s performance somewhat following that of the KCA with little variance. However, in 100% BSP populations, this leads to little adjust- ments of opinions that contain a large degree of bias and so the variance in the performance increases (as can be seen in Figure 6.3).
In general, the results suggest that the end performance of an agent using TRAVOS-R is affected by the configuration of the TRAVOS-R model by the system designer. It is important that the system designer is able to accurately represent the probability of observing a particular signal given a particular relationship. In cases where this is not possible, it is better for the agent designer to use prior information that is composed mainly of uniform parts, rather than completely uniform prior information.
Having examined how different configurations of TRAVOS-R perform, in the next section we examine how its performance changes in a variety of environments, and how it compares with
the TRAVOS mechanism.