Classifying a reporter as honest or lying only based on the information it provides is difficult. Typically, the evaluator’s view in an open MAS is limited (see Section 1.1) and there is usually no way it can confirm whether the ratings it receives are honest and based on the actual observation of the target agent’s performance. Thus, in order to assess the accuracy of such third-party ratings, the evaluator can only compare them with its own observations of the target agent. In our credibility model (Section 6.1), the difference between a third-party rating of the target agent and the performance the evaluator receives from it serves as the measure of the reporter’s accuracy. Based on this measure, the reporter is then classified as lying/inaccurate or honest/accurate. However, this accuracy measure alone is not enough. The difference between a third-party rating and the performance the evaluator observes can be attributed to many things: (1) the variation of the target agent’s performance, (2) the reporter’s (in)ability of making accurate ratings, and (3) the reporter’s intentional manipulation of the rating’s value. Ideally, only when (2) or (3) is the case should the reporter be classified as lying/inaccurate and its future ratings be filtered out (as a result). However, there is no way for the evaluator to tell whether the difference is due to (1) or not because it cannot know the performance the reporter received from the target agent. Therefore, it can only (reasonably) expect the performance variation of providers to be relatively
small and that any inaccuracy greater than the typical performance variation of providers is deemed to be due to (2) or (3). Here, the inaccuracy tolerance threshold ι serves as the borderline to separate between cases (1) and cases (2) or (3) and it needs to be set based on the typical performance variation of providers in an agent’s environment. As discussed at the beginning of the chapter, fixed values for ι can be inefficient if the agent’s designer chooses a wrong value. Moreover, an agent can interact with only a small set of providers in an open MAS whose typical performance variation is different to that of another set or that of the whole open MAS. Hence, in this section, we propose an algorithm to monitor the performance variation of the providers an agent encounters and to adjust the inaccuracy tolerance threshold accordingly.
In order to do so, we first need to calculate the performance deviations of the providers that the evaluator, say, agent a, has encountered. The performance deviation of a provider, say, agentb, in term cis calculated by a as follows:
dv(a, b, c) =
P
ri∈RI(a,b,c)|vi−v|
|RI(a, b, c)|
(7.5) wheredv(a, b, c) is the performance deviation ofb in termc that is observed bya,
RI(a, b, c) is the set of ratings of b in term c that a made from past interactions with b, vi is the value of the rating ri, and v is the mean value of all the rating
values in the set RI(a, b, c). The performance deviation is used to estimate the variations in providers’ performance only when it is calculated from at least two ratings. Otherwise, the deviation from one rating is always 0 and does not give any information regarding the variation of a provider’s performance. Next, the average performance deviation of all the provider agents a encountered at least twice (denoted by the set P) is used as the potential value for the inaccuracy threshold (denoted by ι0):
ι0 =
P
p∈Pdv(a, p, c)
|P| (7.6)
where |P| is the number of agents in the set P. If this number is too small, ι0 may not be representative and the inaccuracy tolerance threshold ι should not be replaced by it completely. Otherwise, the value of ι can fluctuate significantly when an agent has just joined an environment and interacted with a small number of agents. Hence, ι is updated as follows:
ι=
( |P|·ι0+(n
MD−|P|)·ιpre
nMD if |P|< nMD
where nMD is the minimum number of deviation values for ι0 to be considered representative and ιpre is the previous value of ι (i.e. before this update). In
short, if the potential value ι0 is calculated from the deviation values of sufficient agents (i.e. ≥ nMD) then ι0 is assigned to ι. Otherwise, ι is only partly updated based on ι0 and the impact of ι0 on the new value of ι is given by the number of deviation values taken into account (|P|) when ι0 is calculated. The inaccuracy tolerance threshold ι is updated every time an agent has new information about the performance variation of its providers. This means that an ι update happens (after) every time a interacts with a provider agent (i.e. FIRE received a new direct rating) that it has interacted with before (so that ahas at least two ratings of the rated agent to calculate its performance deviation).