Due to the widespread use of reputation systems, research work on them is intensifying and several models have been proposed [JIB07; LLYY09; SS05; MGM06]. Thus, once a reputation system has to be designed sev- eral choices have to be made at different levels of the development pro- cess, before its deployment in a network environment. This calls for a methodology for the analysis and the evaluation of trust and reputation systems that can help researchers and developers in studying, designing and implementing such systems. We address this challenge by propos- ing different kinds of theoretical and software frameworks and tools that, in our opinion, can support the development of trust and reputation sys- tems.
The main contributions of our work can be summarized as follows: 1. Theoretical Framework: a general framework based on Bayesian
decision theory for the theoretical assessment of trust and reputa- tion models.
2. Analysis Methodology: a methodology for analysing reputation systems based on a coordination language.
3. Software Tool: a software environment for network-aware evalua- tion of reputation systems and their rapid prototyping.
Theoretical Framework. Whenever existing or new reputation models have to be analysed, a theoretical framework for the assessment of such models is needed. In this phase it is interesting to study the models on the basis of a small set of simple parameters, such as quantity of available information and decision strategies, while abstracting from implementa- tion and deployment details. To this aim, we propose a general frame- work based on Bayesian decision theory for the assessment of trust and reputation systems. Within our theoretical framework we study how to quantify the confidence in the decisions calculated by the system. We anal- yse how this confidence is related to parameters as decision strategy and number of available ratings. We analyse if there are optimal strategies that maximize confidence when additional information becomes avail- able.
In our analysis, we study the behaviour of trust and reputation sys- tems by relying on the concept of loss function; a loss function evaluates the consequences of possible decisions taken by the system associating a loss to each decision. We quantify the confidence in the decisions calcu- lated by trust and reputation systems in terms of risk quantities based on expected (also known as bayes) and worst-case loss. We study the behaviour of these quantities with respect to the available information, that is the number of available rating values and the decision strategy, in the case of independent and identically distributed observations. We show that there are optimal strategies that maximize confidence as more and more information becomes available. Finally, we study an extention of our framework to a class of rating mechanisms where each rater is charac- terised by a (unobservable, possibly malicious) bias. This can lead the rater to under- or over-evaluate its interactions with the ratees.
Analysis Methodology. Concerning the integration of reputation sys- tems with end-user applications, a methodology for tuning trust and rep- utation models in order to fit to the characteristics of the given network environment is needed. In particular, it is interesting to study whether in the phase of models tuning, the features of the original models are kept. We address such issues by proposing a verification methodology based on the use of the coordination language KLAIM[BBD+03; DFP98]
and related analysis tools [DKL+07; Lor10]. Such approach enables ver-
ification of reputation system specifications. Specifically, it is possible to check whether trust and reputation models meet the expected behaviour, how parties’ initials reputations affect the models and how parties’ be- haviours affect their reputations. In our study, we first define a paramet- ric KLAIMspecification of a reputation system that can be instantiated with different reputation models. Then, we consider a stochastic spec- ification obtained by considering actions with random (exponentially distributed) duration. The resulting specification allows us to perform quantitative analysis of estimation properties of the considered system.
Software Tool. The last issue we address is related to implementation, i.e. to the phase when reputation systems have to be deployed and tested in real network environments. At this stage, real-word implementation details of trust and reputation systems and of the network environment where they have to be deployed have to be taken into account in the eval- uation. We have developed a software tool (NEVER) for network-aware evaluation of reputation systems and for their rapid prototyping through experiments performed according to user-specified parameters. On the one hand, NEVER provides a framework for rapidly developing Java- based implementations of reputation system models and for easily con- figuring different networked execution environments on top of which the reputation systems will run. On the other hand, NEVER can be used for automatically performing experiments on the reputation system im- plementations according to user-specified parameters; this enables the study of their behaviour while executing on given network infrastruc- tures.
Overall our contribution addresses issues related to the study, the de- sign and the implementation of trust and reputation systems. Indeed, we provide theoretical and software tools for the analysis and evaluation of trust and reputation systems, at different stages of their development.