Federation Challenges

Exp3 and Exp4 demonstrated the consequence of when a SAAF controller is limited in performing subject adaptation. Whilst policy adaptation occurred at the point where the impact of violations outweighed the impact of removing trust in the contractor identity provider, Exp1 and Exp2 showed that the attacks could equally be resolved more effectively on an individual scale. However, as demonstrated in a previous paper [6], policy adaptation is necessary in regards to large numbers of subjects committing malicious behaviour (i.e., when a service provider, de- spite mitigating attacks from individuals, is seeing persistent attacks from a given identity provider).

Associated with this is the reliance on an authorisation service’s ability to validate subject credentials. Credential validation enables solutions that manage the trust in identity providers. Without it, SAAF is limited in performing fine grained adaptations against identity providers, resorting to policy adaptation that may impact all subjects from all identity providers.

In regards to control over observation, in the LGZLogistics case study, there are no probes deployed within the contractor identity provider. This highlights the fact that many third party organisations may not provide a complete view of subject attributes, in particular, the release of personal identifiable data. As a result, the model of access generated is representative only of subjects that have requested access, and what valid attributes (post validation) have been used.

Generating a model of access in this fashion is limited, as the model only con- tains a view of active subjects, and does not present a complete view of access (i.e., subjects that have yet to request access will not be modelled). The repercussion of this is that the calculation of impact of solutions against the current modelled state of access may well be higher, due to an incomplete modelled state of access. One potential solution that overcomes the problem of restricted observation of subjects, is a probe at the identity provider that operates in a similar fashion to SAAF’s SimpleSAMLphp effector [7] (see Appendix B.2). Applying the same concepts to a probe, an identity provider could control what subject information is released, how subjects are identified, and which subjects can be observed. This would allow for synchronised models of access within federated environments. However, a potential risk is if an identity provider is hijacked, information sent via a identity provider managed probe could become unreliable.

CHAPTER 5. SIMULATING INSIDER THREAT 160

5.7

Summary

In summary, this chapter has presented an evaluation of the Self-Adaptive Au- thorisation Framework (SAAF) through the simulation of a fictitious case study of insider threat. As part of this evaluation, a malicious changeload has been for- mally defined in the context of authorisation infrastructures in order to describe scenarios of abuse in access control.

The malicious changeload, relevant to the case study, was then executed to stimulate self-adaptation within a federated authorisation infrastructure. A de- ployment of the SAAF prototype was then evaluated in mitigating the malicious changeload under various operational conditions. These included changes to the runtime load of the authorisation infrastructure and the SAAF autonomic con- troller, along with restrictions in available probes and effectors (simulating the presence of a non-cooperating contractor organisation).

The evaluation demonstrated the SAAF prototype’s robustness in handling abuse of access under repeatable conditions, where the prototype was shown to consistently mitigate abuse under normal and high loads. In addition, when faced with limitations in enacting adaptation, the prototype was shown to escalate its selection of policy adaptations in order to overcome failures in subject adapta- tion. Whilst subject adaptation was shown to create minimal impact (in terms of consequence to non-malicious subjects), it was in these conditions that pol- icy adaptation becomes necessary in order to halt the abuse of access. Finally, SAAF has been demonstrated in mitigating the abuse of access in federated envi- ronments, where the use of a domain managed effector has been key to enabling adaptation across multiple management domains.

A limitation in evaluating self-adaptive systems through simulation is the in- ability of dealing with a wide range of changes that are representative of unex- pected subject behaviour, and how subjects may react to adaptation. Whilst case studies of insider threat can provide insight to attack scenarios, they do not consider the runtime consequence of mitigation. To evaluate this, Chapter 6 defines a runtime experiment in which real users are invited to carry out mali- cious behaviour against an organisational resource, protected by a self-adaptive authorisation infrastructure.

Evaluating SAAF through

Gamification

6.1

Introduction

The simulation of insider threat case studies is limited regarding the evaluation of self-adaptive authorisation infrastructures. This is because they would not be able to portray an accurate perception of reality. Simulation has demonstrated par- tial feasibility of the Self-Adaptive Authorisation Framework (SAAF), including how SAAF mitigates malicious behaviour under prescribed conditions. However, simulation can only evaluate a fraction of the scope of change and types of abuse representative of the real-world.

An important step in evaluating SAAF is demonstrating its ability to mitigate abuse of access when faced with uncertainty. Moreover, it is necessary to evalu- ate the consequence of self-adaptation in terms of how human users respond to the presence of a feedback loop. In light of a feedback loop, users may change their behaviour, for instance, to mask their malicious activity. Such change is unpredictable, resultant of intelligent user interaction, and therefore challenging to simulate.

Given SAAF’s experimental status, to consider deploying SAAF in a real or- ganisation would be inherently risky. Therefore, this chapter presents an approach whereby gamification [60] is used to emulate a real-world environment. Gamifica- tion is the use of online games to solve complex problems and generate meaning- ful data as a consequence of human player participation. It is a crowd sourcing technique to capturing large volumes of data by using the premise of a game to

CHAPTER 6. EVALUATING SAAF THROUGH GAMIFICATION 162

motivate human participation.

For evaluating SAAF, gamification is used for generating diverse and unpre- dictable data from real user activity. In particular, it enables the observation of SAAF mitigating cases of abuse at runtime, and the observation of user activ- ity post mitigation. As such, the success of mitigation can be validated, along with evaluating the consequence of self-adaptation by analysing user response to mitigation.

The contribution of this chapter is an approach to evaluating self-adaptive systems through gamification [60]. A key feature of the approach is the ability to observe user activity pre- and post-adaptation, in order to evaluate the runtime consequences of self-adaptive systems. Gamification is demonstrated in evaluat- ing the Self-Adaptive Authorisation Framework (SAAF) by way of deploying an online game as a protected resource within an authorisation infrastructure. Hu- man participants are assigned a set of access rights related to the authorisation of actions within the game. Participants of the game are then invited to choose to act honestly or dishonestly. Dishonest activity is viewed as synonymous to malicious behaviour, requiring mitigation.

The rest of this chapter is structured as follows. In Section 6.2 the objectives and scope of the online experiment are presented. Section 6.3 describes the design of an online game, in which diverse and unpredictable behaviour can be observed. Section 6.4 discusses the deployment of the game in a self-adaptive authorisation infrastructure. Section 6.5 describes the phases and execution of the experiment within the game environment. Section 6.6 discusses the results of the experiments. Section 6.7 discusses the evaluation approaches presented in this thesis, identify- ing limitations. In Section 6.8, a summary of the chapter is provided. Finally, Appendix C contains additional results of the experiment.