Experiment Execution

The experiment is executed over four phases, whereby human participants attempt to beat the game of Snakes and Ladders in as few turns as possible:

• Control - The game is released to a closed set of participants to observe honest play, for validation of detectors;

• Phase 1 - The game is released within the School of Computing, University of Kent, requesting participants to play the game honestly or dishonestly;

• Phase 2 - The game is released externally, advertised via academic and research community mailing lists, in addition to external Universities, re- questing participants to play the game honestly or dishonestly;

• Phase 3 - The game is again released internally within the School of Com- puting, University of Kent, requesting participants to play the game honestly or dishonestly.

In each phase, participants are provided the same guidance (Appendix C.1) in the form of a participant declaration that described the purpose of the experiment, and a brief overview of how the game works. It is expected that participants would interpret the rules of the game as conveyed by the game interface itself.

Each phase is run for a period of time in which the game is accessible to par- ticipants. The length of time is dictated by player activity, whereby a phase is complete once a player succeeds against the SAAF controller. Moreover, com- pletion of a phase indicates that a player has been successful in performing an unknown attack that the controller was not able to detect.

At the end of each phase, the SAAF controller is updated to account for any unknown attacks that have been successful in beating the SAAF controller.

CHAPTER 6. EVALUATING SAAF THROUGH GAMIFICATION 178

This exemplifies SAAF’s ability to be extended in order to cope with previously unknown attacks, as well as promote additional challenges for participants within future phases.

Experiment Variables

Each phase is subject to a set of independent, dependent, and control variables. Independent variables are indicative of environment change, and driven by human participation. Dependent variables measure environment change, which refer to the consequence of human participation. For example, the performance of SAAF, violations detected, unknown attacks performed, the state of the access control, and game usage statistics.

Control variables denote the configuration of the authorisation infrastructure and the SAAF controller. These include the SAAF controller’s perception of behaviour (behaviour policy), available solutions to the controller, the availability of probes and effectors, and configuration of the game environment. Control variables remain fixed throughout phases of the experiment. The only exception is that once a phase has ended and an unknown attack has been identified, detectors are updated in order to enable the SAAF controller to detect the unknown attack as a violation in the next phase. Lastly, each phase of the experiment builds upon its predecessor, meaning that the state of access at the end of phase 1 is the initial state of access for phase 2. This is important as it portrays the SAAF controller operating within a live and evolving environment.

Phase Progression

The experiments were conducted over a period of 7 months, as to obtain a wide range of data. Over the course of each experiment phase, violations (known attacks) were detected and mitigated, preventing malicious players from persisting with dishonest play.

A small number of unknown attacks were successful in enabling a player to beat the game in an unexpected way, resulting in the player obtaining what should have been an impossible score (e.g., completing the game in 0 or 1 turns). Post- phase analysis identified how the player was able beat the game without being detected or mitigated by the controller.

• The control phase was conducted over a period of 1 week where ten players were observed and asked to play a number of games in conformance to the

rules of snakes and ladders. During this time, all players completed their games within a legal number of turns per game (i.e., greater than the least amount of turns possible when playing the game honestly). The SAAF con- troller was configured to detect violations only, albeit no adaptation would take place in order to identify the extent of unintended violations. As a result, a number of low level violations were detected due to genuine player mistakes.

• Phase 1 was conducted over a period of 1 month. It resulted in a single player account successfully performing a code injection attack, in which the game’s resource probe could not detect and notify the SAAF controller. The player had changed their starting square within the game from square 1 to square 63, allowing the player to finish in one turn after a single dice roll. The resource probe failed to report this activity due to limitations in what it could detect. As such, the trace log of the successful player’s activity was analysed and used to create additional detectors within the resource probe.

• Phase 2 was conducted over a period of 5 months. It resulted in a single player account successfully performing a code injection attack, which the SAAF controller failed to detect. The code injection attack was similar to that of the attack observed in phase 1. Despite the probe identifying and logging the attack, two faults were identified through post-phase analysis. This resulted in several false negatives, and ultimately had enabled the suc- cessful malicious player to avoid mitigation.

The first fault identified that a mis-configuration between phase 1 and phase 2 had caused the controller to miss 13 potential violations, depicting similar attacks to the one that was successful in phase 1. To prevent this from occurring in the following phase, all known attacks were simulated within phase 3’s production environment, prior to initiation, in order to confirm correct configuration.

The second fault was associated to prolonged execution (in this case, 5 months), where the resource probe crashed as a result of a critical memory error. This fault was overcome using microrebooting [25], whereby sub- components of a system (e.g., the self-adaptive authorisation infrastructure) were scheduled to restart periodically.

CHAPTER 6. EVALUATING SAAF THROUGH GAMIFICATION 180

accounts successfully performing a code injection that was not detected by the game resource probe or by the SAAF controller. Post-phase analysis identified that two player accounts beat the game in a single turn via iden- tifying the endpoints of the AJAX routines that handled logging and policy enforcement. Instead of playing the game, the player accounts had sent fal- sified data directly to these endpoints to mimic player activity. In this case, it was identified that both accounts had triggered a single move with no associated dice roll. The resource probe contained no detectors to identify such a scenario, and as such, could not inform the SAAF controller of the behaviour. To prevent repetition of this type of attack, the resource probe required additional detectors to identify similar trace behaviour.