Limitations of the Study

Possible study limitations include the following:

Small Study Size: As with the other studies in this dissertation, the results presented are from a feasibility study that is small both in terms of number of participants and amount of data. This has a strong effect on the reported error rates and the deter- mination of statistical significance.

Unbalanced Datasets: Again, as with the other studies in this work, the datasets were un- balanced in terms of number of owner and rest–of–world patterns. This also has an effect on the FAR and FRR (and thus EER) rates.

6.8. Summary 127 Same Classifier: The posterior probabilities for each combined pattern came from the same classifier, although it is possible to combine posterior probabilities of different classi- fiers. It is not expected that the source of the probabilities will make a significant difference, but tests should be performed to examine this.

6.8

Summary

This chapter has detailed the study designed to test the viability of a multimodal biometric using the classifier output from the Keystroke Dynamics and Speaker Verification studies. This study compared two probability-based score-level fusion techniques to determine which provided the lowest overall error rates. Combining the multimodal biometrics using two score-level fusion techniques resulted in an overall reduction in EER, and AUC increase, although these differences were not statistically significant and thus may be attributed to chance. These results support other research that has concluded that multimodal biometric combination is a viable method for achieving lower error rates, and justify the inclusion of multimodal biometrics in the Transparent Authentication Framework.

128

Chapter 7

Transparent Authentication

Perceptions Study

This chapter describes the Transparent Authentication Perceptions Study1, which was per- formed in order to gain perspective regarding users’ willingness to use, and opinions on the design of, a transparent authentication mechanism on a mobile device. This chapter provides the study design details, including details of participants, materials, and methodology as well as reports on the study findings. A discussion of the impact of the study’s results and its role in the framework presented as part of this research rounds out the chapter.

7.1

Study Goals

Alternative authentication methods have been widely researched over the last decade, but rarely deployed outside a lab setting. The reasons for this vary depending on the features provided by such systems, but may be attributed to lack of user knowledge or a misunder- standing of user wants and needs. The consequence is that researchers do not fully under- stand how users will use, bypass or accept new security mechanisms. It is therefore beneficial to determine during the design of such systems whether users would be willing to use the system and what functionality they find important. The feasibility studies reported in Chap- ters 4, 5 and 6 have shown that behavioral biometrics, particularly multimodal, show promise as the basis for the decision–making in a transparent authentication system. The outstanding question is whether mobile device users would choose to use such a method to protect their devices and data. The first purpose of this study is thus to determine whether the participants feel a transparent authentication method on a mobile device provides adequate security, and whether they would consider using it on their own mobile devices.

7.2. Study Design 129 The second purpose of this study is to employ user opinions and suggestions to inform the design of the Transparent Authentication Framework presented in this dissertation. Including user-requested functionality in the final Framework ensures that it is not simply a product of a research-focused endeavor. The findings of this study provide justification for further research into transparent authentication for mobile devices. In this way the user, an important stakeholder, has been consulted and their suggestions considered in the design phase of the Framework.

This study attempts to answer several research questions that are related to the study’s goals: 1. What are the participant’s opinions of, and reactions to, using a transparent authenti-

cation method on a mobile device?

2. What is the participant’s perceived level of security while using a mobile device that employs transparent authentication?

3. Do the participants find the transparent authentication method easy or difficult to use? 4. Do participants find transparent authentication generally helpful or mostly a hindrance? 5. Would participants choose to use a transparent authentication method on their own

mobile device, if one were available?

6. How do participants react to barriers blocking them from completing their intended tasks, in terms of frequency?

To determine the answers to these questions, an iPhone application was developed that pre- sented participants with a series of tasks to complete. Such a study would normally require a fully-functional transparent authentication method on the device. Such a method does not yet exist, thus the study was designed as a Wizard of Oz study [171], in which the partici- pant assumes that the authentication method is running and receives feedback based on their actions to confirm this. In actuality, the application reacted to predetermined actions and triggers; no transparent authentication method was actually implemented.

7.2

Study Design

The Transparent Authentication Perceptions (TAP) study is a lab-based, between-groups study [152, p. 74] in which 30 participants were asked to complete seven tasks using an Apple iPhone provided by the experimenter for the duration of the study. The seven tasks were divided into three security levels: Low, Medium, and High, that represented the level of device confidence the device must have before the task is allowed.

7.2. Study Design 130

7.2.1

Participants

The 30 participants ranged in age from 20 to 58 years old (median = 26.5, mean = 29.4). All participants were mobile device owners currently living in the United Kingdom, and thus had experience with the UK mobile phone network. 60% of the respondents were Android users with various handset models, 13% were iPhone users, 10% used a Blackberry and the remaining 17% used a feature phone (i.e., non-smart phone). 17% of the participants were female and 83% were male. Participants were recruited using convenience sampling meth- ods. Specifically, they were recruited through a combination of email invitations, requests for participation to university classes, and word-of-mouth from other participants to friends and family.

Each study participant was randomly and evenly assigned to either the All, Some or None category. The random nature of this selection is essential in order to avoid study bias and to distribute other possibly confounding influences on the study outcome across all three participant groups.

7.2.2

Apparatus and Materials

Each participant used an Apple iPhone 4 with iOS version 5.1.1 that was provided for their use during the study. It was pre-loaded with the study application and preset with the partici- pant’s randomly chosen category and a starting device confidence of Low for all participants. Since the device was provided by the experimenter, it was possible to control the operating system version, as well as other applications and data on the device. This limited the ef- fects of other authentication methods, applications and data that may have interfered if the participant’s own device had been used, as was done in the keystroke dynamics and speaker verification studies. The experimenter recorded the interviews, with participant permission, using the Voice Memo application on another iPhone.

After the participants had interacted with the transparent authentication application, they were asked a series of questions in a semi-structured interview in order to collect their opin- ions on the security levels, perception of barriers and needs for transparent authentication. The interview responses provided answers to the questions that drove this research.

7.2.3

Procedure

The study began with a short questionnaire designed to elicit the participant’s age range, gen- der and whether they currently own and use a smartphone. The participant was then given a short introduction to transparent authentication, and introduced to the Apple iPhone and the custom application they would use for the experiment. The participant was told that a

7.2. Study Design 131 transparent authentication method was running on the device. This description included a discussion of keystroke dynamics and speaker verification as behavioral biometrics, and the role of explicit authentication (i.e., challenge questions) in transparent authentication. The participant was instructed on how to turn off the authentication and answer the challenge question to override the transparent authentication should they wish to at any point during the experiment. These steps were taken to build a mental model of the intended transpar- ent authentication method, although the actual working of the application depended on the category to which the participant had been assigned.

Since this study was designed in a Wizard of Oz style, there was no authentication system of any type running on the device; the required device functionality was allowed or disallowed based on the settings entered by the experimenter using the interface shown in Figure 7.1. Each participant began the study at the “Low” security level and the category to which they were randomly allocated. Participants were then given an information sheet that outlined the steps for each task they were to perform, as well as instructions on how to answer their chal- lenge question and how to turn off the transparent authentication method. A custom iPhone application was designed for this experiment that was unlike Apple’s usual icon approach to tasks and applications; this was done intentionally so that the participant had a sense of using something different than the usual Apple interface.

Figure 7.1: TAP Study setup screen. This was not seen by the study participants. Upon launching the study application, the participant was prompted via an alert box to set the answer to their challenge question (see Figure 7.2a). The challenge question was provided as a backup to the transparent authentication method. In the case where the current device confidence is too low to allow access to a task, the challenge question would be used to authenticate the user and raise the device confidence to the next level.

7.2. Study Design 132 screen that the participant was able to attempt each task. The participants were given a detailed instruction sheet that they used to complete each task so that each participant com- pleted the tasks in the same order and using the same methods. This was important so that the user perceptions of security and task difficulty were not affected by the order of events. The order of the tasks were from low to high security, and affected whether or not the explicit authentication method was required.

(a) Challenge question alert (b) TAP application Tasks screen

Figure 7.2: Screenshots of the starting screens for the TAP application.