SAND2015-1850C
There’s Hope for Johnny:
Comparing Automatic and Manual Encryption in Email
t* t t
Scott Ruoti, Jeff Andersen, Travis Hendershot,
* t t
Yung Ryn Choe, Daniel Zappala, Kent Seamons
Brigham Young University
t{ruoti, andersen, hendershot} @ isrl.byu.edu, {zappala, seamons} @ cs.byu.edu
*
Sandia National Laboratories
yrchoe@sandia.gov
ABSTRACT
Usable, secure email remains an open problem. Recent research examined manual encryption, the practice of allowing users to view encrypted ciphertext, and found that it helped users better understand secure email and avoid mistakes. In this work, we formally test whether manual encryption affects usability, users’ understanding of secure email, and users’ ability to avoid mistakes. We test these hypotheses using two versions of the Private WebMail (Pwm) system, one that supports manual encryption and one that is fully automatic. Our results demonstrate that after accounting for confounding usability problems, manual encryption does not have a significant effect on usability, users’ understanding of secure email, or users’ ability to avoid mistakes. Additionally, our results show that our versions of Pwm score high on the System Usability Scale (SUS), falling in the 85th to 90th percentile of a large number of systems tested with SUS. We also find that participants are eager to embrace secure webmail and begin using Pwm regularly.
1. INTRODUCTION
Usable, secure email is still an open problem more than 15 years after Whitten and Tygar’s seminal paper, Why Johnny Can’t Encrypt [16]. Recently, there is a renewed interest in secure communications (including email) motivated in part by the recent revelations of broad * Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energys National Nuclear Security Administration under contract DE-AC04- 94AL85000
Copyright is held by the author/owner. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee.
Symposium on Usable Privacy and Security (SOUPS) 2015, July 22-24,
2015, Ottawa, Canada.
Internet surveillance. The Electronic Frontier Foundation (EFF) has published a secure messaging scorecard as the first phase of a campaign to promote the development of tools (including email) that are secure and usable for ordinary users.1 However, to the best of our knowledge, only three laboratory usability studies for secure email have appeared in the research literature since Whitten and Tygar’s work [8, 12, 10].
In a recent book, Garfinkel and Lipford summarize the work on secure email and identify a key open problem in this area: whether automatic and transparent encryption improves the usability of encrypted email or messaging systems [6]. Here, automatic encryption refers to both hiding the key management details that proved to be so problematic in the Whitten and Tygar’s study, as well as hiding ciphertext from end users. The alternative approach discussed by Garfinkel and Lipford is manual encryption, which refers to showing the user ciphertext to help them understand that encryption has occurred.
Two recent studies on this issue contain mixed results. A study by Fahl et al. [5] examined manual and automatic encryption for Facebook messaging and found that both manual and automatic message encryption fared equally well in terms of usability, but did not explore how they affected users’ understanding of what had occurred. A study by Ruoti et al. [10] compared Private WebMail (Pwm), an automatic encryption system that is tightly integrated with Gmail in the browser, with MP, a manual encryption system that operates as a stand-alone desktop application and requires users to cut-and-paste ciphertext into their email. During a laboratory user study, participants using MP’s manual encryption made fewer mistakes and answered more questions correctly on a quiz that tested their understanding of the system. However, the results of the Ruoti study are inconclusive because there are significant differences between Pwm and MP that introduce confounding factors, thus raising questions about the better scores seen with MP.
In this work, we formally test three hypotheses regarding automatic and manual encryption: automatic and manual encryption differ in their usability, manual encryption does better than automatic encryption at helping users understand secure email, and manual encryption does 1 https://www.eff.org/secure-messaging-scorecard
better than automatic encrypt at helping users avoid mistakes. To test these hypotheses, we use a modified version of Ruoti et al.’s Pwm system. We first fix several existing usability problems with Pwm, providing a fairer comparison between automatic and manual encryption and reducing confounding factors. We then test our hypotheses by conducting an A/B test using two identical versions of Pwm that differ only in their support for automatic and manual encryption. We evaluate the hypotheses based on usability scores derived via the System Usability Scale (SUS), a questionnaire that assesses user understanding, and quantitative measures of how often users sent sensitive information without encryption.
Our results demonstrate that after accounting for confounding usability problems, manual encryption does not have a significant effect on usability, users’ understanding of secure email, or users’ ability to avoid mistakes. In addition, our improved version of the Private WebMail system scores an 80.0 on the System Usability Scale (SUS), rating in the “excellent” category for usability and receiving an “A” grade. This score is in the 85th to 90th percentile of a large number of systems tested with SUS [11]. We also find that participants are eager to embrace secure webmail and most want to start using Pwm.
2. RELATED WORK
Whitten and Tygar [16] conducted the first formal user study of a secure email system (i.e., PGP 5). They found serious usability issues with key management and users’ understanding of the underlying public key cryptography. The majority of the users were unable to successfully send encrypted email in the context of a hypothetical political campaign scenario.
Garfinkel and Miller [8] repeated the Johnny experiment with a slightly modified scenario in the context of automatic key management. They focused on S/MIME and its ability to include the sender’s public key along with the email message. They implemented co-Pilot, a client-side email tool in Outlook Express that supports Key Continuity Management (KCM) with S/MIME. Once the user obtains the key from a sender of an email message (trust on first use), the system will alert the user if they ever receive an email message from that sender that is signed or encrypted with a different key. The results showed that automatic key management was more usable than the manual key management in the original Johnny experiment. However, the study revealed that the tool “was a little too transparent” in how well it integrated with Outlook Express. Some users failed to read the instructions associated with visual indicators.
Sheng et al. [12] conducted a small pilot study by repeating the Johnny experiment to determine whether PGP 9 had made usability advances in the eight years since PGP 5 was tested. Even though improvements had been made, some users still struggled with key management. The encryption and decryption had become so transparent that users were unsure if a message they received had actually been encrypted.
Ruoti et al. [10] conducted a series of user studies with Private WebMail (Pwm), a secure email prototype that tightly integrates with Gmail. Even though results showed the system to be quite usable, they found that some users
made mistakes and were hesitant to trust the system since the automatic encryption was too transparent. They also conducted a study comparing Pwm to a desktop application (MP) that supported manual encryption. They found that participants using MP’s manual encryption made fewer mistakes and answered more questions correctly on a quiz that tested their understanding of the system. However, the results are inconclusive because in the Ruoti study there are significant differences between the two systems that introduce confounding factors, thus raising questions concerning how to interpret the results of the study. Their results provided a direct impetus for our research, as we conduct an A/B test of a modified version of Pwm to formally comparing manual and automatic encryption.
Fahl et al. [5] explored manual and automatic encryption in the design of a Facebook chat system. Their user study demonstrated that users preferred automatic key management, but found no significant difference between automatic and manual encryption. They raised the issue that transparency could impact users’ feelings of trust and recommended the issue be addressed in future work.
Garfinkel and Lipford [6] recently published a book that gives an overview of results from the usable security field, including lessons learned and challenges ahead. In their discussion of secure email, they focus on the issue of automatic, transparent encryption (see pages 53-54). They give a detailed overview of the findings from Ruoti et al. [10] that raised concerns about the transparency of automatic encryption and contrast that with the favorable results for automatic encryption found by Fahl et al. [5]. The results in our paper are aimed at addressing this open problem described therein.
3. VALIDATING MANUAL ENCRYPTION
The goal of our research is to determine whether manual encryption is able to increase usability, improve understanding and help users avoid mistakes when sending secure email. More specifically, we are interested in determining if these benefits always exist or only when other usability issues are present, as in Ruoti et al.’s former work [10]. Based on these benefits, we form three sets of hypotheses.
Increased Usability
H0 Automatic and Manual encryption have similar usability.
H1 Automatic and Manual encryption differ in their
usability.
Improved Understanding
H2 Automatic and Manual encryption do equally well at helping users understand secure email.
H3 Manual encryption does better than automatic encryption at helping users understand secure email. Helped Users Avoid Mistakes
H4 Automatic and Manual encryption do equally well at helping users avoid mistakes.
H5 Manual encryption does better than automatic encryption at helping users avoid mistakes.
□ * Encrypted Sensitive Information - This is a plaintext greeting giving context to the message. This message has been encrypted using Pwm
Figure 1: An encrypted email in the inbox
Since this research is verifying the claims made by Ruoti et al., we decided to use their Private WebMail (Pwm) system to test our hypotheses. After obtaining the source code for Pwm, we modified it and created two versions: a version that uses automatic encryption and a version that uses manual encryption. Other than these two differences, the systems were otherwise identical. This allows us to conduct an A/B test that specifically examines the benefits of manual encryption as compared to automatic encryption.
The remainder of this section describes Pwm, explains the differences between the automatic and manual encryption versions of Pwm, and details other modifications we made to Pwm's interface to remove usability issues that might otherwise have been confounding factors in our study.
3.1 Private WebMail (Pwm)
Pwm implements secure email through tight integration with Gmail. When users read or compose secure email, Pwm replaces portions of Gmail's interface with Pwm's own secure interface. Pwm's interfaces are styled differently than Gmail's, providing a clear demarcation of which information is being protected by Pwm.
All secure email sent using Pwm includes instructions on how to setup Pwm for first-time users (see Appendix D.) Pwm's design includes a key escrow server, so anyone can receive a secure email message without having taken any prior steps, such as creating and publishing a public key. The setup instructions direct new users to the Pwm website, where they are able to add a bookmarklet to their browser's bookmark storage.2 The new user then returns to Gmail and clicks on the Pwm bookmarklet to run Pwm. The benefits of using a bookmarklet to run Pwm are that users do not need installation permission on the machine and also avoid any worries related to installing extensions [10]. The drawback is that users are required to click on the Pwm bookmarklet each time they reopen Gmail.
When Pwm is running, secure email messages are automatically decrypted for users. The decrypted contents of the message are displayed in place of the instructions and ciphertext that were in the email message's body (see Figure 2). Optionally, the sender of the message can also include a plaintext greeting with the encrypted email message. This message is shown to the user above the decrypted contents of the email message. Encrypted email messages in the user’s inbox are marked as Encrypted (see Figure 1).
By default, Pwm does not encrypt all email sent by a user, but instead requires that they manually enable encryption on a per-message basis (see Figure 3). Once enabled, Pwm replaces the compose interface with its own interface (see Figure 4). It also annotates the to and subject fields to describe how Pwm handles these two fields. Finally, it adds an area where users can include a plaintext greeting that 2A bookmarklet is a browser bookmark that contains executable JavaScript. This JavaScript is run on the page that users are viewing when the bookmark is clicked.
Figure 2: The read interface for an encrypted email.
Figure 3: Gmail compose interface before enabling encryption
appears before the encrypted part of their email message. Pwm’s key management is handled by a key escrow server. Authentication is done using email-based identity and authentication [7, 15]. Pwm is able to retrieve users’ email and is able to complete authentication without ever prompting users for input.
3.2 Automatic and Manual Encryption
The automatic encryption version of Pwm encrypts and sends email as soon as the user clicks Send encrypted (see Figure 4). We created a manual encryption version that splits this operation into two distinct steps. In the first step, instead of clicking the Send encrypted button, the user instead clicks the Encrypt button. Upon doing so, the user’s email message is encrypted and both the instructions for decrypting the email message and the encrypted ciphertext are shown to the user inside Gmail’s compose interface (see Figure 5). This allows the user to confirm that encryption has actually taken place. In the second step, the user then clicks Gmail’s Send button to send the encrypted email.
3.3 Modifications to Pwm’s Interface
Ruoti et al.’s work indicated that manual encryption improved users’ understanding and helped them avoid mistakes, but it also noted several issues with Pwm’s
Figure 4: Pwm’s compose interface
Figure 5: Ciphertext shown to users after manual encryption
design that might have contributed to these problems. To better isolate the benefits of manual encryption we have fixed these potentially confounding usability issues.
3.3.1 Delayed Encryption
One concern expressed by a significant portion of participants in the studies by Ruoti et al. was that Pwm encrypted email so quickly that they were unsure if it had really done anything. Also, participants indicated the encryption process was so transparent that they were unsure who could read their encrypted email message.
To address both of these problems we added an artificial delay after users click the Send encrypted or Encrypt buttons. For each email recipient, users are shown a message lasting 1.5 seconds that states the email is being encrypted for that user (e.g., “Encrypting for bob@gmail.com”). This helps users understand who will be able to read the encrypted email and also lets them feel that something substantial has happened during the encryption process. Because most email messages have a small number of recipients, this artificial delay does not
significantly impact the overall email experience.
3.3.2 Compose Interface
In order to prevent the contents of users’ drafts from being readable by Gmail, Pwm provides an encrypted-mode compose interface, which is overlayed on top of Gmail’s own interface when encryption is enabled. To better conform to the flow of composing email messages in Gmail (moving from top to bottom), we moved the button which enables encryption to the very top of Gmail’s compose interface (see Figure 3).3
When encryption is enabled, we modified the placeholder text for the recipients and subject fields, explaining how Pwm uses these two fields (see Figure 4). We also added informative text to the top of the compose interface to make clear the current email message’s encryption status (see Figure 4 and Figure 5). Furthermore, in the automatic encryption version we modified Gmail’s Send button to read Send unencrypted, to ensure that users knew that Gmail’s messages are not encrypted by default. We did not make the same change for the manual encryption version as the same Send button is used to send both encrypted and unencrypted email messages.
Finally, we added functionality allowing users to compose plaintext greetings that are included with the encrypted email (see Figure 2 and Figure 4). Although Ruoti et al.’s Pwm mentioned this functionality, their interface had no place for users to compose greetings. We include the greeting as it can help engender trust in a new user that receives an encrypted email, giving them confidence to setup and use Pwm. It also provides context for encrypted email messages before they are decrypted.
3.3.3 Look and Feel
We designed a new website for helping users install Pwm.4 This website was created using the Bootstrap Freelancer theme,5 and has a more professional look and feel than the original Pwm website. Additionally, we standardized the look and feel of Pwm to use the same colors and styles as the website, giving a consistent experience to all Pwm-based user activity.
3.3.4 Tutorials
To improve the user experience, we also added tutorials to Pwm. These tutorials appear the first time a user runs Pwm, reads an encrypted email, or composes an encrypted email. The tutorials are intended to help users understand how secure email through Pwm works. Participants have the option to skip the tutorials and can also choose to rewatch tutorials at their convenience.
The tutorials provide step-by-step instructions on how to use Pwm. Each step uses simple language and instructs about a single feature of Pwm (see Figure 6). Some tutorial steps require explicit action from users before they can move on to the next tutorial step, helping users internalize correct behavior (see Figure 7).
3This button was originally on the bottom of Gmail’s compose interface, next to the Send button.
4 [URL redacted at this time]
5http: //startbootstrap.com/template-overviews/ freelancer/
Figure 6: Style of the tutorial window
Figure 7: Tutorial waiting for action from the user
Introduction.
This tutorial is shown to users the first time they run Pwm. It informs users that Pwm will help protect their email and tells them how to identify whether Pwm was running.
Read.
The first time users read an encrypted email message, they are shown this tutorial. The tutorial shows users how to identify an encrypted email, explains the plaintext greeting, and identifies which portions of the email message were encrypted. Additionally, it informs them that email messages encrypted with Pwm are provided authenticity and confidentiality (even from Gmail).
Compose.
The first time users compose an email message while Pwm is running, they are given the option to watch a tutorial describing how to compose encrypted email. This tutorial teaches users the correct order of operations for composing an encrypted email. It also informs them about who can read their message, the purpose of the optional greeting, and where to type sensitive information.
4.
METHODOLOGY
We conducted an IRB-approved user study comparing automatic encryption with manual encryption. This section gives an overview of the study and describes the scenario design, user tasks, study questionnaire, survey development, and limitations.
4.1 Study Setup
The study ran for two weeks, beginning Tuesday, February 17, 2015 and ending Tuesday, March 3, 2015. In total, 52 individuals participated in our study. Participants were randomly assigned to test either the automatic or manual encryption version of Pwm. Participants took a
minimum of 30 minutes and a maximum of 60 minutes to complete the study. We compensated each participant $10 USD for their efforts.
Studies were conducted in a room that had been set up specifically for this study. When participants first entered the room, they were read a brief introduction to the study (see Appendix A.1). For the remainder of the study, all instructions were written and provided to them, either via a printed information sheet or via email. Participants completed all tasks on a virtual machine, ensuring that the computer started in the exact same state for all participants.
Two study coordinators were involved in the study. One coordinator sat in the same room as the participant and sat within the participant’s peripheral vision. This coordinator was instructed to avoid prompting participants and to only assist participants if they had not made any progress within five minutes. The second coordinator sat in another room and corresponded with the participant over email as part of the study tasks.
4.1.1
Quality Control
We removed one participant from the results of the survey. This participant’s Gmail configuration was such that Pwm did not work. In the process of trying to address the problem, this participant interacted with the coordinators significantly more than any other participant. Additionally, they were the only participant to not use their own email address. We chose to discard this participant’s results as the two preceding effects may have biased their responses. For the remainder of this work we report results based on the 51 remaining participants.
4.1.2 Participant Demographics
We recruited Gmail users for our study at a local university. Participants were evenly split between genders: male (25; 49%), female (26; 51%). Participants skewed young: 18 - 24 years old (45; 88%), 25 - 34 years old (3; 6%), 35 - 44 years old (2; 4%), 55 years or older (1; 2%).
We distributed posters broadly across campus to avoid biasing our results by any particular major. The poster is available in Appendix C. All participants were affiliated with the university,6 with the overwhelming majority being undergraduate students: undergraduate students (44; 86%), graduate students (2; 4%), faculty and staff (5; 10%). Participants had a variety of majors, including both technical and non-technical majors. No major was represented by more than three participants, with the vast majority only having a single participant.
Participants were asked how often they logged into Gmail on the web. Most participants indicated they checked their email on the web many times a day: many times a day (39; 76%), once a day (3; 6%), 2-3 times a week (2; 4%), once a week (2; 4%), 2-3 times a month (2, 4%).
4.2 Scenario Design
During the course of the study, participants were given two scenarios to complete: being hired for a new job and sending credit card information to a spouse.
Both of these scenarios were designed to simulate situations that participants could realistically imagine 6 We did not require this affiliation.
using secure email to complete. Participants completed tasks for the first scenario before beginning the second scenario. Prior to beginning each scenario, participants were provided with a written description of the scenario (see Appendix A.2). This description included information that participants should send in place of their own personal information.7 Participants were asked to treat this sensitive information with the same care as they would treat their own information.
1. Being hired for a new job. Participants were told that they had recently returned from an interview at National Citadel, a fictitious company created for this study. As part of this scenario they had to submit receipts in order to be reimbursed for travel expenses, accept an offer of employment, and complete several hiring tasks.
2. Sending credit card information to a spouse. In this scenario participants were told that their spouse had texted them asking for login information to a credit card website. Participants were also told that they wanted to send this information encrypted over email, but that their spouse had never before used secure email. Participants were asked to send their spouse the requested information, taking whatever steps they felt necessary to ensure their spouse could read the encrypted email.
4.3 Task Design
Based on the two scenarios, we designed several tasks according to two goals. First, we wanted to have realistic tasks that users could envision completing as part of the scenarios. Second, we wanted tasks that used as many email features as possible. Additionally, we designed the tasks without considering Pwm’s feature set, as we wanted to avoid biasing the tasks to reflect favorably on Pwm.
Tasks were completed entirely using email, and participants used their own email accounts. Participants were presented with tasks sequentially. If participants accidentally sent sensitive information without encryption they were notified of their mistake in an email and asked to resend the information using encryption. The exact wording of the task emails is given in Appendix A.3.
Being hired for a new job.
Task 1. Participants received an email from National Citadel containing instructions on how to be reimbursed for expenses from their recent interview. Participants were told to send their Social Security number and a picture of their receipts. They were informed that this information must be encrypted as per National Citadel’s policies. This email also instructed users to set up and use Pwm to encrypt their email messages.8
This task was designed to test whether participants were able to set up and use Pwm having only been provided 7We took this approach to avoid situations where sensitive information might have been leaked to Gmail.
8These instructions were generated by Pwm and are shown in Appendix D
with instructions that might be reasonably expected from a company requesting information to be encrypted.9 Task 2. Participants were first asked to close their browser and then reopen Gmail. They were informed that this simulated several weeks passing after the completion of task 1. Participants were then sent an encrypted email that contained an offer letter. They were asked to reply with their acceptance. They were also asked to CC their new manager.
This task was designed to test whether participants would remember how to enable Pwm. It also tested whether they could use Pwm to CC a new recipient.
Task 3. Participants were sent an email instructing them to email information to a background check company. They were instructed to encrypt the information.
This task was designed to test whether users could enable encryption, either by forwarding the request for information or by composing a new email message.
Task 4. Participants were instructed to send bank account information to National Citadel’s payroll department. They were not reminded to encrypt this information.
This task was designed to test whether users would remember to encrypt information if they were not explicitly prompted to do so. Unlike the preceding tasks, if they sent information without encryption, we still considered this task complete.
Sending credit card information to a spouse.
Task 5. As described in the scenario, participants sent login information to their “spouse” using Pwm. It was left up to the user to decide how to best prepare their spouse for receiving their first Pwm message.
This task was designed to see how participants would induct a new person into using secure email. Regardless of what instructions were sent, we considered this task complete when the information had been sent.10
Task 6. Participants received another email from their spouse asking them for additional credit card information. This request was not encrypted and did not instruct the participant to send the additional credit card information encrypted.
This task was designed to test whether users would remember to encrypt information if they were not explicitly prompted to do so. Unlike most of the preceding tasks, if they sent information without encryption, we still considered this task complete.
To test hypotheses H4 and H5, we measure the number of mistakes participants make while attempting these tasks. 9This task also shows that we designed the tasks around normal email usage and not Pwm. Pwm does not support attachments and only allows for images to be inserted inline with the email body. This task could potentially be confusing to users as they cannot use their normal work-flow for attaching an image.
10Pwm includes instructions by default, and since they were sufficient to help the participant start using Pwm, it was reasonable to assume that the participant believed they would be sufficient to help their spouse start using Pwm.
4.4 Study Questionnaire
We administered our study using the Qualtrics web-based survey software. Before beginning the survey, participants were read an introduction by the study coordinator and asked to answer a set of demographic questions. Participants then completed the six study tasks, following which they were asked to complete a questionnaire regarding their experience. The full text of this questionnaire can be found in Appendix B.2.
To test hypotheses H0 and H1, regarding the usability of each version of Pwm, we had participants complete the ten System Usability Scale (SUS) questions [3, 4]. Answers to these questions are used to derive each version’s SUS score, a single numeric score from 0, the least usable, to 100, the most usable, that provides a rough estimate of the version’s overall usability. Recent research has shown that SUS scores are effective for comparing systems across different study populations and is the usability statistic used in Ruoti et al.’s original work [10]. Moreover, Tullis and Stetson compared SUS to four other usability metrics (three standard metrics from the usability literature and their own proprietary measure) and determined that SUS gives the most reliable results [14].
After completing the SUS questions, participants were asked several questions designed to ascertain whether they would want to use Pwm in their day-to-day lives. These included questions on whether they wanted to have all or only some of their email encrypted, whether they wanted to begin using Pwm, and whether they would feel comfortable using Pwm with their family and friends. Participants were also asked to describe what they liked about Pwm, what they would change about Pwm, and how Pwm could be more applicable to their own lives.
To test hypotheses H2 and H3, we ask participants questions to examine how well they understood the cryptographic properties of Pwm. To test understanding of confidentiality, they were asked to indicate which parties could read a message encrypted with Pwm. Similarly, participants were asked whether Pwm provided authenticity and integrity for secure email messages composed with Pwm. Each question was asked in language that would be approachable to users and did not require a technical background. For each property, participants were given the option to indicate that they were unsure whether that property was provided by Pwm.
The survey also asked participants how they preferred to send sensitive information. They indicated whether they had ever used secure email before, and if so for what and why. Participants were asked whether they would be more likely to use email to send sensitive information if secure email was available to them. Then they were asked whether they would be willing to pay for secure email, and if so, how much.
Finally, we asked participants how they would prefer to have secure email implemented. The options were,
Integrated tightly into a web-based email system. • Integrated tightly into a desktop/mobile application. • A desktop/mobile application that has a separate
interface for encrypting messages.
A browser extension that has a separate interface for encrypting messages.
A web page that has a separate interface for encrypting messages.
A email provider that only supports encrypted email. Participants were asked to rank these options according to their preferences, with the options initially being ordered randomly. Participants were also asked why they preferred their top-ranked preference.
4.5 Survey Development
We began the design of our study before completing the implementation of Pwm. As mentioned earlier, we designed the study to test secure email first and Pwm second. This study development approach allows us to feel more confident that participants were interacting with Pwm in a realistic fashion. To help ensure that this goal was met, we included researchers who are not involved in our secure email research to help design the scenarios and tasks. After creating scenarios and tasks that were acceptable to all parties, we then conducted a pilot study with three participants. We did not identify any significant flaws during the pilot study.
4.6 Limitations
While our studies included students with a diverse set of majors and technical expertise, it would be beneficial for future studies to verify our results using non-student populations. Gmail users may also not be representative of the general population’s preferences regarding secure email, and further studies should be conducted with users of other email systems.
Our study is short-term and is not necessarily representative of how participants would use secure email over a longer period of time. Once our implementation of Pwm has been proven to be sufficiently usable and secure in short-term studies, it is important that it then be analyzed in a more comprehensive long-term study. This could potentially reveal additional information about the advantages and disadvantages of manual encryption.
Our study is a lab study and has limitations common to all studies run in a trusted environment [9, 13]. While there are indications that some participants treated the provided sensitive information as they would their own (e.g., refusing to email the provided social security number), there is still no guarantee that participants’ reactions mimic their real life behaviors. Additionally, our studies did not test participants’ ability to resist attacks.
5.
RESULTS
In this section we report the quantitative results from our user study. First, we report data related to our hypotheses. Second, we report other quantitative data related to participants’ willingness to adopt secure email, preferences for how secure email is implemented, and prior experience using secure email.
5.1 Usability
We evaluated Pwm using the System Usability Scale (SUS). The automatic encryption version of Pwm had a SUS score of 79.1 and the manual encryption version had a SUS score of 81.1. This difference was not statistically significant (two tailed student t-test, unequal variance — p = 0.43). Further breakdown of the SUS scores, along
40
60
70
80
90
Percentiles_______ 15_______________35 j_________ j 65 j j 85 j 95______________________ 100
F D c B A A+
OK Good Excellent Best
Not acceptable Marginal Low M.Hiqh Acceptable
40
50
60
70
80
90
100
SUS Score
Figure 8: Adjective-based ratings to help interpret SUS scores Confidentiality Authenticity Integrity
Automatic 27 23 1 3 17 1 9 22 0 5 85% 4% 11% 63% 4% 33% 81% 0% 19% Manual 24 20 1 3 15 0 10 17 1 6 83% 4% 13% 63% 0% 42% 71% 4% 25% Overall 51 43 2 6 32 1 19 39 1 11 84% 4% 12% 63% 2% 37% 76% 2% 22%
Table 2: Participants’ understanding of Pwm’s confidentiality, authenticity, and integrity properties
C o u n t M ea n S ta n d ar d D ev ia ti o n M in O’ Med ia n CO a Max Automatic 27 79.1 9.6 62.5 71.3 77.5 86.3 97.5 Manual 24 81.1 9.0 62.5 76.9 81.3 87.5 97.5 Overall 51 80.0 9.2 62.5 75 80 87.5 97.5 Ruoti et al. 72 74.2 13.7 42.5 67.5 72.5 85 100
Table 1: SUS scores
with SUS scores from Ruoti et al.’s earlier study, can be found in Table 1. Based on these results, we fail to reject the null hypothesis H0 for usability.
To give greater context we leverage the work of several researchers. Bangor et al. [2] analyzed 2,324 SUS surveys, and derived a set of acceptability ranges that describe whether a system with a given score is acceptable to users in terms of usability. Bangor et al. also associated specific SUS scores with adjective descriptions of the system’s usability. Using this data, we generated ranges for these adjective ratings, such that a score is correlated with the adjective it is closest to in terms of standard deviations. Sauro et al. [11] also analyzed SUS scores from Bangor et al. [1], Tullis et al. [14], and their own data. They calculate the percentile values for SUS scores and assign letter grades based on percentile ranges. The above contextual clues are presented in Figure 8.
Using this context, Pwm’s SUS score of 80.0 is rated as having “excellent” usability and given an “A” grade. It falls between the 85th and 90th percentile. We used Ruoti et al.’s SUS results from their earlier studies and calculated that their implementation of Pwm had an average SUS score of 74.2. This score is rated as having “good” usability,
given a “B” grade, and falls between the 70th and 80th percentile. The difference between our implementation and that of Ruoti et al. is statistically significant (two tailed student t-test, unequal variance — p < 0.01).
5.2 Understanding
We asked three questions to determine whether each participant understood the confidentiality, authenticity, and integrity properties provided by Pwm. The responses indicated whether each participant correctly understood the principle, had some misunderstanding, or was unsure. Table 2 summarizes these results.
In all three cases, the difference between manual and automatic encryption was not statistically significant: Confidentiality X2[2, N = 51] = 0.03, p = 0.98 Authenticity X2[2, N = 51] = 1.10,p = 0.58 Integrity X2[2, N = 51] = 1.56, p = 0.46
Based on these results, we fail to reject the null hypothesis H2 for understanding.
5.3 Avoiding Mistakes
During the study, we recorded all instances of a participant taking an action that deviated from the study parameters. Using this data, we identified several instances where a participant’s actions leaked sensitive information. These results are reported by task in Table 3.
Tasks two, five, and six had no mistakes, and the differences in the remaining three tasks were not statistically significant.
Task 1 X2[1,N = 51] = 2.85, p = 0.09 Task 2 X2[1,N = 51] = 0.00, p =1.00 Task 3 X2[1,N = 51] = 0.003,p = 0.95
their fourth to sixth choice. 5 O O i— (M OO ^ lO ©
^
CO^ ^ ^ ^ ^
CO CO CO CO CO&&&&&&
H H H H H H Automatic 27 0 0 1 0 0 0 Manual 24 4 0 0 1 0 0 Overall 51 4 0 1 1 0 0Table 3: Number of participants who sent sensitive information without encryption
Of the four mistakes in Task 1, only one was related to manual encryption: the participant had encrypted their message correctly, but after encryption used Gmail’s compose interface to modify the encrypted email and add the sensitive information. In the second mistake, the participant had also encrypted their message correctly, but then reloaded the page, which deactivated Pwm, and used Gmail’s compose interface to modify the encrypted draft and add the sensitive information. The remaining two mistakes were from participants who never attempted to install Pwm and sent the sensitive information using Gmail.
Based on these results, we fail to reject the null hypothesis H4 for avoiding mistakes.
5.4
Other results
5.4.1 Acceptability of Pwm
Participants were asked for their opinions about the need for secure email and also asked if they would be willing to use Pwm in their day-to-day lives. These responses are summarized in Figure 9.
Participants are unanimous (51; 100%) in wanting the ability to be encrypt sensitive email, but only a quarter of participants (12; 24%) want all email encrypted. More than half of the participants (27; 53%) disagree with encrypting all email.
Over four-fifths of participants (42; 83%) want to start using Pwm. More than nine out of ten people (47; 92%) agree that their friends and family could easily start using Pwm, and three-fourths (37; 73%) indicate they would use Pwm to send encrypted email to their friends and family.
The majority of participants agree that Pwm protects their email (35; 69%). The rest are not sure (16; 31%). No one disagreed with the statement that Pwm protects their email.
5.4.2 Implementation Preferences
Pwm is an implementation of secure email that tightly integrates with existing web apps. We asked participants to rank other potential methods for implementing secure email. Their reported preferences are given in Figure 10. Nearly all participants (44; 87%) prefer a system that is tightly integrated with their existing webmail solution: most preferred (28; 55%) or second most preferred (16; 31%). The next most popular option is having secure email implemented into an existing desktop or mobile application: most preferred (10; 20%) or second most preferred (17; 33%). The remaining implementation methods had the majority of participants rank them as
5.4.3 Sending Sensitive Information
Half of the participants (25; 49%) indicated that they have sent sensitive information over email. Common types of information include bank account information, social security numbers, passwords, and job specific information. Of the participants who have sent sensitive information over email, most indicated doing it infrequently: less than once a month (19, 76%), once a month (4; 16%), 2-3 times a week (1; 4%), many times a week (1; 4%).
Participants were asked to indicate which forms of communication they preferred to use to send sensitive information. Most participants indicated they preferred to share sensitive information over the phone or in person: over the phone (37; 73%), in person (36; 71%), using email (10; 20%), in a cellular text message (6; 12%), using a secure website (2, 4%), using a fax (1, 2%). Four-fifths of participants indicated that they would be willing to use encrypted email to send sensitive data: willing (40; 79%), depends on who and what (8; 16%), unwilling (3; 6%).
5.4.4 Willingness to Pay
Participants were undecided about whether they were willing to pay for encrypted email: willing(1; 2%), it depends (35; 69%), unsure (4; 8%), unwilling (11; 22%). The participants that were unsure mentioned several factors that would determine if they were willing to pay for secure email: the cost, whether the cost was one-time or recurring, and whether secure email was required for their job. Excluding the participants that were unwilling to pay for email encryption, the remaining participants (40; 79%) were willing to pay on average $5 USD per month (median — $5 USD, mean — $6.105 USD, mode — $2 and $5 USD).
5.4.5 Prior Use of Secure Email
Three participants (3; 6%) had previously used secure email. One had used PGP, another had used Lotus Notes’ built-in encryption, and the third had used a secure email prototype but wasn’t sure what it was called. The participant who had used PGP described his experience, saying,
“Had to generate a key pair, configure the plugin, then distribute the public key (and no one I used it with could figure out how to unencrypt anyway, so I abandoned it). ”
6. DISCUSSION
In this section we begin with a discussion of automatic and manual encryption. We follow this by discussing participant experiences, opinions, and preferences regarding secure email. Finally, we detail lessons learned about our implementation of Pwm.
In this section, participants are referred to as R1 - R52, with the number corresponding to the order in which they participated in our study.
6.1
Automatic and Manual Encryption
For each set of hypotheses, we failed to reject the null hypothesis. This indicates that by adding additional features to automatic encryption (e.g., tutorials, delayed
I want to be able to encrypt all of my email. I want to be able to encrypt sensitive email. I want to start using Pwm.
My friends and family could easily start using Pwm.
I would use Pwm with my friends and family. Pwm protects my email.
Strongly Agree ^|Agree Neither Agree nor Disagree (Disagree Strongly Disagree
Figure 9: Participants’ opinions regarding Pwm (number of participants)
encryption, improved task flow) we are able to replicate the benefits of manual encryption. Additionally, several participants who used the manual encryption were surprised that their messages were not automatically sent. R8 and R29 stated, respectively,
“[...] I had to click to encrypt it and then click to send. I liked the thought of it being one action. ”
“The one thing I would change is the interface of the email after you click encrypt. The first time I encrypted a message, it seemed to disappear. I wasn't sure if it had been sent until I realized that I had to click the send button. I think you could improve this if you made a big blue or green bar in the email that said, ”This contains your encrypted message” and then if you clicked on it you could see what you wrote. ”
Still, this is not an indictment of manual encryption. Our results demonstrate that manual encryption does not negatively impact usability, user understanding, or mistake avoidance. While other features were able to provide the same benefits as manual encryption, we still believe that manual encryption is a valid tool that system designers could choose.
6.1.1
Understanding
Our results demonstrated that most participants understood what cryptographic features were provided by Pwm. Still, there was a significant number of participants that indicated they were unsure about the protections provided by Pwm. This indicates that Pwm could do more to instruct and aid users in their understanding of how their email is protected.
Additionally, care should be taken in evaluating our results on understanding. The questions on authenticity and integrity asked users whether Pwm provided these cryptographic properties, but did not verify this understanding with additional questions. It is possible that participants answered yes to these questions in response to the study’s trusted environment [13]. For example, R26
indicated that Pwm would warn him if the message had been modified in transmission, which, while a true statement, was not something experienced by any participant or mentioned in the tutorials:
“Yes, but Pwm will notify you if it has been motified [sic]. ”
Still, there is evidence that at least some of the participants truly understood these properties. Participant R20 stated the following about integrity:
“Define ‘modified,’ ... it probably can’t be changed in a meaningful way, but could be garbled or corrupted (just by altering the encrypted bits)”
6.2 Acceptability of Pwm
Participants seemed to be interested in encrypting their email. While over half rejected the idea of having all of their email encrypted, every single participant was interested in encrypting their sensitive email. Additionally, all but nine participants wanted to start using Pwm. Furthermore, participants overwhelmingly believed Pwm could be used with their friends and family.
Examples of participants’ positive experiences with Pwm were consistently attested to in their responses. For example, R26, R39, and R42 expressed, respectively,
“Pwm was very simple, but definitely carried a [sic] aura of confidence in actually protecting your information. Now, how well it does is something that I cannot determine due to my lack of knowledge, but it was very well put together. It was very concise and user friendly and did not require esoteric knowledge to operate. I would definitely feel more comfortable sending sensitive information over email if I were using Pwm versus just sending it via an email provider. ”
“I liked how I could encrypt sensitive information like bank account information,
Integrated tightly into a web-based email system.
Integrated tightly into a desktop/mobile application.
A browser extension that has a separate interface for encrypting messages. A email provider that only supports encrypted email.
A web page that has a separate interface for encrypting messages.
A desktop/mobile application that has a separate interface for encrypting messages.
1st 2nd 3rd 4th 5th 6th
60
Figure 10: Participants’ ranking of secure email implementation alternatives (number of participants)
credit card, and other things. It wasn’t that hard to use. I didn’t have to download anything; all I had to do was just save a bookmark and then click on it. It was really easy to use. ”
“I liked how it made encrypting important information so easy. The tutorial was fast and easy. I like that it is easy and convenient to use with day to day emails. I liked that the background was blue so I knew when it was encrypting. ”
Based on these responses and our implementation’s high SUS score, the highest SUS score for secure email systems in the literature, we believe that our implementation of Pwm is usable enough to potentially see widespread adoption. Although significant changes still need to be made to Pwm (e.g., work with more email providers, bug fixes) and a long-term study needs to be conducted to verify our results, it is encouraging to see that users are ready and willing to embrace encrypted email through Pwm.
6.3 Implementation Preferences
Our results indicate that participants are most interested in secure email systems that integrate with existing email systems (see Figure 10). There may exist some bias created by asking this question after allowing participants to use a tightly integrated system. Nevertheless, participants’ free response answers to why they selected their top preference give an indication that these are their true preferences. Most participants cited ease-of-use and the ability to continue to use existing email systems and applications as their prime reasons for preferring tight integration. For example, participants R3 and R26 stated, respectively,
“I want to be able to choose the mail client I want and still have the protection I need”
“I think that an encryption system is more likely to be applied and enjoyed if a user, such
as myself, does not have to access another completely separate application every time I want to send or read an encrypted email. This would become very tedious and time consuming, and I think many people might just give up or find another option. The way it was employed in the scenarios was pretty good, and I think integrating it into your actual, individual account on a web email system like email would make that even better!”
Although only asked why they preferred their top option, a third of participants (18; 35%) explicitly stated that they did not like the idea of switching between multiple applications. R30 and R35 shared, respectively,
“Having a separate app/web page/browser extension would be a hassle. I would prefer the tightly integrated approach like what we did in the scenarios. ”
“I chose my first option for simplicity. It was easier to have the encryption program integrated into the email I was already using. Having to copy and paste through a different interface would be more challenging, especially when initially trying to learn the program. ”
More research should be done to verify whether these preferences hold true across a larger population. If that is the case, then more research emphasis should be placed on solutions that tightly integrate with existing systems.
6.4 Mixed mode email
The optional plaintext greeting that can accompany email encrypted by Pwm is intended to help email senders give confidence to recipients who have never used Pwm that the email is authentic and not spam. Surprisingly, during our study we noted that a small, but significant number of users would include a plaintext greeting in many
Gmail's
Pwm's
Icon
Icon
Figure 11: Gmail’s and Pwm’s icons for attaching images.
of their encrypted email messages. For example, in Task 4 eight participants (8; 16%) including a greeting stating that they were sending their direct deposit information. Interestingly, this was actually a feature that seven participants (7; 14%) listed as one of their favorite features of Pwm. R12 and R51 stated, respectively,
“It wasn’t rigid, I could write part of a message and have the other parts encrypted if I wanted. It was very clear what was encrypted and what wasn’t [...]”
“That is lets me chose when to encrypt and, when not to. It’s also nice to have the option of writing a message before the encrypted part so others know it’s not spam. ”
6.5
Lessons Learned
The following are lessons we learned about our implementation of Pwm during the study.
6.5.1 Instructing New Users
While Pwm automatically includes instructions on how to setup and use Pwm, most participants were unaware of this. During Task 5, participants would often spend several minutes trying to open an old Pwm message to grab the instructions, just to have the message immediately decrypted and the instructions disappear. It would be helpful to make it more clear that these instructions are always included. Perhaps we could even add functionality that allows users to explicitly add these instructions.
6.5.2 Look and Feel
Quite a few participants indicated enjoying the look and feel of Pwm. They also indicated that having a color-scheme that was distinct from Gmail helped them more easily understand what information was encrypted and what information was in the clear. This intuitive understanding potentially helped participants avoid mistakenly entering sensitive information where it would not be protected. Additionally, participants mentioned that it helped them feel more confident in the system. Thus it is clear that when designing tightly integrated systems, it is important to have a distinct look and feel.
6.5.3 Icon Selection
One potential area of concern when integrating tightly with existing webmail systems is that participants may become confused if the secure email system’s interface does not match the underlying webmail system’s interface. In Task 1, participants were asked to attach an image to their message. Pwm’s button that exposes this functionality differs from Gmail (see Figure 11), and we noticed that
participants became confused when they were unable to find Gmail’s attachment icon. Many participants spent several minutes trying to find the button that would allow them to attach an image, often completely ignoring the button that would actually allow them to do so. The failed search behavior exhibited here was surprisingly uniform across participants. When asked about this, R17 indicated,
“I think like in my normal Gmail interface there’s just like an attach button, and I find it at the bottom, and so it kind of threw me off, cause that’s usually how I’m used to attaching them, so I almost had to like start over and try it again and then I figured it out. I dunno, I think it was just that I was trying to do it weird. ”
This issue illustrates the need to match visual cues when presenting users with an alternative interface. Explanatory text is not always sufficient.
6.5.4 Tutorials
We recorded the number of participants that completed each of the tutorials. The video recording for one participant’s session was corrupted, and we were unable to determine if they had completed the tutorials. The results of the remaining 50 participants demonstrate that participants were willing to watch tutorials: watched introductory tutorial (46; 92%), watched tutorial on reading secure email (46; 92%), watched the tutorial on composing secure email (27; 54%).
These result surprised us, as we expected most users to ignore the tutorials. In their user studies, Ruoti et al. reported that nearly all participants ignored a tutorial video that was displayed on their Pwm website next to Pwm’s setup instructions [10]. This result indicates that participants are willing to watch tutorials, though two criteria seem to be crucial: they appear in-page as participants need them and they contain simple and direct wording.11
7. CONCLUSION
We examined the open question on how manual encryption affects the usability of secure email. Based on this question, we formulated three hypotheses that examined how automatic and manual encryption differed in terms of usability, users’ understanding of secure email, and users’ ability to avoid mistakes, respectively. We then tested our hypotheses by conducting an A/B test using two versions of the Private WebMail (Pwm) system [10] that differed only in their support for automatic and manual encryption.
Our results demonstrate that after accounting for confounding usability problems, manual encryption does not have a significant effect on usability, users’ understanding of secure email, or users’ ability to avoid mistakes. In addition, our improved version of Pwm scores an 80.0 on the System Usability Scale (SUS), rating in the 11We believe less participants watched the compose tutorial,
which appeared the first time participants clicked on GMail’s “Compose” button, because it was not clear they needed to see it at this point and because the tutorial drew attention to the option that they could skip it. Better tutorial design could possibly address this issue.
“excellent” category for usability and receiving an “A” grade. This score is in the 85th to 90th percentile of a large number of systems tested with SUS [11].
All participants in our study indicated that they were interested in encrypting sensitive email. Four-fifths of participants expressed a desire to begin using Pwm. Moreover, nearly all participants believed that their friends and family could easily start using Pwm. Although it is necessary to verify these results using a long-term user study, participants’ responses still give hope that we are significantly closer to the goal of providing ordinary users with a usable system to exchange secure emails.
8. REFERENCES
[1] A. Bangor, P. Kortum, and J. Miller. An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction,
24(6):574-594, 2008.
[2] A. Bangor, P. Kortum, and J. Miller. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3):114-123, 2009.
[3] J. Brooke. SUS — a quick and dirty usability scale. In Usability Evaluation in Industry. CRC Press, 1996. [4] J. Brooke. SUS: A retrospective. Journal of Usability
Studies, 8(2):29-40, 2013.
[5] S. Fahl, M. Harbach, T. Muders, M. Smith, and U. Sander. Helping Johnny 2.0 to encrypt his Facebook conversations. In Proceedings of the Eighth Symposium on Usable Privacy and Security, page 11. ACM, 2012.
[6] S. Garfinkel and H. R. Lipford. Usable security: History, themes, and challenges. Synthesis Lectures on Information Security, Privacy, and Trust, 5(2):1-124, 2014.
[7] S. L. Garfinkel. Email-based identification and authentication: An alternative to PKI? IEEE Security
& Privacy, 1(6):20-26, 2003.
[8] S. L. Garfinkel and R. C. Miller. Johnny 2: a user test of key continuity management with S/MIME and Outlook Express. In Proceedings of the First Symposium on Usable Privacy and Security, pages 13-24. ACM, 2005.
[9] S. Milgram and E. Van den Haag. Obedience to authority, 1978.
[10] S. Ruoti, N. Kim, B. Burgon, T. Van Der Horst, and K. Seamons. Confused Johnny: when automatic encryption leads to confusion and mistakes. In Proceedings of the Ninth Symposium on Usable Privacy and Security, page 5. ACM, 2013.
[11] J. Sauro. A practical guide to the system usability scale: Background, benchmarks & best practices. Measuring Usability LLC, 2011.
[12] S. Sheng, L. Broderick, C. Koranda, and J. Hyland. Why Johnny still can’t encrypt: evaluating the usability of email encryption software. In Proceedings of the Second Symposium On Usable Privacy and, Security - Poster Session, 2006.
[13] A. Sotirakopoulos, K. Hawkey, and K. Beznosov. “I did it because i trusted you”: Challenges with the study environment biasing participant behaviours. In SOUPS Usable Security Experiment Reports (USER)
Workshop, 2010.
[14] T. S. Tullis and J. N. Stetson. A comparison of questionnaires for assessing website usability. In
Usability Professional Association Conference, pages 1-12, 2004.
[15] T. W. Van Der Horst and K. E. Seamons. Simple authentication for the web. In Third International Conference on Security and Privacy in
Communications Networks (SecureComm), pages 473-482. IEEE, 2007.
[16] A. Whitten and J. D. Tygar. Why Johnny can’t encrypt: A usability evaluation of PGP 5.0. In 8th
A. PWM USER STUDY
A.1 Introduction
At the beginning of each user study the following was read to the participant by the study coordinator.
Welcome to our Gmail study. I am the study coordinator and am here to assist you as needed.
In this study, you will be using Gmail to complete several scenarios. In each scenario you will play the role of another person. I will provide you with information about this person. During the scenarios, please use this provided information and not your own personal information. Please protect any sensitive information for this person just as if it was your own.
During the course of the study we will record what is happening on your screen. This video will not be seen by anyone besides the researchers and will be destroyed once our research is complete. We will not collect any personally identifying information. Any data, besides the screen recording and answers to the study survey, will be deleted automatically upon your completion of the study.
You will receive $10.00 as compensation for your participation in this study. The expected time commitment is approximately 30 minutes. If you have any questions or concerns, feel free to ask me. You can end participation in this survey at any time and we will delete all data collected at your request.
You may now proceed with the survey on the left-most computer. I will remain in the room to observe the study and also to answer any questions you may have.
A.2 Scenarios
These were the two scenarios used in the study. The first scenario covers Task 1 - Task 4 and the second scenario covers Task 5 and Task 6.
A.2.1 Scenario 1
In this scenario, you have applied for a job with National Citadel. Last weekend they flew you out for a final interview. They told you they would email you instructions for getting your expenses reimbursed.
All information you will need to complete this scenario is provided below.
During this scenario you will receive and respond to several emails. Until you receive an email with a confirmation code, please continue to check your inbox for new email messages. Be aware that sometimes there will be a slight delay (30-60 seconds) before you receive an email.
At this time, please log into your email account where you should see an email message from National Citadel. If you don’t see such a message, ask the study coordinator for help.
APPENDIX
Scenario 1 Persona
1 Social Security Number 834-23-1339 1 State of residence 1 [redacted] 1 1 Date of birth | February 20, 1988 | 1 Bank account number 2365-9814-4907 1 Bank routing number 097-154-228
In this scenario, you have received a text message from your spouse (spouse® [redacted]) asking for help logging in to your credit card website. Your spouse has asked you to email him/her the account username and password.
This information is sensitive, so you want to encrypt it. You know that your spouse has never used encrypted email before. Please do whatever you think you would do in real life to send them this information encrypted with Pwm.
All information you will need to complete this scenario is provided below.
During this scenario you will receive and respond to several emails. Until you receive an email with a confirmation code, please continue to check your inbox for new email messages. Be aware that sometimes there will be a slight delay (30-60 seconds) before you receive an email.
A.2.2 Scenario 2
Scenario 2 Persona
| Account username | family343 | | Account password | b@nkp@ssword | | Credit card number 4716-9364-5318-3222 | Credit card CCV 992
A.3 Task Emails
A.3.1 Task 1
Initial email. Encrypted: Yes
From: finances@nationalcitadel.com Subject: Receipt Reimbursement
Greeting: Hi [participant’s name], thank you for interviewing with National Citadel this week. In order to process your expense reimbursement, please reply to this email with your Social Security Number and a picture of your receipts for your purchases. Company policy requires that you send us this information encrypted. We use Pwm to encrypt email. This email includes directions for setting up Pwm. After setting up Pwm, you will be able to encrypt the required information. Thanks, -Jen Cobb Encrypted Body: Now that you are running Pwm, you can simply reply to this message, and your information will automatically be encrypted.
If the participant encrypts the requested data. Encrypted: No
From: emailstudy@[redacted] Subject: Task Complete
Body: Hi [participant’s name], congratulations on completing your first task. Please continue to watch your inbox for more communications from National Citadel. -ISRL Research
If the participants does not encrypt the requested data. Encrypted: No
From: finances@nationalcitadel.com Subject: Re: Receipt Reimbursement
Body: Hi [participant’s name], it looks like you didn’t send those details securely. Please resend with encryption enabled. Thanks, -Jen Cobb
Participants instructed to close browser and then reopen Gmail.
Encrypted: No
From: emailstudy@[redacted] Subject: Scenario Instructions
Body: Before we proceed with the scenario, please close Chrome. This simulates time passing after your interview. After Chrome is shut down, you may re-open Chrome and navigate to Gmail. Thanks, -ISRL Research
Initial email. Encrypted: Yes
From: hiring@nationalcitadel.com Subject: National Citadel Offer
Greeting: This encrypted email contains details about your employment offer with National Citadel. Please decrypt to view.
Encrypted Body: Congratulations [participant’s name]! We were very impressed with your performance in the interviews and are excited to extend a full-time offer. See below for the details. Please look over the offer and, if it is acceptable, reply to us so we can proceed with the hiring. Also, please CC your acceptance to your manager, Kaylee Clark, at kclark@nationalcitadel.com. We look forward to your reply. Regards, -Rory Tam
A.3.2 Task 2
^J^National Citadel
[TODAY'S DATE]Dear [PARTICIPANTS NAME],
On behalf of National Citadel (the "Company"), I am very pleased to offer you the position of Project Manager. This letter clarifies and confirms the terms of your employment with the Company Start Date and Salary
Unless we mutually agree otherwise in writing you will commence employment on June 1, 2015 ("Start Date"). Your salary will be $6,000 00 per month payable in accordance with the Company's standard payroll practice and subject to applicable withholding taxes. Background Check
This offer is contingent on the successful completion of a background check.
This offer and all terms of employment stated in this letter will expire March 24, 2015.
JOHN, we are very excited about the possibility of you joining us. I hope that you will accept this offer and look forward to a productive and mutually beneficial working relationship Please let me know if I can answer any questions for you about any of the matters outlined in this letter
Sincerely -Zoe Walker Recmiting Manager
If the participant encrypts the requested data. Encrypted: Yes
From: hiring@nationalcitadel.com Subject: Re: National Citadel Offer
Encrypted Body: Hi [participant’s name], we received your acceptance and are excited to begin the hiring process. As part of the onboarding procedure, you will be receiving emails from our Background Check and Payroll departments. Please reply to these emails with the requested information, so we can proceed with your hire. Congratulations once again on joining our team. Regards, -Rory Tam
If the participants does not CC Kaylee Clark. Encrypted: Yes
From: hiring@nationalcitadel.com Subject: Re: National Citadel Offer
Encrypted Body: Hi [participant’s name], it looks like you didn’t CC your acceptance to your manager, Kaylee Clark, at kclark@nationalcitadel.com. Please resend your acceptance and be sure to CC her. Thanks, -Rory Tam If the participants does not encrypt the requested data. Encrypted: Yes
From: hiring@nationalcitadel.com Subject: Re: National Citadel Offer
Greeting: There was a problem with your reply. Please decrypt this message to view.
Encrypted Body: Hi [participant’s name], it looks like you didn’t encrypt your response. Please resend your acceptance and be sure to encrypt it. Thanks, -Rory Tam
If the participants does not CC Kaylee Clark and does not encrypt the requested data.
Encrypted: Yes
From: hiring@nationalcitadel.com Subject: Re: National Citadel Offer
Greeting: There was a problem with your reply. Please decrypt this message to view.
Encrypted Body: Hi [participant’s name], it looks like you didn’t CC your acceptance to your manager, Kaylee Clark, at kclark@nationalcitadel.com. Also, you seem to have sent your reply insecurely. Please resend your acceptance; be sure to CC her, and be sure to use encryption. Thanks, -Rory Tam
A.3.3 Task 3
Initial email. Encrypted: No
From: hiring@nationalcitadel.com
Subject: National Citadel Background Check
Body: Hello [participant’s name], we have received your hiring details and are proceeding with a background check. Please fill out the following details and forward them to our Background Check provider, backgroundcheck@[redacted]. As a reminder, please encrypt all sensitive communications with other entities regarding your employment with National Citadel.
- Full name - Date of birth
- Social Security Number - State of residence Thanks, -Simon Turner
If the participant encrypts the requested data. Encrypted: Yes
From: backgroundcheck@[redacted]
Subject: Re: Fwd: National Citadel Background Check Encrypted Body: Hi [participant’s name], thanks for forwarding your background check details. We’ll get started right away. National Citadel will inform you when your check is complete. Regards, -Inara Sanchez
If the participants does not encrypt the requested data. Encrypted: Yes
From: backgroundcheck@[redacted]
Subject: Re: Fwd: National Citadel Background Check Greeting: Hi [participant’s name], it looks like you sent us these
