Applying GEMS to user behavior during social engineering attacks

Although GEMS was originally conceived for modeling human errors which lead to ac- cidents, researchers have observed GEMS is also useful for modeling human errors which lead to security failures [13, 21]. In particular, we argue that GEMS is useful for under- standing human behavior during social engineering attacks on the Internet. By casting social engineering attacks in the context of GEMS, we can hopefully apply lessons learned by human reliability specialists to help users resist these attacks.

Generally, for an accident to occur, unsafe acts must combine with risky or unusual environmental conditions. During a social engineering attack, the presence of an adversary is the risky condition, and the adversary’s goal is to intentionally trigger human errors, typically through a combination of environmental manipulation and active encouragement. Current Web ceremonies such as password authentication create a lose-lose situation for users. A common modus operandi of adversaries is to mimic trusted Web sites, and if the charade succeeds, an adversary gains the opportunity to exploit strong-but-wrong and inadvisable rules. Alternatively, if the user is suspicious, she is all but forced to proceed at the knowledge-based level to ultimately decide if the site is trustworthy. Attacks are an exceptional situation for most users, and users are ill-equipped to address them. Users are most likely to make mistakes at the knowledge-based level, and studies suggest that many users have erroneous mental models of Web security mechanisms and have a wide variety of faulty defense strategies for phishing attacks [24, 25, 135]. For example, a user in one phishing study believed that she could enter her username/password to authenticate a Web site, assuming that only the legitimate site would be able to respond correctly [24].

In summary, we hypothesize that social engineering attacks on the Internet succeed mostly due a combination of two tactics: 1) tricking users into applying strong-but-wrong and inadvisable rules, and 2) forcing users to make security decisions at the knowledge- based level, where they are most likely to make mistakes. To help users resist social engi- neering attacks on the Internet, we argue that ceremonies should condition users to apply rules that serve them well and help them thwart attacks. In the next section, we introduce the notion of a conditioned-safe ceremony, an approach for designing ceremonies that takes

these observations into consideration.

3.3

Conditioned-safe ceremonies

One natural response to the weaknesses of current Web ceremonies such as challenge questions and passwords is to design ceremonies which try to eliminate user condition- ing, click-whirr responses, and rule-based decision making. This approach is problematic. Rule-based decision making is fundamental to human behavior: it helps us complete rou- tine tasks quickly and easily. Users may be willing to invest extra time and effort to learn a new security mechanism, but if they cannot learn how to use it efficiently, they will become frustrated and disable or circumvent the offending mechanism [41, 46, 141]. Some degree of conditioning may be necessary for the psychological acceptance of security mechanisms by users.

Since users will tend to adopt rules for completing a ceremony that minimize conscious effort, we should not fight users’ tendencies to use rule-based decision making, but take advantage of these tendencies to help users resist social engineering attacks. We should prudently design ceremonies to condition rules that benefit security rather than undermine it. Towards achieving this goal, we introduce the notion of a conditioned-safe ceremony. A conditioned-safe ceremony is one that deliberately conditions users to reflexively act in ways that protect them from attacks. We propose two design principles for building conditioned-safe ceremonies:

• Conditioned-safe ceremonies should only condition safe rules, i.e., rules that are harmless to apply in the presence of an adversary.

• Conditioned-safe ceremonies should condition at least one immunizing rule, i.e., a rule which when applied during an attack causes the attack to fail. We discuss im- munizing rules further in Section 3.3.1.

These principles also have important consequences on what conditioned-safe ceremonies should not do:

• Conditioned-safe ceremonies should not condition rules that require users to decide whether it is safe to apply them. Since many users are unreliable at recognizing risky situations, users should not need to refrain from conditioned behavior to resist attacks.

• Conditioned-safe ceremonies should not assume users will reliably perform actions that: 1) the ceremony has not conditioned her to perform, or 2) are voluntary. Satis- ficing users will learn to omit optional and voluntary actions, so ceremonies should not rely upon users to perform such actions.

For example, a ceremony should not condition the rule “if (legitimate looking login form) then (enter username/password)” because it causes a security failure when applied in the presence of an adversary. To determine if it is safe to apply this rule, a user must first verify the URL bar, the site’s SSL certificate, and other security indicators. Burdening users with these decisions is unsatisfactory. Ideally, in a conditioned-safe ceremony, a user should be able to resist an attack even if she has no idea she is at risk and performs the same actions as she usually performs under benign conditions.

However, user behavior is unpredictable and an adversary may try to trick users into deviating from their normal, conditioned behavior in a way that causes a security failure. Conditioned-safe ceremonies need safeguards to resist these attacks. In the human reli- ability community, designers often introduce constraints called forcing functions to help prevent errors in safety-critical environments. We argue that forcing functions can also be useful for conditioned-safe ceremonies, and we discuss them further in the next section.