Building a Modern Security Engineering

(1)

Building a Modern Security

Engineering Organization

[email protected]

(2)

Who is this guy anyway?

•  Built and led the Etsy Security Team

– Spoiler alert: what this presentation is about

•  Recently co-founded Signal Sciences to

productize eﬀective AppSec approaches

(3)

This talk is a collection of lessons learned from building and adapting a security

(4)

For security teams, the world has changed in fundamental ways:

(5)

–  Code deployment is now near-instantaneous

–  Merging of development and operations

(6)

–  Code deployment is now near-instantaneous

–  Merging of development and operations

means more people with production access

(7)

(8)

(9)

(10)

Etsy pushes to production 30 times a day

(11)

Constant iteration in production via feature ﬂags, ramp ups, A/B testing

(12)

But doesn’t the rapid rate of change mean things are less

(13)

Actually, the opposite is true

(14)

They key to realize is vulnerabilities occur in all development methodologies

…But there’s no such thing as an out-of-band patch in continuous deployment

(15)

They key to realize is vulnerabilities occur in all development methodologies

…But there’s no such thing as an out-of-band patch in continuous deployment

(16)

Compared to:

“We’ll rush that security ﬁx. It will go out … in about 6 weeks.”

(17)

What makes continuous deployment safe?

(18)

(19)

(20)

The same culture of graphing and monitoring inherent to continuous

deployment can be used for security too

(21)

Surface security info for everyone, not just the security team

(22)

(23)

“Don’t treat security as a binary event” - @ngalbreath

(24)

Building a (k-‐)rad culture

(25)

In the shift to continuous deployment, speed increases by removing

organizational blockers

(26)

Trying to make security a blocker means you get routed around

(27)

Instead, the focus becomes on

incentivizing teams to reach out to security

(28)

Keys to incentivizing conversation:

– Don’t be a jerk. This should be obvious, but

empathy needs to be explicitly set as a core part of your teams culture.

(29)

– Don’t be a jerk. This should be obvious, but

empathy needs to be explicitly set as a core part of your teams culture.

– Make realistic tradeoﬀs. Don’t fall in to the

trap of thinking every issue is critical.

•  Ex: Letting low risk issues ship with a reasonable remediation window buys you credibility for

when things actually do need to be addressed immediately.

(30)

– Coherently explain impact. “This would

allow all our user data to be compromised if the attacker did X & Y” paints a clear picture, where “The input validation in this function is weak” does not.

(31)

– Coherently explain impact. “This would

allow all our user data to be compromised if the attacker did X & Y” paints a clear picture, where “The input validation in this function is weak” does not.

– Reward communication with security

team. T-Shirts, gift cards, and high ﬁves all work (shockingly) well.

(32)

– Take the false positive hit yourself. Don’t

send unveriﬁed issues to dev and ops teams. When issues come in, have the secteam verify and make ﬁrst attempt at patch.

– Scale via team leads. Build relationships with technical leads from other teams so they make security part of their teams

(33)

– Take the false positive hit yourself. Don’t

send unveriﬁed issues to dev and ops teams. When issues come in, have the secteam verify and make ﬁrst attempt at patch.

– Scale via team leads. Build relationships

with technical leads from other teams so they make security part of their teams

(34)

(35)

Startups begin with a simple access control policy: Everyone can access

everything

(36)

As organization grow there will be more pressure to institute access policies

(37)

The key to remember is don’t take away capabilities

(38)

Methodology:

1.  Figure out what capability is needed

2.  Build an alternate way to perform the

needed function in a safe way

3.  Transition the organization over to the safe

way

(39)

Methodology:

2.  Build an alternate way to perform the needed function in a safe way

3.  Transition the organization over to the safe

way

(40)

Methodology:

3.  Transition the organization over to the safe way

(41)

Methodology:

3.  Transition the organization over to the safe way

(42)

EX: SSH access to production systems

(43)

Security policy goal: Eliminate unneeded access to production systems

–  Why do developers do it? Ex: To view error logs

–  Build alternate approach: Send the logs to

central logging service (ex: elasticsearch, splunk, etc)

–  Publicize the new tooling to the organization

–  After majority of transition, alert on any logins to

(44)

(45)

(46)

(47)

(48)

Speciﬁcally, some thoughts on:

–  Bug Bounties

(49)

(50)

Bug bounties are tremendously useful. If you’re not working towards launching one,

(51)

Common concerns about launching a bounty:

1.  Budgetary concerns. Money is almost

never the main motivation for researchers, you can launch a bounty with just a hall of fame and still get great submissions.

2.  Risk of inviting attacks. You’re already

getting attacked continuously, you’re just not getting the results.

(52)

1.  Budgetary concerns. Money is rarely the

main motivation for participants, you can launch a bounty with just a hall of fame and still get great submissions.

2.  Risk of inviting attacks. You’re already

getting attacked continuously, you’re just not getting the results.

(53)

1.  Budgetary concerns. Money is rarely the

main motivation for participants, you can launch a bounty with just a hall of fame and still get great submissions.

2.  Risk of inviting attacks. It’s the Internet.

You’re already getting pentested

continuously, you’re just not receiving the report.

(54)

The ultimate goals of a bug bounty are threefold:

1.  Incentivize people to report issues to you

in the ﬁrst place

2.  Drive up cost of vulnerability discovery and

exploitation for attackers

3.  Provide an external validation of if your

security program is working (or not)

(55)

in the ﬁrst place

3.  Provide an external validation of if your

security program is working (or not)

(56)

in the ﬁrst place

3.  Provide an external validation of where

your security program is working (and where it’s not)

(57)

Before you launch, record what vulnerability classes you expect to see and what you don’t.

Compare this against the issues actually reported.

(58)

Before you launch, record what vulnerability classes you expect to see and what you don’t.

Compare this against the issues actually reported.

(59)

Keep metrics on:

– Number of bugs reported and severities

– Time to remediation of reported issues

You want both of these metrics to trend down

over time

(60)

Practical considerations:

– Inform all teams before bounty launch,

especially non-engineering teams

•  Ex: Customer Support

– Attacks will start almost immediately

For Etsy bug bounty launch, time from

(61)

– Inform all teams before bounty launch,

especially non-engineering teams

•  Ex: Customer Support

– Attacks will start almost immediately

For Etsy bug bounty launch, time from announcement to ﬁrst attack: 13min

(62)

– Your ﬁrst 2-3 weeks will be intense. Have as

many people as you can dedicated to triage and response

(63)

– Operationally review any helper systems for

scaling problems beforehand

•  When 10-100x traﬃc hits helper systems your security team uses, what falls over?

– Money almost never the overriding factor,

hall of fame is

–  Researchers are generally great to interact

(64)

scaling problems beforehand

– Money is almost never the main motivation

for bounty participants, hall of fame credit is

–  Researchers are generally great to interact

(65)

scaling problems beforehand.

– Money is almost never the main motivation

for bounty participants, hall of fame credit is

–  Key to great researcher interaction is

(66)

XXX

(67)

Problems with “pentesting” are well understood in the oﬀensive community

(68)

Pentests typically result in a list of

enumerated known vulnerabilities to be patched, not data on how a real attacker

would operate against a given environment

(69)

Attack simulations should be done to learn how attackers are likely to achieve goals

against your organization

NOT to show compromise is possible (spoiler alert: it is.)

(70)

Use this attack data to focus where/how to build detection mechanisms

(71)

From an organizational side, attack simulations compliment vulnerability

(72)

Four keys to eﬀective attack simulations: 1.  Goal oriented

•  “Obtain domain admin”, “read the CEOs email”, “view credit card data”, …

•  Ask attack team for input on goals, they’ll come up with ones you didn’t think of

2.  Full ganization in scope

•  Have attack team call a contact if they’re about to do something risky

– several week simulat

(73)

Four keys to eﬀective attack simulations: 1.  Goal oriented

•  “Obtain domain admin”, “read the CEOs email”, “view credit card data”, …

•  Ask attack team for input on goals, they’ll come up with ones you didn’t think of

2.  Full organization in scope

•  Have attack team call a contact if they’re about to do something risky

–  Ex: Instead of throwing an exploit that lands “most of

the time”, grant access to the target system with temporary credentials

(74)

Four keys to eﬀective attack simulations:

3.  Simulate realistic compromise patterns

•  Start the attack team on a:

–  standard laptop/desktop to simulate phishing/clientside

compromise

–  database or web server to simulate SQL injection/RCE

•  0days aren’t cheating, they’re reality. Attack team

should be encouraged to use them.

–  Break simulation down into iterations:

•  Don’t spend the full engagement time on only round of testing, once one team achieve goal(s), then swap in new attack team to achieve the same goal(s)

–  Ex: We try to run 3-4 iterations per several week

(75)

Four keys to eﬀective attack simulations:

3.  Simulate realistic compromise patterns

•  Start the attack team on a:

–  standard laptop/desktop to simulate phishing/clientside

compromise

–  database or web server to simulate SQL injection/RCE

•  0days aren’t cheating, they’re reality. Attack team

should be encouraged to use them.

4.  Break simulation down into iterations:

•  Don’t spend the full engagement time on only round

of testing, once one team achieve goal(s), then swap in new attack team to achieve the same goal(s)

–  Ex: We try to run 3-4 iterations per several week

(76)

The project output should be attack chains

showing how attack team went from A->B->C to achieve goals, what steps they took and

(77)

Just as importantly, what steps they didn’t

take

Ex: “We didn’t try to ﬁnd internal network diagrams on your wiki because zone transfers were enabled so we could got enough data about your network from that”

(78)

Remember, the goal is to simulate realistic

attack behaviors and patterns that can be

(79)

In addition, simulate varying attack proﬁles from quick & loud to quietly maintaining

(80)

Over multiple iterations learn what

behaviors overlap between attackers and what strong signals of lateral movement in

(81)

TL;DR

(The section formerly known as “Conclusions”)

(82)

•  Adapt security team culture to DevOps

and continuous deployment by:

– Surfacing security monitoring and metrics

– Incentivize discussions with the security

team

– When creating policy, don’t take away

capabilities

•  Drive up attacker cost through bug

bounty programs, countering phishing, and running realistic attack simulations

(83)

Building a Modern Security Engineering