Hobbled Penetration Testing:
The Disconnect Between Testing and
Real Attacks
Jason Wood Principal Security Consultant Secure Ideas
Background Info
Principal Security Consultant at Secure Ideas
Penetration Tester, Security Engineer & Systems Administrator
− Web environments
− Infrastructure
− Windows, Linux, UNIX
− Went to security in 2006
3
How Penetration Tests Are Viewed
5
How Pentesters Are Viewed
7
How Pentesters View Everyone Else
9
Time and money is spent deploying defenses
− Do our defenses work?
− Does our staff know how to use them?
− Can we see where attackers are going?
− Do we know how to respond to the attackers?
− What’s the impact of weaknesses being exploited?
QA testing for our defenses and controls
Opportunity to learn as well as test
Potential “Live fire” exercise
The Role of Penetration Tests (as I see it)
Different definitions of what a pen test is
Narrowly scoped engagements
The quality of testing and reporting varies wildly
Fixing flaws may or may not happen
Pen tests are finding flaws that should have been discovered MUCH earlier
Little collaboration
May be a checkbox to be completed
Frequently do not reflect how organizations are breached
The State of Penetration Testing
11 Discover systems and vulnerabilities
Validate vulnerabilities
Analyze and prioritize weaknesses
………End of Vulnerability Assessment………. Combine vulnerabilities into a single plan of attack Exploit the vulnerability to see where it leads
− More functionality and control of a system?
− Able to access data?
− Stepping stones to the next set of vulnerabilities
− Show what can be done
Vulnerability Assessment vs
Penetration Test
Test focused only on a single application or system
Important systems and avenues of attack are off limits
− (Frequently off limits to internal security too)
Doesn’t remotely resemble a real attack Why?
− Time and resource limitations
− Business impact
− Politics
− Fear
13
Decide how to simulate the attack(s)
− What access would the bad guy get?
− Where would they be on the network?
− What privileges would they have?
− How can we emulate that?
Decide on a sample of systems
Are there windows where testing is less risky?
CHEAT!
Compensating for Narrow Scoping
Pen testers sitting in a room with no contact with security or operations
Hiding that pen testing is being done from other staff Managers don’t know that a test is planned
As a result…
− Testers end up not producing what the customer needs/wants
− Operations and development feel ambushed
− IT staff starts responding to alerts and burns time they didn’t expect − Users are upset if there is unexpected disruptions
− Turf wars ensue
15
Plan to walk before running
− The first pen tests are almost always awful
Be Prepared!
− Decide what you want to accomplish
− Have systems and applications available and ready
− Determine who should participate and how
− Expect and make requests to ensure your testers are prepared!
− Have a plan to monitor and verify your defenses are working
Build a Testing Strategy
Collaborate
− Internally
− Externally
Plan to up the difficulty over time
Build relationships with penetration testing firms
More Strategy
17
What do we want out of security testing?
What should internal security be responsible for?
And when does it need to be done?
When should third party pen testing be done?
What do we know about attackers?
Can we simulate likely attacks?
Questions to Ask Ourselves
What do we not know?
What do we fear happening?
What previous testing has been done?
How can we build on previous testing to increase
our ability to detect and respond?
How do we measure progress over time?
And Some More Questions
19 Collaborative Start
Responding Less Notice
No Notice Active Defense
Build a Progression of Difficulty
Lots of companies to choose from
Number of employees may have little impact on quality
Discuss insurance and appropriate legal work
Be up front with your goals and what you need from the engagement
Are they willing to work with you on how to conduct the test?
Do they have good suggestions on how to improve on your plans?
21
Work closely with the testing team
Dedicate staff to trying to follow what the testers are doing
− Where are the they visible and where do they disappear?
Use it as a training opportunity
Chatter back and forth during the test
− We saw X and Y. Was that you? What did you do?
There may be system disruptions so prepare accordingly
− We don’t know what we don’t know
Expect that there may be a daunting list of things to fix
Combine Red Team With Blue Team
Phishing attacks and compromised desktops are a major source of breaches
Disrupting operations is not acceptable to the business
Prepare a typical desktop(s) to use during the testing
“Compromise” the workstation by using client side exploits
on purpose
Determine what can be done from this point
− What can the testers accomplish from here?
− What can they access?
23
A vendor has remote access and/or systems within our network
Can’t test their systems because they don’t want to
participate
Determine what access do those systems have and create similar systems to use at initial footholds
− Vendor has VPN to subnet X with Y access
− “Compromise” the sample system and see how things develop
Scenario: The Third Party System
The testers find a serious flaw on an important system
It is determined that it is too risky to exploit
Determine what access a successful exploit would grant
Create a way to simulate that access
− Running as a service account or administrator account
− Testers then use that access as a foothold to test from
Scenario: Vulnerability on a Fragile
System
25
Penetration testing can be more effective for us by:
− Being prepared with goals and a plan
− Collaborating to foster learning and create less animosity
− Avoid letting people play the blame game This is a drill, not a real breach
− Building appropriate simulations to address situations that are deemed unacceptable or too risky
Try to keep it real and not too little or too much access
− Expect that it will take time and multiple engagements Not all tests need to be done by a third party
Take Aways
Jason Wood [email protected]
@Jason_Wood