What your pen-tester won t tell you...

(1)

What your pen-tester won’t tell you...

Michael Kemp, Xiphos Research Labs mk@xiphosresearch.com

Introduction: whoami

• UK based company co-founder (security research, software, and yes, penetration testing)

• Over 12 years professional experience in IT security / pen testing (more if you count a wayward youth)

• Have managed clients and pen test teams

• Speaker at global conferences – last minute replacement here • Work with customers to improve their security

• Set up XRL to do something a little bit different • Am strangely unnerved by men in suits

(2)

Introduction: Vote for Pedro

Introduction: Why this Talk

• Because things don’t work right at the moment

• The penetration testing “industry” is maturing – which means that sales is getting slicker, as is the potential for abuse

• Because if you hire a charlatan – it’s your customers that will suffer • Because I need to get this off my chest

• Because I like breaking things and would like to still be doing it in 10 years • Because it *may* be helpful

(3)

Introduction: Penetration Testing (defined)

• Pen-testing means many things to many people • Is part of a risk and compliance program • Is a vital component of a secure SDLC

• Is a mechanism for discovering whether your protections are as good as you like to think they are

• Has a history

• Pen-testing is a mechanism for evaluating the potential attack surface available to malicious external or internal attackers

• Involves thinking and acting like a criminally inclined attacker without wandering off scope of having criminal intent

• Now a much broader spectrum that ‘classic’ network pen-tests, and encompasses attacks not just against technology (e.g. physical sec and SE)

• Regardless of the colour of the box – the goal is the same (to increase defensive posture)

(4)

• Prior to the 1990’s pen-testing was a dark art, largely unknown in the private sector

• In 1993, Dan Farmer and Weitse Venama wrote the seminal paper “Improving the security of your site by breaking into it”

• Tools and services began to emerge from under the radar and be more widely utilised

• Illegitimate attacks obviously pre-date pen-testing as a service offering (MIT model railway club & phreaking)

• Pen-testing is now widely utilised and understood?

Introduction: The Pen-Testing ROI

• The ROI of pen-testing is hard to gauge – let’s look at some figures though.... • In 2008 tier 1 & 2 merchants were required to perform pen-testing according to PCI

• According to Cybersource, 2008 was also the worst year for credit card losses • There has been a down-trend since – but why?

• If everyone still has losses (even when pen-testing is mandated) what’s the point of pen-testing?

(5)

Click the Start Button

• Vulnerability scanners are cheap • Scanners are thorough

• Scanners are *everywhere* • All pen-testers use scanners • Some *only* use scanners

• Is the value in the findings or the interpretation? • It doesn’t take a pen-tester to pen-test (badly) anymore

• Automated scanners are dumb

• If you rely on them – you will *not* find everything •Example A – Hospitality

(6)

• As a client you get a report with “risk” ratings (more on reports later)

• Risk is determined by the tool (sometimes by a pen-tester cherry picking results from a tool)

• No environment consideration, no consideration of asset value, limited consideration of architecture

• No consideration of goal orientated testing (more about scoping later too) • Any value apart from the “look ma, we’ve been ‘pen-tested’”?

• Headline issue: Tools do not a pen test make! • Great for finding low hanging fruit

• What about pen-testers that write there own?

• Either available elsewhere (because they have published) or still in beta (and they hope to release)

• Or it’s a script composed to make testing easier • But what about pen-testers who find 0-day?

(7)

The 0-day illusion

• Many pen-test providers make much of providing zero days • “We can find things Nessus can’t”

• True – in custom client apps • False – in vendor kit

• Why? Because the exploit market is alive and well...

• Your penetration tester may find zero day, you’ll never know (it’s worth more)

Most Certs are paper

• Prior to 2000, most penetration testers were self taught

• No methodologies, but lots of step by step examples of how to use tools • Lots and lots of companies now offer training in penetration testing • Certification does not mean experience

• With 80+ hours in a boot camp anyone can learn to use nmap and become a “certified” hacker

• Not to devalue proper education

• You can teach risk, you can teach tools, you can teach law – can you really teach skewed logic?

(8)

Most Certs are paper

• C|EH et al are largely redundant (brain dumps and study guides) • CHECK / CREST is of value – it actually tests something

• Degree courses *can* be of value • Experience is what counts •Example B – Legacy kit

Why your pen-tester loves Social Engineering

• It works • Always • It always works

(9)

• Consider the following:

•Company X hires Pen-Test Pete. As part of the sales process, SE is promised. Pen-Test Pete phones up the helpdesk – what do the helpdesk do?

• Social Engineering *does* have a point • Great for staff awareness

• Great for checking procedures

• Humans *are* always the weakest link...

• It’s not really penetration testing in the classical sense • It is what attackers *sometimes* do

(10)

The Great Wall of Nonsense

• Physical security assessment is now mandated by PCI-DSS

• Usually consists of an ‘audit’ (e.g. Walking around with a clipboard looking for cameras)

• Is it of use?

•Example C – The Retailer

The Great Wall of Nonsense

• Lock-picking is fun if you are mechanically inclined • Real attackers don’t pick locks

• Consider C I Host

• Based in Chicago, in 2008 data centre was robbed

• How? Two masked men pistol whipped the lone IT staffer and stole all the assets • If a criminal really wants your assets – locks and cameras aren’t a deterrent

(11)

Everyone Hates Reports

• The word reporting strikes terror into the heart of your pen-tester • Reports fit the same format for *all* clients, and breakdown as follows

• Executive Summary (we found X vulns, please hire us again) • “Risk” rating

• Issue Description • “Remediation”

• Technical Summary (did we mention we found X vulns, hire us again)

• Most reports are useless – Why? • Consider the following:

“A flaw was found with within mod_isapi which would attempt to unload the ISAPI dll when it encountered various error states. This could leave the callbacks in an undefined state and result in a segfault. On Windows platforms using mod_isapi, a remote attacker could send a malicious request to trigger this issue, and as win32 MPM runs only one process, this would result in a denial of service, and potentially allow arbitrary code execution.“

(12)

• Let’s dissect that:

• A “vulnerability” exists in the mod_isapi module on Apache, which if an attacker sends a certain request results in DoS or potential code execution

• Only affects Windows installs of Apache • Only affects one module

• No public exploit code • It’s patched

• Not that scary

• The vendor issue description (and one in the automated tool) sounds scary • Doesn’t actually determine risk (e.g. This is only an issue if the module is enabled on Windows, and if the attacker knows the unpublished request)

• No environmental analysis

• Is it an issue if it’s on a segregated network segment? • No exploit to play with

(13)

Why Metasploit has made your Pen-tester lazy

• In 2003, US researcher H.D.Moore launched the Metasploit project

• Public repository of weaponised exploit code relating to public vulnerabilities • If there is a module in MSF, a pen tester can use the exploit to target specific vulnerabilities in client estates

• How?

•search platform X | use exploit X | exploit

• If it works they potentially get control of a target

• If manual searching is too difficult, your pen-tester can always use Fast-Track • Released in 2008, FT is a front end for MSF

• You give it a target, it scans for potential vulnerabilities (finds open ports) and then search the MSF repository and launches exploits

• Totally point and click approach to getting root on vulnerable systems • No knowledge of pen-testing (or indeed systems under test) required

(14)

• Arron Finnon has done some fascinating research on Metasploit • He has found that many IDS (including Snort) are runed to detect MSF implemetations of exploits

• If an attacker uses the ‘root’ exploit prior to implementation in MSF it can bypass many IDS

• What pen-tester is going to compile an exploit though? (UNIX make cmd takes work...)

Oooo Shiny....

• Nearly all pen-testers secretly want to be full time security researchers • Nothing wrong with that apart from the ‘race to advisory’

• Vulnerability purchase programs have stopped some of that

• Still lots of pen-testers wanting to be known for researching the latest tech • If technology is old though, doesn’t mean it should be looked at

• Just because pen-tester A has made press, doesn’t make them a good tester • It may make them an *excellent* researcher with media savvy

(15)

Hall of Fail

“Hey,

Does anyone have a comprehensive audit program/checklist for physical security? I would want something that maps up to the PCI DSS standards (although this data doesn’t process payment data it is highly sensitive and thus meets the same security requirements). It isn’t a data centre we are auditing, more a physical centre that wipes our disks on our behalf. A few of the physical security audit programs I checked out through a Google search weren’t up to much. Any such programs that you use and would be willing to share would be great, right up to the policies, risk assessments, BIA, logs and physical controls.”

Hall of Fail

“I have a quick question for you. I'm preparing to perform a Pen test for a HIPPA compliance requirement. The client had asked if there is a way for me to compare my findings against a HIPPA industry average. (i.e. The client is compared to other health care providers and is either better or worse than the average in the industry).

(16)

Hall of Fail

• What sort of vulnerabilities were around ten years ago? • Missing patches

• System defaults • Weak credentials

• Insecure vendor systems and devices • Bad architecture

• Human weakness

Hall of Fail

• What sort of vulnerabilities are around now? • Missing patches

• System defaults • Weak credentials

• Insecure vendor systems and devices • Bad architecture

(17)

Hall of Fail

• Has pen-testing changed anything?

• The external perimeter is a lot tighter – Internally most organisations are not good

• Actual attackers (unless state funded) are getting lazy • RFI and The Sun

• Talking of RFI – Example C – The other retailer

Making things Better

• Define what you want – Scoping is essential

• Attackers don’t have scopes, but they also don’t send you bills

• Ensure that your pen-testers know what you want, and accept your goals, and respond accordingly

• Ensure that all penetration testing activities are repeatable (many will claim to be working to a methodology – make sure they are)

• Accept that all pen-testing has limits – make sure they aren’t intelligence • Test for a goal – Not just for compliance (if it is for PCI-DSS make sure the goal is PAN and card data)

(18)

Let’s go Shopping

• Ensure that any pen-tester you hire actually uses blended manual attacks (e.g. Vuln scanners will find blank passwords & SQLi – does your pen tester use these to compromise the DB, extract data, and pivot off the DB – or just tell you what the tool found?)

• Select your pen-tester – Know the skills and biographies of the people that will be testing you

• Ensure that your pen-tester has experience, not just paper – what have they done before? How is that relevant to you?

• Get fixed costs and time-scales in case the scope changes over time • Don’t trust testimonials

• Ensure that risk is based around goal • Ensure that reporting is relevant

• Ensure that pen-testing is repeatable – make it part of a systematic approach to security management

• Remember the pen-tester is not the enemy – if you let them know your concerns, your architecture, your workflows, and your policies – they can ensure they work (or not) – Play nice with them and they will play nice with you

(19)

• Change the rules – Outside in is great, but largely pointless • Ensure that your pen-tester has the skills they claim

• Make sure real risk is addressed, not the FUD external attacker

• Consider hiring internal teams – calling the pen-tester for third party sanity checks

• Penetration testing is part of a process of security – not a get out jail free card • If you want a tick in the box, get “data breach” insurance (and hope for the best) or hire an inept pen-test company

Thanks for Listening

• Questions?

+44(0)121 233 2479 mk@xiphosresearch.com www.xiphosresearch.com