Practical examples of Big Data, security analytics and visualization

(1)

Practical examples of Big Data, security

analytics and visualization

Jeff McGee, Data Scientist

(2)

Objective

• Identify problems in

Security that could be

solved with better

analytics

• Discuss recent efforts

on Big Data and

Visualization

• Share examples of how

HP’s Cyber Defense

Center has leveraged

these capabilities

(3)

(4)

Good guys are making things less predictable

Challenge: There is more noise

• Mobile

• Bring your own device

• Virtual machines and “the Cloud”

• SAAS

• New sources of logs

(5)

And bad guys know how to stay inside the bell curve.

Challenge: There is less signal

Unknown:

Harder to detect

• New behavior

• Goes to an approved place

• Works encrypted

• Authorized Use

• Inside of baseline

• Outside monitored infrastructure

• Matches a signature

• Goes to a bad place

• Works in the clear

• Unauthorized Use

• Outside of baseline

• Within monitored infrastructure

(6)

Solutions to big data problems

Let’s take techniques originally built for other

domains and apply them to security:

• Map-reduce

• Columnar Data Stores

• Machine Learning

(7)

Tools and technologies

• Hadoop – Framework for distributed computing

• Vertica – Columnar database

• Tableau – Visualization software

• Numpy/Scikit-learn – Machine learning tools

ArcSight

Vertica

analytic

platform

Hunt

teams

Security intelligence

(8)

Overview: The Vertica analytic platform

Rapid iterative conversations

with your data

Real time Analytics

Purpose built for Big Data from the first line of code

Store & Analyze PBs

Ingest 30 TB/hour

Proven Scalability

Works with Hadoop, R

Ecosystem of Visualization

Tools, SDKs and Community

Open & Extensible

Efficient compressed storage

Scale-out architecture

Easy to setup & manage

Low TCO

Private Cloud

Public Cloud

Appliance

Software Only

(9)

Security visualization

Practical examples

Router Operating System /Informational Network-based IDS/IPS /Suspicious Network-based IDS/IPS /Recon Network-based IDS/IPS /Informational Network-based IDS/IPS /Compromise Network Monitoring /Informational Network Monitoring Firewall Firewall /Normal Firewall Firewall Content Security Applications /Informational Applications bust6

Category Device Type

Applications Content Security Firewall Host-based IDS/IPS Mainframe Network Monitoring Network-based IDS/IPS Operating System Router VPN

(10)

(11)

Category Significance

Informational/Error /Hostile /Compromise /Suspicious /Normal /Informational/Warning /Recon /Informational 0M 10M 20M 30M 40M 50M 60M 70M 80M 90M 100M Co Network-based IDS/IPS

Network-based IDS/IPS Network-based IDS/IPS

Network-based IDS/IPS Network-based IDS/IPS Host-based IDS/IPS Network Monitoring Operating System Applications Applications Applications Firewall Firewall Firewall VPN fromAfarSourcePt

Count of Destination Port

0 20,000,000 40,000,000 66,854,010

0 66,854,010

Count of Destination Port

(12)

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13 Router Operating System /Informational Network-based IDS/IPS /Suspicious Network-based IDS/IPS /Recon Network-based IDS/IPS /Informational Network-based IDS/IPS /Compromise Network Monitoring /Informational Network Monitoring Firewall Firewall /Normal Firewall Firewall Content Security Applications /Informational Applications bust6

Applications Content Security Firewall Host-based IDS/IPS Mainframe Network Monitoring Network-based IDS/IPS Operating System Router VPN

Proportional relationships

(13)

(14)

Category Device Type

Applications Content Security

Database Firewall Host-based IDS/IPS

Mainframe Network Monitoring

Network-based ID.. OperatingSystem

Policy Man agement Router Security Mangement VPN 0M 100M 200M 300M 400M 500M 600M 700M Cou DeviceSeveritybyDevice

Count of Device Severity for each Category Device Type. The view is filtered on Category Device Type, which keeps 13 of 20 members.

(15)

Category Outcome / Category Significance

/Attempt /Failure /Success

/Comp /Informational /Informational/Error /Normal /Recon /Suspicious Null /Compromise /Informational /Informational/Alert /Informational/Error /Informational/Warning Null /Compromise /Informational /Informational/Alert /Informational/Error /Informational/Warning /Normal /Recon /Suspicious 0M 50M 100M 150M 200M 250M 300M 350M Cou destHostnameAttemptFailSuccess

Count of Destination Host Name for each Category Significance broken down by Category Outcome. The data is filtered on Destination Host Name, which excludes Null. The view is filtered on Exclusions (Category Outcome,Category Significance) and Category Outcome. The Exclusions (Category Outcome,Category Significance) filter keeps 35 members. The Category Outcome filter excludes Failure.

(16)

(17)

Category Outcome / Category Significance

/Attempt /Failure /Success

/Comp /Informational /Informational/Error /Normal /Recon /Suspicious Null /Compromise /Informational /Informational/Alert /Informational/Error /Informational/Warning Null /Compromise /Informational /Informational/Alert /Informational/Error /Informational/Warning /Normal /Recon /Suspicious 0M 50M 100M 150M 200M 250M 300M 350M Cou destHostnameAttemptFailSuccess

Count of Destination Host Name for each Category Significance broken down by Category Outcome. The data is filtered on Destination Host Name, which excludes Null. The view is filtered on Exclusions (Category Outcome,Category Significance) and Category Outcome. The Exclusions (Category Outcome,Category Significance) filter keeps 35 members. The Category Outcome filter excludes Failure.

(18)

(19)

(20)

Keep in mind this is demo data – however a quick internet search shows this domain has a

reputation as a bullet proof server, delivering malware. Our visualization shows us it’s been

accessed every day for the last 30 days.

(21)

Bullet proof

servers

White spac

e

(22)

Hunt teams

Use case 1

(23)

Device Receipt Time

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 Cat _Network 0K 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..

count of device severity 30 dall all scatter

Device Severity High Medium Unknown Very-High

(24)

Jun 3 Jun 8 Jun 13 Jun 18 Jun 23 Jun 28 Jul 3 Jul 8 Jul 13 Jul 18 Hour of Device Receipt Time [2014]

Cat Network 0K 20K 40K 60K 80K 100K 120K 140K Co IPS Events Device Severity High Medium Unknown Very-High

30 days

(25)

(26)

fromVictim

Jun 7 Jun 12 Jun 17 Jun 22 Jun 27 Jul 2 Jul 7 Jul 12 Minute of Device Receipt Time [2014]

Network 0 100 200 300 400 500 600 Cou tenacle Category Technique /Exploit/Vulnerability /Policy/Breach

/Traffic Anomaly/Network Layer /Traffic Anomaly/Network Layer/Flow /Traffic Anomaly/Network Layer/IP Fragments

(27)

Hunt teams

Use case 2

(28)

(29)

Source addresses

Des

tin

at

ion

s

Sonar trend

(30)

Hunt teams

Use case 3

(31)

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 32 VPN Network-based IDS/IPS Network Monitoring Host-based IDS/IPS Firewall Applications fromAfar2infoOnly

Applications Content Security Database Firewall Host-based IDS/IPS Mainframe Network Monitoring Network-based IDS/IPS Operating System Policy Management Security Mangement VPN