Cyber Security With Big Data

(1)

Cyber Security With Big Data

Fast. Complete. Cost-Eﬀec1ve.

Harry J Foxwell, PhD

Principal Consultant

Oracle Public Sector

Oct 2015

(2)

Safe Harbor Statement

The following is intended to outline our general product direcOon. It is intended for

informaOon purposes only, and may not be incorporated into any contract. It is not a

commitment to deliver any material, code, or funcOonality, and should not be relied upon

in making purchasing decisions. The development, release, and Oming of any features or

funcOonality described for Oracle’s products remains at the sole discreOon of Oracle.

Oracle ConﬁdenOal – Internal/Restricted/Highly Restricted 2

(3)

Pressures on Tradi1onal Tools

Growing Data Volume

& Variety

Demand for PredicOve AnalyOcs

Mobile Users

(4)

Big Data Issues

• Ingest Rates (Velocity)

–

 

Simultaneous ingest and query cause conﬂicts and delays in the system

• Data Reten1on (Volume)

–

 

Insuﬃcient periods of Ome for idenOﬁcaOon of trends in data

–

 

Requirement is for mulO-year storage and retenOon = mulO-PB

• Poor Query Performance

–

 

Hours-Days for results, reducing/limiOng number of queries that can be executed

–

 

Challenge for the analyst to “visualize’ the problem and reﬁne query dynamically

• Queries performed on separate compuOng pladorms due to poor performance

–

 

Need hierarchy of systems to conduct queries (

?

)

10/8/15 HHB Systems LLC 4

(5)

SIEM Deﬁciencies

• Ingest Rates

– 

Unable to capture all events in a medium-sized network

– 

Systems struggle to execute queries while collecOng data

• Data Reten1on

– 

_{Log ﬁle sizes are increasing dramaOcally}

– 

Need data over long Omeframes to ﬁnd the Advanced Persistent Threat

• Poor Query Performance

– 

Hours to days for queries to execute

– 

_{Analysts not able to conceptualize results and re-query}

• Results

– 

_{Missed events & trends}

– 

More analysts required, but focused on basic event ID, instead of cyber intel

– 

Excessive system architecture costs

(6)

SIEM Deﬁciencies, Example Use Case

• Ingest Rates

–

 

Can only capture <4,500 Events Per Second (EPS) and

require 70,000 EPS

• Data RetenOon

–

 

Only retain 14 days of full data (limited data for 30 days) and

require 5 years

• Poor Query Performance

–

 

Analysts logger searches take up to 10 hours for a 7 day query

• Limited Scalability

–

 

Unable to accommodate all potenOal users of the system

Scaling up Exis1ng Logger Architecture Would Cost $22.5M!

(7)

Resul1ng Eﬀects

• PotenOally missed events – or longer remedia1on 1mes due to delays

• Some incidents are missed due to lack of storage capacity

• Takes longer to iden1fy a trend due to storage requirements

• Less produc1ve staﬀ and more analysts required

• More personnel focused on lower levels (event idenOﬁcaOon and

validaOon) rather than on cyber intelligence

• Slow system performance

• Costly HW and SW soluOons

10/8/15 7

(8)

Today’s Data Architectures

Data

Warehouse

BI Tools and Dashboards

Custom/Advanced

Analy1cs

OLTP

Systems

(9)

Data

Sources

Acquiring and Using More & Varied Data

Big Data

Ecosystem

NoSQL DB

Data

Warehouse

BI Tools and Dashboards

High Volume

Distributed File

System

Custom/Advanced

Analy1cs

(10)

Data

Sources

Discovering Valuable Data Anywhere

Big Data

Ecosystem

NoSQL DB

Data

Warehouse

BI Tools and Dashboards

High Volume

Distributed File

System

Custom/Advanced

Analy1cs

Knowledge Discovery Engine

Informa1on Discovery

(11)

“There are a lot of commonly held beliefs about big data that need

to be challenged, with the ﬁrst being that you simply adopt

Hadoop and are good to go. The problem is that Hadoop is a

technology, and big data isn't about technology.

Big data is

about business needs. In reality, big data should include

Hadoop and rela1onal [databases] and any other

technology that is suitable for the task at hand.

”

–

 

Ken Rudin, Head of AnalyOcs, Facebook

Oracle ConﬁdenOal – Internal 11

Oracle’s Big Data Strategy

Big Data = Hadoop + NoSQL + Rela1onal…

(12)

2015 O’Reilly Data Science Survey

• Four tools—SQL, Excel, R, and Python

–

 

remain at the top for the third year in a row

• Spark (and Scala) use has grown

• R is now used by more data professionals

–

 

who otherwise tend to use commercial tools

–

 

hqp://duu86o6n09pv.cloudfront.net/reports/2015-data-science-salary-survey.pdf

(13)

Big Data Analy1cs Challenge

Separate data access interfaces

(14)

Oracle Big Data Management

Preserving investment with transparent Big Data access

14

NoSQL

(15)

Oracle Big Data SQL

15

Massively Parallel SQL Query across Oracle, Hadoop and NoSQL

Oracle Database 12c

Oﬄoad Query to

Exadata Storage Servers

Small data subset

quickly returned

Hadoop & NoSQL

Oﬄoad Query to

Data Nodes

SQL

data

subset

SQL

(16)

Oracle’s Open Source Support

(17)

Sun Oracle X4-2L Servers with

per server

:

• 2 * 8 Core Intel Xeon E5 Processors

• 64 GB Memory

• 48TB Disk space (864 TB of storage in a Full Rack)

Integrated Software (3.0):

• Oracle

Linux

• Oracle

Java

VM

• Cloudera Distribution of Apache

Hadoop

(CDH) 5.0

• Cloudera Manager

5.0 and Options

• Apache

Spark

• Oracle

R

Distribution

• Oracle

NoSQL

Database

(18)

Cyber POC Architecture

Protected

Network

1. Copy of all

data packets

Port Mirror

2. Log Files

Oracle Big Data and

Rela1onal Data Stores

Security Analyst

Predic1ve Aqack RecogniOon,

Session and File AnalyOcs

(19)

Data

Sources

Enterprise Data Architecture

Knowledge Discovery Engine

Big Data

Ecosystem

NoSQL DB

Rela1onal

Data

BI Tools and Dashboards

Real-Time

Recommenda1ons

High Volume

Distributed File

System

Custom/Advanced

Analy1cs

Machine Learning Algorithms

Informa1on Discovery

(20)

SO

U

RC

ES

DATA RESERVOIR

DATA WAREHOUSE

Oracle Database

Oracle Industry

Models

Oracle Advanced

Analy1cs

Oracle Spa1al & Graph

Big Data Appliance

Big Data

Discovery

Oracle Event

Processing

Cloudera Hadoop

Oracle Big Data SQL

Oracle NoSQL

Oracle R Advanced

Analy1cs for Hadoop

Oracle R Distribu1on

Oracle Database

In-Memory,

Mul1-tenant

Oracle Industry

Models

Oracle Advanced

Analy1cs

Oracle Spa1al & Graph

Exadata

Oracle

GoldenGate

Time Decisions

Oracle Real

Oracle Big

Data

Connectors

Oracle Data

Integrator

ANALYTICS

Exaly1cs

Oracle Business

Analy1cs

Enterprise

Business Intel

Self-Service

Discovery

Data Mashup

Oracle Big

Data SQL

Endeca Info

Discovery

OBIEE

Oracle Big Data Management System

(21)

Example: Packet Analy1cs On Big Data Appliance

• Tested several

diﬀerent cluster sizes

• Processed data sets

from 1-100 TB

• Completed full

analysis of

100TB of

data in 16 hours

Real-Time Analysis of

a 14Gbs data stream

(22)

Example: ArcSight on Eng System Results

• Achieved 70,000 EPS during POC

• OpOmized later for over 115,000 EPS or

25x more data

• Exadata compression produced

14x storage improvement

• Queries returned

27x-6,500x faster

• _{Logger query that previously took 9 days is now back in <2 mins}

• ~90% Cost Reduc1on

(Saving over $20M)

(23)

Reduce the Incident Handling Life Cycle

• Enhance prepara1on and data availability to analysts

• Enable faster detec1on due to enhanced event correlaOon and processing

• Facilitates collabora1on and analyOc support for containment

• Enables enhanced valida1on of architecture to idenOfy other similar aqack

locaOons

• Captures and retains data for corporate history and community

collaboraOon reducing post incident acOvity resources and Omelines

10/8/15 Page 23

(24)

How Oracle Engineered Solu1ons Support Incident Response

• Accelerates mission 1melines, reducing threat 1melines

• Enables precision responses

• Enables a new class of real 1me strategies and analy1cs

• Speeds dispatch to mgmt, enables analy1cs and automa1on

• Enables more dynamic organiza1onal management and response

• Integrate over longer period to catch low and slow aoacks & track

campaigns

• Roadmap for maturing the incident response capability-

– 

provides scalability and ﬂexibility for work ﬂow and collabora1on

• How the program ﬁts into the overall organizaOon –

– 

_{enables integra1on SOC/CERT/ Focused Intelligence}

10/8/15 Page 24

(25)

Enables focus on Cyber Intelligence

10/8/15 25

(26)

(27)

(28)

Net Collector

Filtering, Aggrega1ng, Reconstruc1ng Network Packets

• Integrated

low-cost

probes

• Up to 40Gbps at each probe

• Embedded real-Ome NoSQL data store

• Intrusion Detec1on System & Deep

Packet InspecOon capability

• ExtracOon of metadata, content and staOsOcs

• Smart data transportaOon

(29)

Net Aggregator

Managing and Analyzing the Data

• Mul1ple data models for diﬀerent

types of analysis

• Uniﬁed system metadata

• Real-Ome and batch processing

• IntegraOon with External

Repositories and AnalyOcs via

open APIs

(30)

Analysis Desktop

Rich visualiza1on and Informa1on discovery

• Network (graph) & map view

• Guided navigaOon

• Data & text analyOcs

• Interac1ve dashboards

• ReporOng and publishing

• Session ReconstrucOon

•  

3 rd

Party Analy1cs

(31)

Chaos Becoming Order

Data that was:

•   Previously disjointed

•   Difficult to access

•   Hard to understand

Is now:

•   Presented in COPs

•   Single sign on

•   Visually driven

•   Discovery learning

Endeca InformaOon Discovery

Chaos Becoming Order

(32)

Oracle Big Data Discovery

Changing the Game for Agile Government Innova1on on Big Data

Proﬁle

Easily add data and

see it automaOcally

and conOnuously

cataloged, enriched

and related

Find

Use familiar

guided search

across massive

amounts of

diverse data

Understand

Know what’s

important from

diagnosOc analysis

of millions of data

characterisOcs

Transform

Powerful tools to

quickly clean up

and wrangle

dirty data so it’s

ready to go

Discover

Uncover

valuable new

insights

Collaborate

Publish, share and

evolve as you learn

more

Predict

Use new

insights to

deﬁne and

reﬁne predicOve

models

32

(33)

Breakthrough InnovaOon:

Oracle Big Data Discovery

33

Radically

simplify

data prep and Analysis process

(34)

Extending Data Management…

Big Data Management System

NoSQL

• Scale Agency

–

 

Meet mobile challenges

–

 

Accelerate developer agility

–

 

Scale-out economically

–

 

Serve data faster

• _Run Agency

–

 

Integrate exisOng systems

–

 

Support mission-criOcal tasks

–

 

Protect exisOng expenditures

–

 

Insure skills relevance

Rela1onal

Hadoop

• _{Change Agency}

–

 

Disrupt fraudsters

–

 

Improve supply chains

–

 

Leverage new paradigms

–

 

Exploit new analyses

(35)

Overcoming Barriers to AdopOon of New Technologies

ConﬁdenOal 35

INTEGRATION

SKILLS

SECURITY

Engineered

Systems

_All Data

SQL on

Database

Security on

All Data

(36)

Exadata

Big Data

Appliance

Engineered for Big Data Management at Scale

Intelligent Storage

Smart Scan

Storage Indexing

Advanced Compression

Easy Upgrades

Easy ConsolidaOon

Engineered System for

Oracle Database

Security

- AuthenOcaOon

- AudiOng

- EncrypOon

High Availability

Easy Upgrades

Rapid Provisioning

Engineered System for

Hadoop & NoSQL

Integrated Enterprise Management

Engineered

Data

Connectors

(37)

Govern & Secure All your Data

Hadoop, NoSQL & Rela1onal

• ExisOng Capability

–

 

AuthenOcaOon through Kerberos

–

 

AuthorizaOon through Apache Sentry

–

 

AudiOng through Oracle Audit Vault

–

 

EncrypOon for Data-at-Rest

–

 

Network EncrypOon

• Big Data SQL adds

–

 

Advanced Security on Hadoop & NoSQL

• Masking and RedacOon

–

 

Virtual Private Database

• _{Fine-grain Access Control}

Oracle ConﬁdenOal – Internal 37

(38)

Unstructured Data

NoSQL DB

HDFS Storage

Rela1onal Data

Log File Data

(RDBMS)

SIEM Interface

Security

Admin

Data

Packets

Internal Network

Devices/Souware

Enterprise Cyber Data Architecture

Common Opera1ng Picture

Big Data

Connectors

Discovery

Aoack

S

O

A