Cloud-based Log Analysis and Visualization

(1)

Cloud-based Log Analysis and Visualization

mobile-166 My syslog

DeepSec 2010, Vienna, Austria

(2)

Logging as a Service

Raffael (

Raffy

) Marty

2

• Founder @

• Chief Security Strategist and Product Manager @ Splunk

• Manager Solutions @ ArcSight

• Intrusion Detection Research @ IBM Research

• IT Security Consultant @ PriceWaterhouse Coopers

Applied Security Visualization Publisher: Addison Wesley (August, 2008)

(3)

Agenda

3

•Introduction

•Beaver Challenge

•The Cloud

•Visualization

•Visualization Tools

•Visualization in the Cloud

•Visualization Use-Cases

(4)

Raffael Marty - @zrlram

The Public Cloud

4

IaaS

- Infrastructure

PaaS

- Platform

SaaS

- Software

LaaS

- Logging

(5)

Visibility and Big Data

(6)

Visibility

6

•

Monitoring

-

Performance

-

Availability

-

Ephemeral Infrastructure

•

Security

-

New Threats

-

New Vulnerabilities

-

Different Risk Distribution

IaaS

- Similar to before

PaaS

- Lack of Infrastructure

SaaS

- Blind?

Application Instrumentation

and Logging

(7)

Big Data

7

•

NoSQL

•

Distributed data stores

•

Distributed queues

•

Map reduce

•

ETL

(Extract, Transform, Load)

•

...

(8)

Information Visualization

8

•

Better tools and capabilities

•

Across disciplines

•

More instrumentation

•

Dichotomies

(9)

Open Your Eyes

(10)

Information Visualization?

A picture is worth a thousand log records.

Inspire Pose a New Question Explore and Discover Support Decisions Communicate Information Increase Efficiency Answer a Question 10

(11)

Visualization

Tools

(12)

Reporting vs. Visualization

12

•

Reporting Libraries

-

HighCharts

-

Flot

-

Google Chart API

-

Open Flash Chart

-

HTML5

•

Visualization Libraries

-

TheJIT

-

Graphael

-

Protovis

-

ProcessingJS

-

Flare

(13)

HighCharts

13

•

Click-Through

•

On load

-

near real-time updates

•

Zoom

•

AJAX data input via JSON

(14)

Google Visualization API

•

JavaScript

•

Based on DataTables()

•

Many graphs

•

Playground

- http://code.google.com/apis/ajax/playground 14 http://code.google.com/apis/visualization/interactive_charts.html

(15)

ProtoVis

•

JavaScript based visualization library

•

Charting

•

Treemaps

•

BoxPlots

•

Parallel Coordinates

•

etc.

15

http://vis.stanford.edu/protovis/

(16)

TheJIT

•

JavaScript InfoVis Toolkit

•

Interactive

•

Link Graphs

16

(17)

Processing

•Visualization library •Java based

•Interactive (event handling) •Number of libraries to -draw in OpenGL -read XML files -write PDF files •Processing JS -JavaScript -HTML 5 Canvas -Web IDE 17 http://processing.org/ http://processingjs.org/

(18)

Data

Visualization

in the

Cloud

(19)

LaaS - Logging as a Service

19

• Log collection

• all data in one place

• Log storage and management

• index, storage, archive

• Extremely fast log search across all your data

• data source agnostic (no parsers)

• innovative Web shell

• API log access

• oAuth authentication • always on

Benefits

• No installation • Easy configuration • No maintenance • Great scalability • 7x24 availability • Pay as you go

(20)

AfterGlow Cloud

20 Grapher Loggly JSON CSV DOT Graph

(21)

Visualization

Use-Cases

(22)

Old Skewl

22

(23)

Traditional Style

23

(24)

The Analysis Approach

24

Overview first

Zoom

Details on

demand

(25)

NetFlow Visualization

•

Treemap

•

Protovis.JS

•

Size: Amount

•

Brightness: Variance

•

Color: Sensor

•

Shows: Scans -

bright spots

• Thanks to Chris Horsley

(26)

Firewall Treemap

(27)

Firewall Log

Port Source IP Destination IP

(28)

IDS Signature Tuning

(29)

(30)

IDS Sig Tuning - Treemap

30 Hierarchy: Source Destination Signature Number of Events Color: Service

(31)

IDS Sig Tuning - Treemap

31 Hierarchy: Source Destination Signature Number of Events Color: Priority

(32)

IDS Sig Tuning - Treemap

32 Hierarchy: Signature Source Service (Port) Color: Priority

(33)

Visualization

Resources

(34)

Share, discuss, challenge, and learn about security

visualization.

http://secviz.org

•

List:

secviz.org/mailinglist

•

Twitter:

@secviz

(35)

Applied Security Visualization

•

Bridging the gap between security and visualization

•

Hands-on, end to end examples

•

Data processing and analysis

Chapters

•

Visualization

•

Data Sources

•

From Data to Graphs

•

Perimeter Threat

35

Addison Wesley (August, 2008) ISBN: 0321510100

•

Compliance

•

Insider Threat

(36)

36

about.me/raffy

http://www.highcharts.com/

http://code.google.com/apis/ajax/playground

http://code.google.com/apis/visualization/interactive_charts.html

JavaScript based visualization library

JavaScript InfoVis Toolkit

http://thejit.org/

http://processing.org/http://processingjs.org/

http://secviz.org