Splunk Search Pro Tips

(1)

Dan Aiello

Principal Cyber Security

Engineer, MITRE

Dan Aiello, Principal Cyber Security Engineer

Splunk Search Pro Tips

(2)

Agenda

§

My background

§

Comments

§

Search by index

§

In the year 2000

§

Red Card

§

Watch Lists

§

Search Job Inspector

§

More fun with subsearches

§

Carrier signal

§

Imaginary events

(3)

My Splunk background

§

4 years Splunk experience

§

SOC is primary user base

§

6 indexers

§

350 GB data/day

§

90 indexes

(4)

/ / N o c o m m e n t

comments

(5)

Comment your Splunk search

sourcetype=access_combined_wcookie

| eval COMMENT="This is my comment"

or

sourcetype=access_combined_wcookie

| rename COMMENT -‐> "This is my comment"

(6)

rename vs. eval for comments?

§

In practice, it does not seem to matter

§

In a few odd circumstances, I have seen rename be

faster than eval

(7)

(8)

(9)

Searching by index and sourcetype

Specifying an index in

your search speeds it up

This difference is less

pronounced in Fast Mode

(10)

I n t h e y e a r 3 0 0 0

(11)

Get with the times

§

Timestamps are extremely important for Splunk data

§

Detected at index time, set forever

§

Cannot be fixed if they’re wrong

§

Common errors:

–

Incorrect time zone interpretation

(12)

Past, present, future

If this search ever returns events, you have timestamp

problems

1

:

index=* earliest=+30m latest=+9y

This requires some tweaking, depending on your

expected delay:

index=* | eval delta=_indextime-‐_time | where

delta>300

1

Or a flux capacitor

2 2

Or a TARDIS

(13)

y o u r a p p r o x i m a t e w a i t t i m e i s …

red card

(14)

Calculate average delay proxy logs

index=main

| eval delta = _indextime -‐ _time

| timechart span=1h avg(delta)

(15)

(16)

Calculate average delay proxy logs

Solution:

*/5 * * * * wget http://testdomain.zzz

index=main

testdomain.zzz

| eval delta = _indextime -‐ _time

| timechart span=1h avg(delta)

Search terms

Duration

index=main

453 s

(17)

b e t t e r t h a n g r e p - f

watchlists

(18)

Watchlist examples

§

Known “evil” IP addresses

§

Known “evil” domain names

§

List of your DMZ web servers

(19)

(20)

Conventional way

to use watchlists

This is essentially

grep -F

(21)

(22)

What’s the difference?

Both return the same events

(23)

What’s the difference?

(24)

Saving time on a search

can be important for large

or frequent searches

Small watchlists and

large datasets make

this difference greater

(25)

Just the subsearch

How does it work?

(26)

Subsearches

implicitly end with

| format

Add it explicitly to

see what’s

(27)

[ | inputlookup ip_watchlist.csv | search

type=malicious | fields clientip ]

…after the subsearch is evaluated becomes this:

( ( clientip="131.178.233.243" ) OR

( clientip="212.58.253.71" ) OR … )

(28)

(29)

Why is

( ( clientip="131.178.233.243" ) OR

( clientip="212.58.253.71" ) OR … )

Better than

| lookup ip_watchlist.csv clientip | search

type=malicious

(30)

… e x p l a i n s i t a l l

(31)

This icon means there’s

some debugging message

you should examine

Inspect Job

(32)

(33)

Debugging message

(34)

The slowest parts of a Splunk

search are usually field extraction

(35)

Approximate order of operations for searches

1.

Search index for keywords

2.

Read matching events from disk

3.

Extract fields (as necessary)

4.

Match keywords to fields (as necessary)

5.

Filter (e.g. additional “where” or “search” pipes)

(36)

(37)

A stitch in time saves nine

lookup

subsearch

2

21 Check index for keywords

39,000

2,700 Read matching events from disk

39,000

2,700 Extract fields (i.e. regex)

39,000

2,700 Match keywords to fields

39,000

2,700 Filter

* This is illustrative and approximate, not precise

(38)

sourcetype=access_

combined_wcookie

( ( clientip="131.178.2

33.243" ) OR

( clientip="212.58.253.

71" ) OR … )

lookup method reads

and regexes all this data

subsearch method

reads only this data

(39)

(40)

(41)

Field name mismatch with subsearch

For lookup and subsearch, sometimes fields need to be

renamed

lookup method

| lookup watchlist.csv foo AS bar

subsearch method

[ | inputlookup watchlist.csv

| rename foo AS bar ]

(42)

If you rename a field to

“query”, you can search

anywhere in the event rather

(43)

Large subsearches

43

If your watchlist is >10000

lines, the subsearch method

(44)

Large subsearches

Add “|format”

explicitly to fix it!

Warning is

gone

We have

(45)

(46)

What’s the problem with watchlists?

When they don’t alert, is it because:

§

the watchlist is broken?

(47)

Label your test domain as

type=test in the watchlist

And adjust your

subsearch accordingly

(48)

Label your test domain as

type=test in the watchlist

And adjust your

subsearch accordingly

Add test cases to your watchlist

What’s so great about that?

We could have just used

(49)

With your wget, you know

precisely how many to

expect and you can alert

only when it’s erroneous

(50)

w e l a n d e d o n t h e m o o n

imaginary events

(51)

(52)

How is this helpful?

You can add to a

watchlist inline

(53)

Creating data on the fly with timestamps

53

This will also

provide a current

(54)

(55)

Overall lessons

§

Read Splunk Search Manual

§

Try multiple methods

§

Use the Job Inspector

(56)