Passwords are for Chumps

(1)

David Veuve

SE, Splunk

Passwords are

for Chumps

(2)

Who Am I?

!   David Veuve – Sales Engineer for Major Accounts in

Northern California

! dveuve@splunk.com

!   Former Splunk Customer (For 3 years, 3.x through 4.3)

!   Security Guy

!   Primary Author of Splunk Search Usage app

(3)

Agenda

3

!   Why Single Sign On (SSO)?

!   SeUng up SSO on Windows

!   SeUng up SSO on Linux

!   SeUng up SSO via SAMLv2

!   A liWle something extra

!   Wrap up

!   All conﬁg ﬁles (where possible for Windows) will be posted to

(4)

Disclaimer

During the course of this presenta[on, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cau[on you that such statements reﬂect our current expecta[ons and

es[mates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in the this presenta[on are being made as

of the [me and date of its live presenta[on. If reviewed a`er its live presenta[on, this presenta[on may not contain current or accurate informa[on. We do not assume any obliga[on to update any forward-‐looking statements we may make. In addi[on, any informa[on about our roadmap outlines our general product direc[on and is subject to change

at any [me without no[ce. It is for informa[onal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obliga[on either to develop the features or func[onality described or to

(5)

What is Wrong with Passwords

5

!   Diminish adop[on

!   Dispropor[onately discourage the users you really want

–  Execu[ves/Managers, Business Users

!   Fundamentally insecure

(6)

Detail: Passwords are Fundamentally Insecure

!   People write them on post-‐

it notes

!   People create simple ones

!   People type them into

phishing websites

!   People reuse them across

many websites

(7)

Beneﬁts of Single Sign On

7

!   Easier adop[on

!   More secure

!   Facilitates High Availability

–  Search Head Pooling works beWer with SSO enabled

(8)

Limita[ons of Splunk SSO

!   Single Sign On depends on an external proxy that will handle the

authen[ca[on piece, and then pass the username in an HTTP header to Splunk

!   Even with Single Sign On handling authen[ca[on, we s[ll need an

LDAP connec[on to assign users to individual roles. This is not

typically an issue for internal deployments, but is a greater issue for SAML deployments

(9)

Single Sign On -‐ Deﬁni[on

9

!   Single sign-‐on (SSO) is mechanism whereby a single ac6on of user

authen6ca6on and authoriza6on can permit a user to access all

computers and systems where he has access permission, without the need to enter mul6ple passwords

–  hWp://www.opengroup.org/security/sso/

!   In prac[ce: Users are automa[cally logged in without typing

(10)

Common Single Sign On Methods

!   Ac[ve Directory

–  AD has supported SSO via NTLM and others for years

!   Kerberos

–  Core to Ac[ve Directory and widely used in Linux / OSX

!   SAML

–  Commonly used for online systems

!   Smart Card (or One Time Password)

–  Can be implemented by one of the above, or a hook into Ac[ve Directory to intercept and service authen[ca[on accounts

(11)

How to Decide Which Method

11

!   Windows Server Environment:

–  Windows Authen[ca[on

–  Easiest setup in my experience

!   Linux Server Environment:

–  Kerberos –  S[ll easy

! Splunk hosted via external cloud (or with 3rd party SSO such as Okta,

PingIden[ty, etc.):

–  SAML

–  Most Challenging approach !   3rd Party Proxy / Load Balancer

(12)

(13)

Splunk Setup Steps

13

1.  Set up LDAP Authen[ca[on

2.  Map LDAP Groups

3.  Update server.conf

(14)

LDAP Conﬁgura[on

!   Frequently done by Splunk Users

–  hWp://docs.splunk.com/Documenta[on/Splunk/6.1.3/Security/ConﬁgureLDAPwithSplunkWeb

(15)

server.conf and web.conf Setup

15

! server.conf

–  trustedIP Indicates that the local splunkd will trust the user coming from splunkweb

ê  (Remember that indexers implicitly trust the search head, so this only happens on the search head)

! web.conf

–  trustedIP Indicates that splunkweb will trust the user coming from your upstream proxy/other device

–  SSOMode Indicates whether local logons are allowed

(16)

Security Quick Tip

!   Limit the number of trusted IPs you have conﬁgured on splunkweb,

as they will be able to masquerade as any user

!   If you have tools.proxy.on = true, and see your worksta[on’s IP

address in /debug/sso, turn oﬀ tools.proxy.on and don’t add every worksta[on to the trustedIP list

(17)

Demo – Splunk Setup

(18)

(19)

Demo – server.conf

19

! server.conf – Refers to the local splunkd

–  Remember that splunkweb running on the same box will communicate with splunkd via 127.0.0.1

(20)

Demo – web.conf

! web.conf – Refers to the local splunkweb

–  SSOMode

ê  Permissive – allows either SSO or direct access to splunkd

ê  Strict – SSO only (cannot log in with local auth seUngs – if locked out, must modify via conf ﬁles)

–  trustedIP

ê  IP of Proxy

–  remoteUser

ê  Parameter containing username

–  tools.proxy.on

ê  Required for old versions of Apache. This is turned on in a bunch of examples, but for none of the systems I’ve used has it actually been necessary

(21)

(22)

Core Technologies at Play

!   Func[oning Splunk Install

!   Ac[ve Directory Infrastructure

!   IIS Web Server (2012 R2 in my test, but known to work at least through

2008)

–  Plarorm addons:

ê  ARR

–  hWp://www.iis.net/downloads/microso`/applica[on-‐request-‐rou[ng

ê  ISAPI Module

ê  ISAPI Filters Module

–  Free Third Party

ê  ISAPI_Rewrite3

–  hWp://www.isapirewrite.com/

(23)

High Level Process

23

1.  Conﬁgure Authen[ca[on for IIS Site

2.  Conﬁgure Reverse Proxy for IIS Site

3.  Conﬁgure URL_Rewrite to empty Accept Encoding

–  Workaround for UI quirk

(24)

Windows Authen[ca[on Diagram

!   Users will hit the IIS Server, which will

authorize them via Integrated Windows Authen[ca[on

!   Requests will then be proxied to Splunk

! Splunk will perform authoriza[on via

LDAP Groups

!   Users will get a seamless authen[ca[on

and authoriza[on experience, and be greeted by the Splunk page!

(25)

Challenges

25

!   By default, Splunk will use gzip encoding, but that doesn’t work with

IIS ARR rou[ng rules. As a result, we need to store the original

Accept Encoding in a header, wipe it, and then replace it. That will be seen in the example

!   IIS does not support wri[ng the authen[cated user informa[on into

a header. This is why we need the external ISAPI_Rewrite3 Lite

module. Fortunately, we can use the free Lite module by oﬄoading the rou[ng

!   (Neither of these issues exist on Linux, or should exist on 3rd party

(26)

Why Third Party (ISAPI_Rewrite3 Lite)

!   ISAPI_Rewrite3 by Helicon is a great way to port conﬁgura[ons over

from Apache

!   In par[cular, it allows us to set a header a`er the authen[ca[on

part completes, which is not possible out of the box with IIS

!   There are two versions of ISAPI_Rewrite3 – free and commercial

–  For this conﬁgura[on, we only need the free version. The commercial version adds addi[onal proxy capabili[es which are delivered by IIS ARR

(27)

Demo – Enabling Authen[ca[on

(28)

(29)

Demo – Conﬁgure URL Rewrite

(30)

(31)

Demo -‐ Helicon

(32)

(33)

Demo – Successful Logon

(34)

Troubleshoo[ng

! Wireshark – Verify that communica[on to your search head has the

proper ﬁeld populated

!   Debug page

–  hWp://YourIISServer/debug/sso

!   IIS Detailed Debug Logs

–  By default, IIS will only show you the major error code (e.g., 500). If you turn on detailed logs, it will also show the more detailed logs, e.g.:

ê  HTTP Error 500.52 -‐ URL Rewrite Module Error. Outbound rewrite rules cannot

(35)

Troubleshoo[ng with Wireshark

35

!   Capture relevant traﬃc (port 8000)

!   Then look for the actual headers

(36)

Troubleshoo[ng with Debug SSO

!   Great source for ensuring

your seUngs are correct

!   Look par[cularly for the SSO

Mode, trustedIPs and the Remote user HTTP Header. This has to be the same as what is seen inWireshark

!   Hopefully your setup will

(37)

Troubleshoo[ng with IIS Logs

37

!   By default IIS logs aren’t very helpful. While troubleshoo[ng,

turn on detailed logs for your site

!   Just click on Error Logs, then Edit Feature SeUngs, then

(38)

(39)

Core Technologies

39

!   Working Splunk Installa[on

!   Linux Kerberos

!   Apache Web Server

–  mod_auth_kerb –  mod_proxy

–  mod_rewrite

(40)

High Level Process

!   Create AD Service Account

!   Create keytab

!   Conﬁgure Linux Host Kerberos

!   Conﬁgure Apache to use mod_auth_kerb

!   Conﬁgure Apache to revers proxy using mod_proxy

(41)

Linux Authen[ca[on Diagram

41

!   Users will hit the Apache Server, which

will authorize them via Kerberos to AD

LDAP Groups

!   Users will get a seamless authen[ca[on

and authoriza[on experience, and be greeted by the Splunk page!

(42)

Challenges

!   Biggest challenge with this approach is that there are many diﬀerent

sets of instruc[ons on the internet. This approach, end to end, worked in my environment

(43)

Demo – Create AD User

43

(44)

Demo – Create Keytab

!   Copy-‐paste from internet. Note that this will reset the password

! ktpass -‐princ {PRINCIPAL NAME} -‐mapuser {username@fqdn} -‐

crypto {YourChoice} -‐ptype KRB5_NT_PRINCIPAL -‐pass {LookAtMyLongPassword} -‐out {Path\to\keytab}

(45)

Demo -‐ Conﬁgure Linux Host Kerberos

45

!   Change the realm to your local realm

!   Note that this should probably match your

users’ desktop conﬁg – i.e., if they log into mydomain.local and you’re hos[ng this site on mydomain.com, you will need to conﬁgure IE/Firefox/Chrome to try a kerberos Auth

(46)

Demo -‐ Conﬁgure Apache to use auth_kerb

!   Change the realm and AuthName

to your local realm/domain FQDN

!   Conﬁgure the Krb5KeyTab to

where you copied the ﬁle over from your domain controller

!   KrbMethodK5Passwd allows users

without kerberos to authen[cate via password

!   Require valid-‐user tells Apache

(47)

Demo -‐ Conﬁgure Apache to Reverse Proxy

47

!   This leverages and requires mod_proxy to work,

but is a preWy straighrorward conﬁg beyond that

!   The last two lines are the heart of the

conﬁg – behind the scenes, take anything going to myserver/* and send a parallel request to hWp://127.0.0.1:8000/*

!   If moun[ng your web path at a diﬀerent

directory, consider the root_endpoint seUng

(48)

Demo – Conﬁgure Remote User Header

!   Unlike with Windows, here we can leverage a simple conﬁg to insert

the remote user into the REMOTE-‐USER header

!   In seUng this up, I tried several aWempts to get the remote_user

(49)

Demo – PuUng it all together

49

!   All the conﬁgura[on for my

environment lives in /etc/hWpd/ conf.d/splunksso.conf

(50)

Troubleshoo[ng

!   Paralleling the Windows troubleshoo[ng, there are three great tools

for troubleshoo[ng on Linux:

–  Apache Logs (hey, it’s super easy to Splunk those!) –  Debug SSO Splunk Endpoint

(51)

Troubleshoo[ng with Apache Logs

51

!   Make sure your keytab is in the right path!

(52)

Troubleshoo[ng with Debug SSO

Mode, trustedIPs and the Remote user HTTP Header. This has to be the same as what is seen in tcpdump

(53)

Troubleshoo[ng with tcpdump

53

!   Great to verify that the reverse

proxy actually works and that the seUngs are correct

!   Look par[cularly for the the Remote

(54)

(55)

Core Technologies

55

!   Working Splunk Installa[on

!   Linux Host (CentOS 6.0 for this demo)

–  yum install xmlsec1 xmlsec1-‐openssl xmlsec1-‐openssl-‐devel openssl hWpd mod_ssl

–  Install EPEL on your RHEL-‐type box to get the xmlsec1s –  Lasso

!   Apache Web Server

–  mod_auth_mellon

!   SAMLv2 Iden[ty Provider

–  Recommend that to get started, you leverage a known working partner such as Okta (used here) or PingIden[ty. Then adapt to your own SAMLv2

(56)

High Level Process

!   Install host dependencies

!   Set up Iden[ty Provider (e.g., Okta/PingIden[ty/etc.)

!   Set up mellon conﬁg

!   Set up mod_auth_mellon conﬁg

!   Based almost completely on Paul Stout’s excellent guide:

hWp://blogs.splunk.com/2013/10/09/splunk-‐sso-‐using-‐saml-‐ through-‐okta/

(57)

SAMLv2 Authen[ca[on Diagram

57 !   Users will hit the Okta Server, which will

authorize them and then forward them (via POST) to the Splunk server, which does not have to be accessible to Okta (can be behind the VPN)

LDAP Groups

!   Users will get a seamless authen[ca[on and

authoriza[on experience, and be greeted by the Splunk page!

(58)

Challenges

!   The provided versions of mod_auth_mellon / lasso only work for

hWpd 2.2. There will be a conﬂict if you try to install on 2.4, and when I tried a newer version of mod_auth_mellon (0.7.0 instead of 0.5.0) it never worked, and never errored out

–  Recommend that you set up ﬁrst on 2.2 (RHEL or equivalent 5.x or 6.x, verify with hWpd -‐v) as it’s a known working version

!   SAMLv2 is a notoriously ﬁnicky setup with lots of moving parts.

Recommend that you start with a known working combina[on (e.g., Okta has a no-‐limit free version for a single app), then make incremental changes to move to your own implementa[on

(59)

On Groups

59

!   The major downside to SAMLv2 in Splunk is that it will only handle

authen[ca[on. You will s[ll need to set up groups to handle authoriza[on, which would require an LDAP connec[on

(60)

Demo – Install Host Dependencies

! wget hWp://dl.fedoraproject.org/pub/epel/6/x86_64/epel-‐release-‐6-‐8.noarch.rpm

!   rpm -‐ivh epel-‐release-‐6-‐8.noarch.rpm

!   yum install hWpd xmlsec1 xmlsec1-‐openssl xmlsec1-‐openssl-‐devel mod_ssl openssl

!   Disable or tune selinux (/etc/selinux/conﬁg)

!   Set your hostname to match your principal name (e.g., splunk.dvsplunk.com)

! wget hWps://dev.entrouvert.org/redhat/6/RPMS/x86_64/lasso-‐2.3.6-‐1.el6.x86_64.rpm

! wget hWps://dev.entrouvert.org/redhat/6/RPMS/x86_64/

mod_auth_mellon-‐0.5.0-‐1.el6.x86_64.rpm

!   rpm -‐ivh lasso-‐2.3.6-‐1.el6.x86_64.rpm

(61)

Demo – Set up Iden[ty Provider (IdP)

61

!   Very easy with Okta

–  Add Applica[on –  Provide URL

(62)

Demo – Grab IdP Metadata

(63)

Demo – Set up Mellon Conﬁg

63

!   Paul Stout’s previously-‐linked-‐to guide includes a handy script that

(64)

Demo – Set up mod_auth_mellon

!   The Mellon conﬁg is preWy

straighrorward, and very copy-‐pasteable

!   For an explana[on of the

ProxyPass conﬁgura[on, please see the Linux Conﬁg sec[on

(65)

Troubleshoo[ng

65

!   The recommended troubleshoo[ng tools for this conﬁgura[on are

iden[cal to those for normal Linux systems:

–  Apache Logs (hey, it’s super easy to Splunk those!) –  Debug SSO Splunk Endpoint

(66)

Troubleshoo[ng with Apache Logs

!   Make sure your keytab is in the right path!

(67)

Troubleshoo[ng with Debug SSO

67

Mode, trustedIPs and the Remote user HTTP Header. This has to be the same as what is seen in tcpdump

(68)

Troubleshoo[ng with tcpdump

!   Great to verify that the reverse

proxy actually works and that the seUngs are correct

!   Look par[cularly for the Remote

(69)

(70)

Splunk Search Usage

! Splunk Search Usage Analysis and Adop[on Tracking, with security reports

!  

(71)

(72)

Wrap Up

!   Three Op[ons for Single Sign On:

–  Windows Web Server – Easy –  Linux Web Server – Easy

–  SAML – Achievable, recommend a packaged solu[on if you need this (e.g., Okta, PingIden[ty, etc.)

!   SSO gives you more security, greater adop[on, and less headache

!   You can probably set this up in your environment in < 1 hr

!   Check out the Splunk Search Usage app to beWer understand users

(73)

Conﬁg Files – GitHub

73

!   That was a lot of material, right?

!   Get all the conﬁgs here: hWp://www.davidveuve.com/go/conf-‐sso

hWp://xkcd.com/565/

(74)

Passwords are for Chumps

David Veuve

Disclaimer

What is Wrong with Passwords

Beneﬁts of Single Sign On

Single Sign On -­‐ Deﬁni[on

Common Single Sign On Methods

How to Decide Which Method

Splunk Setup Steps

Security Quick Tip

Demo – Splunk Setup

Core Technologies at Play

High Level Process

Challenges

Why Third Party (ISAPI_Rewrite3 Lite)

Demo – Enabling Authen[ca[on

Troubleshoo[ng with Wireshark

Troubleshoo[ng with IIS Logs

High Level Process

Challenges

Demo -­‐ Conﬁgure Linux Host Kerberos

Demo -­‐ Conﬁgure Apache to use auth_kerb

Demo -­‐ Conﬁgure Apache to Reverse Proxy

Demo – Conﬁgure Remote User Header

Troubleshoo[ng

Troubleshoo[ng with tcpdump

High Level Process

Challenges

On Groups

Demo – Set up Iden[ty Provider (IdP)

Troubleshoo[ng

Troubleshoo[ng with Debug SSO

Troubleshoo[ng with tcpdump

Wrap Up

Conﬁg Files – GitHub

Single Sign On -‐ Deﬁni[on

Demo -‐ Conﬁgure Linux Host Kerberos

Demo -‐ Conﬁgure Apache to use auth_kerb

Demo -‐ Conﬁgure Apache to Reverse Proxy