Intro to AppDynamics with SSL

(1)

(2)

1. SSL Introduction 2. SSL in Java

3. SSL in AppDynamics

Agenda

(3)

(4)

What is SSL/TLS?

• Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), are cryptographic protocols that provide security for communications over networks such as the Internet.

• TLS and SSL encrypt the segments of network connections at the Transport Layer end-to-end.

It’s ok to keep saying “SSL”—everybody does it!

(5)

What Security is Provided?

• Endpoint Authentication • Unilateral or Bilateral • Communication Confidentiality • For preventing • Eavesdropping • Tampering • Message Forgery

(6)

History of SSL/TLS

• Secure Sockets Layer (SSL)

• Developed by Netscape Corp • Versions 1, 2, and 3

• Transport Layer Security (TLS)

• Successor of SSL

• IETF standards track protocol, based on SSL 3.0

• Last updated in TLS 1.2 • TLS 1.3 is in draft status

Protocol Year SSL 1.0 n/a SSL 2.0 1995 SSL 3.0 1996 TLS 1.0 1999 TLS 1.1 2006 TLS 1.2 2008 TLS 1.3 TBD

(7)

SSL Handshake

Client

Browser connects to website

Web Server

(8)

SSL Handshake

Client

Server responds with Certificate and key

Web Server

(9)

SSL Handshake

Client Web Server

CA

Client verifies certificate with CA

(10)

SSL Handshake

Client Web Server

Random Key

Client sends a random key to

server encrypted with the public key

(11)

SSL Handshake

Client Web Server

Random Key

(12)

SSL Handshake

• All data encrypted with the server’s public key can only be decrypted by the server’s private key

• The randomly generated key was:

• Randomly generated by the client • Encrypted with the server’s public key

• Only the Server and the Client would know the key, and unless they share it no one else would know

• Encrypting the data with the random key secures the data from prying eyes

(13)

Version Vulnerabilities

• SSL is old. Use TLS instead.

• The last SSL was released back in 1996 • POODLE

• Affects SSL 3.0 (and TLS 1.0+ depending on the vendor) • Heartbleed

• OpenSSL bug rather than defect in SSL spec, but everyone uses OpenSSL so it’s bad news. This has been patched, but not everyone has upgraded

• FREAK

• A weakness in some implementations of SSL/TLS that may allow an attacker to decrypt secure communications between vulnerable clients and servers. Fixed in newer OS and browser releases, March 2015

• Logjam

• Allows a man-in-the-middle attacker to downgrade vulnerable TLS connections to 512-bit export-grade cryptography. This allows the attacker to read and modify any data passed over the connection. A server/website is vulnerable if it supports the DHE_EXPORT ciphersuites or if it uses small parameters for DHE

• SHA1

• Allows attackers to generate and install a fake certificate, if enough computing resources are applied. Browsers will stop accepting SHA1 certificates in 2017

(14)

Certificates

(X.509 aka PKIX) (Identity) Certificate

• A data structure including a public key for an entity and the identity of that entity

• Plus some other information related to the entity and/or the CA • All signed by a (generally) different entity called a Certificate

Authority (CA).

• Some companies or agencies have internal CAs

(15)

Certificate Authority (CA)

• Entity that issues digital certificates

• A trusted third party by the owner and the party relying upon the certificate

• Issues the root certificate

(16)

Certificate Chain

Root Certificate

• The final authority to validate the certificate

• There are dozens of well known CAs included in browsers: VeriSign, GoDaddy, DigiCert, etc.

• Some companies and agencies have their own root certificate

(17)

Certificate Chain

Intermediate Certificate

• Most CAs now operate in a hierarchical fashion, where the root key is not used to directly issue user certificates

• Instead the root CA and its root (private) key is used to sign

certificates for several intermediate or subordinate CAs, each of which has their own keypair

• Each intermediate CA can then issue user certs, or sometimes a second level of intermediate certs--this can be extended to several levels

• If the certificate was not issued by a trusted CA, the connecting device (e.g., a web browser) will then check to see if the certificate of the issuing CA was issued by a trusted CA, and so on until either a trusted CA is found (at which point a trusted, secure connection will be established) or no trusted CA can be found (at which point the device will usually display an error).

(18)

AppDynamics.com Certificate

Alias name: cn_appdynamics_com__o__appdyna Creation date: Oct 14, 2015

Entry type: trustedCertEntry

Owner: CN=appdynamics.com, O="AppDynamics, Inc.", L=San Francisco, ST=CA, C=US Issuer: CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US

Serial number: 301d0badd79504e2d3ca9180797d4cc

Valid from: Thu Dec 05 19:00:00 EST 2013 until: Mon Feb 08 07:00:00 EST 2016

Certificate fingerprints:

MD5: 9D:96:9D:E8:D6:7F:92:B2:3C:2A:67:FB:C1:7A:B3:D2

SHA1: EA:02:EB:98:63:CF:C1:27:4E:8C:9E:2B:F8:13:A9:73:77:F8:C3:48

SHA256: DE:62:4C:DC:09:E4:F4:99:EB:B4:82:71:31:A9:60:4E:09:43:F6:6C:B5:E7:D5:FE: E6:F7:88:60:0F:32:21:C6

Signature algorithm name: SHA256withRSA Version: 3

(19)

DigiCert Root Certificate

Alias name: cn_digicert_sha2_secure_server Creation date: Oct 14, 2015

Entry type: trustedCertEntry

Owner: CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US

Issuer: CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US

Serial number: 1fda3eb6eca75c888438b724bcfbc91

Valid from: Fri Mar 08 07:00:00 EST 2013 until: Wed Mar 08 07:00:00 EST 2023

Certificate fingerprints:

MD5: 34:5E:FF:15:B7:A4:9A:DD:45:1B:65:A7:F4:BD:C6:AE

SHA1: 1F:B8:6B:11:68:EC:74:31:54:06:2E:8C:9C:C5:B1:71:A4:B7:CC:B4

SHA256: 15:4C:43:3C:49:19:29:C5:EF:68:6E:83:8E:32:36:64:A0:0E:6A:0D:82:2C:CC:95: 8F:B4:DA:B0:3E:49:A0:8F

Signature algorithm name: SHA256withRSA Version: 3

(20)

(21)

JSSE

• JSSE = Java Secure Socket Extension is the default Java package • Was optional package before JDK 1.4. Now it’s bundled in the JDK • Lots of old, fragile implementations out there < Java 7

• JSSE is getting better, but requires newer JREs

(22)

Keystore

• A Java KeyStore (JKS) is a repository of security certificates, either authorization certificates or public key certificates

• Contains public/private keypairs

• The private key is accompanied by certificate chain for the corresponding public key

• Decryption based on private key

(23)

Truststore

• Stores certificates of parties you trust • Contains self-signed certs

• Copied from Java’s own cacerts.jks

• Handles the case where certs are signed by the internal CA

(24)

java.security File

The contents of $JAVAHOME/lib/security/java.security define the JVM’s SSL stack

● Security providers

○ There are default providers but they can be overridden, e.g. PKCS#12, NSS, FIPS, PKCS#11

● Keystore type

○ keystore.type=jks ● Policy files

(25)

Using keytool

• Built-in Java tool to work with certificates and keystores

• Command line • Lots of options

• Easy to feel overwhelmed or confused

• See the keytool cookbook at the end

keytool -list -v -keystore keystore.jks

keytool -import -alias <alias> -keystore keystore.jks -file <Path_to_Cert>

(26)

(27)

SSL Architecture

• Controller and EUM can be secured

• Can be handled by the app server, but this adds some overhead • Can be offloaded by Apache Web Server, Load Balancer

• The documentation lists keytool steps

• Copy/paste introduces potential errors

• Intimidating for people unfamiliar with keytool

(28)

Supported Versions

Java

• Controller: TLSv1.2 • Java Agent:

• For Java 8 SE applications, TLSv1.2.

• For Java 7+ applications, TLSv1.2. You can change the agent to use SSLv3 if desired.

• For Java 5/6 applications, SSLv3. You must enable SSLv3 or TLSv1.0 connection on the Controller if the agents connect directly to the

Controller.

• Agent needs to trust the Certificate of the Controller (cacerts.jks)

.NET

• More straightforward because you don’t use keytool

(29)

Import a New Cert Into the Controller

1. Stop the Controller app server

2. export PATH=$PATH:<CONTROLLER_HOME>/jre/bin

3. cd <CONTROLLER_HOME>/appserver/glassfish/domains/domain1/config 4. cp keystore.jks keystore.jks.original_DATETIME

5. keytool -delete -alias s1as -keystore keystore.jks -storepass changeit 6. keytool genkeypair alias s1as keyalg RSA keystore keystore.jks

-keysize 2048 -validity 1825 -storepass changeit

• First and Last name equals the “CN”. First and Last name equals reference number, if that’s how your CA works.

7. keytool certreq alias s1as keystore keystore.jks storepass changeit -file AppDynamics.csr

8. Send over the CSR

9. Import root cert, if needed

• keytool import alias <alias|privatecaroot> keystore keystore.jks -storepass changeit -file <Path_to_Root_or_Intermediate_Cert>

10. keytool -import -trustcacerts -alias s1as -keystore keystore.jks -storepass changeit -file mycert.cer

11. Start the Controller app server and verify HTTPS

Docs here

(30)

Import a New Cert Into the EUM Server

1. cd <eum_home>/eum-processor

2. ../jre/bin/keytool genkey keyalg RSA validity 3560 alias 'eumprocessor'

-keystore bin/mycustom.-keystore

3. ../jre/bin/keytool certreq keystore bin/mycustom.keystore file /tmp/eum.csr

-alias 'eum-processor'

4. Send the CSR to the CA for signing

5. ../jre/bin/keytool -import -trustcacerts -alias myorg-rootca -keystore

bin/mycustom.keystore -file /path/to/CA-cert.txt

6. ../jre/bin/keytool -import -keystore bin/mycustom.keystore -file

/path/to/signed-cert.txt -alias 'eum-processor’ 7. Add these to bin/eum.properties

• processorServer.keyStorePassword=mypassword

• processorServer.keyStoreFileName=mycustom.keystore

8. Restart the EUM Server

Docs here

(31)

Verifying the Keystore

Require (at least) 3 aliases in the Controller’s keystore.jks • s1as

• reporting-instance • glassfish-instance

(32)

Verifying the Keystore

Match AuthorityKeyIdentifier to SubjectKeyIdentifier

• These must match as you move through the certificate chain

AuthorityKeyIdentifier [ KeyIdentifier [ 0000: B5 45 F2 CF 83 6E 5F 27 54 57 B2 06 49 7F C0 FC .E...n_'TW..I... 0010: 00 6E F7 FA .n.. ] ] SubjectKeyIdentifier [ KeyIdentifier [ 0000: B5 45 F2 CF 83 6E 5F 27 54 57 B2 06 49 7F C0 FC .E...n_'TW..I... 0010: 00 6E F7 FA .n.. ] 32

(33)

I/O error: Remote host closed connection during handshake; nested exception is javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

I/O error: Connection reset; nested exception is java.net.SocketException: Connection reset

• The server doesn't trust the client, client certificate not in server truststore

• The client is sending the wrong certificate to the server • Agent communicating to the Controller not on TLSv1.2

• Possibly downgrade the Controller’s SSL/TLS settings

Possible Agent Exceptions

(34)

Possible Agent Exceptions

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

• Truststore is not found

Caused by: sun.security.validator.ValidatorException:

PKIX path building failed: sun.security.provider.certpath.

SunCertPathBuilderException: unable to find valid certification path to requested target

• Server certificate not found in truststore • Server certificate expired or revoked

(35)

Possible Agent Exceptions

[Thread-2] 22 Oct 2015 15:54:07,070 INFO XMLConfigManager - Trying secure protocol:TLS

● The agent is trying to connect over TLSv1.0 so you need to

downgrade the Controller’s security

● Same error can happen for the agent trying to connect over SSL

(36)

Debugging SSL

curl –v https://controller.example.com:8181

* Trying 54.192.161.211...

* Connected to controller.example.com (10.10.1.200) port 443 (#0) * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate: controller.example.com

* Server certificate: DigiCert SHA2 Secure Server CA * Server certificate: DigiCert Global Root CA

> GET / HTTP/1.1

> Host: controller.example.com > User-Agent: curl/7.43.0

> Accept: */*

(37)

Debugging SSL

openssl s_client -connect google.com:443

CONNECTED(00000003)

depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA verify error:num=20:unable to get local issuer certificate verify return:0 ---Certificate chain 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com

i:/C=US/O=Google Inc/CN=Google Internet Authority G2

1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2

i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

---Server certificate

---BEGIN CERTIFICATE---<it’s a long base64 chunk...> ---END

CERTIFICATE---subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com

issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2

---No client certificate CA names sent

---SSL handshake has read 4049 bytes and written 456 bytes

---New, TLSv1/SSLv3, Cipher is AES128-SHA Server public key is 2048 bit

Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : AES128-SHA Session-ID: F3350870CBDBF3CCA87E16F5976E65009427E33DBE41BC8725E82 BFA79B4B40E Session-ID-ctx: Master-Key: CE88A94AF5A29B76D7268FF0E8714A898EE168EE1AE4EF3D15627 C74E4CE7D76602662D4356D17B98770DF1D3E0EA13F Key-Arg : None Start Time: 1447099801 Timeout : 300 (sec) Verify return code: 0 (ok)

(38)

---Debugging SSL

-Djavax.net.debug=SSL

• Docs for Java 6, 7, 8

Command line options

• -Djavax.net.ssl.keyStore=

• -Djavax.net.ssl.keyStorePassword= • -Djavax.net.ssl.trustStrore=

• -Djavax.net.ssl.trustStrorePassword=

(39)

(40)

Resources

AppDynamics Specifics • Controller SSL Settings

• Controller Security Parameters • Enable SSL for Java

• Java agent SSL Configuration Properties • Enable SSL for .NET

• EUM Server SSL Settings

General Info

• Mozilla intro to PKI

• Mozilla intro to SSL

• Mozilla TLS and SSL

• JSSE 6, 7, 8

(41)

Keytool Cookbook

Create Keystore, Keys and Certificate Requests

Generate a keystore and key pair

keytool -genkey -alias mydomain -keyalg RSA -keystore keystore.jks -storepass password

Generate a certificate signing request (CSR) for an existing Java keystore

keytool -certreq -alias mydomain -keystore keystore.jks -storepass password -file mydomain.csr

Generate a keystore and self-signed certificate

keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.jks -storepass password -validity 3652

(42)

Keytool Cookbook

Import Certificates

Import a root or intermediate CA certificate to an existing keystore

keytool import trustcacerts alias root file Thawte.crt keystore keystore.jks -storepass password

Import a signed primary certificate to an existing keystore

keytool -import -trustcacerts -alias mydomain -file mydomain.crt -keystore keystore.jks -storepass password

Export Certificates

Export a certificate from a keystore

keytool -export -alias mydomain -file mydomain.crt -keystore keystore.jks -storepass password

(43)

Keytool Cookbook

List/View Certificates

Print a stand-alone certificate

keytool -printcert -v -file mydomain.crt

List which certificates are in a keystore

keytool -list -keystore keystore.jks -storepass password

Verbose list which certificates are in a keystore

keytool -list -v -keystore keystore.jks -storepass password

List a particular keystore entry using an alias

keytool -list -v -keystore keystore.jks -storepass password -alias mydomain

(44)

Keytool Cookbook

Delete Alias

Delete an alias from a keystore

keytool -delete -alias mydomain -keystore keystore.jks -storepass password

Rename Alias

Rename an existing alias

keytool -changealias -alias domain -destalias newdomain -keystore keystore.jks

Change Passwords

Change a keystore password

keytool -storepasswd -new new_storepass -keystore keystore.jks -storepass password

Change a private key password

keytool -keypasswd -alias client -keypass old_password -new new_password -keystore client.jks -storepass password

(45)