DOCUMENTUM CONTENT SERVER CERTIFICATE BASED SSL CONFIGURATION AND TROUBLESHOOTING

(1)

White Paper

Abstract

This White Paper explains configuration for enabling Certificate based SSL for secure communication to Connection Broker and Content Server. It also describes steps to troubleshoot issues faced during configuration. March 2013

DOCUMENTUM

®

_{CONTENT SERVER CERTIFICATE}

BASED SSL CONFIGURATION AND

TROUBLESHOOTING

(2)

The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and

specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

(3)

Executive summary

Content Server and Connection Broker support connections in both native and secure modes. For secure connections, Anonymous SSL is used by default. Support for Non Anonymous or Certificate based SSL for communication is new feature introduced in Content Server to further enhance communication security. For using Certificate based SSL, Content Server, Docbroker and Clients needs to be configured after installation. This paper explains in detail configuration and troubleshooting steps. All the components (Server, Docbroker and Clients) should use same mode for secure communication. Mixed environments are not supported, i.e. either all the components have to use Anonymous SSL or Non Anonymous SSL for communication. For Certificate based SSL, only supported cipher is AES128-SHA for maintaining backward compatibility with RSA libraries.

Audience

The audience for this white paper comprises personnel responsible for the Installation, configuration and deployment of Content Server in production environments. This document is intended for internal EMC teams, partners and customers.

Configuration

This section describes the steps required to configure different components of the system (Docbroker, Content Server, Clients) to use certificate based SSL for communication. Following needs to be configured:

1. Docbroker as SSL server (Content Server & DFC as SSL clients)

2. Content Server as SSL server (DFC as SSL client) and SSL Client (Docbroker as SSL Server)

3. DFC as SSL client

To enable clients to use Certificate based SSL communication for secure connections to Docbroker and Content Server, we need to install Docbroker and Content Server in secure or 'native & secure' mode. After installation is complete, all the services needs to be stopped (Docbroker, Content Server & Method Server) and then system needs to be configured for Certificate based SSL manually. Currently Content Server installer do not support automatic configuration of Certificate based SSL, so this configuration needs to be done manually after installation.

(5)

Docbroker

Create Docbroker Keystore

For configuring Docbroker as SSL server, a private key needs to be created for Docbroker and Certificate Signing Request (CSR) needs to be generated using private key. This CSR then needs to be presented to and signed by Certificate Authority (CA) for generating Docbroker's public certificate. Finally, we need to create a Keystore for Docbroker to store its private key and public certificate.

All the above steps are described here in detail. In this paper OpenSSL tool is used for managing Public Key Infrastructure (PKI). Below command can be used to generate Docbroker's private key and CSR. It will generate private key (brokerkey.pem) and CSR (brokercsr.pem) in PEM (text) format. CSR (brokercsr.pem) can then be presented to some CA for generating Docbroker's public certificate.

openssl req -newkey rsa:2048 -days 365 -subj

"/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=broker.dctm.emc.com" -keyout brokerkey.pem -out brokercsr.pem

Loading 'screen' into random state - done Generating a 2048 bit RSA private key

...+++ ...+++

writing new private key to 'brokerkey.pem' Enter PEM pass phrase:

Verifying - Enter PEM pass phrase: ---

For the purpose of this paper, we will show the configuration using self-signed certificates instead of CA signed certificates. Below command will generate Docbroker's private key (brokerkey.pem) and self-signed public certificate (brokercrt.pem), both in PEM format. Option -x509 is used to self-sign the public certificate.

openssl req -newkey rsa:2048 -x509 -days 365 -subj

"/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=broker.dctm.emc.com" -keyout brokerkey.pem -out brokercrt.pem

(6)

Docbroker will look for private key and public certificate in Keystore which should be in PKCS #12 format. There is no restriction on Keystore's filename or extension. It should be placed in $DOCUMENTUM/dba/secure as Docbroker will look in this directory for locating its keystore. To generate Docbroker's keystore (broker.p12) by storing key and self-signed public certificate, use below command:

openssl pkcs12 -export -out broker.p12 -inkey brokerkey.pem -in brokercrt.pem -name broker

Loading 'screen' into random state - done Enter pass phrase for brokerkey.pem: Enter Export Password:

Verifying - Enter Export Password:

When executed, it will ask for key’s and Keystore's password. Option -name is specified to provide an alias for keys in keystore.

Create Docbroker Keystore's password file

Keystores are secured by making them password protected. Docbroker Keystore's password is encrypted and stored in a file which will be used by Docbroker for accessing Keystore. Docbroker will look in directory $DOCUMENTUM/dba/secure for locating password file. There is no restriction on password file's name or extension. Below command generates password file with keystore's password:

dm_encrypt_password -encrypt <password> -file <file> e.g dm_encrypt_password -encrypt broker -file broker.pwd

dm_encrypt_password utility is installed with Content Server and will work on machines on which Content Server is installed.

Docbroker configuration

Docbroker will use properties in Docbroker’s configuration file (<Docbroker>.ini) for resolving Keystore and password file names. Property keystore_file is used to specify Keystore file name and keystore_pwd_file is used to specify file name in which Keystore's password is stored.

(7)

Put below mandatory properties in Docbroker’s configuration file (<Docbroker>.ini) in $DOCUMENTUM/dba directory, for the files that we generated in previous steps. keystore_file=broker.p12

keystore_pwd_file=broker.pwd

Start Docbroker

Docbroker should start successfully on secure port (1490).

Server

Create Server Keystore

Similar to Docbroker, private key and public certificates needs to be generated for Server and stored in Server's Keystore from where Server will access them.

Below command can be used to generate Server's private key and CSR. openssl req -newkey rsa:2048 -days 365 -subj

"/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=server.dctm.emc.com" -keyout serverkey.pem -out servercsr.pem

For generating private key (serverkey.pem) and self-signed public certificate (servercrt.pem), use above command with –x509 option as below:

openssl req -newkey rsa:2048 -x509 -days 365 -subj

"/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=server.dctm.emc.com" -keyout serverkey.pem -out servercrt.pem

Loading 'screen' into random state - done Generating a 2048 bit RSA private key ...+++

.+++

writing new private key to 'serverkey.pem' Enter PEM pass phrase:

Verifying - Enter PEM pass phrase: ---

(8)

These keys generated should now be stored in Keystore. Server Keystore should be in PKCS #12 format. There is no restriction on Keystore's filename or extension. It should be placed in $DOCUMENTUM/dba/secure where server will look to locate its Keystore. Generate Server's Keystore (server.p12) containing private key and self-signed public certificate using below command:

openssl pkcs12 -export -out server.p12 -inkey serverkey.pem -in servercrt.pem -name server

Loading 'screen' into random state - done Enter pass phrase for serverkey.pem: Enter Export Password:

Verifying - Enter Export Password:

Create Server Keystore's password file

Server Keystore's password is encrypted and stored in a file in directory $DOCUMENTUM/dba/secure. There is no restriction on password file's name or extension. Below command generates password file with Keystore's password:

dm_encrypt_password -encrypt server -file server.pwd

Create Server Trust-Store

When acting as SSL client to Docbroker, Server needs to verify public certificate sent by Docbroker. For verifying Docbroker's public certificate, Docbroker’s public certificate or CA certificate which was used to sign Docbroker's public certificate needs to be stored in a Keystore. This Keystore will act as Server's trust-store and server will verify Certificates sent by Docbroker against certificates in this keystore. Server expects trust-store to be in PKCS #7 binary format. There is no restriction on trust-store's name or extension. It should be placed in location $DOCUMENTUM/dba/secure. Below command generates trust-store by importing Docbroker's certificate.

openssl crl2pkcs7 -nocrl -certfile brokercrt.pem -outform der -out server-trust.p7b

(9)

Server configuration

Server resolves file names of Server Keystore, trust-store and Keystore’s password files by reading properties from Server configuration file (server.ini). Property keystore_file is used to specify the name of Server's keystore, keystore_pwd_file to specify password file name and truststore_file to specify trust-store name. All the below properties are mandatory.

Put below properties in Server Configuration file (server.ini): keystore_file=server.p12

keystore_pwd_file=server.pwd truststore_file=server-trust.p7b

Start Server

Server should start successfully and errors related to connection to Docbroker should not be displayed.

DFC

Create DFC Trust-Store

When DFC will be acting SSL client to Docbroker or Server, it will use a trust-store to store trusted certificates. Certificates sent by Docbroker or Server will be validated against the certificates in DFC trust-store. DFC trust store should be in JKS (Java Keystore) format. There is no restriction on store's name or extension as well as location.

We will convert the Docbroker & Server public certificates that were generated in text format to binary format using OpenSSL. Then public certificates of Docbroker and Server in binary format will be imported to DFC trust-store.

Below commands converts certificates from PEM to DER format:

openssl x509 -outform der -in brokercrt.pem -out brokercrt.der openssl x509 -outform der -in servercrt.pem -out servercrt.der

(10)

Following commands create DFC trust-store (dfc.keystore) by importing Docbroker's & Server's public certificates:

keytool -importcert -keystore dfc.keystore -file brokercrt.der -alias broker

keytool -importcert -keystore dfc.keystore -file servercrt.der -alias server

DFC configuration

DFC will look in dfc.properties file to resolve trust-store location, name and password. Property dfc.security.ssl.truststore is used to specify trust-store's path and dfc.security.ssl.truststore_password to specify trust-store's password. Trust-store's password can be given in plain text or encrypted format. In dfc.properties, put entries for DFC trust-store and trust-store's password:

dfc.security.ssl.truststore = c\:/secure/dfc.keystore dfc.security.ssl.truststore_password = password

DFC trust-store password can be encrypted using DFC utility as below: java com.documentum.fc.tools.RegistryPasswordUtils <password> E.g.

java com.documentum.fc.tools.RegistryPasswordUtils password MmKZP8KOlvrE6ndUolyl6Q==

The encrypted password generated by this command can be copied and pasted to properties file. For this command to execute successfully, dfc.jar should be in java classpath.

There is an additional property dfc.security.ssl.use_existing_truststore that is mutually exclusive to above two properties. When this property is specified, there is no need to put above two properties. In this case, Java Keystore will act as DFC trust-store or trust-trust-store can be specified using JVM parameter javax.net.ssl.trustStore. For secure connections to Server and Docbroker, put the value of property dfc.session.secure_connect_default as secure. Same property controls behavior of connections to both Docbroker and Server.

(11)

secure.dfc.session.secure_connect_default = secure

Troubleshooting

This section describes some of the common issues faced during configuration and steps to troubleshoot.

Docbroker startup fails

1. Check if Docbroker Keystore and Keystore's password files are present in $DOCUMENTUM/dba/secure

2. Check if entry for above two files is there in Docbroker configuration file (<Docbroker>.ini)

3. Check if the format of Docbroker Keystore is PKCS #12.

Below commands should list the keys in Keystore if it’s in PKCS #12 format, else error will be displayed.

Using OpenSSL:

openssl pkcs12 -info -in <keystore>

Using Keytool:

keytool -list –v -storetype pkcs12 -keystore <keystore>

4. Check if password in Keystore password file is correct. For testing, password can be put in plain-text (without encryption).

(12)

1. Check if Server Keystore, Server Keystore password and Server trust-store files are present in $DOCUMENTUM/dba/secure

2. Check if entry for above three files is there in Server configuration file (server.ini)

3. Check if the format of Server Keystore is PKCS #12. See commands in Docbroker startup fails section to verify.

4. Check if server trust-store is in PKCS #7 binary (der) format. For verifying, check if below command dumps it successfully: openssl pkcs7 -in <Keystore> -inform der

E.g. openssl pkcs7 -in server-trust.p7b -inform der

Server not able to connect to Docbroker

1. Check whether Docbroker Keystore has proper key and public certificate. Use below command to print the keys in keystore:

Using OpenSSL:

openssl pkcs12 -info -in <keystore>

Using Keytool:

keytool -list -storetype pkcs12 -keystore <keystore>

2. Check whether Docbroker is sending proper certificates.

Below command will start a simple client which tries to connect to SSL Server and displays certificate chain sent by Server:

openssl s_client -showcerts -debug -connect <SSL_Server_IP>:<SSL_Server_Port>

(13)

E.g. openssl s_client -showcerts -debug -connect 10.8.53.24:1490

3. Check if Server's trust-store contains Docbroker's public certificate or CA certificate chain used to sign Docbroker's public certificate

Use below command to display all certificates in trust store:

openssl pkcs7 -in <trust-store> -inform der -print_certs -text

Clients not able to connect to Docbroker

1. Check if proper entries are present in dfc.properties and trust-store file exists. 2. Check if DFC trust-store contains Docbroker's public certificate or CA certificate

used to sign Docbroker’s public certificate.

To dump Trust-store's contents (no need to specify storetype as default type is JKS):

keytool -list -keystore <keystore> -storepass <storepass>

Clients not able to connect to Server

1. Check if proper entries are made in dfc.properties and trust-store file exists. 2. Check if DFC trust-store contains Server's public certificate or CA certificate

used to sign Server’s public certificate.

To dump Trust-store's contents (no need to specify storetype as default type is JKS):

keytool -list -keystore <keystore> -storepass <storepass>

3. Check if dfc.properties has right value for trust-store password. For verification, password can be given in plain-text.

(14)

If Sever and Docbroker are configured in secure mode with Certificate based SSL and second Docbase is created, installer will hang for some time in middle during step where docbase tries to connect to existing Docbroker. This issue occurs as Docbase will try to connect using Anonymous SSL by default and since mixed mode is not supported, it won’t be able to connect. Follow below steps to resolve this issue:

1. While the installer is hanging, open server.ini of the new Server 2. Add the proper options for Certificate based SSL

3. Start/restart the new Server Miscellaneous

1. To verify that connection is secure and check encryption algorithm used, use jvm parameter -Djavax.net.debug=all, when starting the client.

Conclusion

Documentum support for Certificate based SSL communication to Connection Broker and Content Server improves security in Documentum products. This paper describes how to configure deployment to use this feature. For further information on installation, Content Server Enterprise Edition Installation Guide can be referenced.

References

1. Content Server Enterprise Edition Installation Guide

2. OpenSSL Documentation: http://www.openssl.org/docs/apps/openssl.html

3. Keytool Documentation (Java SE7):

http://docs.oracle.com/javase/7/docs/technotes/tools/windows/keytool.html

Appendix

(15)

Table 1 Content Server Error Messages

Error Message Possible Cause

[DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_init(). Error (705) SSL error(10100) in call to SSL_CTX_new(). Winsock error: 10100."

[DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_open(). Error (501) Network Library not initialized. Extended network error: 0."

• Server Keystore missing • Server Keystore different

format than PKCS #12 • Server Keystore corrupt • Server Keystore password

file contains wrong password

• Server Keystore password file missing

[ERROR] [AGENTEXEC 3088] Detected during program initialization: Command Failed:

connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DM_SESSION_E_RPC_ERROR]error: "Server communication failure"

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

• Server Keystore empty

[DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_init(). Error Unknown error code 10008 (_nl_error_ = 0). Extended network error: 0."

• Server trust-store missing • Server trust-store corrupt

[DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_init(). Error Unknown error code 1795 (_nl_error_ = 0). Extended network error: 0."

• Server trust-store wrong format

[DM_DOCBROKER_E_NETWORK_ERROR]error: "An error occured performing a network operation: (Unknown error code 112 (_nl_error_ = 0)). Network specific error: (Extended network error: 0)."

[DM_DOCBROKER_E_CONNECT_FAILED_EX]error: "Unable to connect to DocBroker. Clients please check your dfc.properties file for a correct host. Server please check your server.ini or target

• Server trust-store empty • Docbroker's keystore

(16)

attributes in server config. Network address: (INET_ADDR: family: 2, port: <port>, host: <host_name> (<host_ip>, 1835080a))."

connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker “<docbroker_name>:<port>“ failed

[DM_SESSION_E_RPC_ERROR]error: "Server communication failure"

javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error:

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

• DFC trust-store missing

javax.net.ssl.SSLHandshakeException:

sun.security.validator.ValidatorException: PKIX path building failed:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target sun.security.validator.ValidatorException: PKIX path building failed:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

• DFC trust-store missing Docbroker's Certificate

(17)

connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DM_SESSION_E_RPC_ERROR]error: "Server communication failure"

javax.net.ssl.SSLHandshakeException:

sun.security.validator.ValidatorException: PKIX path building failed:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target sun.security.validator.ValidatorException: PKIX path building failed:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

• DFC trust-store missing Server's Certificate

java.net.SocketException:

java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.DefaultSSLContextImpl)

java.io.IOException: Invalid keystore format

• DFC trust-store different format

(18)

java.security.cert.CertificateException: Unable to initialize, java.io.IOException: DerInputStream.getLength(): lengthTag=127, too big.

java.io.IOException: DerInputStream.getLength(): lengthTag=127, too big.

• DFC trust-store corrupt

java.io.IOException: Keystore was tampered with, or password was incorrect

java.security.UnrecoverableKeyException: Password verification failed

• DFC trust-store wrong password