S
ECURONIX
UEBA 5.0
Securonix Proprietary Statement
This material constitutes proprietary and trade secret information of Securonix, and shall not be disclosed to any third party, nor used by the recipient except under the terms and conditions prescribed by Securonix.
The trademarks, service marks, and logos of Securonix and others used herein are the property of Securonix or their respective owners.
Securonix Copyright Statement
This material is also protected by Federal Copyright Law and is not to be copied or reproduced in any form, using any medium, without the prior written authorization of Securonix.
However, Securonix allows the printing of the Adobe Acrobat PDF files for the purposes of client training and reference. Information in this document is subject to change without notice. The software described in this document is furnished under a license agreement or nondisclosure agreement. The software may be used or copied only in accordance with the terms of those agreements. Nothing herein should be construed as constituting an additional warranty. Securonix shall not be liable for technical or editorial errors or omissions contained herein. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or any means electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's internal use without the written permission of Securonix.
Copyright 2017 © Securonix, Inc. All rights reserved.
Contact Information
Securonix, Inc.
14665 Midway Rd. Ste. 100, Addison, TX 75001 www.securonix.com
855.732.6649
Revision History
Date Product Version Description
CONTENTS
Troubleshooting
4
Monitoring the Application
4
Troubleshooting the Application
5
Database Overview
5
MySQL Query Management
9
Alerts and Common Issues
11
Securonix Application Issues
11
Activity Import Issues
24
UI Issues
30
Troubleshooting
This guide describes how to troubleshoot the Securonix Enterprise 5.0 application. For more troubleshooting
tips and tricks, see the
Securonix Support Forums
.
For issues not found in this guide or in the Forums, contact Securonix Support [email protected].
Monitoring the Application
The following components are critical to monitoring:
System Resources
l
Java 8
l
java -version
l
Tomcat process monitoring-Tomcat 8
lservice tomcat status
l
ps –ef |grep tomcat
l
MySQL Memory/CPU-Mysql 5.6.34
lsudo service mysql status
lDisk Usage
ldf -sh
ldf -u
lUI URL monitoring
lService/Port monitoring
lLog Monitoring
l/Securonix/Tomcat/logs/Securonix.log
l/Securonix/Tomcat/logs/catalina.out
lJob Monitoring
l/Securonix/Securonix_home/import/success
l/Securonix/Securonix_home/import/failed
Some common scenarios in which alerts may trigger:
Issue Solution
Long-running jobs result in high tomcat heap usage on the master node.
Ensure that the behavior job has been configured to run incrementally, which will accrue behavior baselines on a daily basis.
High MySQL Memory/CPU usage on the master could be caused by long-running queries or multiple copies of the same queries fired over and over.
1. Stop Tomcat.
2. Kill the queries.
3. Restart Tomcat.
Issue Solution High disk/Java usage may indicate a problem in the ingestion
of logs by Securonix.
High disk usage import/success, import/failed indicates that housekeeping jobs may not be regularly clearing the processed logs.
OutOfMemory exceptions in catalina.out indicate that the heap limits set on the tomcat JVM have been reached.
This can be resolved by analyzing the usage trends in Sitescope.
A restart may be necessary to return Tomcat to normal operation after encountering this exception.
Troubleshooting the Application
To troubleshoot any errors, first look at the various steps in which data flows:
1. Scripts
2. Analyzers/Preprocessors (If Any)
3. Securonix Ingestion Job
Logging/file output at multiple levels assists in troubleshooting:
l
/Securonix/Securonix_home/conf/applicationcontext.xml
l/Securonix/Tomcat/conf/Catalina/localhost/Profiler.xml
lSecuronix/Tomcat/logs/catalina.out
lSecuronix/Tomcat/logs/securonix.log
lSecuronix/securonix_home/import/success
lSecuronix/securonix_home/import/failed
lMySQL – information_schema.processlist
For example, if we find a certain policy not producing the expected results:
1. Examine line filters and input configurations to check if the events are not being filtered out.
a. Check files in /Securonix/Securonix_home/import/failed to see if any relevant lines have failed the
import
2. If a Analyzer is associated with that datasource, check /Securonix/Tomcat/logs/securonix.log and
/Securonix/Tomcat/logs/catalina.out for information on the preprocessing.
Database Overview
Note:
Important tables, foreign keys, and their description are provided in a separate document.
Troubleshooting
Migration Components
The following components are involved during a migration process in Securonix. Migration typically involves
moving over new data source, user, and other Securonix entity-based import configurations from one
environment to another ( ex: UAT to Prod).
l
ConfigResourceParsers
l
Table primarily used for migration of Activity Import configurations, policies, and behavior
configurations between environments.
l
Contains import, policy and behavior settings as XML.
l
Connection Types
aka
Data sources
(From
Configure > Connection Types
)
l
These contain information on the configuration for retrieving the data - e.g. File location, Splunk
connection credentials
l
Other configurations
l
User import settings
: Attribute mapping, encryption setting and other rules set up in user imports
lConfigxml
: Additional XML configuration present in configxml table
l
Lookup Tables
(Using MySQL dump),
Workflows
l
Lookup tables and workflows can be migrated from one environment to another using MySQL
dump/restore.
l
The dump must be restored on both master & child nodes.
lPatches & Connectors
l
These are performed by an exact copy of the relevant directories and files. Instructions will be
provided with each new package that must be applied.
Migration of CRPs
The following steps must be followed to update the exported configurations in the configresourceparsers
table to the most recent version. As of now, selectively exporting certain policies/behavior is not possible
through the UI. Select all behavior/policies while performing the export, and run the SQL provided below to
remove unwanted items.
1. Save the following configurations via UI (Select all options):
a. Line Filters from
Add Data > Activity > [Datasource] > Line Filters > Save Template
.
b. Policies from
Analytics > Policy Violations >[Sort by Datasource] > Save Template
.
c. Policies from
Analytics > Activity Outliers > Save Template
.
d. Behavior Configuration from
Analytics > Behavior Profiles > Save Template
.
2. Create a new datasource from
Add Data > Activity > New Datasource
.
a. Select resource type that has been migrated.
3. SQL to remove unwanted items:
a. Delete Behavior Profile.
b. Update configresourceparsers set bprofilexml=updatexml(bprofilexml,"//bprofileconfig
[id/bpname='PROFILE_NAME']", "") where resource_type='RESOURCE_TYPE';
c. Delete Policy
d. Update configresourceparsers set policyxml=updatexml(policyxml,"//policyMaster
[name='POLICY_NAME']","") where resource_type='RESOURCE_TYPE';
e. Show Behavior Profiles
f. Select extractvalue(bprofilexml, "BprofileconfigList/bprofileconfig/id/bpname") from
configresourceparsers where resourcetype='RESOURCE_TYPE';
g. Show Policies
h. Select extractvalue(policyxml, "PolicyMasterList/policyMaster/name") from configresourceparsers
where resourcetype='RESOURCE_TYPE';
MySQL Query Management
Running MySQL queries are displayed using the following:
show [full] processlist;
They can also be viewed via the table information_schema.processlist:
mysql> desc information_schema.processlist;
E x a m p l e :
+---+---+---+---+---+---+
| Field | Type | Null | Key | Default | Extra |
+---+---+---+---+---+---+
| ID | bigint(21) unsigned | NO | | 0 | |
| USER | varchar(16) | NO | | | |
| HOST | varchar(64) | NO | | | |
| DB | varchar(64) | YES | | NULL | |
| COMMAND | varchar(16) | NO | | | |
| TIME | int(7) | NO | | 0 | |
| STATE | varchar(64) | YES | | NULL | |
| INFO | longtext | YES | | NULL | |
+---+---+---+---+---+---+
8 rows in set (0.00 sec)
Using a select on this table, restrict the output to a specific user, DB, or Query.
In case of long-running queries holding up the system
, run the following command to get the process
ids and kill them:
mysql -N -u operations -p -e "select id from information_schema.processlist where user='operations' and info like 'select%' and and command <>'Sleep'" | cut -f1 | sed -e 's/^/kill /' | sed -e 's/$/;/' > /tmp/kill.sql
mysql –u operations –p < /tmp/kill.sql
Note:
In the example above, the query used to fetch the process IDs is limited to select queries
running through the webuser user account, as well as those which are active (not sleeping).
Troubleshooting
Note:
Ensure that the binlog dump query (run by the user replica, with Info field “binlog dump”) is
not killed – this will kill the replication.
Tomcat has its own layer of query management. Killing running queries from MySQL may cause
errors in the application , requiring a tomcat restart.
Killing insert/update/delete queries is not recommended. These trigger time-consuming rollbacks,
as well as resulting in loss of data.
Alerts and Common Issues
The following describes common alerts and issues within the Securonix Enterprise application and how to
resolve these issues.
Securonix Application Issues
ERROR connection. DatasourceConnectionProvider: Could not find datasource:
java: /comp/env/jdbc/securonixDS
Severity 1
Details
Securonix application after starting tomcat does not come up. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
Typically, this issue occurs when the Profiler.xml file is accidentally deleted from the /Tomcat/conf/Catalina/localhost folder. This file acts as the connection between the application and the database.
Solution
1.
Validate that all the important Securonix configuration files exist in the paths below:a. Profiler.xml (/Securonix/Tomcat/conf/Catalina/localhost/)
b. application-context.xml (/Securonix/Securonix_home/conf)
2. Replace the file in the folder from the backup files. Example:
$ scp secitprtpprd04:/ [backuplocation]/Profiler.xml]/Securonix/Tomcat/conf/Catalina/localhost/
3. Verify that Tomcat services are not already running:
$ ps -ef|grep tomcat4. When the copy has completed, restart the application:
$ service tomcat restartValidation
Details
1.
After restarting, navigate to (/Securonix/Tomcat/conf/Catalina/localhost/) and verify Profiler.xml exists.$ ls /Securonix/Tomcat/conf/Catalina/localhost/
2.
Tail the logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Ensure no errors exist in logs.
3.
Browse the application on the UI: https://hostname:8443/ProfilerThis application should now load without any issues.
404 Error after loading the Securonix URL in the browser
Severity 2
Details
When loading the Securonix URL, Http 404 error is shown. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
SolutionDetails
Clear browser cache and reload the page. If this does not resolve the issue proceed with steps below: Validate using the following steps:
1. Check if the Securonix tomcat service is up and running using the following command:
$ ps -ef | grep tomcat(Linux Based Systems) ( This should show a pid for tomcat )
2. Check for MySQL service:
$ ps -ef | grep mysql( This should show a pid)
3. Validate that the Profiler.xml file located under /Securonix/Tomcat/conf/Catalina/localhost exists.
This file is essential as it provides the connection between the application and the database.
4.
Validate the parameters set within the JDBC URL. Specifically, check the name of the database in the jdbc URL.Ensure it's pointing to the correct DB.$
vi/Securonix/Tomcat/conf/Catalina/localhost/Profiler.xml
Jdbc URL: jdbc:mysql://[Node_IP]:3306/[database_
name]?autoReconnect=true&rewriteBatchedStatements=
true&useUnicode=true&characterEncoding=UTF-8&maxReconnects=10
5.
If both these services are up and running and the parameters in the Profiler.xml are correct and page is still not coming up, Check Securonix and catalina logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Please refer this document to fix any other errors that might appear. Validation
1. Browse the application on the UI
https://hostname:8443/Profiler
This application should now load without any issues
2.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Application throwing Java Heap Space Error /OutOfMemoryException
Severity 2
Details
Errors in the Securonix log include Java heap Space and out of memory error messages. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
Details Solution
1.
Check the Java memory parameters. These are defined in CATALINA_OPTS (specified in startup.sh (/Securonix/Tomcat/bin/startup.sh) file of tomcat).$ vi /Securonix/Tomcat/bin/startup.sh
2.
Check parameters -Xms8g -Xmx50g. If these values are small, you can increase these values depending on the available RAM size. Ex: 256 GB Ram can have heap size min and max of 70 to 90G. However, increasing these values is contextual and may not always be necessary.A large file size at the time of activity import getting loaded into memory may cause Java heap space For imports in such cases, check that the split file functionality is enabled (tag is found in application-context.xml file found in /Securonix/securonix_home/conf folder).
$ vi /Securonix/securonix_home/conf/application-context.xml Find additional information about OutOfMemoryExceptions in the following resource: Understand the OutOfMemoryError Exception
3. If you need to change any parameters above, stop Tomcat on all nodes beginning with the child
nodes then the master:
$ service tomcat stop4. Perform the necessary changes as described above.
5. Start tomcat on all nodes beginning with Master node then child nodes:
$ service tomcat startValidation
1. Browse the application on the UI
https://hostname:8443/Profiler
This application should now load without any issues
2.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Tomcat/Securonix does not start properly
Severity 1
Details
Securonix application after starting tomcat does not come up. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
/Securonix/Tomcat/logs/catalina.out
Details Solution
1.
Typeps -ef |grep tomcatto determine how many tomcat instances are up.2. If multiple instances are up perform the following: kill -9 {PID} of the tomcat instance(s) from step 1.
3. Restart Tomcat:
$ service tomcat restartOR
1.
Check if the Node database is locked due to:a.
Select and insert statements being run at the same time on the same table Login to mysql:$ mysql –u root –p database_name Show processlist;
b.
Offline/live database dump occurringIn either case, you will have to wait for the process to complete before Tomcat can be restarted. Validation
1. Browse the application on the UI
https://hostname:8443/Profiler
This application should now load without any issues
2.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Communications link failure with auto reconnect JDBC URLs
Severity 1
Details
Communications link failure with auto reconnect JDBC URLs prevents job import. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
Details Solution
Details
To prevent connection timeouts, ensure all database connection URLs, use the following four parameters: l
autoReconnect=true
l
rewriteBatchedStatements=true
luseUnicode=true
l
characterEncoding=UTF-8
This must be changed in several places:a.
Profiler.xml $ vi /Securonix/Tomcat/conf/Catalina/localhost/Profiler.xml url="jdbc:mysql://<host>:<port>/<database>?autoReconnect=true&rewriteBatchedStatement=true& useUnicode=true&characterEncoding=UTF-8" testWhileIdle="true" testOnBorrow="true" testOnReturn=“true" validationQuery="SELECT 1" validationInterval="30000"b.
Hibernate Templates (on Master and Child)a.
Master Nodes: $ vi /Securonix/securonix_ home/conf/hibernate/hibernate.cfg.<childname>.xml <property name="hibernate.connection.url">jdbc:mysql://<childhost>:<port>/<database>?autoReconnect=true&am p;rewriteBatchedStatements=true&useUnicode=true&characterEncoding=UTF-8</property>b.
Child Nodes: $ vi /Securonix/securonix_ home/conf/hibernate/hibernate.cfg.master.xml <property name="hibernate.connection.url">jdbc:mysql://<masterhost>:<port>/<database>?autoReconnect=true&a mp;rewriteBatchedStatements=true&useUnicode=true&characterEncoding=UTF-8</property>c.
Database Connections in Securonix UI:In each of the Database Connections between the Master and the Child, and the archive connections, ensure that the four parameters are in the DB URL:
1.
ClickConfigure->Connection Types. JDBC URL:jdbc:mysql://<masterhost>:<port>/<database>?autoReconnect=true&rewriteBatchedStatements=tru e&useUnicode=true&characterEncoding=UTF-8
Details
2. If you need to change any parameters above first stop tomcat on all nodes beginning with
child nodes then master:
$ service tomcat stop3. Perform the necessary changes as described above.
4. Start Tomcat on all nodes beginning with Master node then child nodes:
$ service tomcat startValidation
1. Browse the application on the UI
https://hostname:8443/Profiler
This application should now load without any issues
2.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Error after upgrading nodes - "Out of resources/Too many files open" error
Severity 2
Details
A limitation may be set on the number of concurrent open files that your MySQL maybe allowed to keep open. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
Error:
7:15:03,453 ERROR [JDBCExceptionReporter] Out of resources when opening file './securonix_node1/activityfreqnwtime15#P#week3.MYD' (Errcode: 24 -Too many open files) 17:15:03,456 ERROR [GrailsRuntimeConfigurator] [RuntimeConfiguration] Unable to load beans from resources.groovy org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.ExceptionInInitializerError at grails.spring.BeanBuilder.invokeBeanDefiningClosure (BeanBuilder.java:723) at grails.spring.BeanBuilder.beans(BeanBuilder.java:573) at grails.spring.BeanBuilder.invokeBeanDefiningClosure (BeanBuilder.java:723) at grails.spring.BeanBuilder.beans(BeanBuilder.java:573) at grails.spring.BeanBuilder.invokeMethod(BeanBuilder.java:519) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
at com.securonix.application.hibernate.util.DbUtil.executeHQLQuery (DbUtil.java:1187)
Solution
1.
Step 1
: Stop Tomcat on all nodes beginning with child nodes then master:
$ service tomcat stop2.
Step 2: Edit the limits.conf file shown below:$ sudo nano /etc/security/limits.confsecuronix soft nofile 24000 securonix hard no file 32000
3.
Step 3
: Stop mysql on all nodes:
$ service mysql stop4.
Step 4: Under the [mysqld] section of my.cnf, add the following parameter:$ vi /usr/my.cnfopen_files_limit = 100000
5.
Step 5
: Start MySQL on all nodes:
$ service mysql start6.
Step 6
: Start Tomcat on all nodes beginning with Master node then child nodes:
$ service tomcat startDetails Validation
1.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Error in QRTZ Scheduler
Severity 2
Details
Error shows up on application startup. securonix.log shows the following errors:
ERROR [QuartzSchedulerListener] Error in Quartz scheduler: Couldn't acquire next trigger: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=5' at line 1
org.quartz.JobPersistenceException: Couldn't acquire next trigger: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_ SELECT_LIMIT=5' at line 1 [See nested exception:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_
LIMIT=5' at line 1] at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger (JobStoreSupport.java:2794) at org.quartz.impl.jdbcjobstore.JobStoreSupport$36.execute (JobStoreSupport.java:2737) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock (JobStoreSupport.java:3768) at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger (JobStoreSupport.java:2733) at org.quartz.core.QuartzSchedulerThread.run (QuartzSchedulerThread.java:264)
Find more details regarding the errors in the following logs: l
/Securonix/Tomcat/logs/securonix.log
l
/Securonix/Tomcat/logs/catalina.out
SolutionDetails
1.
Update mysql-jdbc driver(.jar file).This file is located in Tomcat/lib folder and Profiler/WEB-INF/lib folder. Take a backup of the jar file:
$ cp /Securonix/Tomcat/lib/(mysql-jdbc driver).jar /Securonix/archive/ (mysql-jdbc driver).jar.bkp
$ cp /Securonix/Tomcat/webapps/Profiler/WEB-INF/lib/(mysql-jdbc driver).jar /Securonix/archive/(mysql-jdbc driver).jar.bkp
Copy the new jar files to the locations below:
$ cp /path/to/jarfile /Securonix/Tomcat/lib/(mysql-jdbc driver).jar $ cp /path/to/jarfile /Securonix/Tomcat/webapps/Profiler/WEB-INF/lib/ (mysql-jdbc driver).jar
2. Stop Tomcat on all child nodes beginning with child nodes then master:
$ service tomcat stop3. Restart Tomcat on master:
$ service tomcat restart4.
Start Tomcat on child nodes:$ service tomcat startThe version of the jar file in both the folders should be the same. Validation
1.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Job Misfired
Severity 2
Details
This error shows up in logs when a job has missed its schedule and runs at a later time. The job usually continues to completion.
Find more details regarding the errors in the following logs: l
/Securonix/Tomcat/logs/securonix.log
l
/Securonix/Tomcat/logs/catalina.out
SolutionDetails
1. Allow the job to run and monitor the logs for any subsequent errors
2. If there are any errors refer this document to take the necessary course of action or contact with
Securonix Support [email protected] for a solution.
3. If the job takes longer time to complete, manual intervention is required to delete the job and
reschedule the job.
Validation
1.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Null Pointer Exception
Severity 1
Details
This error shows up in logs for different reasons.
Find more details regarding the errors in the following logs: l
/Securonix/Tomcat/logs/securonix.log
l
/Securonix/Tomcat/logs/catalina.out
Solution1.
Investigation
: Tail the logs and identify the root cause of the issue.
2. Take corrective action if it’s due to data anomaly or configuration issue. For any other issues contact
Securonix Support [email protected].
Validation
1.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
SecuronixDB not configured
Severity 2
Details
This error shows up in logs only during the initial Securonix application configuration. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
Solution1.
On the Securonix UI, clickConfigure -> Connection Types.If it exists, verify that Profiler.xml exists at /Securonix/Tomcat/conf/Catalina/localhost.
If the Profiler.xml file is missing replace it from the backup location on StandBy Master and restart Tomcat:$ service tomcat restart
If the SecuronixDB connection type does not exist, create it based on the step by step directions provided on the UI.
2.
Under Connection Types, check if the SecuronixDB connection type exists.3. Test connection to ensure it is successful.
Validation
1.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
2. Check that the SecuronixDB connection type is visible on UI under
Configure->Connection Types
.
3.
Test connection to ensure it is successful.Invalid HQL
Severity 2
Details
This error shows up in logs when an HQL-based policy runs. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
SolutionDetails
1. Identify the policy which is causing the invalid HQL errors in logs.
2. Check the HQL for the policy and correct it based on HQL syntax.
3. Convert the HQL to SQL and test the validity of the query.
4. Schedule or run the policy.
Validation
1.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Activity Import Issues
Errors while connecting to Splunk: Unable to connect to Splunk
Severity 2
Details
Securonix to splunk connection at the time of configuring an activity import throws error as below Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
Error:
javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.Handshaker.activate(Handshaker.java:470) at sun.security.ssl.SSLSocketImpl.kickstartHandshake (SSLSocketImpl.java:1438) at sun.security.ssl.SSLSocketImpl.performInitialHandshake (SSLSocketImpl.java:1308) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343) at sun.net.www.protocol.https.HttpsClient.afterConnect (HttpsClient.java:563) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect (AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream (HttpURLConnection.java:1092) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream (HttpsURLConnectionImpl.java:250)
Details
1. Go to Java installation folder
2. Open {JRE_HOME}/lib/security/java.security -file in text editor:
3.
Backup java.security file with data prior to editing the file.$ vi {JRE_HOME}/lib/security/java.security
4. Go to the last line
5. Comment out the following line "jdk.tls.disabledAlgorithms=SSLv3"
6. Restart Tomcat:
$ service tomcat restartValidation
1.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
2.
Step 2: On Securonix UI App, clickAdd Data -> Activity. Click the name of the data source that has a Splunk source.Scroll to the bottom of the pop-up screen and clickRefresh Splunk Searches.
3. Ensure Splunk searches display without errors in logs.
Activity Import Failed (Permission denied errors)
Severity 1
Details
Activity import fails as input log files have different ownership as compared to theTomcat owner/application owner. This results in the application being unable to pick up log files from the in folder. The following errors are often seen when this issue occurs:
securonix.log.2015-09-09: java.io.FileNotFoundException: /securonix_ home/import/success/tacacs-output1.txt (Permission denied)
securonix.log.2015-09-09: java.io.FileNotFoundException: /securonix_ home/import/success/tacacs-output-20150909003001.log (Permission denied) securonix.log.2015-09-09: java.io.FileNotFoundException: //securonix_ home/import/success/tacacs-output-20150909023001.log (Permission denied) securonix.log.2015-09-09: java.io.FileNotFoundException: /securonix_ home/import/success/tacacs-output-20150909043001.log (Permission denied)
Find more details regarding the errors in the following logs: l
/Securonix/Tomcat/logs/securonix.log
l
/Securonix/Tomcat/logs/catalina.out
SolutionThe tomcat/application ownership was changed from the default ‘root’ ownership to a different owner. If the input files have root ownership, they are not picked by the application during import.
1.
Change the ownership of the input files to the owner of the Tomcat to resolve the issue:$ chown -R securonix.securonix [Securonix filepath]
Note: Consult Securonix Support [email protected] before changing permissions for the files related to the Securonix application on the platform.
Validation
1.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
2.
Access the path or location while logged in to console as the Tomcat user. You should not experience any issues or errorsDate/Time Mapping Missing error during activity import
Severity 1
Details
Error logs show DateTime missing errors during the activity import. Find more details regarding the errors in the following logs:
l
/Securonix/Tomcat/logs/securonix.log
l/Securonix/Tomcat/logs/catalina.out
SolutionDetails
For the Activity Import to be successful, the format of the date time value coming in to the activity feed must be correct. If the format is specified incorrectly in Step 2 of Activity Import (Line Filter), the data won't be imported into Securonix. Ensure the format is configured correctly as in the following image:
For example, if the input data feed displays the Date as 4/21/2015 and Time as 12:45:00, the format should be Date: MM/dd/yyyy and Time: HH:mm:ss.
Refer to the following link for more options:Class SimpleDateFormat. Validation
1.
Test the job again to ensure no errors exist in logs: Alerts and Common IssuesError in the Activity Import: Required Data is missing during activity import
Severity 1
Details
Error logs show required data missing during the activity import:
10:18:39,243 ERROR [QuartzJob] [JobId:238] Error in job execution -ActivityImport_pp test_2015_7_30_10_18_37
com.securonix.application.exception.matcher.MatcherReaderException: com.securonix.application.exception.matcher.MatcherReaderException: Null value encountered, required data missing
Find more details regarding the errors in the following logs: l
/Securonix/Tomcat/logs/securonix.log
l
/Securonix/Tomcat/logs/catalina.out
SolutionFor the Activity Import to be successful, the following fields cannot be null. This error can occur in the following scenarios:
1. Activity log data field that is mapped to the transaction string is null.
2. Activity log data field that is mapped to the account name (AccountID) is null.
These two fields are required and cannot be NULL. Validation
1.
Test the job again to ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
Job Misfired: Required data is missing during activity import
Severity 1
Details
Error logs show required data missing during the activity import:
10:18:39,243 ERROR [QuartzJob] [JobId:238] Error in job execution -ActivityImport_pp test_2015_7_30_10_18_37
com.securonix.application.exception.matcher.MatcherReaderException: com.securonix.application.exception.matcher.MatcherReaderException: Null value encountered, required data missing
Find more details regarding the errors in the following logs: l
/Securonix/Tomcat/logs/securonix.log
l
/Securonix/Tomcat/logs/catalina.out
SolutionFor the Activity Import to be successful, the following fields cannot be null. This error can occur in the following scenarios:
1. Activity log data field that is mapped to the transaction string is null.
2. Activity log data field that is mapped to the account name (AccountID) is null.
These two fields are required and cannot be NULL. Validation
1.
Test the job again to ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.out
UI Issues
New Policy Category Widget Not Showing Up
Severity 1
Details
Policy Category widget does not show up on the security dashboard on the left-hand pane even though there are relevant policy violations against it
Solution
If special characters are included in the policy name, the widget won't appear on the left navigation pane, even though it shows up on the Threats screen.
1. When creating a new policy category, ensure the name does not include any special characters (eg '-')
.
Details Validation
1.
When you have modified the policy to exclude special characters, ensure it appears on the security dashboard.Emails Not Being Sent Out from the Server
Severity 1
Details
Emails are not sent out from the Securonix server Solution
1.
Check and configure for the following parameters underConfigure->Settings->SMTP Server Settings:a. SMTP Port
b. Email credentials
c. Hostname
2. Click
Save
and
Send Test Email
to confirm the parameters are set correctly.
Validation
When you receive the test email, emails are being sent out correctly.
Preview not shown when connected to DB
Severity 1
Details
When getting data (Activity/Access/User) from a database, the preview does not show any data and instead displays the error 'error getting preview from the database'.
Solution
Newline characters may be included in the query being used in Securonix.
1. Rewrite the query replacing all newline characters with space.
Validation
Click preview after modifying as above and the resulting preview should load on screen.
Securonix Application unable to connect to an SQL server instance
Severity 1
Details
When trying to connect to a SQL Server, an error ' Login Failed. The login is from an untrusted domain and cannot be used with Windows authentication'.
Cause: The account used to connect to the SQL server is going through Windows (Active Directory) authentication trying to connect to the SQL Server.
Solution
1. Use the following format in your connection URL:
jdbc:jtds:sqlserver://IP:1433/database_
name;useNTLMv2=true;domain=windows_domain_name
Validation
1. Enter the connection details as above.
2. Click Test Connection to ensure Connection Successful message appears.
3.
Ensure no errors exist in logs:$ tail -123f /Securonix/Tomcat/logs/securonix.log $ tail -123f /Securonix/Tomcat/logs/catalina.ou
Other Errors in Log
User XYZ not found in directory
Severity 1
Details
Error logs show ‘User XYZ not found in directory’
12:36:44,728 ERROR [CustomLDAPAuthAuthenticator] Error - User admin not found in directory.
12:36:44,802 DEBUG [LoginController] in LoginController => authfail => customMessage => User authentication failed.
12:36:44,811 DEBUG [AuditUIUtil] in auditLog() => username => admin 12:36:44,811 DEBUG [AuditUIUtil] in auditLog() => action => authfail , controller => LOGINCONTROLLER , userAction => LOGIN , auditMessage => User authentication failed. , status => ERROR
12:36:44,901 DEBUG [LoginController] authfail BadCredentialsException -Authentication Failed. Check the username/password you entered and try again.
12:36:44,907 DEBUG [AuditUIUtil] in OPERATION === title = , desc = Failed Login
12:36:44,907 DEBUG [AuditUIUtil] in executeAuditLog()...=> currentUserName => admin Auth name => admin , HostAddress => 10.144.180.112
controllerConstant => LOGIN_CONTROLLER, controller => LOGINCONTROLLER, action => LOGIN remote add => 10.144.180.112
12:36:44,907 DEBUG [AuditUIUtil] checkAudit() parameters ==> title => , action => authfail , controller => LOGINCONTROLLER , status => ERROR 12:36:44,919 DEBUG [AuditUIUtil] customMessage ==> User authentication failed..
Cause: The account trying to login does not exist.
Find more details regarding the errors in the following logs: l
/Securonix/Tomcat/logs/securonix.log
l
/Securonix/Tomcat/logs/catalina.out
SolutionIf this user must be granted access, contact the Securonix Admin to allow access.
For all users whose login is unexpected, check /Securonix/Tomcat/logs/securonix.log for more details related to this access as shown above.
Validation
No validation steps required.
