The Log Management menu allows the user to deal with large volumes of device-generated log messages by defining relevant collection policies performed by an intelligent tool called the Log Collector.
Figure 12 The Log Collector’s Role
The Log Collector:
collects all desired event data from the system, application and device event logs, giving the most comprehensive security record possible. This makes possible the correlation of events gathered from all relevant internal and external systems, providing real-time end-to-end session tracking and monitoring.
can receive data directly as a Log Collectorless connection on systems where the
deployment of the Log Collector is not possible or desired. For example, routers which do not allow Log Collector installation can instead send Syslog event data directly to the SMP.
converts collected event data into a standardized IDMEF format according to pre-defined and custom rule-sets. This data standardization enables the SMP to directly compare fields within events collected from different systems and services.
transmits converted events by SSL/TLS to the SMP for further processing. The encryption process first compresses the data for optimal bandwidth utilization. In case of network failure or congestion, spooled events are sorted and transmitted in priority order.
can extract event information from multiple log formats and protocols, including: RDEP, WMI, Syslog, OPSEC, proprietary databases, flat-files and APIs.
Supported Collection Method per Log Collector Platform
Table 3 Supported collection types for each Log Collector platform - 32 bit
Table 4 Supported collection types for each Log Collector platform - 64 bit
User Guide 25
* Supported Log Sources are: 2003, 2003 R2, 2008, XP, Vista
** You must install the ‘compat-libstdc++’ compatibility module to make it work properly.
*** If 32-bit SE linux is enabled, log collection cannot start. You need to set SELINUX=permissive in /etc/selinux/config.
Log Process
This part of the documentation allows you to have a schematic view of log processing from its process by the Log Collector to its display within the Web Console or reporting interface through the correlation and aggregation engines.
Schema
Figure 13 Log Process
Description
1.A Log Collector collects and processes logs obtained via syslog, database connection, through the direct reading of a file, etc. The process:
a.either ignores the log,
b.or sends raw log and/or event data in an IDMEF format via a safe channel.
For a more detailed diagram, please refer to The Log Collector.
2.Once the object has been received by the SMP, the server extracts the various components (raw log and/or elementary event).
a.If the object is a raw log: it is redirected towards the database dedicated to raw logs, then its process stops. For legal reasons, no action is performed on it, from reception to archiving process. You can view this via a search screen (forensic dedicated to raw logs).
b.If the object is an event: it is forwarded on to the Event Preparer.
3.The Event Preparer has only one function: enriching the event according to the information contained in the database regarding the DNS name or IP address.
a.If the event contains two kinds of information, then they will both be inserted into the database (if this is not already the case).
b.If the event contains no information of this kind, then it is ignored.
c.If the event contains one of these two kinds of information, this tool evaluates whether it can add another kind of information to it according to its database.
4.The main function of the Totaling Engine is for statistic analysis. It allows the counting of events. In other words, it means that each event coming from the Asset Database is sent to both the Aggregation Engine and the Totaling Engine.
The log contains evaluation rules to count different elementary events according to criteria predefined by the user.
The aim of this step is to calculate for example the number of alerts coming from various sources received by the firewall per day. It is then possible to draw up a precise report without generating calculations regarding aggregation, correlation, etc. It allows the processing of minor events with a high interest regarding the counting.
5.The Aggregation Engine’s main aim is to produce fewer events and then take less amount of space on the disk. It behaves nearly in the same way as the Correlation Engine. Rules evaluate events in order to generate actions. Usually, when there is an action, the engine is separating one event from another. Here are the possibilities:
a.To be able to ignore an event (skip).
b.To fuse this event (or not) with other ones according to specific criteria (definition of common fields, grouped fields, deleted fields). This is the only way to proceed with continuous treatment of an aggregated event.
By default, it is also possible to store a elementary event in a dedicated database. This is especially useful for event storage and searching. For a more detailed diagram, please refer to section The Aggregation Process.
User Guide 27
6.Once events have been processed by the Aggregation Engine, they are not only sent to the Correlation Engine but also stored in a dedicated database (aggregated events).
The data contained in this database are available from within the Web Console. The aggregated events details are based upon the elementary events database.
7.The aim of this step is to correlate various aggregated events and/or alerts. This will give you a more general view of the different actions to perform (creation of an alert most often).
Alerts are created and stored in a dedicated database allowing report edition, visualization (main view of the console) and forensic search.
For a more detailed diagram, please refer to section The Correlation Process.
Adding a Log Source Automatically: Wizard
The wizard is a tool intended at adding and configuring a log source in an easy and friendly way.
The log source is the element which generates security events or logs that will be collected by the Log Collector e.g. a firewall, a proxy, an IDS, a web application, an Operating System, a database...
Here is a description of the log collection by the Log Collector.
The Log Collector
The wizard is composed of three main screens where you must:
configure the log source, configure the Log Collector, define the connection type.
The Log Collector selects a log among the list of logs pending to be processed.
The Log Collector:
extracts the necessary data to create an elementary event in IDMEF format.
transforms it to create a raw log.
Both raw logs and elementary events enter a filter where a Collection Policy will be applied. Several options are possible:
nothing is sent
only elementary events are sent: a small number or only the main ones or all those with a Taxonomy or all of them
both are sent: all the elementary events with a Taxonomy + small number of raw logs or all the elementary events with a Taxonomy + raw logs or all the elementary events + all types of raw logs
all types of elementary events + all types of raw logs are sent
The element (either the elementary event or raw log or both) is sent to the SMP via a safe channel (SSL).
User Guide 29
Opening the Wizard
You can access the wizard by clicking on Log Management > Add a log source menu entry.
The Welcome page is displayed. It lists important information about what you must gather before starting the wizard such as the:
Log source type (e.g. CheckPoint, Squid...), Log source host IP or DNS address,
Name of the SMP Log Collector that will collect the log,
Connection parameters to the log source (e.g. installation folder, login, password, domain...).
You must click Configure a Product to start the wizard.
Note: If you work with an LMI (LX/ST/MX) server, then click on Configure a Forwarder and follow the procedure described in the UCM User Guide.