Using pattern databases

To classify messages using a pattern database, include adb_parser()statement in your syslog-ng configuration file using the following syntax:

Declaration:

parser <identifier> {db_parser(file("<database_filename>"));};

Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.

Example 13.1. Defining pattern databases

The following statement uses the database located at/opt/syslog-ng/var/db/patterndb.xml. parser pattern_db {

db_parser(

file("/opt/syslog-ng/var/db/patterndb.xml") );

};

To apply the patterns on the incoming messages, include the parser in a log statement:

log {

source(s_all);

parser(pattern_db);

destination( di_messages_class);

};

Note

The default location of the pattern database file is/opt/syslog-ng/var/run/patterndb.xml. Thefileoption of thedb-parserstatement can be used to specify a different file, thus different db-parser statements can use different pattern databases. Later versions of syslog-ng will be able to dynamically generate a main database from separate pattern database files.

Using pattern databases

Example 13.2. Using classification results

The following destination separates the log messages into different files based on the class assigned to the pattern that matches the message (for example Violation and Security type messages are stored in a separate file), and also adds the ID of the matching rule to the message:

destination di_messages_class {

For details on how to create your own pattern databases see Section 13.5.3, The syslog-ng pattern database format (p. 186).

13.2.1. Using parser results in filters and templates

The results of message classification and parsing can be used in custom filters and file and database templates as well. There are two built-in macros in syslog-ng OSE that allow you to use the results of the classification: the .classifier.classmacro contains the class assigned to the message (for example violation, security, or un-known), while the.classifier.rule_idmacro contains the identifier of the message pattern that matched the message.

Example 13.3. Using classification results for filtering messages

To filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in a log statement.

filter fi_class_violation {

Filtering on theunknownclass selects messages that did not match any rule of the pattern database. Routing these messages into a separate file allows you to periodically review new or unknown messages.

To filter on messages matching a specific classification rule, create a filter that checks the .classifier.rule_id macro. The unique identifier of the rule (for examplee1e9c0d8-13bb-11de-8293-000c2922ed0a) is theidattribute of the rule in the XML database.

Pattern database rules can assign tags to messages. These tags can be used to select tagged messages using the tags()filter function.

Using parser results in filters and templates

Note

Starting with version 3.2, syslog-ng OSE automatically adds the class of the message as a tag using the .classifier.<message-class>format. For example, messages classified as "system" receive the.classifier.system tag. Use thetags()filter function to select messages of a specific class.

filter f_tag_filter {tags(".classifier.system");};

The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.

Example 13.4. Using pattern parsers as macros

For example, you want to parse messages of an application that look like"Transaction: <type>.", where <type> is a string that has different values (for example refused, accepted, incomplete, and so on). To parse these messages, you can use the following pattern:

'Transaction: @ESTRING::.@'

Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, for example:

'Transaction: @ESTRING:TRANSACTIONTYPE:.@'

After that, add a custom template to the logpath that uses this template. For example, to select everyacceptedtransaction, use the following custom filter in the log path:

match("accepted" value("TRANSACTIONTYPE"));

Note

The above macros can be used in database columns and filename templates as well, if you create custom templates for the destination or logspace.

Use a consistent naming scheme for your macros, for example,APPLICATIONNAME_MACRONAME.

13.2.2. Downloading sample pattern databases

Sample pattern databases are available at the BalaBit Download page. Note that even though these pattern databases contain over 8000 rules for more than 200 applications and devices, they are only samples and experimental databases that are not officially supported and may or may not work in your environment.

The syslog-ng pattern databases are available under the Creative Commons Attribution-Share Alike 3.0 (CC by-SA) license. This includes every pattern database written by community contributors or the BalaBit staff. It means that:

■ you are free to use and modify the patterns purposes;

■ when redistributing the pattern databases you must distribute your modifications under the same license;

■ and when redistributing the pattern databases, you must make it obvious that the original syslog-ng pattern databases are available here.

For legal details, the full text of the license is available here.

Downloading sample pattern databases