AlienVault Creating a Data Source Plugin

How to create a data source plugin

CONTENTS

CONTENTS

1.

1.

INTRODUCTION

INTRODUCTION

...

...

...

...

...

...

...

...

...

...

4

4

2.

2.

TYPES

TYPES

OF

OF

DATA S

DATA S

OURCE

OURCE

PLUGINS ...

PLUGINS ...

...

...

...

...

4

4

2.1.

2.1.

Detector

Detector

Plugins

Plugins

... 5

... 5

2.2.

2.2.

Monitor Plugins ... 23

Monitor Plugins ... 23

3.

3.

HOW TO

HOW TO

CREATE A

CREATE A

CUSTOM D

CUSTOM D

ATA SOURCE

ATA SOURCE

PLUGIN

PLUGIN

...

...

...

...

23

23

3.1.

3.1.

Exchange W

Exchange W

eb SMTP

eb SMTP

server logs

server logs

... 24

... 24

3.2.

3.2.

Creation of the plugin configuration file

Creation of the plugin configuration file

exchangews.cfg 

exchangews.cfg 

 ... 24

 ... 24

3.3.

3.3.

Create the database file

Create the database file

exchangews.sql 

exchangews.sql 

 ... 26

 ... 26

3.4.

3.4.

 Activate data source plugins

 Activate data source plugins

...

...

26

26

3.5.

3.5.

Files

Files

.local ...

.local ...

... 32

... 32

5.

5.

HOW TO

HOW TO

USE CUSTOM FUNCTION

USE CUSTOM FUNCTION

IN DATA

IN DATA

SOURCE PLUGINS ...

SOURCE PLUGINS ...

33

33

APPENDIX A

-APPENDIX A -

RECOMMENDATIONS BEFORE

RECOMMENDATIONS BEFORE

CREATING A

CREATING A

NEW

NEW

PLUGIN

PLUGIN

...

...

35

35

APPENDIX B

-APPENDIX B -

LIST

LIST

OF D

OF D

ATA SOURCE

ATA SOURCE

PLUGINS ...

PLUGINS ...

...

...

37

37

B.1.

B.1.

Database

Database

Plugins

Plugins

... 37

... 37

B.2.

B.2.

Log

Log

Plugins ...

Plugins ...

37

37

B.3.

B.3.

Monitor Plugins ... 38

Monitor Plugins ... 38

B.4.

B.4.

Remote

Remote

Plugins

Plugins

... 39

... 39

B.5.

B.5.

SDEE Plugins ... 39

SDEE Plugins ... 39

B.6.

1.

INTRODUCTION

The objective of this document is to explain how to create plugins supported by AlienVault

USM.

 A plugin is a software component that adds a specific feature to AlienVault USM. Plugins are

used to improve the collection capabilities of the AlienVault Sensors and to indicate to the

system, how to understand and to collect events generated by each application and device.

Sensors receive events from remote hosts using the Syslog, WMI or any other protocols. The

sensors use the Collection Plugins (also called Data Source connectors) in ord er to support the

maximum possible number of applications and devices.

For any system that consumes logs, it is needed a parser to read those logs and extract

information from them into standard information fields (u sername, IP addresses, etc.).

 AlienVault does this via Agent plugin that defines how to collect events from the application or

device as well as how events should be normalized before sending them to the AlienVault

USM central Server. Log Normalization is essentially breaking down a log message into

common fields.

It is necessary to enable a plugin in order to indicate to the system that must collect events

generated by an application or device. Plugins may be pre-configured by AlienVault or defined

by users.

 AlienVault plugins are text configuration files and have the extension *.cfg. These files are

located in

/etc/ossim/agent/plugins 

 in

the Sensor’s file system.

2.

TYPES OF DATA SOURCE PLUGINS

There are 2 types: monitor and detector:



Detector. These plugins receive logs, information and extract events from them. They

process text log information from log files created by RSyslog collection system; and from

log data retrieved from remote systems via one of the remote collection protocols such as

SDEE and SFTP. These plugins can be:



Database. They monitor a file in external databases.



Logs. They monitor a file, usually receiving data through syslog.

Remote Logs. They monitor a file in a remote appliance.



SDEE (Security Device Event Exchange). CISCO device logs.



WMI (Windows Management Instrumentation). They collect remotely Microsoft

Windows events and data in an agent

-

less way.



Monitor. These plugins request information from systems, checking the status of the things

syslog like normal logs and they are often used to correlate log events into alarms by

matching events against the current status of systems.

2.1. Detector Plugins

2.1.1.

DATABASE PLUGINS

It is easier to understand how this type of plugin works by means of an example:

"" #$% &'()* +,-./01&2 34567897:;<=>? +)@8A762 BC3*;:*B*)B@' *8(D4*;C*E E@5')*;:(B(D(E* E@5')*9BC3*;FEEG4 E@5')*973; E@5')*93@'B;HHI= 5E*'; 3(EEJ@':; :D; E4**3;=I 3'@)*EE; EB('B;8@ EB@3;8@ +EB('B9G5*'C2

G5*'C;KE*4*)B &L# < 3)7MN@JO5FD*' A'@F 3)7B'()* (E 3)7 LN,-N PQ 3)7MN@JO5FD*' :*E)K

'*6*R3;

5E*':(B(<;STHU

4@6; V7'5E STHU :*B*)B*: @8 STWUX 3(BYZ ST[U ST\U

+G5*'C2

G5*'C;KE*4*)B 3)7MN@JO5FD*'X 3)7M-]*8B$4(EEX 3)7M&*RB,(B(X

3)7M/3347)(B7@8O(F*X 3)7MO&0E*'O(F*X 3)7M1@678O(F*X 3)7M$#0X 3)7MN*(:E A'@F 3)7B'()* (E 3)7 LN,-N PQ 3)7MN@JO5FD*'K '*6*R3; '*A;I 3456789E7:;< 5E*'8(F*;ST\U 5E*':(B(<;STWU 5E*':(B(W;STHU 5E*':(B(H;ST[U 5E*':(B([;ST=U 5E*':(B(\;ST^U 5E*':(B(=;ST?U 4@6;ST<UXSTWUXSTHUXST[UXST\UXST=UXST^UXST?U

The fields related to database fields are an example for mssql. In case of having mysql, it must

be indicated.

Indicate the point to start to capture. It must be a query to obtain the last event identified by a

sequence number. In this case will be:

E*4*)B &L# < 3)7MN@JO5FD*' A'@F 3)7B'()* (E 3)7 LN,-N PQ 3)7MN@JO5FD*' :*E)

The last “RowNumer ” is obtained from a table.

 A query for getting all values is needed. The same field used in the “start query” must be

selected as first element.

G5*'C;KE*4*)B 3)7MN@JO5FD*'X 3)7M-]*8B$4(EEX 3)7M&*RB,(B(X

A'@F 3)7B'()* (E 3)7 LN,-N PQ 3)7MN@JO5FD*'K

Regexp 

 field must be empty:

'*6*R3; '*A;I

3456789E7:;<

$2 is the second element in the query. In this example is the value of pci.EventClass

5E*'8(F*;ST\U 5E*':(B(<;STWU 5E*':(B(W;STHU 5E*':(B(H;ST[U 5E*':(B([;ST=U 5E*':(B(\;ST^U 5E*':(B(=;ST?U 4@6;ST<UXSTWUXSTHUXST[UXST\UXST=UXST^UXST?U

2.1.2.

LOGS

This is an example of a log plugin:

+,-./01&2 34567897:;<\=H +)@8A762 *8(D4*;C*E BC3*;:*B*)B@' E@5')*;4@6 4@)(B7@8;_]('_4@6_@3B*8*BM4@6

)'*(B*9A74*;A(4E*

3'@)*EE;

EB('B;8@ " 4(58)Y 345678 3'@)*EE JY*8 (6*8B EB('BE EB@3;8@ " EY5B:@J8 345678 3'@)*EE JY*8 (6*8B EB@3E EB('B53; EY5B:@J8; +@3B*8*B ` E3(F :*B*)B*:2 '*6*R3;Kabc#d:@F(78efghifBbc#dgN$9%#ef%#V[ifBjf+fghf2jfBf+bc#dE')9F(74efghif2 fBf+bc#d:EB9F(74efghif2MjK *]*8B9BC3*;*]*8B 3456789E7:;< :*]7)*;S'*E@4]bTgN$9%#iU E')973;S'*E@4]bTgN$9%#iU 5E*':(B(<;ST:@F(78U 5E*':(B(W;STE')9F(74U 5E*':(B(H;ST:EB9F(74U

Plugins extract events (SIDs) from logs by matching each line in the log according to a

regular expression, and then normalizing out data fields from the text. So when the

following log message arrives:

.*D ? <IZI>ZI= 6@46@BY( EEY:+W[[^W2Z .(74*: 3(EEJ@': A@' :674 A'@F <>WM<=?M=M=> 3@'B HH>>W EEYW

It matches the following SID from the SSH plugin.

The information in a log entry to be normalized into field is specified in the regular

expression

_:

!"#"$%;Kbc# &'()"* fJSHUfEhf:S<XWUfEf:f:Zf:f:Zf:f:ifEhbc# &'+)* fghiMjEEYMj.(74*: bc# &),%"* 35D47)k*Cl3(EEJ@':l8@8*i A@'fEhbc# &-./0* 78](47:

5E*'icfEjbc# &1+"!* fghifEMjA'@FfEhbc# &+!2* fghifEMj3@'BfEhbc# &+%0!)* f:S<X\UiK

 And these values are normalized out of it:

,(B* ; .*D ? <IZI>ZI= E')973 ;<>WM<=?M=M=> 0E*'8(F* ; :674

The level of information that can be extracted from a log source is dependent on the level

of detail in the plugin. The more SIDs defined, the greater the ability to extract meaning

from processed logs.

2.1.3.

REMOTE LOGS

This is an example of a remote log plugin:

m /47*8](54B 345678

m /5BY@'Z /47*8](54B &*(F (B :*]*4n(47*8](54BM)@F m #45678 EEY`'*F@B* 7:Z[IIH ]*'E7@8Z IMIM<

m 1(EB F@:7A7)(B7@8Z WI<H`I=`I\ <<Z[H m m /))*3B*: 3'@:5)BEZ m @3*8DE: ` @3*8EEY \M[ m @3*8DE: ` @3*8EEY \M\ m @3*8DE: ` @3*8EEY \M= 1

The bolded fields in the regexp indicate that the matching text will be mapped to information fields during normalization.

m @3*8DE: ` @3*8EEY \M^ m @3*8DE: ` @3*8EEY \M? m @3*8DE: ` @3*8EEY \M?3W m @3*8DE: ` @3*8EEY \M> m ,*E)'73B7@8Z m

m gEY bg*)5'* gY*44i 7E ( 3'@6'(F A@' 4@66786 78B@ ( '*F@B* F()Y78* m (8: A@' *R*)5B786 )@FF(8:E @8 ( '*F@B* F()Y78*M

m 0N1Z YBB3Z__JJJM@3*8EEYM)@F m

m T%:Z EEYM)A6X] <M<W WI<I_IH_WH <=Z[WZ<? o5(8F(4E -R3 T m m +,-./01&2 34567897:;[IIH :EB973;f9$.pb345678`:*A(54BEXE*8E@'i :EB93@'B;WW +)@8A762 BC3*;:*B*)B@' *8(D4*;C*E +01!2"3!"40)"560# 4@)(B7@8;_]('_4@6_(5BYM4@6 )'*(B*9A74*;A(4E* 3'@)*EE;EEY:

EB('B;8@ EB@3;8@ EB('B53;_*B)_787BM:_EEY EB('B EY5B:@J8;_*B)_787BM:_EEY EB@3 70+)3 1+"!3!00)  %(++8'3 !"('9663/(6+" +EEY ` .(74*: 3(EEJ@':2 *]*8B9BC3*;*]*8B

'*6*R3;KbfgQg1Lp9,/&-ifEhbc#dE*8E@'e+afE2jiMjcEEYMjc.(74*: 3(EEJ@': A@' bc#d5E*'efghifEhA'@FfEhMjcbc#dE')ef%#V[iMjc3@'BfEhbc#dE3@'Bef#LN&iK 3456789E7:;< :*]7)*;S'*E@4]bTE*8E@'iU :(B*;S8@'F(47q*9:(B*bT<iU E')973;STE')U :EB973;S'*E@4]bTE*8E@'iU E')93@'B;STE3@'BU 5E*'8(F*;ST5E*'U

Entries marked in bold must always appear because they are used for connecting to remote

host:

2.1.4.

SDEE (SECURITY DEVICE EVENT EXCHANGE)

SDEE

is a standard that specifies the format of messages and protocol used to

communicate events generated by security devices. This protocol is used in the Cisco

Systems IPS Sensor 5.0. AlienVault support this type of logs collection. AlienVault USM

captures events from:



Cisco Network Prevention Systems (IPS)

Cisco Network Detection Systems (IPS)

Cisco Switch IDS



Cisco IOS routers with the Inline Intrusion Prevention System (IPS) functions

Cisco IDS modules for routers



Cisco PIX Firewalls



Cisco Catalyst 6500 Series firewall service modules (FWSMs)

Cisco Management Center for Cisco security agents



CiscoWorks Monitoring Center for Security servers

If you have your own update package from your vendor, you can populate the AlienVault

database with the new signatures.

Go to

/usr/share/ossim/scripts/

to update the plugin sid information:

3CBY@8 )'*(B*$7E)@%#gg7:F(3M3C %Lg`g[<=`$1%M3k6MRF4 ,-1-&- .NLr 345678 st-N- 7: ; K<\>^K"

,-1-&- .NLr 3456789E7: JY*'* 34567897: ; K<\>^K"

%Og-N& %O&L 345678 b7:X BC3*X 8(F*X :*E)'73B7@8i V/10-g b<\>^X <X u$7E)@` %#guX u$7E)@ %8B'5E7@8 #'*]*8B7@8 gCEB*Fui"

%Og-N& %O&L 3456789E7: b34567897:X E7:X )(B*6@'C97:X )4(EE97:X 8(F*X 3'7@'7BCX '*47(D747BCi V/10-g b<\>^X \>?=X O011X O011X u$7E)@`%#gZ r7)'@E@AB p,% p%. #('E786 V548*'(D747BCuX HX [i"

2

This protocol is used in the Cisco Systems IPS Sensor 5.0 to replace Remote Data Exchange Protocol (RDEP).

%Og-N& %O&L 3456789E7: b34567897:X E7:X )(B*6@'C97:X )4(EE97:X 8(F*X

3'7@'7BCX '*47(D747BCi V/10-g b<\>^X \>?[X O011X O011X u$7E)@`%#gZ %- $Lr LDo*)B $@:* -R*)5B7@8uX HX [i"

%Og-N& %O&L 3456789E7: b34567897:X E7:X )(B*6@'C97:X )4(EE97:X 8(F*X 3'7@'7BCX '*47(D747BCi V/10-g b<\>^X \>?\X O011X O011X u$7E)@`%#gZ v57)kB7F* N&g# $@8B*8B`&C3* -R)*EE7]* 1*86BYuX HX [i"

%Og-N& %O&L 3456789E7: b34567897:X E7:X )(B*6@'C97:X )4(EE97:X 8(F*X

3'7@'7BCX '*47(D747BCi V/10-g b<\>^X <><\>X O011X O011X u$7E)@`%#gZ p'**8 ,(F Q@5BY -E)@'B g@ABJ('* 03:(B* $Y*)kuX <X [i"

%Og-N& %O&L 3456789E7: b34567897:X E7:X )(B*6@'C97:X )4(EE97:X 8(F*X 3'7@'7BCX '*47(D747BCi V/10-g b<\>^X <>[I<X O011X O011X u$7E)@`%#gZ r7)'@E@AB #5D47EY*' .74* #('E786 V548*'(D747BCuX HX [i"

This is script generates the needed SQL information to update AlienVault database. Write

the following to insert information:

3CBY@8 )'*(B*$7E)@%#gg7:F(3M3C %Lg`g[<=`$1%M3k6MRF4 e E:**MEG4 @EE7F`:D d E:**MEG4

If you want to update cross-correlation information:

3CBY@8 )7E)@%#gLEr(3M3C %Lg`g[<=`$1%M3k6MRF4

'*34()* 78B@ 3456789'*A*'*8)* ](45*E b<\>^X <<I>X HII<X Hi" '*34()* 78B@ 3456789'*A*'*8)* ](45*E b<\>^X <<I>X HII<X Hi" '*34()* 78B@ 3456789'*A*'*8)* ](45*E b<\>^X <<I>X HII<X Hi" '*34()* 78B@ 3456789'*A*'*8)* ](45*E b<\>^X <<I>X HII<X Hi" '*34()* 78B@ 3456789'*A*'*8)* ](45*E b<\>^X W<\=X HII<X <i" '*34()* 78B@ 3456789'*A*'*8)* ](45*E b<\>^X W<\^X HII<X Hi" '*34()* 78B@ 3456789'*A*'*8)* ](45*E b<\>^X W<\^X HII<X Hi" '*34()* 78B@ 3456789'*A*'*8)* ](45*E b<\>^X W<\^X HII<X Hi" MMM

MMM

3CBY@8 )7E)@%#gLEr(3M3C %Lg`g[<=`$1%M3k6MRF4 e E:**`@EMEG4 @EE7F`:D d E:**`@EMEG4

Do not forget to restart ossim-server in order to update AlienVault Server

cache.

Follow the instructions below to configure AlienVault Agent and collect events from SDEE

capable device:

1.

 Add SDEE reference to this file: /etc/ossim/agent/config.cfg 

2.

Edit this file: /etc/ossim/agent/plugins/cisco- ips.cfg 

+,-./01&2 34567897:;<\>^ +)@8A762 BC3*;:*B*)B@' *8(D4*;C*E E@5')*;E:** E@5')*973; 5E*'; 3(EEJ@':; E4**3;\ 3'@)*EE; EB('B;8@ EB@3;8@

3.

Insert the credentials: your “ source_ip”, “user ” and “ password ” data.

4.

Restart AlienVault Agent for receiving data from SDEE device.

Keep in mind the following points:



Each time a new session begins with a SDEE device, a Subscription ID will be provided. If

in order to continue collecting from the device. The AlienVault Agent closes the session

automatically, but if not, you should do it manually.



The latest Subscription ID can be found here:  /etc/ossim/agent/sdee_sid.data

Execute the following:

3CBY@8 _5E'_EY('*_@EE7F_E)'73BE_)4@E*g,--E*EE7@8M3C g5DE)'73B7@8%,



This closes the last session. If you still have problems, execute the following:



The agent debugging can also turn on, stopping the current agent and starting it manually

on verbose mode:

@EE7F`(6*8B `]



You should get something like this:

WI<W`I\`I^ I\Z<\Z[IX>W\ /6*8B +,-P0p2Z dcRF4 ]*'E7@8;K<MIK *8)@:786;K0&.` ?Kced*8]Z-8]*4@3* RF48E;KYBB3Z__JJJM)7E)@M)@F_)7:E_WII=_I?_)7:**K RF48EZ*8];KYBB3Z__JJJMJHM@'6_WIIH_I\_E@(3`*8]*4@3*K RF48EZE:;KYBB3Z__*R(F34*M@'6_WIIH_I?_E:**K RF48EZ)7:;KYBB3Z__JJJM)7E)@M)@F_)7:E_WII=_I?_)7:**Ked*8]Zt*(:*'edE:Z@@D%8A@ed E:ZE*EE7@8%:eH^HH>^)WA?I(^>W([IW>AD))I):IW^*\d_E:ZE*EE7@8%:edE:Z'*F(78786` *]*8BEeId_E:Z'*F(78786` *]*8BEed_E:Z@@D%8A@ed_*8]Zt*(:*'ed*8]ZP@:CedE:Z*]*8BEed_E:Z*]*8BEed_*8]ZP@:Ce d_*8]Z-8]*4@3*e

2.1.5.

WMI (WINDOWS MANAGEMENT INSTRUMENTATION)

They collect remotely Microsoft Windows events and data in an agent-less way.

This is an example of a WMI plugin:

+,-./01&2

34567897:;<\<?

+)@8A762

*8(D4*;C*E E@5')*;JF7 )'*:*8B7(4E9A74*;_*B)_@EE7F_(6*8B_JF79)'*:*8B7(4EM)E] E4**3;<I 3'@)*EE; EB('B;8@ EB@3;8@ +EB('B9)F:2 )F:;JF7) `0 Lgg9sr%90g-NwLgg9sr%9#/gg __Lgg9sr%9tLg& Kg*4*)B

[email protected]*XN*)@':O5FD*' A'@F s78HW9O&1@6-]*8B sY*'* 1@6A74* ; u/3347)(B7@8uK l Y*(: `8 H l B(74 `8 < l )5B `A W `: fl

'*6*R3;

+)F:2

)F: ; JF7) `0 Lgg9sr%90g-NwLgg9sr%9#/gg __Lgg9sr%9tLg& Kg*4*)B

$@F35B*'O(F*X-]*8B$@:*X1@6A74*Xr*EE(6*XN*)@':O5FD*'Xg@5')*O(F*X&7F*s'7BB*8X0E *' A'@F s78HW9O&1@6-]*8B sY*'* 1@6A74* ; u/3347)(B7@8u (8: N*)@':O5FD*' e Lgg9$L0O&-NK l )(B EB('B9'*6*R3;ab+afl2hiflbf:hiflb+afl2hifl '*6*R3;Kabc#dECEB*F98(F*e+afl2hiflbc#d3456789E7:ef:hiflbc#d4@6A74*e+afl2hiflb c#dF*EE(6*e+afl2hiflbc#d'*)@':85FD*'e+afl2hiflbc#dE@5')*8(F*e+afl2hiflbc#dB7F *J'7BB*8e+afl2hiflbc#d5E*'8(F*eMjiTK E')973;S'*E@4]bTIiU 3456789E7:;ST<U 5E*':(B(W;STWU 5E*':(B(H;STHU 5E*':(B([;ST[U 5E*':(B(\;ST\U 5E*':(B(=;ST=U 5E*'8(F*;ST^U

2.1.5.1.

PREPARING WINDOWS

1.

Create a new limited user for not using an administrator account for remote

connections and make the installation much more secure.

2.

For this example, the user “ wmiuser ” and password “wmi ” have been created.

3.

Configure DCOM to allow the user access to the computer remotely.

4.

Grant remote launch to DCOM and activation permissions for our user:

a)

Run Dcomcnfg  by selecting Run on the Start  menu and typing in Dcomcnfg . Then

click OK.

b)

Open “ Administrative Tools”

and expand “Component Services”. Click the

secondary button of the mouse over “ My Computer ” and select “Properties”.

3

c)

Click on “COM Security ” tab. Then click on Edit Limits  under “ Access

Permissions”:

e)

Click OK.

f)

Click Apply.

g)

Click OK.

5.

Run Dcomcnfg   by selecting Run  on the Start   menu and typing in Dcomcnfg . Then

click OK.

6.

Open “ Administrative Tools”

 and expand “Component Services”. Click the secondary

button of the mouse over “ My Computer ” and select “Properties”.

7.

Click on “COM Security ” tab and then click on Edit Limits  under “Launch and

 Activation Permissions”.

4

9.

Enter the user name and click OK.

10.

In the “Launch and Activation Permisson ” screen, click the options: “ Remote Launch”,

“Local Activation” and “Remote Activation”. Then click OK.

12.

Click Apply.

13.

Click OK.

2.1.5.2.

CONFIGURING ALIENVAULT USM

Before the activation of wmi plug-ins, it is necessary to create a file having Windows IPs

and credentials.

1.

Create a wmi_credentials.csv  file.

]7F _*B)_@EE7F_(6*8B_JF79)'*:*8B7(4EM)E]

2.

 Add ips, users and password with following formats:

<W^MIMIM<X5E*'X3(EE

<W^MIMIMWX:@F(78_5E*'X 3(EE <W^MIMIMHX:@F(78_5E*' X3(EE

2.2. Monitor Plugins

These plugins are used to execute actions in sensors in correlation time through directives. For

instance, the 2005 plugin monitor is used in these 2 files:

8B@3`F@87B@'M)A6 E*EE7@8`F@87B@'M)A6

This is used from monitor directives like:

_*B)_@EE7F_E*']*'_(47*8](54B`(BB()kEMRF4Z d'54* BC3*;KF@87B@'K 8(F*;K/V r@'* BY(8 <I E*)E 3*'E7EB*8)*K '*47(D747BC;K\K A'@F;K<ZgN$9%#K

B@;K<Z,g&9%#K 3@'B9A'@F;K<ZgN$9#LN&K 3@'B9B@;K<Z,g&9#LN&K

 %61#-.:-'3;<==>; %61#-.:+-'3;<@A; )@8:7B7@8;K6*K ](45*;K<IK 78B*'](4;K<\K B7F*9@5B;KHIK (DE@45B*;KB'5*Ke

3.

HOW TO CREATE A CUSTOM DATA SOURCE PLUGIN

This section explains how to create a custom plugin to process Exchange Web Server logs

through the SIEM engine.

3.1. EXCHANGE WEB SMTP SERVER LOGS

The log file used for the following hands

on exercise can be downloaded from here:

*R)Y(86*JEM4@6

Once the file has been downloaded, open it to see the logs we are going to parse. Here are

some sample lines:

WI<<`<I`I> I\ZIIZ<> <M<M<M< H=/[W<=I gr&#gV$< r--`#,$ <>WM<=?M<MW I v0%& ` H=/[W<=I W[I =W<> =? [ I gr&# ` ` ` `

<M<M<M<I ` <M<M<M> +<<_L)B_WI<<Z<HZ<=Z[I `I=II2 Kt-1L `ch<M<M<M> gr&#K W\I [=

3.2. CREATION OF THE PLUGIN CONFIGURATION FILE

1.

Global plugin configuration settings:



Copy the file “ssh.cfg  “ and name this new one as “ exchangews.cfg ”.



Change the “ plugin

 

id “ field (use 9001 as it is part of the user range that goes up to

10000).



Change location to point to the log file “ /var/log/exchangews.log ”



Delete startup and shutdown fields. These fields are not going to be used (there is no

application associated with this plugin).



Create a new translation table

, as it is shown below. This step is optional:

 A translation table is used for translating a string to a number in order to use it as plugin_sid. It is necessary to include the function {translate($field_to_translate)} for using a translation table.

RRRR;=

,-./01&9;>>>>

2.

Create new rules, filling up the fields below. Create two regular expressions to parse the

data, because there are two different formats in the log file.

B"$27(.#"8+ 5 C"."!-2 !16"D

mWI<<`<I`I> I\ZIIZ<\ <M<M<M< H=/[W<=I gr&#gV$< r--`#,$ <>WM<=?M<MW I t-1L ` hH=/[W<=I W\I I [? <H I gr&# ` ` ` `

mWI<<`<I`I> I\ZIIZ<= <M<M<M< H=/[W<=I gr&#gV$< r--`#,$ <>WM<=?M<MW I r/%1 ` h.NLrZhdB*EBnE(F34*<M)@Fe W\I I \^ [\ I gr&# ` ` ` `

*]*8B9BC3*;*]*8B '*6*R3;Kbc#d:(B*ef:S[U`f:SWU` f:SWUfEf:SWUZf:SWUZf:SWUifEbc#dE')973ef%#V[ifEbc#d5E*':(B(WefghifEbc#dY@EB8(F *efghifEbc#d5E*':(B(HefghifEbc#d:EB9%#ef%#V[ifEf:fEbc#dBC3*efJhiK :(B*;S8@'F(47q*9:(B*bT:(B*iU 3456789E7:;SB'(8E4(B*bTBC3*iU :EB973;S'*E@4]bT:EB973iU E')973;S'*E@4]bTE')973iU Y@EB8(F*;STY@EB8(F*U 5E*':(B(W;ST5E*':(B(WU 5E*':(B(H;ST5E*':(B(HU B"$27(.#"8+ 3 C"."!-2 !16" < EFG9 H0!4()D

m<M<M<M<I ` <M<M<M> +<<_L)B_WI<<Z<HZ<=Z[I `I=II2 Kt-1L `ch<M<M<M> gr&#K W\I [=

m<M<M<M<I ` <M<M<M> +<<_L)B_WI<<Z<HZ<=Z[< `I=II2 Kr/%1 ` ch.NLrZhdx*7BYnB*EB:@F(78M)@Fe gr&#K W\I [=

*]*8B9BC3*;*]*8B '*6*R3;Kbc#dE')973ef%#V[ifE` fEbc#d:EB973efghifEf+bc#d:(B*ef:f:f_fJSHUf_f:S[UZf:f:Zf:f:Zf:f:ifE` f:S[Uf2fEfKbc#dBC3*efJhiK :(B*;S8@'F(47q*9:(B*bT:(B*iU 3456789E7:;SB'(8E4(B*bTBC3*iU :EB973;S'*E@4]bT:EB973iU

E')973;S'*E@4]bTE')973iU

3.

Check regular expressions with logs inside the file “ /var/log/exchangews.log ”. There are

several utilities on the Internet to test regular expressions written in Python. It is

recommended to use one of these utilities to check that the created regular expressions

match the logs.

3.3. CREATE THE DATABASE FILE

1.

Create a file using the following examples:

%Og-N& %O&L 345678 b7:X BC3*X 8(F*X :*E)'73B7@8i V/10-g b>II<X <X u*R)Y(86*JEuX u-R)Y(86* -`F(74 s*D E*']*'ui"

%Og-N& %O&L 3456789E7: b34567897:X E7:X )(B*6@'C97:X )4(EE97:X 8(F*X 3'7@'7BCX '*47(D747BCi V/10-g b>II<X <X O011X O011X u*R)Y(86*JEZ t-1Lu XHX Wi"

%Og-N& %O&L 3456789E7: b34567897:X E7:X )(B*6@'C97:X )4(EE97:X 8(F*X 3'7@'7BCX '*47(D747BCi V/10-g b>II<X >>>>X O011X O011X u*R)Y(86*JEZ p*8*'7) *R)Y(86* *]*8Bu XHX Wi"

2.

Insert file values into the database in the server box.

)(B *R)Y(86*JEMEG4 l @EE7F`:D

3.

 Apply changes in SIEM.

_*B)_787BM:_@EE7F`E*']*' '*EB('B

3.4. ACTIVATE DATA SOURCE PLUGINS

Choose one of the following options to activate plugins:



Through command line console

Through web

3.4.1.

ACTIVATE PLUGINS THROUGH COMMAND LINE CONSOLE

1.

Open a console terminal application and connect to the AlienVault System by running

the following command:

EEY '@@Bn%#9(::'*EE

IP_address 

 refers to the default IP of your appliance.

2.

Next, it appears a screen which includes the main menu:

3.

By using the arrow keys on the keyboard, select the option “ Configure Sensor ”. Accept

the selection (<OK >) by pressing Enter  key.

4.

Select the option “Configure Data Source Plugins ”. Accept the selection (<OK >) by

pressing Enter  key.

5.

Select the plugins to activate. To move between them use the arrow keys on the

keyboard and select/deselect it by pressing the Space Bar   on the keyboard. Accept

the selection (<OK >) by pressing Enter   key. It is possible to select several plugins.

 Accept the selection (<OK >) by pressing Enter  key.

6.

The ‘Configure Sensor’ window appears. Move from < OK > to <Back > by using the

Tab key on the keyboard. Back to the AlienVault Setup Screen.

7.

Select the option “ Apply all changes”. Accept the selection (<OK >) by pressing Enter 

key.

8.

 Apply all changes (<Yes>) by pressing Enter  key.

10.

Once the process finishes, the following screen appears:

11.

Press Enter  key. The AlienVault Setup screen appears.

12.

Move from <OK > to <Exit > by using the Tab key on the keyboard. Press Enter  key.

3.4.2.

ACTIVATE PLUGINS BY WEB

To activate plugins by the web, the instructions below should be followed:

1.

Use a web browser access to your AlienVault console at https://your_ip/

2.

Write a valid IP address in the navigation bar of a web browser.

3.

Enter a valid user name and password and click on Login.

5.

The following window appears:

6.

Click on one of the “ Node Name” then, on Sensor Configuration link, and finally on

Collection link. A table appears:

This table displays 2 columns. The left column shows plugins that are enabled and the

right column shows plugins that are available to be enabled.

To pass an item from one side to the other, drag and drop the item or use the links [+]

or [-] which are next to each item.

7.

To make all changes take effect, click the APPLY CHANGES button.

3.5. FILES .LOCAL

Whenever a plugin file is going to be changed, it is recommended to copy the

filename.cfg 

 into

another file named

filename.cfg.local 

. Make all necessary changes in the .local file and keep

the .cfg file without any change. By copying the file, you preserve the original plugin version

over the updated version. The new updates will never overwrite your plugin customization.

Keep in mind that the original plugins can be modified by AlienVault when an

update process has been done.

5.

HOW TO USE CUSTOM FUNCTION IN DATA SOURCE PLUGINS

The instructions below should be followed:

1.

Insert a custom function field in the plugin .cfg file:

+)@8A762

)5EB@F9A58)B7@8E9A74*;_*B)_@EE7F_(6*8B_345678_EEY9)5EB@F9A58)B7@8EM)A6

2.

Create a function file having in mind that a function must start with “ Start Function

<func 

 

name>” and must end with “ End function”:

gB('B .58)B7@8 4@69Y*44@ :*A 4@69Y*44@bE*4AiZ '*B5'8 Kt*44@ 4@6yK -8: .58)B7@8 gB('B .58)B7@8 4@69Y*44@9:(B( :*A 4@69Y*44@9:(B(bE*4AX:(B(iZ '*B5'8 Kt*44@ 4@6Z wEK w :(B( -8: .58)B7@8

3.

Edit the plugin rules to use the function by using two points:

+IIW< ` EEY ` .(74*: 3(EEJ@':2

m .*D ? <IZI>ZI= 6@46@BY( EEY:+W[[^W2Z .(74*: 3(EEJ@': A@' :674 A'@F <>WM<=?M=M=> 3@'B HH>>W EEYW *]*8B9BC3*;*]*8B '*6*R3;KbfJSHUfEhf:S<XWUfEf:f:Zf:f:Zf:f:ifEhbc#dE*8E@'efgjiMjEEYMj.(74*: 3(EEJ@': A@' bc#d5E*'efghifEhA'@FfEhMjcbc#dE')ef%#V[iMj3@'BfEhbc#dE3@'Bef:S<X\UiK 3456789E7:;< E*8E@';S'*E@4]bTE*8E@'iU :(B*;S8@'F(47q*9:(B*bT<iU E')973;STE')U

:EB973;S'*E@4]bTE*8E@'iU E')93@'B;STE3@'BU

5E*'8(F*;ST5E*'U

5E*':(B(<;IJ60#:7"660KLM

5E*':(B(W;IJ60#:7"660:'()(KN1+"!LM

It is not possible to apply a built

instance, translate(:log_hello()) ), as the last function which are executed are

the custom ones.

So translate will receive :log_hello() as a simple chain of chars.

APPENDIX A - RECOMMENDATIONS BEFORE CREATING A NEW

PLUGIN

Follow the following recommendations before creating a new plugin:



Copy a log as big as possible.



Extract events from the log using consecutive grep - v , until the command does not return

anything.



Use grep to check individually every event. Try to find different values that a same event

may take.



Discard repeated events.



Look for event patterns in order to group them using some identifier such as the same field

distribution, for instance.



Take into account that your target will be identifying individual events using a  plugin_sid ,

you may need to think what translates you will be using.



For every event, find out the number of times it is repeated within the log, using the

following command to count the lines.

J) `4



Think if it's worth using a single regex for an event or if several can be grouped together

without making the regex very complex.



Only capture the fields that are going to be used in correlation later on.



Create a rule with a generic regex at the end to capture any remaining event.



Choose the right pre

-

check, keeping in mind that it applies a first filter to the events.



Make sure the rules are alphabetically ordered, starting with 0001 and finishing with 9999,

creating 0002, 0003... groups, leaving room for future expressions.



The rules are loaded and applied in alphabetical order, so events captured by a rule will

not be processed by the rules loaded after that one. Rule order must be chosen carefully

to avoid event masking due to generic rules being loaded before specific ones.



The SQL does not need the sids to be correlative. Gaps can be left in order to make it

more maintainable.



Be careful if you add a custom function into a plugin; or if you access to a proprietary

APPENDIX B - LIST OF DATA SOURCE PLUGINS

B.1. Database Plugins

B.2. Log Plugins

airlock.cfg aix-audit.cfg aladdin.cfg allot.cfg

alteonos.cfg amun-honeypot.cfg apache.cfg apache-syslog.cfg arpalert.cfg arpwatch.cfg artemisa.cfg aruba.cfg

aruba-6.cfg ascenlink.cfg avast.cfg axigen-mail.cfg bind.cfg bit9.cfg bluecoat.cfg bro-ids.cfg

cisco-3030.cfg cisco-ace.cfg cisco-acs.cfg cisco-acs-idm.cfg cisco-asa.cfg cisco-asr.cfg cisco-fw.cfg cisco-ids.cfg cisco-ips-syslog.cfg cisco-nexus-nx-os.cfg cisco-pix.cfg cisco-router.cfg cisco-vpn.cfg cisco-wlc.cfg citrix-netscaler.cfg clamav.cfg clurgmgr.cfg courier.cfg cyberguard.cfg dhcp.cfg

dionaea.cfg dovecot.cfg dragon.cfg enterasys-rmatrix.cfg exchange.cfg extreme-switch.cfg extreme-wireless.cfg f5.cfg

f5-firepass.cfg fidelis.cfg fortigate.cfg fortiguard.cfg fortimail.cfg fw1-alt.cfg fw1ngr60.cfg gfi.cfg

glastopng.cfg heartbeat.cfg honeyd.cfg hp-eva.cfg iis.cfg

imperva-securesphere.cfg

intrushield.cfg ipfw.cfg iphone.cfg iptables.cfg ironport.cfg isa.cfg  juniper-srx.cfg juniper-vpn.cfg kismet.cfg linuxdhcp.cfg

modsecurity.cfg monit.cfg motorola-firewall.cfg mwcollect.cfg nagios.cfg nepenthes.cfg nessus.cfg nessus-detector.cfg netgear.cfg netkeeper-fw.cfg netkeeper-nids.cfg netscreen-firewall.cfg netscreen-igs.cfg netscreen-manager.cfg netscreen-nsm.cfg nfs.cfg

nortel-switch.cfg ntsyslog.cfg openldap.cfg optenet.cfg oracle-syslog.cfg osiris.cfg ossec.cfg ossec-idm.cfg ossec-idm-single-line.cfg ossec-single-line.cfg ossim-agent.cfg p0f.cfg

pads.cfg paloalto.cfg pam_unix.cfg panda-as.cfg pf.cfg postfix.cfg prads.cfg prads_eth0.cfg proxim-orinoco.cfg pureftpd.cfg radiator.cfg radware-ips.cfg raslogd.cfg realsecure.cfg rrd.cfg rsa-secureid.cfg sap.cfg sendmail.cfg serviceguard.cfg shrubbery-tacacs.cfg sidewinder.cfg siteprotector.cfg siteprotector-snmp.cfg sitescope.cfg

smbd.cfg snare.cfg snare-idm.cfg snare-mssql.cfg snare-msssis.cfg snort_syslog.cfg sonicwall.cfg sophos.cfg spamassassin.cfg squid.cfg squidGuard.cfg ssh.cfg stonegate.cfg stonegate_ips.cfg storewize-V7000.cfg sudo.cfg

suhosin.cfg suricata-http.cfg symantec-ams.cfg symantec-epm.cfg syslog.cfg tacacs-plus.cfg tarantella.cfg tippingpoint.cfg token-rsa.cfg trendmicro.cfg usbudev.cfg vandyke-vshell.cfg vmware-esxi.cfg vmware-vcenter.cfg vmware-workstation.cfg vplus.cfg

vsftpd.cfg vyatta.cfg W2003DNS.cfg watchguard.cfg webmin.cfg websense.cfg wuftp.cfg

B.3. Monitor Plugins