Linux Syslog Messages in IBM Director
Ever want those pesky little Linux syslog messages (/var/log/messages) to forward to IBM Director? Well, it’s not built in, but it’s pretty easy to setup. You can forward syslog messages from an IBM Director Agent (Level-2 agent) to it’s Director server in three (3) easy steps:
1. Create a FIFO (more about FIFOs later).
2. Configure the syslog daemon to send messages to the FIFO.
3. Write a script to read the FIFO and generate Director events.
Easy, huh? If you’re interested, just continue reading.
Create a FIFO
So what is a FIFO? A FIFO (for First In First Out) is also called a named pipe. Now if you’ve worked with Linux or other Unix systems, you’re quite familiar with pipes:
ls /usr/sbin/ | more
The pipe (“|”) takes the output of the first command (“ls /usr/sbin/”) and “pipes” it to the second command (“more”). A FIFO merely takes this pipe concept and gives it a name.
A FIFO appears in the file system although it does not take up space in the file system.
You create a FIFO using the “mkfifo” command, specifying the location of the FIFO:
mkfifo /opt/ibm/director/logwatch
I chose the /opt/ibm/director directory since this FIFO is specifically for IBM Director, but you can put it anywhere.
Configure syslog
Telling the syslog daemon to send messages to the FIFO is pretty simple as well.
Syslog is configured through the /etc/syslog.conf file. Each line tells syslog what to do with messages and consists of two entries, called “selectors” and “actions”. The
selectors are made up of providers and priorities separated by a period (.). For example:
“mail.*” means all mail messages, “news.crit” means critical news messages and
“*.warning” means all warnings. Actions are basically regular files to record the message in, Linux consoles to post messages to, remote machines to notify or FIFOs to write to.
(See the syslog.conf manpage for more details on configuring the syslog daemon.) To get the syslog daemon to send warning, critical and error messages to our new FIFO, simply add the following line to your /etc/syslog.conf and restart the daemon:
*.warning;*.crit;*.err |/opt/ibm/director/logwatch
Event Handler script
Now all we need is a script to forward these messages as IBM Director events. Here’s a
simple script to do just that:
#!/bin/bash
# logwatcher.sh
# A simple script to generate IBM Director events from incoming messages
true=0
while [ $true ]; do read line
Genevent /TYPE:"Director.Linux.LogWatcher" /TEXT:"$line"
done
Figure 1 - logwatcher.sh script
Now all this script does read input and generate an IBM Director event for each line. To use run the following command:
logwatcher.sh < fifo
Now every line written to the FIFO will be sent as an IBM Director event.
A Better Mousetrap
The one limitation in the above script is the fact that the severity of the event is not included in the Director event. Since the priority of the message is not included when syslog generates the message, the best way I’ve thought to include severities in the messages is to use multiple FIFOs:
prw-r--r-- 1 root root 0 Jul 4 16:16 logwatch.critical prw-r--r-- 1 root root 0 Jul 4 15:02 logwatch.fatal prw-r--r-- 1 root root 0 Jul 4 16:18 logwatch.harmless prw-r--r-- 1 root root 0 Jul 4 16:19 logwatch.minor prw-r--r-- 1 root root 0 Jul 4 16:06 logwatch.unknown prw-r--r-- 1 root root 0 Jul 7 09:46 logwatch.warning
Figure 2 - Sample FIFOs
I created a FIFO for each priority event I want to forward to Director.
# LogWatcher FIFOs
*.=info;*.=notice |/opt/ibm/director/logwatch.info
*.=warning |/opt/ibm/director/logwatch.warning
*.=err;*.=crit |/opt/ibm/director/logwatch.critical
*.=emerg |/opt/ibm/director/logwatch.fatal
Figure 3 - Sample syslog.conf entries
I then created a script that you could tell which FIFO to listen to and generate Director
events of the proper severity.
#!/usr/bin/perl
# Logwatcher
# Version 1.0
#
# Written by Paul Engel
my $fifo_root = "/opt/ibm/director/logwatch";
my $logfile = "/var/log/logwatcher";
my $pidfile = "/var/run/logwatcher.pid";
sub watcher {
my $severity = shift;
my $dir_sev = $severity;
$dir_sev =~ s/^(.)/\U$1/;
my $fifo = "${fifo_root}.${severity}";
defined(my $pid = fork) or die "Can't fork watcher process
$severity: $!";
if ($pid){
return $pid }else{
open STDERR, ">>$logfile";
open LOG, ">>$logfile";
select LOG;
$| = 1;
open FILE, "<$fifo" or
die "Unable to open FIFO $fifo: $! ";
while (1) {
while (<FILE>) { chomp;
my $date = `date`;
chomp $date;
print "\n";
print "---\> $date \<--- \n";
my $cmd = "Genevent
/TYPE:\"Director.Linux.LogWatcher\" /TEXT:\"$_\" /SEV:$dir_sev
>> $logfile";
system $cmd;
print "====================================\n";
} }
close FILE;
close LOG;
close STDERR;
} }
sub close_watchers {
my $signal = shift;
print "Shutting down watchers...";
kill $signal, @pids;
unlink $signal;
print " [done]\n";
}
defined(my $pid = fork) or die "Can't fork main daemon process: $!";
if ($pid){
open FILE, ">$pidfile" or
die "Unable to open PID file $pidfile: $! ";
print FILE "$pid\n";
close FILE;
}else{
our @pids;
print "Starting watchers...";
push (@pids, watcher('critical'));
push (@pids, watcher('fatal'));
push (@pids, watcher('harmless'));
push (@pids, watcher('minor'));
push (@pids, watcher('unknown'));
push (@pids, watcher('warning'));
print " [done]\n";
$SIG{HUP} = \&close_watchers;
$SIG{INT} = \&close_watchers;
$SIG{KILL} = \&close_watchers;
$SIG{TERM} = \&close_watchers;
$SIG{QUIT} = \&close_watchers;
waitpid($pids[$#pids],0);
}
Figure 4 - logwatcher.pl script
OK, so this is a little more complicated. First, I used Perl since I could get away from file redirection. Basically this script forks processes to watch each of the FIFOs listed. It then records the process ID (PID) of the controlling daemon. When that daemon is shut down (with a KILL or INT signal) it first shuts down all the child watchers. Also, events are written to the /var/log/logwatcher file including errors and successful message from the Genevent command.
Starting the logwatcher
To make life easier, I created an init script to start and stop Logwatcher. Save this in your /etc/init.d directory and you can simply use the following call to start Logwatcher:
/etc/init.d/logwatcher start
#!/bin/bash
# Logwatcher init script
# Version 1.0
#
# Written by: Paul Engel
pidfile="/var/run/logwatcher.pid"
start() {
if [ -e "$pidfile" ]; then
echo "PID file found ($pidfile)!"
echo Logwatcher may be running echo Startup Aborted!
else
echo "Starting LogWatcher "
/opt/ibm/director/logwatcher.pl fi
}
stop() {
if [ -e "$pidfile" ];then
echo Stopping LogWatcher kill `cat $pidfile`
rm $pidfile else
echo Logwatcher does not appear to be running.
fi }
status() {
if [ -e "$pidfile" ]; then
echo "Logwatcher appears to be running (PID: `cat
$pidfile`)"
else
echo "Logwatcher doesn't appear to be running"
fi }
case "$1" in
start) start;;
stop) stop;;
status) status;;
restart) stop sleep 5 start ;;
*)
echo "usage: $0 {start|stop|restart}"
exit 1 ;;
esac
Figure 5 - logwatcher init script
What you get
Now events will start flowing to Director. They will look something like this:
Figure 6 - Logwatcher event in Director Event Log
