LOG- UND EVENTMANAGEMENT

(1)

OSMC 2013

– 23.10.2013

LOG- UND EVENTMANAGEMENT

(2)

AGENDA

￭

_{Kurzvorstellung}

￭

_Einführung

￭

_Architektur

￭

_Installation

￭

_{Routing und Filterung von Events}

￭

_{Interfaces & API}

￭

_{Integration in Nagios und Icinga}

￭

_{Eventkorrelation mit EDBC}

(3)

(4)

KURZVORSTELLUNG NETWAYS

• Firmengründung 1995

• Open Source seit 1997

• 40 Mitarbeiter

• Spezialisierung in den Bereichen

Open Source Systems

Management und Open Source

Datacenter Infrastructure

(5)

NETWAYS KOMPETENZEN

• Monitoring & Reporting

• Configuration Management

• Service Management

• Knowledge Management

• Backup & Recovery

• High Availability & Clustering

• Cloud Computing

• Load Balancing

• Virtualization

• Database Management

OPEN SOURCE

SYSTEMS MANAGEMENT

OPEN SOURCE

DATA CENTER

(6)

PuppetCamp 2013/2014

• 28. November München

• 11. April Berlin

• CfP für Berlin läuft noch

Open Source Datacenter Conference

• 08. – 10. April 2014

• 125 Teilnehmer (2012)

• Datacenter | Automation | DevOps

• CfP läuft bis zum 31. Dezember 2013

(7)

(8)

LOGS

Logs

-> Fluss an

unstrukturierten

Daten

Oct 4 16:57:24 web sshd[25828]: Received disconnect from 10.10.0.31: 11: disconnected by user

(9)

EVENTS

Event

-> Fluss an

strukturierten

Daten

Event {

Time: Oct 4 16:57:24

Process: sshd

State: Received disconnect from 10.10.0.31

Client: 10.10.0.31

}

(10)

LOG & EVENTMANAGEMENT

(11)

TOOLS

￭

_{Nagios & Icinga Addons}

• check_logfiles

• NagTrap

• EventDB

• EDBC

￭

_{Logmanagement-Tools}

• Graylog

• Fluentd

• Logstash

(12)

LOGSTASH

(13)

(14)

LOGSTASH

￭

_{Logmanagement auf Basis von JRuby}

￭

_{Konfigurierbare “Pipe”}

￭

_{Flexible Plugin-Architektur für}

• Input

• Filter

• Output

￭

_{Standardplugins für alle gängige Protokolle}

￭

_Webinterface

(15)

Outputs

• amqp • boundary • circonus • cloudwatch • datadog • datadog_metrics • elasticsearch • elasticsearch_http • elasticsearch_river • email • exec • file • ganglia • gelf • gemfire • google_cloud_storage • graphite • graphtastic • hipchat

LOGSTASH - IO

Inputs

• amqp • drupal_dblog • elasticsearch • eventlog • exec • file • ganglia • gelf • gemfire • generator • graphite • heroku • imap • irc • log4j • lumberjack • pipe • rabbitmq • redis • relp • s3 • snmptrap • sqlite • sqs • stdin • stomp • syslog • tcp • twitter • udp • unix • varnishlog • websocket • wmi • xmpp • zenoss • zeromq • http • irc • jira • juggernaut • librato • loggly • lumberjack • metriccatcher • mongodb • nagios • nagios_nsca • null • opentsdb • pagerduty • pipe • rabbitmq • redis • riak • riemann • s3 • sns • sqs • statsd • stdout • stomp • syslog • tcp • udp • websocket • xmpp • zabbix • zeromq

(16)

INSTALLATION - LOGSTASH

￭

_{Download - http://logstash.net}

(17)

ARCHITEKTUR

Shipper

(18)

REDIS

￭

_{NoSQL in memory auf Basis von C}

￭

_{Unterstützung verschiedener “Datentypen”}

• strings

• hashes

• lists

• sets and sorted sets

￭

_{Support für verschiedene Replikationsszenarien}

￭

_SAUSCHNELL

$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q SET: 122556.53 requests per second

GET: 123601.76 requests per second LPUSH: 136752.14 requests per second LPOP: 132424.03 requests per second

(19)

INSTALLATION - REDIS

￭

_{Download - http://redis.io/download}

￭

_make

￭

_{make test}

￭

_{make install}

￭

_{/usr/local/bin/redis-server}

(20)

ELASTICSEARCH

￭

_{Schemafreier RESTful Suchserver auf Basis von Java}

￭

_{Basierend auf Lucene Core}

￭

_{“Vergleichbar” mit Apache Solr}

￭

_{Verteilte Architektur durch}

• Shards

• Replicas

• Gateways

(21)

INSTALLATION - ELASTICSEARCH

￭

_{Download – http://elasticsearch.org/download/}

￭

_{Entpacken des Archives}

(22)

(23)

ÜBERSICHT

Shipper

(24)

KONFIGURATION - LOGSTASH - SHIPPER

￭

_{Übermittlung von Logs an Logstash}

• Logstash

• Lumberjack

• Syslog

• Log4J

• Gelf

• File-Read

• u.v.a.m.

(25)

KONFIGURATION - LOGSTASH - SHIPPER

￭

_{Konfiguration}

input { file { path => "/root/osmc/demodata/access.log.1” type => "apache-access" } } output { stdout { debug => true } redis { host => "127.0.0.1" data_type => "list" key => "logstash.apache" } }

￭

_{java -jar logstash-current.jar agent -f}

logstash_shipper.conf

Shipper

(26)

KONFIGURATION - LOGSTASH - INDEXER

￭

_{Konfiguration}

input { redis { host => "127.0.0.1" type => "redis-input"

# these settings should match the output of the agent data_type => "list" key => "logstash.apache” } } output { stdout { debug => true } elasticsearch { host => "127.0.0.1" } } Shipper Shipper Shipper

(27)

KONFIGURATION - LOGSTASH – INDEXER - APACHE

￭

_{Konfiguration für Apache-Logs}

input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” format => "json_event" } } filter { if [type] == "apache-access" { grok {

match => [ "message", "%{COMBINEDAPACHELOG}" ] } } } output { elasticsearch { host => "127.0.0.1” } } Shipper Shipper Shipper

(28)

KONFIGURATION - LOGSTASH – INDEXER - GEOIP

￭

_{Konfiguration für Geo-Daten}

input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” } } filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG}" } geoip { source => "clientip" add_tag => ["geotag"] } } output { elasticsearch {host => "127.0.0.1”} } Shipper Shipper Shipper

(29)

(30)

(31)

(32)

(33)

KIBANA - DEMO

(34)

(35)

REALTIME LOGANALYSE

￭

_{Analyse verschiedener Quellen in Realtime}

￭

_{Prüfung auf Patterns und States}

• Facilitites

• Regex

• Programs

(36)

ÜBERSICHT LOGSTASH UND ICINGA

Search & Storage Webinterface Indexer

Icinga –Web Icinga - Commandpipe

(37)

KONFIGURATION - LOGSTASH – INDEXER - ICINGA

￭

_{Konfiguration für Icinga-Alert}

input { … } filter { if [type] == "syslog" {

grok {match => [ "message", "%{SYSLOGBASE}" ] } grep {

match => [ "message", "Error" ] drop => false add_tag => "nagios-update" add_field => [ # "nagios_host", "%{@source_host}", "nagios_host", "localhost", "nagios_service", "Logstash", "nagios_level", "2”] }}} output { elasticsearch {host => "127.0.0.1”} nagios { commandfile => "/var/lib/icinga/rw/icinga.cmd" }} Shipper Shipper Shipper

(38)

LOGSTASH – ICINGA - DEMO

(39)

(40)

EDBC - EINFÜHRUNG

￭

_{EDBC – EventDB Correlator}

• Receptors

Eingangskanäle für verschiedene Eventquellen

• Processors

Verarbeitet konfigurierte Filterregeln

• Chains

Verbindet verschiedene Receptoren und Processoren

zu komplexeren Prozesseketten

(41)

EDBC - ARCHITEKTUR

Event A

Event B

Event C

Clearing

Event A

Receptor

Aggregator

Acknowledge Group

Persister

Filter-Chain Cleared Aggregate / No match

(42)

EDBC - BEISPIEL

[example-aggregator]

class: processor

type: aggregation

matcher: message REGEXP 'The server (?P<HOSTNAME>\w+) just

went down. Errorcode (?P<CODE>\d+)'

aggregateMessage: Server $HOSTNAME is down (Code : $CODE)

($_COUNT events)

datasource: @mysql

(43)

(44)

(45)

STATSD & GRAPHITE

￭

_StatsD

• Netzwerkdaemon auf Basis von UDP

• Bucket -> Value -> Flush

• Entkoppelte Zwischenaggretion für Statisik

￭

_Graphite

• Graphing-Framework bestehend aus

• Whisper (Datenbank)

• Carbon (Engine)

(46)

INSTALLATION – STATSD - NODEJS

￭

_{apt-get install make python g++ checkinstall}

￭

_{mkdir nodejs && cd nodejs}

￭

_{wget -N http://nodejs.org/dist/node-latest.tar.gz}

￭

_{tar xzvf node-latest.tar.gz && cd `ls -rd node-v*`}

￭

_checkinstall

(47)

INSTALLATION – STATSD

￭

_{wget https://github.com/etsy/statsd/archive/master.zip}

￭

_{unzip master.zip}

(48)

MONITORING - STATSD

￭

_{Status Informationen}

• echo stats | nc 127.0.0.1 8126

• echo health | nc 127.0.0.1 8126

￭

_{Timer- und Counterinfo}

• echo counters | nc 127.0.0.1 8126

• echo timers| nc 127.0.0.1 8126

(49)

INSTALLATION – GRAPHITE

￭

_{Download der Sources}

• git clone

https://github.com/graphite-project/graphite-web.git

• git clone

https://github.com/graphite-project/carbon.git

• git clone

https://github.com/graphite-project/whisper.git

(50)

INSTALLATION – GRAPHITE

￭

_{Installation Whisper}

pushd whisper

sudo python setup.py install

popd

￭

_{Installation Carbon}

pushd carbon

sudo python setup.py install

popd

￭

_{Konfiguration Carbon}

pushd /opt/graphite/conf

cp carbon.conf.example carbon.conf

(51)

INSTALLATION – GRAPHITE - WEBAPP

￭

_{Check Dependencies Graphite webapp}

pushd graphite-web

python check-dependencies.py

popd

￭

_{Installation Graphite webapp}

pushd graphite-web

python setup.py install

popd

￭

_{Konfiguration Apache}

(52)

ÜBERSICHT STATSD UND GRAPHITE

Search & Storage Webinterface Indexer

Graphite Statsd

(53)

KONFIGURATION - LOGSTASH – INDEXER - STATSD

￭

_{Konfiguration für Statsd}

input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list"

key => "logstash.apache” format => "json_event” add_field=> ["sitename","www.icinga.org"] }

}

filter { if [type] == "apache-access" { grok {match => [ "message", "%{COMBINEDAPACHELOG}" ] } }} output {

stdout { debug => true } if [type] == "apache-access" { statsd { host => "localhost" port => 8125 namespace => "logstash" debug => false increment => "apache.%{sitename}.response.%{response}” count => ["apache.%{sitename}.bytes", "%{bytes}"]

} } elasticsearch {host => "127.0.0.1”}} Shipper Shipper Shipper

Broker Indexer Search & Storage Webinterface

(54)

GRAPHITE - DEMO

(55)

(56)