Centinel: Streaming Data Handler. September 07 th, 2015

(1)

Centinel:

Streaming Data Handler

(2)

An Overview

(3)

Centinel: Streaming Data Handler

The Centinel project aims at providing a distributed, reliable framework for efficiently collecting, aggregating and sinking streaming data across Persistence DB and stream analyzers (example: Graylog, Elastic search, Spark, Hive etc.). This framework enables SDN applications/services to receive events from multiple streaming sources (example: Syslog, Thrift, Avro, AMQP, Log4j, HTTP/REST etc) and execute actions like network configuration/batch processing/real-time analytics.

Core features of Centinel framework are:

 Stream collector - Collecting, aggregating and sinking streaming data  Log Service - Listen log stream events coming from log analyzer

 Log Service - Enables user to configure rules (example: alerts, diagnostic, health, dashboard etc.)  Log Service - Performs event processing/analytics

 User Interface - Enable set-rule, search, visualize, alert, diagnostic, dashboard etc.

 Adaptor - Log analyzer plug-in to Graylog and a generic data-model to extend to other stream analyzers (Logstash etc.)

 REST Services - Northbound APIs for Log Service and Steam collector framework  Leverages - TSDR persistence service, data query, purging and elastic search

(4)

Centinel: High Level Architecture

(5)

Centinel: Functional Overview

Feature Details

Streams

 Mechanism to route messages into categories in real time while they are processed like stream for audit logs( install bundle etc.)

 Rule configuration includes message, level, source etc.

 Streams are generated by GrayLog server as per user-defined rules. Event-handler module handles streams from GrayLog server and persist it into Centinel

Alerts

 Log alerts :

 Alerts get generated based on specific event matching in real-time

 Alert condition types : Message count condition, Field value condition, Field string value condition  Alerts get cleared if specified condition does not persist. Alert check interval time is configurable,

default is 60 seconds.

 Common alert operations like manual acknowledge , delete, filter , sort etc. will be supported Search

and Analyze

 Support for search query language.  Time range for search can be specified  Visualization includes histogram

(6)

Centinel: Functional Overview (continued)

Dashboard

 Build pre-defined views on data by adding widgets. Domain expert can define search query and save results on dashboard.

 Search result type : Search result counts, Search result histogram charts, Field value charts,Quick value results

Diagnostics

 Enables , user to specify/configure group of log messages (events or specific conditions ) as single rule. Order of messages and condition for each messages can also be configured.

 Notification will be persisted on db if log messages are coming out of order/sequence

 On clicking notification on UI, popup window will open which displays expected order and received order of log messages.

 User will be able to scroll popup window to see individual received log message.

 Event processor has intelligence to verify sequencing of messages and generate diagnostic reports  Success notification is displayed when all specified conditions are true.

Health  User specify list of messages with conditions for feature.

 Feature health will be changed to Critical, Major, Warn on reception of all messages specified

(7)

Centinel: Project links

https://wiki.opendaylight.org/view/Project_Proposals:Centinel

Project Proposal

https://wiki.opendaylight.org/view/Centinel:Main

Project Main page

Release plan

https://wiki.opendaylight.org/view/Centinel:Beryllium_Release_Plan

https://lists.opendaylight.org/pipermail/centinel-dev/

Project Email archives

Email

[email protected]

(8)

Design Approach

8

(9)

AAA: Authentication, Authorization & Accounting

ALTO: Application Layer Traffic Optimization

AuthN: Authentication

BGP: Border Gateway Protocol

CAPWAP: Control and Provisioning of Wireless Access Points

COPS: Common Open Policy Service

DIDM: Device Identification and Driver management

DLUX: OpenDaylight User Experience

DDoS: Distributed Denial Of Service

DOCSIS: Data Over Cable Service Interface Specification

FRM: Forwarding Rules Manager

GBP: Group Based Policy

IoTDM: Internet of Things Data Broker

LACP: Link Aggregation Control Protocol

LISP: Locator/Identifier Separation Protocol

MAPLE: Maple Programming

NIC: Network Intent Proposal

OVSDB: Open vSwitch DataBase Protocol

OPFLEX: Extensible Policy Protocol

Legend

“LITHIUM”

AAA- AuthN Filter

OpenDaylight APIs (REST)

OpenFlow Enabled Devices

DLUX OpenStack VTN Coordinator Neutron SDNI Wrapper

DDoS Protection

Open vSwitches Additional Virtual & Physical Devices Topology Processing DIDM MD-SAL / Yangtools GBP Service SFC DOCSIS Abstraction VTN Manager Plugin20C LISP Service BGP PCEP OVSDB OVSDB NETCONF PCMM/C OPS SNBI BGP LISP PCEP Plugin20C SNMP OpenFlow Neutron Service SDNI Aggregator Persistence L2 Switch TCP-MD5 SXP USC Discovery IoTDM IoT LACP MAPLE ALTO CAPWA P Reservation TSDR VPN Service NIC USC Manager OPFLEX Topology Inventory FRM Network Applications Orchestrations and Services

NB APIs Applications Plugin Services Controller platform SB interfaces & protocols plugins

PCEP: Path Computation Element Protocol

PCMM: Packet Cable MultiMedia

Plugin2OC: Plugin To OpenContrail

SDNI: SDN Interface (Cross-Controller Federation)

SFC: Service Function Chaining

SNBI: Secure Network Bootstrapping Infrastructure

SNMP: Simple Network Management Protocol

SXP: Source-Group Tag eXchange Protocol

TSDR: Time Series Data Repository

TTP: Table Type Patterns

USC: Unified Secure Channel

VTN: Virtual Tenant Network

(10)

Introduction and Data Models

Centinel framework enables SDN applications/services to receive events from multiple streaming sources (example: Syslog, Thrift, Avro, AMQP, Log4j, HTTP/REST etc) and execute actions like network configuration/batch processing/real-time analytics.

The entire architecture can be divided into the following modules:

 User interface  Centinel NB APIs

 Stream Analyzer Service (LAAS and Plugin)

 Log Service (“Configure Rules” and “Event handler” Modules)

 Stream Collector Service (Flume and Sqoop based)  Leverage persistence database service

This design document depicts the subset of functionality (Alert) and the same is subject to follow-up add-on

updates.

The stream data handler is a framework for collecting, aggregating stream data across Persistent DB and stream analyzer.

To achieve the same, the entire model could be broken up into two parts:

 Rules – Configurable from Northbound

 Notifications/Events – Output from stream analyzer

10

(11)

Rules

The Northbound of Centinel will allow operators to set Rules for

collecting, aggregating and analysis of streaming data.

The following diagram explains the Centinel data model for log services where in the BaseRule is the basic model object that is imported by BaseAlert which is further used by different types of rules.

BaseRules model object is the basic data model that will be

shared/imported by all the other “Log service” features like Alerts, Streams, Health, Dashboard and any other feature that Centinel

framework may want to support in future.

(12)

Events

12

When a user configures a rule successfully, notifications/Events corresponding to the rule will be supplied by the stream analyzer. The diagram below shows the data model for events (notifications and alert) received from stream analyzer. The base model for

notifications/events is “BaseEvent” that contains event type and stream and is further extended by alarm that provides detailed information about the event that has occurred for

example the stream which raised the alarm, alert condition, feature or module from which alert was raised etc.

(13)

Alerts

The diagram below shows the class diagram for RPCs and listeners. It defines the following interfaces and java classes for handling the Alerts:

 LAAS interface and its implementation that provides log analyzer abstraction layer and also publish notifications.

 Plugin class that provides alarm call backs from the analyzer (grey log server).

 AlertService interface and its configuration Implementation that defines and

implements CRUD on Alert Rules and also write it in md-sal data-store.

 AlertServiceListener class that listens to any data change in md-sal database.  ServiceAggregator and

ServiceEventHandler classes that registers for notification and receive notification published by the LAAS as DTO

 ServicePersistData class that stores the DTO from event handler into the persistent database

(14)

North Bound APIs

14

NotifyAlarm() RPC exposed by LAAS. Plugin use it to notify alarms

SetAlertRule() Set rules for alerts

RemoveAlertRule() Remove rules for alerts

GetAlertRule() Get all alert rules

getAlarms() Retrieve recent 300 alarm.

(15)

Sequence diagram for Rule Handling

Following is the sequence diagram for the Rule handling part.

It describes the flow for “How is a rule for Alert getting

(16)

Sequence diagram for Event Handling

16

Below is the sequence diagram for the Rule handling part.

It describes the flow for processing an Event (Alarm in this case) before it get stored in the persistent data store.

(17)