Elettra DAta analysis Tool: a data webhousing tool for heterogeneous log analysis

(1)

Elettra DAta aNalysis

Tool:

a data webhousing

tool

for

heterogeneous log analysis

Roberto

Roberto PugliesePugliese Stefano

Stefano MaraspinMaraspin Alessio

Alessio CurriCurri

Software for Measurements Software for Measurements

Experiment Division Experiment Division

Sincrotrone

(2)

Outline

Elettra

Virtual

Collaboratory

(EVC)

The portal application

The Collaboration Tools

Usage scenarios

DANT an advanced Log

Analyzer

Software Architecture

DANT at work

(3)

What is the

Elettra

Virtual

Collaboratory

(EVC)?

EVC is an example of virtual laboratory, a system which allows a EVC is an example of virtual laboratory, a system which allows a team of researchers distributed anywhere in the world to perform team of researchers distributed anywhere in the world to perform a a

complete experiment on the equipped

complete experiment on the equipped beamlinesbeamlines and experimental and experimental stations of

stations of ElettraElettra. .

User at

ELETTRA Team Memberat Home Lab

Remote Collaborator

Equipment

Control Data CPU

(4)

EVC in action: a web portal

EVC is based on

the “web portal”

metaphor

EVC supports

four different user

categories:

VisitorsVisitors

Normal usersNormal users

Project leadersProject leaders

(5)

Collaborating to an EVC project

Scientists working

to an EVC project

can use many

project related project related collaboration collaboration tools tools EVC presents an EVC presents an adaptive interface adaptive interface changing to suite changing to suite the category the category and expertise and expertise

level of the user

(6)

Collaboration tools:

scientific visualisation

Scientists can

browse, visualise

and process

remotely

scientific data

in real

-

time

as soon as

the data is

collected

(7)

Collaboration Tools: Remote

Beamline

Control and Supervision

Beamwatch

presents a

synoptic view

of the

beamlines

Autorised

people

can thus operate

remotely on

the

beamline

intrumentation

(8)

Collaboration Tools:

Electronic Notebook

Web application

Web application whichwhich substitutes the Beamline

substitutes the Beamline

LogBook

LogBook registeringregistering meaningful

meaningful beamlinebeamline events

events using a wikiusing a wiki- -weblog

weblog methaphormethaphor Events can be

Events can be enteredentered manually or automatical

manually or automatical- -ly

ly by a programby a program

Texts and images are

automatically indexed

and hence easily

searchable and browsable

(9)

2 mm

Reconstructed stack of 279 slices Volume rendering

EVC usage scenarios: real time

micro

-

tomographic

reconstructions

EVC will be used to allow real

EVC will be used to allow real--time microtime micro--tomographictomographic reconstructions on SYRMEP

reconstructions on SYRMEP beamlinebeamline, as soon as , as soon as data is collected

(10)

A new experimental

station was recently

added to EVC: added to EVC: ELETTRA Control ELETTRA Control Room Room

Available tools for

this experimental this experimental station are: station are: Chat, Scientific Chat, Scientific File Browser, File Browser, OperatorConsole

OperatorConsole, , ENotebookENotebook, , LogAnalizer LogAnalizer

EVC usage scenarios:

remoting

(11)

Collaboration tools:

Advanced

LogAnalizer

Advanced

Advanced LogAnalizerLogAnalizer isis a web application

a web application whichwhich allows

allows toto selectselect variablesvariables from the logfiles

from the logfiles producedproduced by

by different control anddifferent control and supervision systems,

supervision systems,

and to plot them in a

user

user specifiedspecified temporaltemporal interval

interval

Advanced LogAnalizer is

technically a data warehouse, modular both

technically a data warehouse, modular both considering the considering the data loading and the data visualisation (Visual Data

(12)

About DANT...

Modular, web

-

based Log Analysis Tool

Capable of storing a large amount of

heterogeneous data into a unique,

homogeneous data warehouse (thus

facilitating information retrieval).

Allows (remote) data analysis

Doesn't require specific hardware or

software to run.

It can run properly on any

PC with a moderate amount of RAM and

CPU speed (IE 256MB 350Mhz).

(13)

DANT Goals

Problem Context

ElettraElettra Scientific Instruments generate Log files with different Scientific Instruments generate Log files with different

structures and store them in different locations. This obviously structures and store them in different locations. This obviously doesn't make any analysis easy.

doesn't make any analysis easy.

Current log files often includeCurrent log files often include significativesignificative redundancies and thus redundancies and thus

are easily prone to inconsistencies or

are easily prone to inconsistencies or unnecesaryunnecesary large storage large storage requirements.

requirements.

Goals

Collect all the instruments data logs into a unique, reliable and Collect all the instruments data logs into a unique, reliable and

efficient data base system. efficient data base system.

Permit therefore easy data analysis or comparisons, possibly with Permit therefore easy data analysis or comparisons, possibly with

other already available databases and tools (i.e.

(14)

Software Architecture

(15)

(16)

Software Features

-

1

Abstract and Layered development Approach

(Portability and

(Portability and CustomizationCustomization always kept in mind)always kept in mind)

Abstract DB Connections:Abstract DB Connections: DANT can run upon most DANT can run upon most

common DB systems, including for example: Oracle,

common DB systems, including for example: Oracle, MysqlMysql and

and PostgreSQLPostgreSQL..

Only software requirementOnly software requirement is to have ais to have a phpphp engineengine installed installed

on the (server) machine where the tool is intended to be run. on the (server) machine where the tool is intended to be run. Nowadays almost every web server supports

Nowadays almost every web server supports php. php.

Cross Platform Application: being developed as a web Cross Platform Application: being developed as a web

tool, DANT offers full cross

tool, DANT offers full cross plattfromplattfrom support. All of the support. All of the processing and data handling occur on the server side,

processing and data handling occur on the server side,

leaving to the connected clients just the task of displaying

some HTML code in a web browser.

(17)

Software Features

-

2

Low Cost

Entirely based upon Open Source Software.Entirely based upon Open Source Software.

Easy Deployment and Configuration

There's a single configuration file keeping track of all system There's a single configuration file keeping track of all system

variables (IE Database Type, Maximum allocable memory..). variables (IE Database Type, Maximum allocable memory..).

This Configuration file can be additionally generated during This Configuration file can be additionally generated during

installation, so that deployment and usage are straight forward. installation, so that deployment and usage are straight forward.

Modular (thus easily Scalable)

Both Input (indexing) and output (presentation, visualization) Both Input (indexing) and output (presentation, visualization)

extensions to the DB system are built as modules, thus allowing extensions to the DB system are built as modules, thus allowing users to choose (or develop) the most adapt modules to the

users to choose (or develop) the most adapt modules to the situation.

(18)

Input Module Example: BCS Logs

Implemented as a Python script

Receives data either from STDIN, named pipes

(remote Unix

syslog

) or Archived Log Files

Capable of reading and Indexing compressed

data (

gzipped

archives)

Has SMART Hierarchical Indexing capability

(no need to set up when new instruments are

added).

(19)

Sample Use Case

–

Step 1

(20)

Sample Use Case

–

Step 2

(21)

Sample Use Case

–

Step 3

(22)

Advanced Analysis

Combined Plots Drawing Capability

Used not to plot variable values in a timeline but rather Used not to plot variable values in a timeline but rather relate different variables together in a single plot,

relate different variables together in a single plot,

displaying the result of a certain expression.

Sample Use Case: Measure Machine Efficiency and find Sample Use Case: Measure Machine Efficiency and find out actual

(23)

DANT Status ...

Currently Supported Logs:

Currently Supported Logs: BeamlineBeamline Control System, Control System, Beamline

Beamline Access Control System Storage RingAccess Control System Storage Ring Average archived records Indexing (hourly):

Average archived records Indexing (hourly): 35.00035.000

Maximum Acceptable Fetched Values for a Single Plot

[upon current hardware]:

[upon current hardware]: 35.00035.000 Average Monthly Stored Records:

Average Monthly Stored Records: around 25.000.000around 25.000.000

Probably swap

-

in/swap

-

out of older data will be

required

Record Size:

Record Size: 94bytes94bytes

Average Compression Factor

(24)

…

and future developments

Integration with Other Analysis

Tools/Instruments

Add Java Support (applet analysis)Add Java Support (applet analysis)

PermitPermit RealtimeRealtime AnalysisAnalysis

New Presentation Modules development (IDL)New Presentation Modules development (IDL)

Integration with Other Data Sources

VUO (Virtual User Office)VUO (Virtual User Office)

BPM (BPM (BeamlineBeamline Position Monitor)Position Monitor)

(25)

Acknowledgements

Fulvio

Bille

’, Michele

Turcinovich

,

Roberto

Borghes

, Claudio

Scafuri

,

Massimiliano

Chiardone

, Kristina

Djinovic

from ELETTRA

Doriano

Lamba

, Alberto

Cassetta

from

CNR