Mastering the Velocity Dimension of Big Data

(1)

BigData

Mastering the Velocity

Dimension of Big Data

Emanuele Della Valle

DEIB -‐ Politecnico di Milano

[email protected]

(2)

BigData

Agenda

•

It's a streaming world

•

Mastering the velocity dimension with

informaEon ﬂow processing

•

Hands-‐on on esper and the Event Processing Language

•

Stream Reasoning: adding the variety dimension

•

What about Veracity?

(3)

BigData

It's a streaming World! 1/2

•

Oﬀ-‐shore oil operaEons

•

Smart CiEes

•

Global Contact Center

•

Social networks

•

Generate data streams!

(4)

BigData

It's a streaming World! 2/2

•

What is the expected Eme to failure when that

turbine's barring starts to vibrate as

detected in the last 10 minutes?

•

Is public transportaEon

where the people are?

•

Who are the best available agents to

route all these unexpected contacts

about the tariﬀ plan launched yesterday?

•

Who is driving the discussion

about the top 10 emerging topics ?

E. Della Valle, S. Ceri, F. van Harmelen, D. Fensel It's a Streaming World! Reasoning upon Rapidly Changing InformaEon. IEEE Intelligent Systems 24(6): 83-‐89 (2009)

(5)

BigData

Requirements 1/8

A system able to answer those queries must be able to

•

handle

massive datasets

–  A typical oil producEon plaborm is equipped

with about 400.000 sensors

–  Telecom data is the most pervasive data

source in urban are, in Milano there are

1.8 million mobile users

–  A global contact centre of a Telecom

operator counts 500 millions of clients

–  Facebook alone has 1.1 billion

of acEve users

(6)

BigData

Requirements 2/8

•

process

data streams

on the ﬂy

–  The sensors on typical oil producEon

plaborm generates 10,000 observaEons per minute with peaks of 100,000 o/m

–  The mobile users in Milano generates 20,000 call/sms/data connecEons per minute with peaks of 80,000 c/m

–  A global contact centre receives 10,000 contacts per minute with

peaks of 30,000 c/m

–  Facebook, as of May 2013, observes 3 millions "I like" per minute

(7)

BigData

Requirements 3/8

•

cope with

heterogeneous dataset

–  The sensors on typical oil producEon

have been deployed over 10 years by 10s of diﬀerent producers

–  Tens of data sources are normally

needed to make sense of an urban phenomena

–  A global contact centre consists in 100s of oﬃces owned by diﬀerent subsidiary companies engaged yearly

–  Each social network has its own data model, APIs, …

(8)

BigData

Requirements 4/8

•

cope with

incomplete data

–  10s of sensors and networking links broke down daily

–  Coverage is incomplete

–  Only standard cases are covered by fully

machine processable data records 100s of contacts per minute are manage ad-‐hoc

–  ConversaKons happen outside the social networks, too!

(9)

BigData

Requirements 5/8

•

cope with

uncertain data

–  Sensor out-‐of-‐operaKng range

–  Faulty sensors

–  Agents misunderstand, get Kred, …

–  Irony, sarcasm, …

(10)

BigData

Requirements 6/8

•

provide

reacKve answers

–  detecEon of dangerous situaEons

must occur within minutes

–  recommendaEons to ciEzens must

be performed in few seconds

–  rouEng a contact through each step of

the decision tree must take less than a

second

–  Search autocompleEng may need

to be updated every few minutes

(11)

BigData

Requirements 7/8

•

support

ﬁne-‐grained informaKon access

–  IdenEfy a turbine among thousands

–  Locate a bus among thousands

–  Contact an agent among thousands

–  IdenEfy an opinion maker among

thousands of inﬂuencers for a topic

(12)

BigData

Requirements 8/8

•

integrate

complex domain models

of

–  operaKonal and control process

–  various city aspects

–  contact management, contract types, agent skills, contactor proﬁles, …

–  topics, user proﬁles, …

(13)

BigData

Requirements (wrap up)

•

handle

massive datasets

•

process

data streams

on the ﬂy

•

cope with

•

cope with

incomplete data

•

cope with

uncertain data

•

provide

reacKve answers

•

support

ﬁne-‐grained informaKon access

•

integrate

complex domain models

Volum e' Ve lo ci ty ' Var ie ty ' Ve rac ity ' x x x x x x x x x x h?p://emanueledellavalle.org

13

(14)

BigData

Other domains

•

Intrusion detecEon

•

Fraud DetecEon

•

Emergency Response Services

•

TransportaEon and LogisEcs

•

Supply Chain OpEmizaEon

•

System monitoring

•

Click inspecEon

•

...

(15)

BigData

Mastering the Velocity dimension with

InformaEon Flow Processing (IFP) soluEons

(16)

BigData

Velocity: Say goodbye to DBMS

data

query

reply

The DBMS way

data

query

reply

The IFP way

(17)

BigData

IFP -‐ Gartner

The Gartner hype cycle

(18)

BigData

IFP -‐ Forrester

Forrester’s top 15 emerging tech to watch:

Now to 2018

(19)

BigData

Is there a market beyond hype?

•  Complex Event Processing

Market Worth $3,322M by 2018

(2014 Report by MarketsandMarkets)

•  Major players include:

–  Microsom –  IBM –  Oracle –  SAP –  Tibco –  .... h?p://emanueledellavalle.org

19

(20)

BigData

InformaEon Flow Processing

•  The IFP engine processes incoming ﬂows of informa1on according to a set of

processing rules

–  Processing is “on line”

•  Sources produce the incoming informaEon ﬂows, sinks consume the results of processing, rule managers add or remove rules

•  InformaEon ﬂows are composed of informa1on items

–  Items part of the same ﬂow are neither necessarily ordered nor of the same kind

•  Processing involve ﬁltering,

combining, and aggregaEng ﬂows, item by item as they enter the engine

Sources Sinks

IFP Engine

Informa1on Flows _{-‐-‐-‐-‐-‐} Informa1on Flows

-‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐ Rules -‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐ Rule managers h?p://emanueledellavalle.org

20

(21)

BigData

IFP: A bit of history of two approaches

Traditional DBMS Active DBMS DSMS Event-based Systems CEP h?p://emanueledellavalle.org

21

(22)

BigData

From Passive to AcEve DBMSs

•

Standard DBMSs

–

Purely passive:

Human-‐ac1ve

database-‐passive (HADP)

–

ExecuEon happens only when

asked by clients (through queries)

•

AcEve DBMSs

–

The reacEve behavior moves

(in part) from the applicaEon

to the DB layer…

–

…which executes Event CondiEon AcEon (ECA) rules

(23)

BigData

AcEve DBMSs

•

As a DBMS extension

–

Rules may only refer to the internal state of the DB

•

Closed DB applicaEons

–

Rules may support the semanEcs of the applicaEon,

but external sources of events are not allowed

–

But events may come from external sources …

(24)

BigData

Data Stream Management Systems (DSMS)

•

Data streams are (unbounded)

sequences of Eme-‐varying

data elements

•

Represent:

–

an (almost)

“

conEnuous

”

ﬂow of informaEon

–

with the recent informaEon being more relevant as it

describes the current state of a dynamic system

Eme

(25)

BigData

Data Stream Management Systems (DSMS)

•

The nature of streams requires a

paradigmaEc change*

– 

from persistent data

•

one Eme semanEcs

– 

to transient data

•

conEnuous

* This paradigmaEc change ﬁrst arose in DB community in the late '90s

(26)

BigData

ConEnuous SemanEcs

•

ConEnuous queries registered over streams that

are observed trough windows

_window

input streams

Registered

streams of answer

ConEnuous

Query

Dynamic

System

(27)

BigData

Event-‐based systems

•  Components collaborate by

exchanging informaEon about occurrent events. In parEcular:

–  Components publish noEﬁcaEons about the events they observe, or

–  they subscribe to the events they are interested to be noEﬁed

about

•  CommunicaEon is:

–  Purely message based

–  Asynchronous –  MulEcast –  Implicit –  Anonymous topic=ﬁre* & place=* topic=* & place=1st_ﬂoor

topic=ﬁre alarm & place=*

ﬁre alarm at 1st_ﬂoor

ﬁre training at 1st_ﬂoor

(28)

BigData

Complex Event Processing (CEP)

•

CEP systems adds the ability to deploy

rules

that describe how

composite events can be generated from primiEve (or composite)

ones

•

Typical CEP rules search

for

sequences of

events

–  Raise C if A→B

•

Time is a key aspect

in CEP

Rules -‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐ h?p://emanueledellavalle.org

28

(29)

BigData

Transforming vs. DetecEng Systems

Transforming rules

•  Deﬁne an execuEon plan combining primiEve operators

–  Each operator transforms one or more

input ﬂows into one or more output ﬂows

•  The execuEon plan can be deﬁned:

–  explicitly (e.g., through graphical

notaEon)

–  implicitly (using a high level language)

•  Omen used with homogeneous informaEon ﬂows

–  To take advantage of the predeﬁned

structure of input and output

DetecKng rules

•  Present an explicit disEncEon

between a detec1on and

produc1on phase

–  Detect a situaEon of interest

–  Produce a “complex” event to noEfy that the situaEon has happened

•  Usually, detecEon is based on a

logical predicate that captures

paFerns of interest in the history

of received items

(30)

BigData

Transforming languages

DeclaraKve

•

Specify the expected result

rather than the desired

execuEon ﬂow

•

Usually derive from

relaEonal languages

–  RelaEonal algebra / SQL

ImperaKve

•

Specify the desired

execuEon ﬂow

•

StarEng from primiEve

operators

–  Can be user-‐deﬁned

•

Usually adopt a graphical

notaEon

CQL/Stream:

Select IStream(*) From F1[Rows 5],

F2[Rows 10] Where F1.A = F2.A

h?p://emanueledellavalle.org

30

Arvind Arasu, Shivnath Babu, Jennifer Widom: The CQL conEnuous query language: semanEc foundaEons and query execuEon. VLDB J. 15(2): 121-‐142 (2006)

(31)

BigData

ImperaEve Languages

Aurora (Boxes & Arrows Model)

31

Daniel J. Abadi, Donald Carney, Ugur ÇeEntemel, Mitch Cherniack, ChrisEan Convey, Sangdon Lee, Michael Stonebraker, Nesime Tatbul, Stanley B. Zdonik: Aurora: a new model and

(32)

BigData

DeclaraEve languages

•

Specify a ﬁring

condi1on

as a pa?ern

•

Select a porEon of incoming ﬂows through

–

Logic operators

–

Content / Eming constraints

•

The

ac1on

part uses the selected items to produce new knowledge

TESLA / T-Rex

Define Fire(area: string, measuredTemp: double) From Smoke(area=$a) and last

Temp(area=$a and value>45) within 5 min. from Smoke Where area=Smoke.area and

measuredTemp=Temp.value

Gianpaolo Cugola, Alessandro Margara: TESLA: a formally deﬁned event speciﬁcaEon language. DEBS 2010: 50-‐61

Gianpaolo Cugola, Alessandro Margara: Complex event processing with T-‐REX. Journal of Systems and Somware 85(8): 1709-‐1728 (2012)

(33)

BigData

Hands-‐on Esper and EPL

•

See dedicated slides

(34)

BigData

Adding Variety to Velocity

(35)

BigData

Variety: Syntax changes, semanEcs not

•

Data comes from mulEple sources…

•

… in mulEple formats

•

The “SemanEc Web” is teaching us how to represent

informaEon (knowledge) in a standard way

–

RDF, OWL, Ontologies, SPARQL, ...

•

But the world in not staEc...

(36)

BigData

DSMS/CEP + SemanEc Web = Stream Reasoning

Requirement

DSMS CEP

SemanKc

Web

massive datasets

data streams

incomplete data

noisy data

reacEve answers

ﬁne-‐grained informaEon access

complex domain models

✗

The

Stream Reasoning Era

is coming

(37)

BigData

IFP: A bit of history (ConEnued)

Traditional DBMS Active DBMS DSMS Event-based Systems CEP Semantic Web Stream Reasoning h?p://emanueledellavalle.org

37

(38)

BigData

Research QuesEon

•

is it possible to

make sense in real Kme of

mulKple

,

heterogeneous

,

giganKc

and inevitably

noisy

and

incomplete

data streams

in order

to support

the

decision process

of extremely large numbers of concurrent user?

E. Della Valle, S. Ceri, F. van Harmelen, D. Fensel It's a Streaming World! Reasoning upon Rapidly Changing InformaEon. IEEE Intelligent Systems 24(6): 83-‐89 (2009)

(39)

BigData

Stream Reasoning feasibility (intuiEon)

•

Many relevant reasoning methods are not able to

deal with high frequency data streams

•

However, trade-‐oﬀ exists between the complexity of the reasoning

method and the frequency of the data stream

Complexity

Raw Stream Processing SemanEc Streams DL-‐Lite DL AbstracEon SelecEon InterpretaEon Reasoning Querying Re-‐wriEng Change Frequency PTIME NEXPTIME 104_Hz 1 Hz Complexity vs. Dynamics

Heiner Stuckenschmidt, Stefano Ceri, Emanuele Della Valle, Frank van Harmelen: Towards Expressive Stream Reasoning. Proceedings of the Dagstuhl Seminar on SemanEc Aspects of Sensor Networks, 2010.

AC0

(40)

BigData

PracEcal cases

•

10+ deployments

in Sensor Networks, and Social media analyEcs

BOTTARI

Winner of SemanKc Web Challenge 2011

City Data Fusion

Winner of IBM faculty award 2013

M. Balduini, I. Celino, D. Dell’Aglio, E. Della Valle, Y. Huang, T. Lee, S.-‐H. Kim, V. Tresp:

BOTTARI: An augmented reality mobile applicaEon to deliver personalized and locaEon-‐based recommendaEons by conEnuous analysis of social media streams. J. Web Sem. 16: 33-‐41 (2012)

(41)

BigData

Where to learn more about stream

reasoning

•

General Info

–

h?p://streamreasoning.org

•

W3C Community

–

h?p://www.w3.org/community/rsp/

•

Papers

–

h?p://scholar.google.com/scholar?hl=en&q=stream

+reasoning

41

(42)

BigData

What about Veracity?

(43)

BigData

Support for Uncertainty

•

Many applicaKons

deal

with imprecise (or even

incorrect) data from sources

–  E.g., sensors, RFID, …

•

The ability to

associate a

degree of uncertainty

to

informaEon items increases

expressiveness

–  To the content of items

•  Imprecise temperature reading –  To the presence of an item

(occurrence of an event)

•  Spurious RFID reading

•

Two orthogonal aspects

–  Support for uncertain input

•  Allows rules to deal with/ reason about uncertain input data

–  Support for uncertain output

•  Allows rules to associate a degree of uncertainty to the output produced

•  “Uncertain rules”

(44)

BigData

Where to learn more on

InformaEon Flow Processing

•

PhD course on Stream And Complex Event Processing

–

Lecturers

•  Emanuele Della Valle

•  Gianpaolo Cugola

•  Alessandro Margara

–

More info

•  h?p://bit.ly/PhD-‐course-‐IFP

(45)

BigData

Thank you!

Any QuesEon?

Emanuele Della Valle

DEIB -‐ Politecnico di Milano

[email protected]