Hadoop and Data Warehouse Friends, Enemies or Profiteers? What about Real Time?

(1)

Hadoop and Data Warehouse –

Friends, Enemies or

Profiteers? What about Real

Time?

Kai Wähner

[email protected]

@KaiWaehner www.kai-waehner.de

(2)

Disclaimer

!

These opinions are my own and do not necessarily

represent my employer

(3)

Key Messages

Big Data is not just Hadoop, concentrate on Business Value!

A good Big Data Architecture combines DWH, Hadoop and Real Time!

The Integration Layer is getting even more important in the Big Data Era!

(4)

Agenda

• Terminology

• Data Warehouse and Business Intelligence

• Big Data Processing with Hadoop

• Fast Data Processing in Real Time

(5)

Agenda

• Terminology

• Data Warehouse and Business Intelligence

• Big Data Processing with Hadoop

• Fast Data Processing in Real Time

(6)

Big Data Architecture

DWH / BI

Hadoop

Real Time

Big Data Architecture

(7)

DWH means analyzing structured data

http://www.exforsys.com/tutorials/msas/data-warehouse-database-and-oltp-database.html

(8)

Big Data means analyzing everything

• Store everything

• Even without structure

• Use whatever you need (now or later)

(9)

What is Big Data? The combined Vs of Big Data

V olume

(terabytes,

petabytes)

V ariety

(social networks,

blog posts, logs,

sensors, etc.)

Velocity

(realtime) V alue

X

(10)

Real Time

Wikipedia Definition:

• Real time programs must guarantee response within strict time constraints, often referred to as

"deadlines”. Real time responses are often understood to be in the order of milliseconds, and sometimes microseconds.

• The term "near real time” refers to the time delay introduced, by automated data processing or network transmission.

• The distinction between the terms "near real time" and "real time" is somewhat nebulous and must be defined for the situation at hand.

Hereby, for this talk, I define:

– Real time == response in nanoseconds || microseconds || milliseconds – Near real time == (response time > one second)

(11)

Agenda

• Terminology

• Data Warehouse and Business Intelligence

• Big Data Processing with Hadoop

• Fast Data Processing in Real Time

(12)

Big Data Architecture

DWH / BI

Hadoop

Real Time

Big Data Architecture

(13)

DWH vs. BI

• Data Warehouse (DWH)  Storage

• Business Intelligence (BI)  Analytics

• Both terms are often used as synonym, i.e. when someone talks

about a DWH, this might include analytics

• BI can be used without a DWH

(14)

Typical DWH Process

http://wikibon.org/blog/not-your-fathers-data-analytics/

A DWH is „Business Case driven“:

• Reporting

• Dashboards

• Drill Down Analytics

Different DWH Options:

• Enterprise DWH ( == EDW)

• Department / Project DWH

• Embedded BI (into Applications)

(15)

BI == Reporting + Statistics + Data Discovery

DWH

BI

(16)

BI Visualization

(17)

DWH

• SQL: e.g. MySQL

• MPP: e.g. Teradata, EMC Greenplum, IBM Netezza

– Scale very well (almost linear), very high performance, hardware / software costs

also increase a lot

BI

• Microsoft Excel

• BI Tools: e.g. TIBCO Spotfire, Tableau, MicroStrategy

Hint: Good BI tools

• allow data discovery / visualization using different sources, not just DWH

• are easy to use

Products

(18)

BI Tool Example: TIBCO Spotfire

(19)

DWH - Real World Use Case

http://spotfire.tibco.com/assets/bltef8a0cfc133c4cdf/zipcar.pdf

(20)

Embedded BI - Real World Use Case

https://www.jaspersoft.com/embeddedShowcase/periscope.html

(21)

Problems of a DWH

No flexibility / agility

• Just structured data

• Just some (maybe aggregated) history data

• Just good for already known business cases

Low speed

• ETL is batch, usually takes hours or sometimes even days

• No proactive reactions possible  “too late architecture”

High costs (per GB)

• Just selected data

• Too old data is often outsourced to archives

(22)

DWH vs. Big Data

http://martinfowler.com/bliki/DataLake.html

(23)

Agenda

• Terminology

• Data Warehouse and Business Intelligence

• Big Data Processing with Hadoop

• Fast Data Processing in Real Time

(24)

Big Data Architecture

DWH / BI

Hadoop

Real Time

Big Data Architecture

(25)

Why no longer DWH, but Hadoop?

Hadoop was built to solve problems of RDBMS and DWH…

Benefits of Hadoop:

• Store and analyze all data

– all data == not just selected (maybe aggregated) data – all data == structured + semi-structured + unstructured

 be more flexible, adapt to changing business cases

• Better performance (massively parallel)

• Ad hoc data discovery – also for big data volumes

• Save money (commodity hardware, open source software)

(26)

What is Hadoop?

Apache Hadoop, an open-source software library, is a

framework that allows for the distributed processing of

large data sets across clusters of commodity hardware

using simple programming models. It is designed to scale

up from single servers to thousands of machines, each

offering local computation and storage.

(27)

MapReduce

Simple example:

• Input: (very large) text files with lists of strings, such as:

„318, 0043012650999991949032412004...0500001N9+01111+99999999999...“

• We are interested just in some content: year and temperate (marked in red)

• The Map Reduce function has to compute the maximum temperature for every year

(28)

Hadoop Products

MapReduce

HDFS

Ecosystem

Features included

few many

Apache Hadoop

(29)

Hadoop Ecosystem

(30)

Hadoop Products

MapReduce

HDFS

Ecosystem

Hadoop

Distribution

few many

Apache Hadoop

Packaging

Deployment-Tooling

Support

+

(31)

Hadoop Distributions

(… more available)

EMR

(32)

Hadoop Products

MapReduce

HDFS

Ecosystem

Hadoop

Distribution

Big Data Suite

few many

Apache Hadoop

Packaging

Deployment-Tooling

Support

+ Tooling / Modeling

Code Generation

Scheduling

Integration

+

(33)

Big Data Integration Suite: TIBCO BusinessWorks

(34)

Hadoop Real World Use Case:

Replace ETL to improve Performance

“The advantage of their new system is that they can now look at their

data [from their log processing system] in anyway they want:

• Nightly MapReduce jobs collect statistics about their mail system such as spam counts by domain, bytes transferred and number of logins. Benefit: Improved speed compared to typical ETL.

• When they wanted to find out which part of the world their customers logged in from, a quick [ad hoc]

MapReduce job was created and they had the answer within a few hours. Not really possible in your typical ETL system.”

http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data

( no TIBCO reference)

(35)

• A lot of data must be stored „forever“

• Numbers increase exponentially

• Goal: As cheap as possible

• Problem: Queries must still be possible (compliance!)

• Solution: Commodity servers and „Hadoop querying“

Global Parcel Service

http://archive.org/stream/BigDataImPraxiseinsatz-SzenarienBeispieleEffekte/Big_Data_BITKOM-Leitfaden_Sept.2012#page/n0/mode/2up

Hadoop Real World Use Case:

Storage to reduce Costs

( no TIBCO reference)

(36)

DWH or Hadoop?

DWH Hadoop

Data Structured All data

Maturity Established in Enterprise New concepts

Tooling Installed, good

knowledge and experience

New tools, coding required, business can still use SQL-similar queries or same BI tool

Costs High (per GB) Low (per GB)

(37)

DWH plus Hadoop?

DWH and Hadoop complement each other very well

• Store all data in Hadoop (cheap per GB)

• ETL from Hadoop to DWH (expensive per GB)

• Create specific reports / dashboards in DWH (leverage existing products and knowledge)

• Do Ad Hoc (Big) Data Discovery directly in Hadoop, no DWH needed

Good BI tools support both, DWH and Hadoop!

For example, TIBCO Spotfire has connectors to:

• RDBMS (e.g. MySQL)

• MPP (e.g. Teradata, IBM Netezza, Greenplum)

• Hadoop (e.g. Hive, Impala)

• In-Memory (e.g. TIBCO ActiveSpaces, SAP HANA)

• ...

(38)

• Short term:

Use Hadoop (only) when you can save (a lot of) money or when you can not solve your business problem without Hadoop. A lot of things have to be improved, e.g. governance, security, performance, and tool support.

• Long term:

Hadoop can replace DWH (as you can create a DWH on top of Hadoop with SQL interface as of today)!

• Be aware:

A lot of other options emerged for analyzing big data besides Hadoop, e.g.

- Analytical databases with SQL interface (MemSQL, Citus Data) - Log Analytics (Splunk, TIBCO LogLogic)

- Graph databases (Neo4j, InfiniteGraph) - Cassandra, MongoDB, you name it...

Recommendation DWH vs. Hadoop vs. NoSQL

(39)

Vendors Strategy...

Hadoop vendors push Hadoop as DWH replacement

 Called e.g. „Enterprise Data Hub“ (Cloudera) or „Data Lake“ (Hortonworks)

http://gigaom.com/2013/10/29/clouderas-plan-to-become-the-center-of-your-data-universe/ http://hortonworks.com/wp-

content/uploads/downloads/2013/04/Hortonworks.ApacheHadoopPatternsOfUse.v1.0.pdf

(40)

Vendors Strategy...

MPP / DWH vendors add Hadoop support as complementary

addon to their DWH

 Reason (probably): Market pressure!

 Benefit: One platform (including tooling and support) for DWH and Hadoop („SQL-for-everything“)

(41)

Example: EMC combines DWH and Hadoop

http://wikibon.org/wiki/v/EMC_Integrates_Greenplum_DB_and_Hadoop_with_Pivotal_HD http://www.gopivotal.com/big-data/pivotal-hd

(42)

Example: Teradata combines DWH and Hadoop

http://www.teradata.com/Teradata-Enterprise-Access-for-Hadoop/

http://gigaom.com/2014/04/07/teradata-says-hadoop-is-good-for-business-but-for-how-long/

(43)

Hadoop evolving from Batch to Near Real Time

Hadoop is MapReduce == Batch (== hours, minutes, seconds)

• Good for complex transformations / computations of big data volumes

• Not so good for ad hoc data exploration

• Improvements: Hive Stinger (Hortonworks) etc.

Non-MapReduce processing engines added in the meantime (YARN makes it possible)

• Ad hoc data discovery (== seconds)

• Hive / Pig with Apache Tez replacing MapReduce under the hood for data processing

• New Query engines, e.g. Impala (Cloudera) or Apache Drill (MapR)

MPP vendors (e.g. Teradata, EMC Greenplum) also add own query engines

• Offer fast data exploration (without MapReduce)

• “SQL-for-everything”

Some Hadoop problems remain

• No good, easy tooling (Hadoop ecosystem)  might be solved next years

• Missing maturity (alpha / beta versions)  might be solved next years

• “Commodity hardware” no longer sufficient with these new emerging technologies (for instance: SQL-on- Hadoop solutions require a lot of memory)

• No “real time” (== ms, ns), but “near real time” (> 1 sec)  “too late architecture”

(44)

Agenda

• Terminology

• Data Warehouse and Business Intelligence

• Big Data Processing with Hadoop

• Fast Data Processing in Real Time

(45)

Big Data Architecture

DWH / BI

Hadoop

Real Time

Big Data Architecture

(46)

Real Time: “The Two-Second Advantage”

“ A little bit of the right information, just a

little bit beforehand – whether it is a

couple of seconds, minutes or hours – is

more valuable than all of the information

in the world six months later… this is the

two-second advantage.”

Vikek Ranadivé, Founder and CEO of TIBCO

(47)

The Value of Data decreases over Time

Time Business Event

Data Ready for Analysis

Analysis Completed

Decision Made

$$$$

$$$

$$

$

Action Taken

Event Processing

speeds action and

increases business

value by seizing

opportunities while

they matter

(48)

What is Big Data? The combined Vs of Big Data

V olume

(terabytes,

petabytes)

V ariety

(social networks,

blog posts, logs,

Velocity

(realtime)

X

Fast

Data

(49)

Complex Event / Stream Processing / In-Memory

Concepts

• Streams: Monitoring millions of events in a specific time window to react proactively

• Stateful: Collect, filter and correlate events with state to anticipate outcomes and react proactively

• Transactional: Highly performant transactional event processing

Products vs. Frameworks

• Products are mature, mission-critical, in production, e.g. TIBCO StreamBase, IBM InfoSphere Streams

• Open Source Frameworks, e.g. “Apache Spark” and “Apache Storm”

– Future will tell us about performance, tooling, support, etc.

– Can be combined with Hadoop

– Are complementary to Products such as TIBCO StreamBase

In-Memory

• Can also be used for “big data” (Terabytes possible!)

• Usually complementary, i.e. they can respectively have to be combined with stream processing / complex event processing

(50)

Stream Processing Architecture (Example: TIBCO StreamBase)

TIBCO StreamBase

Continuous Query

Continuous Query Processor

Ad Hoc Query

Alerts

Active Tables

Trading Signal Transaction Cost Orders / Executions

Market Data

Alert Setting

TIBCO Live Datamart

Snapshot AND always-live

updates

Connect to streams

Anticipate opportunities, proactive action

(51)

Example: TIBCO StreamBase Tooling

StreamBase Development Studio

• Visual Development

• Visual Debugging

• Feed Simulation

• Unit Testing

StreamBase Live Datamart

• Real Time Analytics and Visualization

• Ad hoc queries

• Alerts and Notifications

• Web, Mobile and API Integration

(52)

Some Fast Data Use Cases

• Algorithmic trading (trading)

• Fraud detection (finance)

• Predictive sensor analytics (manufacturing)

• Continuous network analytics (telecom)

• Omni-channel sales (retail)

FAST DATA use cases show up everywhere, not just in trading! 

 Let’s take a closer look at one example …

(53)

“The future of retail technology is real-time and event

driven.”

- CIO, leading retailer

(54)

MATCH

PSYCHOLOGICAL ROUTER

Spend 23%

Last Experience 76%

Browser Type 68%

43% 52%

18%

Location 28%

92%

79%

88%

Nice to see you again!

Inventory

(55)

LIVE PROMOTIONS & PRICING PROGRAM, CAMPAIGN & OFFER

MANAGEMENT

The Event-Driven Retail Reference Architecture

EVENT-DRIVEN VIRTUAL CUSTOMER IMAGE EVENT-DRIVEN INVENTORY FABRIC

EXTERNA L

EXTERNA

L CRM INVENTORY WAREHOUS STORE

EVENT-DRIVEN PAYMENTS

WALLET

LOYALTY POINTS

REAL-TIME CUSTOMER INTERACTION

SENTIMENT ANALYTICS &

ALERTING

(56)

Retailers want to treat their stores like warehouses...

Inventory

(from In-Memory)

Demand

(from the ESB)

Cross Sell

Aggression

Action

(dynamic rules)

(57)

Hadoop:

• Storage

• Complex computing (MapReduce)

Real Time:

• Immediate (proactive) reactions

• Monitor streaming data in Real Time

Example:

TIBCO StreamBase and its Apache Flume connector for reading streaming data from Hadoop / HDFS or to send streaming data to Hadoop / HDFS

Real Time plus Hadoop?

(58)

Use Case:

• Predict pricing movement in live bets

Hadoop:

• Store all history information about all past bets

• Use MapReduce to precompute odds for new matches, based on all history data

TIBCO StreamBase:

• Compute new odds in real time to react within a live game after events (e.g. when a team scores a goal)

• Monitor stream data in real time dashboards

Real Time plus Hadoop Real World Use Case

http://www.casestudyu.com/news/2014/04/04/7762652.htm

(59)

“WHEN 5 KEY BOOKIES RAISE

THE SAME ODDS IN A 5-SECOND

WINDOW, BET LESS”

? ?

? ? ? ? ? ? ?

Streaming Algorithm

(60)

Reference Architecture: Streaming Betting Analytics

Event Processing

MONITOR

REAL-TIME ANALYTICS AGGREGATE

HISTORICAL COMPARISON

Predictive odds analytics

Zero Latency Betting Analytics

GLOBAL, DISTRIBUTED INFRASTRUCTURE

Historical odds deviations

B

U

S

BETTING LINES

SCORES

NEWS

HADOOP

Context:

Historical

B

U

S

CACHE CACHE CACHE

Real-Time Analytics

CORRELATE

(61)

Recap: Big Data Architecture

DWH / BI

Hadoop

Real Time

Big Data Architecture

(62)

Off Topic

What about Integration?

(63)

Off Topic

Integration is no talking point in this

session… However:

It gets even more important in the future!

The number of different data sources and technologies increases even more than in the past

– CRM, ERP, Host, B2B, etc. will not disappear

– DWH, Hadoop cluster, event / streaming server, In-Memory DB have to communicate