Big Data Processing: Past, Present and Future

(1)

Big Data Processing:

Past, Present and Future

Orion Gebremedhin

National Solutions Director – BI & Big Data , Neudesic LLC.

VTSP – Microsoft Corp.

[email protected] [email protected]

@OrionGM

(2)

Big Data Processing:

Past, Present and Future

(3)

•  History and Fundamentals of Big Data Processing

•  SQL Server for Big Data, Past, Present and Future

•  Summary

Topics Covered

(4)

Characteristics of Big Data

Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your

database architectures.

To gain value from this data, you must choose

an alternative way to process it.

(5)

Characteristics of Big Data

The Vs of Big Data

•  Volume

• 

40 Zettabytes (43 Trillion Gigabytes) of data will be created by 2020. 300 Times increase from 2005

• 

Most companies in the U.S have at least 100Tb of data

•  Velocity

• 

NYSE captures 1TB of trade information every day

• 

The average modern car has over 100 sensors

•  Variety

• 

Nearly 420 Million wearable health monitors

• 

Over 4 Billion hours of video watched on YouTube everyday

(6)

(7)

History of Big Data

A big data cluster is a highly interconnected platform built from a collection of commodity parts.

(8)

Scale Up vs. Scale Out

Scale up (SMP) Scale out (MPP)

+

(n)

Upgrade components or buy bigger server each time Add nodes to the cluster

Multiprocessor system where processors share resources :

•  Operating System (OS),

•  memory,

•  I/O devices

and connected using a common bus

Multiple processors, each processor using its own OS and memory and communicating with each other using some form of messaging interface

(9)

Notable milestones in Commodity hardware

CDC 6600 by Control Data Corporation. "The 6600 CPU had multiple functional units which could operate simultaneously (i.e., in parallel), allowing the CPU to overlap instructions' execution times“..

http://en.wikipedia.org/wiki/CDC_6600

A Beowulf cluster (1990s) is a computer cluster of what are normally identical, commodity-grade computers networked into a small local area network with libraries and programs installed which allow processing to be shared among them.

http://en.wikipedia.org/wiki/Beowulf_cluster

(10)

Some Applications of Big Data

Big Data supercomputers are pattern explorers.

•  Shopping Patterns

•  Sensor and Intelligent devices Data analytics

•  Social Network associations and suggestions

•  Predictive analytics

•  Crime investigation

(11)

(12)

SQL Server for Big Data

(13)

SQL Server Optimizations

(14)

About Analytics Platform System

! 

SQL Server Parallel Data

Warehouse

Microsoft HDInsight PolyBase

Microsoft Analytics Platform System

(15)

APS Growth Topology

Base Unit Scale Unit Extension Base Unit

(16)

Introducing the Microsoft Analytics Platform System

•  Relational and non- relational data in a single appliance

•  Enterprise-ready Hadoop

•  Integrated querying across Hadoop and PDW using T- SQL

•  Direct integration with Microsoft BI tools such as Microsoft Excel

•  Near real-time performance with In- Memory Columnstore

•  Ability to scale out to accommodate growing data

•  Removal of data warehouse bottlenecks with MPP SQL Server

•  Concurrency that fuels rapid adoption

•  Industry’s lowest data warehouse appliance price per terabyte

•  Value through a single appliance solution

•  Value with flexible hardware options using commodity hardware

(17)

Deployment options and hybrid

solutions

(18)

Provides a single T-SQL query model for PDW and Hadoop with rich features of T-SQL, including joins without ETL

Uses the power of MPP to enhance query execution performance

Supports Windows Azure

HDInsight to enable new hybrid cloud scenarios

Provides the ability to query non- Microsoft Hadoop distributions, such as Hortonworks and

Cloudera

SQL Server Parallel Data

Warehouse Microsoft Azure

HDInsight

PolyBase

Microsoft HDInsight Hortonworks for

Windows and Linux Cloudera

Connecting islands of data with PolyBase

Resul t set Selec

t…

(19)

Microsoft’s modern data warehouse

Data Platform Analytics Platform System SQL Server 2014

Microsoft Azure HDInsight

(20)

Summary

•  Understand your data growth to determine when to

“Scale-Out”.

•  Determine the right tool for the workload you have.

(21)

Big Data Processing: Past, Present and Future

Big Data Processing:

Past, Present and Future

Orion Gebremedhin

National Solutions Director – BI & Big Data , Neudesic LLC.

VTSP – Microsoft Corp.

[email protected] [email protected]

@OrionGM

Big Data Processing:

Past, Present and Future

• History and Fundamentals of Big Data Processing

• SQL Server for Big Data, Past, Present and Future

• Summary

Topics Covered

Characteristics of Big Data

Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your

database architectures.

To gain value from this data, you must choose

an alternative way to process it.

Characteristics of Big Data

The Vs of Big Data

• Volume

40 Zettabytes (43 Trillion Gigabytes) of data will be created by 2020. 300 Times increase from 2005

Most companies in the U.S have at least 100Tb of data

• Velocity

NYSE captures 1TB of trade information every day

The average modern car has over 100 sensors

• Variety

Nearly 420 Million wearable health monitors

Over 4 Billion hours of video watched on YouTube everyday

History of Big Data

A big data cluster is a highly interconnected platform built from a collection of commodity parts.

Scale Up vs. Scale Out

Scale up (SMP) Scale out (MPP)

+

(n)

Notable milestones in Commodity hardware

CDC 6600 by Control Data Corporation. "The 6600 CPU had multiple functional units which could operate simultaneously (i.e., in parallel), allowing the CPU to overlap instructions' execution times“..

A Beowulf cluster (1990s) is a computer cluster of what are normally identical, commodity-grade computers networked into a small local area network with libraries and programs installed which allow processing to be shared among them.

Some Applications of Big Data

Big Data supercomputers are pattern explorers.

• Shopping Patterns

• Sensor and Intelligent devices Data analytics

• Social Network associations and suggestions

• Predictive analytics

• Crime investigation

SQL Server for Big Data

SQL Server Optimizations

About Analytics Platform System

!

!

!

!

!

!

!

!

Microsoft Analytics Platform System

APS Growth Topology

Base Unit Scale Unit Extension Base Unit

Introducing the Microsoft Analytics Platform System

Deployment options and hybrid

solutions

Provides a single T-SQL query model for PDW and Hadoop with rich features of T-SQL, including joins without ETL

Uses the power of MPP to enhance query execution performance

Supports Windows Azure

HDInsight to enable new hybrid cloud scenarios

Provides the ability to query non- Microsoft Hadoop distributions, such as Hortonworks and

Cloudera

Connecting islands of data with PolyBase

Resul t set Selec

t…

Microsoft’s modern data warehouse

Summary

• Understand your data growth to determine when to

“Scale-Out”.

• Determine the right tool for the workload you have.

Questions and Discussion

Questions?

•  History and Fundamentals of Big Data Processing

•  SQL Server for Big Data, Past, Present and Future

•  Summary

•  Volume

•  Velocity

•  Variety

•  Shopping Patterns

•  Sensor and Intelligent devices Data analytics

•  Social Network associations and suggestions

•  Predictive analytics

•  Crime investigation

! 

! 

! 

! 

! 

! 

! 

! 

•  Understand your data growth to determine when to

•  Determine the right tool for the workload you have.