Real Time Big Data Processing

Full text

(1)

Real Time Big Data Processing

Cloud Expo 2014

Ian Meyers

Amazon Web Services

(2)

Global Infrastructure

Compute Storage

AWS Global Infrastructure

Database App Services

Deployment & Administration

Networking

Analytics

(3)

Region

US-WEST (Oregon) EU-WEST (Ireland)

ASIA PAC (Tokyo)

ASIA PAC (Singapore) US-WEST (San Francisco)

SOUTH AMERICA (Sao Paulo)

US-EAST (Virginia) GOV CLOUD

ASIA PAC (Sydney)

Global Infrastructure

(4)

Availability Zone

Global Infrastructure

(5)

Ingestion | Integration

(6)

Elastic Block Store

High performance block storage device

1GB to 1TB in size

Mount as drives to instances with snapshot/cloning functionalities

IMAGE

Availability

99.99%

Durability

99.999999999%

Is a Web Store

Not a file system

No Single Points of Failure Eventually consistent

Paradigm Object store Performance Very Fast

Redundancy Across Availability Zones Security Public Key / Private Key

Pricing $0.095/GB/month

Typical use case

Write once, read many

Limits 100 Buckets, Unlimited Storage, 5TB Objects

Simple Storage Service

Highly scalable object storage for the internet 1 byte to 5TB in size

99.999999999% durability

(7)

Peak Requests: 1.2 Million / Second

14 40

102

262

762

1300

2100

0 500 1000 1500 2000

Q4 2007 Q4 2008 Q4 2009 Q4 2010 Q4 2011 Q4 2012 Today

Objects in S3

Bil lion s

(8)

Simple Storage Service

Highly scalable object storage 1 byte to 5TB in size 99.999999999% durability

Glacier

Long term object archive Extremely low cost per gigabyte

99.999999999% durability

Storage Lifecycle Integration

(9)

Complex Analytics

Unstructured Data

Parallel ETL

(10)

Elastic MapReduce

Managed, elastic Hadoop (1.x & 2.x) cluster Integrates with S3, DynamoDB and Redshift

Install Storm, Spark & Shark, Hive, Pig, Impala &

End User Tools Automatically Support for Spot Instances

Integrated HBase NOSQL Database

Analytics

Elastic MapReduce

Compute Storage

AWS Global Infrastructure Database App Services

Deployment & Administration

Networking

Analytics

(11)
(12)

Analytics Orchestration

Data Pipeline

Automatically Provision EC2 & EMR Resources Manage Dependencies & Scheduling

Automatically Retry and Notify of Success &

Failure Compute Storage

AWS Global Infrastructure Database App Services

Deployment & Administration

Networking

Analytics

Output: S3 file

Path: s3://trend-data/#{year-month-day}.csv Activity: EMR Transform

Hive Query: user-metrics.hql Frequency: Daily

Input: RDS Table

Table: User-Demographics

SQL Precondition: “Select last_update from table“ > #{YY-MM-DD}

Input: DynamoDB Table

Table: User-Event-Data-#{year-month}

Success Notification: metrics@example.com Failure Notification: emr-admin@example.com Delay Notification: : emr-admin@example.com

(13)

Structured Data Management

(14)

Database

DynamoDB

Provisioned throughput NoSQL database Fast, predictable, configurable performance Fully distributed, fault tolerant HA architecture Integration with EMR & Hive

RDS Dynamo

DB

Redshift Elasticache

Compute Storage

AWS Global Infrastructure Database App Services

Deployment & Administration

Networking

Analytics

(15)

Database

Redshift

Managed Massively Parallel Petabyte Scale Data Warehouse

Streaming Backup/Restore to S3

Load data from S3, DynamoDB and EMR Extensive Security Features

Scale from 160 GB -> 1.6 PB Online

RDS Dynamo

DB

Redshift ElastiCache

Compute Storage

AWS Global Infrastructure Database App Services

Deployment & Administration

Networking

Analytics

(16)

Amazon Redshift parallelizes and

distributes everything

Query

Load

Backup

Restore

Resize

Compute

Node

Compute

Node

Compute

Node

Leader

Node

Common BI Tools

JDBC/ ODBC

10GigE Mesh

(17)

V OLUME

VELOCITY

V ARIETY

(18)

V OLUME

VELOCITY

V ARIETY

(19)

GB TB PB

ZB

EB

Big Data is Moving Fast…

• IT Application logs, Infrastructure logs, Metering, Audit logs, Change and Configuration logs

• Web sites / Mobile Apps/ Ads Clickstream, User Engagement

• Sensor data

Weather, Smart Grids, Wearables

• Social Media, User Content 450MM+ Tweets/day

(20)

The Move to Real Time

Analytics Recency Requirements are moving from Daily to

Minute Granularity

Query Engine Approach

(Data Warehouse, YesSQL,

NoSQL databases)

Repeated queries over the

same well-structured data

Pre-computations like

indices and dimensional

views improve query

performance

Batch Engines (Map-

Reduce)

Semi-structured data is

processed once or twice

The “query” is run on the

data. There are no pre-

computations.

Streaming Big Data

Processing Approach

Real-time response to

content in semi-structured

data streams

Relatively simple

computations on data

(aggregates, filters, sliding

window, etc.)

Enables data lifecycle by

moving data to different

stores / open source systems

(21)

Application Services

Amazon Kinesis

Managed Service for Real Time Big Data Processing Create Streams to Produce & Consume Data

Elastically Add and Remove Shards for Performance Use Kinesis Worker Library to Process Data

Integration with S3, Redshift and Dynamo DB Compute Storage

AWS Global Infrastructure Database App Services

Deployment & Administration

Networking

Analytics

(22)

Data Sources

App.4

[Machine Learning]

AWS Endpoint

App.1

[Aggregate &

De-Duplicate]

Data Sources

Data Sources

Data Sources

App.2

[Metric Extraction]

S3

DynamoDB

Redshift App.3

[Sliding Window Analysis]

Data Sources

Availability Zone

Shard 1 Shard 2 Shard N Availability

Zone

Availability Zone

Amazon Kinesis

(23)

Analytics Tooling Integration

S3

Batch Write Files for Archive into S3

Sequence Based File Naming

Redshift

Once Written to S3, Load to Redshift

Manifest Support

User Defined Transformers

DynamoDB

BatchPut Append to Table

User Defined Transformers

Storm

Use Kinesis as a Spout

Provided as Open Source on GitHub

amazon-kinesis-connectors

Kinesis Connectors

S3 Dynamo

DB

Redshift Kinesis

Storm

(24)

EMR Integration with Kinesis

Read Data Directly into Hive,

Pig, Streaming and Cascading

from Kinesis Streams

No Intermediate Data

Persistence Required

Simple way to introduce real time

sources into Batch Oriented Systems

Multi-Application Support & Automatic

Checkpointing

(25)

Integrated Real Time Analytics

(26)

Thank you!

• Join the sessions in our AWS Lab

• Meet the team at our Amazon Web Services Partner

Village

@AWS_UKI for local AWS events & news

@AWScloud for Global AWS News and Announcements

Figure

Updating...

References

Updating...