Oracle Big Data Handbook

ORACLG Oracle Press

Oracle Big ^Data

Handbook

Tom Plunkett Brian Macdonald

Bruce Nelson Helen Sun

Mark F. Hornick Keith Laker

Khader Mohiuddin Debra L. Harding

David Segleau

Gokula Mishra Robert Stackowiak

Mc Graw Hill

Education

New York

Chicago

San Francisco Athens London Madrid Mexico

City

Milan New

Delhi Singapore Sydney

^Toronto

Acknowledgments

^xxi

Introduction xxv

PART I

Introduction

1 Introductionto

Big

^{Data 3}

Big

^{Data 4}

Google's MapReduce ^Algorithm

^and

Apache Hadoop

⁵

Oracle's

Big

^{Data Platform 7}

Summary

¹⁰

2 The Value of

Big

^{Data 11}

Am I

Big Data,

^orIs

Big

^{DataMe? 12}

Big Data,

Little Data—It's Still Me 15

What

Happened?

¹⁶

NowWhat? 17

Reality,

^Check

^Please!

¹⁸

WhatDo You Make of It? ²⁰

Information Chain Reaction (ICR) 21

Big

^Data,

Big ^Numbers, Big

^{Business? 23}

Twitter 24

Facebook 25

Internal Source 25

ICR: Connect ²⁶

ICR:

Change

²⁷

xi

xii

Oracle

Big

^Data

Handbook

Wanted:

Big

^{Data Value 29}

Big

^Data

^Example

^1: Clinical Trial

Research

Within the Healthcare

Industry

³⁰

Example

^2:

Improvements

^{in Car}

Design

^for

Driver

Safety

Within the Automotive

Industry

³¹

Summary

³²

PARTII

Big ^{Data Platform}

3 The

Apache Hadoop

^{Platform 37}

Software vs. Hardware 39

The

Hadoop

Software Platform ³⁹

Hadoop

Distributions and Versions 40

The

Hadoop

Distributed File

System (HDFS)

⁴⁰

Scheduling, Compute,

^and

Processing

⁴³

Operating ^System

^{Choices 45}

I/O and the Linux Kernel 46

The

Hadoop

Hardware Platform 46

CPU

and Memory

⁴⁷

Network 47

Disk 48

Putting

^ItAll

Together

⁴⁸

4

Why

^an

Appliance?

⁵¹

Why

Would Oracle Create

a

Big

^Data

Appliance?

⁵²

What Isan

Appliance?

⁵³

WhatAre the Goals of

Oracle

Big

^Data

Appliance?

⁵⁴

Optimizing

^an

Appliance

⁵⁵

Oracle

Big

^Data

Appliance

^{Version 2 Software 56}

Oracle

Big

^Data

Appliance

^{X3-2 Hardware 58}

Where Did Oracle

Get

Hadoop Expertise?

⁶¹

Configuring

^a

Hadoop ^Cluster

⁶³

Choosing ^the

Core Cluster

Components

⁶⁴

Assembling

the Cluster 66

What Abouta Do-It-Yourself

Cluster?

67

Total Costs ofa

Cluster

69

TimetoValue 73

Howto Build Out

Larger

^{Clusters 75}

Can I

Add

Other Software

to Oracle

Big

^Data

Appliance?

⁷⁵

Drawbacks ofan

Appliance

⁷⁶

5 BDA

Configurations, Deployment Architectures,

^and

Monitoring

⁷⁹

Introduction 80

Big

^Data

Appliance

^{X3-2 Full Rack}

(Eighteen

^{Nodes) 82}

Big

^Data

Appliance

^X3-2 Starter Rack

(Six Nodes)

86

Big

^Data

Appliance

X3-2 In-Rack

Expansion

^(Six

Nodes)

89

Hardware Modificationsto BDA 89

Software

Supported

^on

Big

^Data

Appliance

^{X3-2 90}

BDA Install and

Configuration

^{Process 92}

Critical and Noncritical Nodes 94

Automatic Failover of the NameNode 95

BDA Disk

Storage Layout

⁹⁶

Adding Storage

^{to a}

Hadoop

^{Cluster 99}

Hadoop-Only Config

^and

Hadoop+NoSQL

^{DB 99}

Hadoop-Only Appliance

¹⁰⁰

Hadoop

^and

^NoSQL

^{DB 100}

Memory Options

¹⁰³

Deployment Architectures

103

Multitenancy

^and

Hadoop

in the Cloud 103

Scalability

¹⁰⁵

Multirack BDA Considerations 106

Installing

Other Softwareon the BDA 107

BDA in the Data Center 107

Administrative Network 107

ClientAccessNetwork 108

InfiniBand Private Network 108

Network

Requirements

¹⁰⁹

Connecting

^to Data Center LAN 111

Example Connectivity

Architecture 111

Oracle

Big

^Data

^Appliance

RestrictionsonUse 112

BDA

Management

^and

Monitoring

¹¹³

Enterprise Manager

¹¹⁵

Cloudera

Manager

¹¹⁷

Hadoop Monitoring

Utilities: Web GUI 117

Oracle ILOM 120

Hue 122

DCLI

Utility

¹²³

xiv

Oracle

Big

^{Data Handbook}

6

Integrating

the Data Warehouse and

Analytics

Infrastructure to

Big

^{Data 125}

The Data Warehouseas a Historic Database of Record 126

The Oracle Database as a Data

Warehouse

¹²⁷

Why

the Data Warehouse and

Hadoop

^Are

Deployed ^Together

¹²⁸

Completing

^the

^Footprint:

^Business

Analyst

^{Tools 130}

Building

^Outthe Infrastructure ¹³¹

7 BDAConnectors ¹³³

Oracle

Big

Data Connectors 134

Oracle Loader for

Hadoop

¹³⁶

Online Mode ¹³⁷

Oracle OCI DirectPath

Output

¹³⁹

JDBC Output

¹³⁹

Offline Mode ¹⁴⁰

Oracle

^Data

Pump Output

¹⁴¹

Delimited

^Text

Output

¹⁴¹

Installation of Oracle

Loader

^for

Hadoop

¹⁴²

Invoking

Oracle Loader for

Hadoop

¹⁴³

Input

^{Formats 144}

DelimitedTextlnputFormat

¹⁴⁵

RegexInputFormat

¹⁴⁶

AvrolnputFormat

¹⁴⁶

HiveToAvrolnputFormat

¹⁴⁶

KVAvroInputFormat

¹⁴⁷

Custom

Input

^{Formats 147}

Oracle Loader for

Hadoop

Configuration

^{Files 147}

Loader

Maps

¹⁵⁰

Additional

Optimizations

¹⁵²

Leveraging

^{InfiniBand 152}

Comparison

^to

Apache Sqoop

¹⁵³

Oracle

SQL

Connector for HDFS 153

Installation of Oracle

SQL

Connector for HDFS 157

HIVE Installation 159

Creating

^External

^Tables Using

Oracle

SQL

Connector for HDFS 160

ExternalTable

Configuration

^{Tool 161}

Data Source

Types

¹⁶¹

Configuration

^Tool

Syntax

¹⁶²

Required ^Properties

¹⁶³

Optional Properties

¹⁶⁴

ExternalTable

Tool for

^DelimitedTextFiles 164

Testing

^{DDLwith -noexecute 167}

Adding

^a New HDFS Filetothe Location File 167

Manual External Table

Configuration

^{1 68}

Hive Sources ₁₆₉

ExternalTable

Example

¹⁷⁰

Oracle Data

Pump

^{Sources 171}

Configuration ^Files

¹⁷³

Querying

with Oracle

SQL

Connector for HDFS 175

Oracle R Connector

for Hadoop

¹⁷⁶

Oracle Data

Integrator Application Adapter

^for

Hadoop

¹⁷⁷

8 Oracle

NoSQL

^Database 181

What Isa

NoSQL

^Database

System?

¹⁸²

NoSQL Applications

¹⁸⁴

Oracle

NoSQL

^{Database 185}

A

Sample

^{Use Case 186}

Architecture

188

Client Driver ₁₈₉

Key-Value

^{Pairs 190}

Storage

^{Nodes 192}

Replication

¹⁹³

Smart

Topology

¹⁹⁴

Online Elasticity

¹⁹⁴

No

Single

Point of Failure 195

Data

Management

¹⁹⁵

APIs 195

CRUD

Operations

¹⁹⁶

Multiple Update Operations

¹⁹⁶

Lookup Operations

¹⁹⁶

Transactions 197

Predictable Performance 198

Integration

¹⁹⁹

Installation and Administration 200

Simple

Installation 200

Administration

200

How Oracle

NoSQL

Database Stacks

Up

²⁰¹

Useful

^{Links 202}

PART III

Analyzing Information and Making ^Decisions

9 In-Database

Analytics: ^Delivering

^{FasterTimetoValue 205}

Introduction

²⁰⁶

Oracle's In-Database

Analytics

²⁰⁸

Why Running

In-Database Is So

Important

²¹¹

XVi

Oracle

Big

^{Data Handbook}

Introduction toOracle Data

Mining

and

Statistical Analysis

²¹¹

Oracle's In-Database Advanced

Analytics

²¹³

Oracle Data

Mining

²¹³

Introduction to R ²²³

Text

Mining

²³¹

In-Database

Statistical

Functions 236

Making

Bl Tools Smarter 237

Spatial ^Analytics

²³⁸

Understanding

^the

^Spatial

^{DataModel 239}

Querying

^the

Spatial

^{Data Model 239}

Using Spatial Analytics

²⁴⁰

Making

Bl Tools Smarter 241

Graph-Based Analytics

²⁴²

Graph

^{Data Model 242}

Querying Graph

^{Data 243}

Multidimensional

Analytics

²⁴⁵

Making

^{BlTools Smarterand Faster 246}

In-Database

Analytics:

Bringing

^{It All}

Together

²⁴⁷

Integrating Analytics

^into

Extract-Load-Transform

Processing

²⁴⁷

Delivering ^Guided Exploration

²⁴⁸

Delivering Analytical Mash-ups

²⁴⁹

Conclusion 249

10

Analyzing

^{Datawith R 251}

Introduction to

Open

^{SourceR 252}

CRAN, Packages,

^{and TaskViews 252}

GUIs and IDEs 255

Traditional R and Database Interaction vs. Oracle R

Enterprise

²⁵⁶

Oracle's

Strategic

^R

^Offerings

²⁵⁸

Oracle R

Enterprise

²⁵⁹

Oracle R Distribution 260

ROracle

261

Oracle R Connector for

Hadoop

²⁶¹

Oracle R

Enterprise:

^{Next-LevelView 261}

Oracle R

Enterprise Installation and Configuration

²⁶³

Using

^{Oracle R}

Enterprise

²⁶⁵

Transparency Layer

²⁶⁵

EmbeddedR Execution 276

Predictive

Analytics

²⁹³

OracleR Connectorfor

Hadoop

³⁰⁹

Invoking MapReduce ^Jobs

³¹¹

Testing

^{ORCH R}

Scripts ^Without

^the

Hadoop

^{Cluster 311}

Interacting

with HDFS from R 313

HDFS

Metadata Discovery

³¹⁴

Working

^with

Hadoop Using

the ORCH Framework 316

ORCH Predictive

Analytics

^on

Hadoop

³¹⁷

ORCHhive

319

Oracle RConnector for

Hadoop

^and

Oracle R

Enterprise

Interaction 322

Summary

³²²

11

Endeca

Information

Discovery

³²⁵

Why

Did Oracle Select Endeca? ₃₂₆

ProductSuitesOverview 326

Endeca

Information

Discovery

^{Platform 328}

Major

^{Functional Areas 328}

Key

^{Features 328}

Endeca Information

Discovery

and Business

Intelligence

³³¹

Differencein Roles and Functions 332

Bl

Development

^Process

vs. Information

Discovery Approach

³³³

Complementary

^{But Not Exclusive 334}

Architecture

335

Oracle Endeca Server 336

Oracle Endeca Studio 339

Oracle Endeca Integration

^{Suite 342}

Endeca on

Exalytics

³⁴³

Scalability

^{and Load}

Balancing

³⁴⁴

Unifying

Diverse Content Sets 348

Endeca Differentiator 349

Industry

^{Use Cases 349}

Hands-On with Endeca 351

Installation and

Configuration

³⁵¹

Developing

^{an Endeca}

Application

³⁵³

12

Big

Data Governance 357

Key

Elements of

Enterprise

Data Governance 359

Business Outcome 359

Information

Lifecycle Management

³⁵⁹

Regulatory ^Compliance

^{and Risk}

^Management

³⁶⁰

Metadata

Management

³⁶⁰

Xviii

Oracle

Big

^{Data Handbook}

Data

Quality Management

³⁶¹

Master and

Reference

Data

Management

³⁶¹

Data

Security

^and

Privacy Management

³⁶²

Business Process

Alignment

³⁶²

How Does

Big

^Data

Impact Enterprise

Data Governance? 363

Modeled

Data vs. Raw Data 363

Types

^of

Big

^{Data 366}

Applying

Data Governance to

Big

^{Data 370}

Leveraging Big

Data Governance 373

Industry-Specific

^{Use Cases 377}

Utilities

377

Healthcare 379

Financial Services 380

Retail ₃₈₂

Consumer

Packaged

^Goods

^(CPG)

³⁸³

Telecommunications ₃₈₄

Oil and Gas ₃₈₆

How Does

Big

^Data

Impact

Data Governance Roles? 388

Governance Roles and

Organization

³⁸⁸

An

Approach

^to

Implementing Big

^{Data Governance 389}

13

Developing Architecture

and

Roadmap

^for

Big

^{Data 393}

Architecture Capabilities

^for

Big

^{Data 394}

New Characteristics of

Big

^{Data 394}

Conceptual Architecture Capabilities

^of

Big

^{Data 395}

Product Capabilities

^{and Tools 397}

Making Big

Data Architecture Decisions 399

Architecture

Development

^Process

for

Realizing

Incremental Values ₄₀₀

Overview of Oracle Information

ArchitectureFramework 400

Overview of

Applied

^{OADP for}

Information Architecture 406

Big

^DataArchitecture

Development

^{Process 408}

Impact

^{on Data}

Management

and Bl Processes 415

Traditional Bl

Development

^{Process 415}

Big

^{Data and}

Analytics Development

^{Process 415}

Big

Data Governance 416

Traditional Data Governance Focus ₄₁₇

New Focus for Governance in

Big

^Data 417

Developing

^{Skills and}

^Talent

418

Data Scientist 418

Big

^Data

Developer

⁴¹⁹

Big

^Data

Administrator

₄₁₉

Big

^{Data BestPractices} 419

Align Big

Data Initiative with

Specific

Business Goals ₄₂₀

Ensurea

Centralized

IT

Strategy

for Standards and

Governance

420

Use a Center of

Excellence

to

Minimize

Training

^{and Risk 420}

Correlate

Big

Data with Structured Data 420

Provide High-Performance

and Scalable

Analytical

^{Sandboxes 420}

Reshape

^{the IT}

Operating

^Model 421

Index

₄₂₃