Oracle Data Integration Solutions

(1)

(2)

Oracle

Data Integration Solutions

Joachim Jaensch

- Principal Sales Consultant

(3)

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for

information purposes only, and may not be incorporated into any contract. It is not a

commitment to deliver any material, code, or functionality, and should not be relied upon

in making purchasing decisions. The development, release, and timing of any features or

functionality described for Oracle’s products remains at the sole discretion of Oracle.

(4)

Agenda

4

1

Oracle OpenWorld 2014

Oracle Data Integration Solutions (

DIS

)

Oracle Data Integrator (

ODI

)

Oracle GoldenGate (

OGG

)

Oracle Enterprise Data Quality (

OEDQ

)

Oracle Enterprise Metadata Management (

OEMM

)

Oracle Data Service Integrator (

ODSI

)

Oracle Active Data Guard (

OADG

)

2

3

4

5

6

7

(5)

Oracle Data Integration Solutions (DIS)

Introduction

(6)

Oracle Data Integration Solutions 12c

Deliver real-time data integration for

Cloud

and

Big Data

Cloud

Apps

Database

• Real-time data replication; optimized for

Database 12c and Oracle Exadata

• End-to-end integrated with simplified

deployment

• Unified tooling for both structured data

sources and Hadoop / NoSQL

• Flexible deployment on-premise or in the

Cloud for heterogeneous systems

• Expanded support for 3rd party systems

and Oracle Applications in real-time data

integration and continuous availability

solutions

Oracle Data Integrator

Oracle GoldenGate

Oracle Enterprise Data

Quality

Oracle Data Services

Integrator

Oracle Metadata

Management

Oracle Active

Data Guard

(7)

Oracle Data Integrator

E-LT: Bulk Data Processing and Fast Data Transformation

Big Data

Cloud

Apps

Database

• Certified for leading

technologies to deliver fast

time to value

• High-performance, low cost

of ownership E-LT

architecture

• Lightweight deployment

• Flexible, easy to enrich

functionality

Oracle Data Integrator

High Performance E-LT

Declarative Design

Data Services

Extensible Knowledge Modules

(8)

Industry Leading Performance

Extremely Fast Execution and Reduced Cost

E-LT provides a flexible architecture for

optimized performance on any platform

Benefits

• Leverages set-based transformations

• Improves performance for loading,

no network hop

• Takes advantage of existing infrastructure:

hardware and software

Conventional ETL Architecture

Extract

_Load

Transform

Next Generation Architecture

“

E-LT

”

Load

Extract

(9)

Declarative Design

Improved Developer Productivity For Faster Time to Solution

9

• Easy

and

universal

for designing simple to complex mappings

• Robust

and

Reusable

: Create Mappings once for any physical design

• Quick

to define and refactor maps, high degree of automation

• Extensible

model for any data integration mechanism

• Benefits

• Significantly reduces the learning curve

• Shortens implementation times

• Simpler maintenance

ODI Declarative Design

Define

How

: Knowledge Modules

2

1 Automatically

Generate

Dataflow

Define

What

You Want

(10)

Reverse

Engineer

Metadata

Journalize

(CDC)

Load from

Source to

Staging

Check

Constraints

Integrate,

Transform

Data

Service

Knowledge Modules

Simpler Physical Design and Shorter Implementation Time

10

Oracle

Sqoop

Hive

HBase

Oracle

Merge

SAP ERP

SAP BW

Oracle

Datapump

Oracle

DBLink

JMS

External

Tables

Teradata

Oracle

Spatial

Siebel

eBusines

Suite

IBM DB2

Netezza

SCD

Key Architecture Benefits:

• Faster development and simpler

maintenance using templates

• Easy to extend and add new best

practices

• Enforces predictability and reduces cost

of ownership

Pluggable Knowledge Modules Architecture

(11)



Big Data

: Hive, HBase, Sqoop



Best for Oracle

: Merge, Spatial, Multi-Table

Insert, Optimizer Hints and more



Real-Time

: CDC with GoldenGate



Applications

: E-Business Suite, Siebel,

PeopleSoft, JD Edwards Enterprise One, JD

Edwards World, SAP ERP and SAP BW



Heterogeneous

: Optimizations for all major

RDBMS: IBM DB2, Microsoft SQL Server,

Teradata, Netezza, etc.

Applications

Legacy

Databases

Big Data

Cloud

Extended Connectivity

(12)

• Key Architecture Benefits: 100% Java, Open APIs, fast E-LT

D

A

B

File

C

C$_0

C$_1

LKM

IKM

I$

E$ (Errors)

CKM

IKM

RKM

JKM

Check-Load

Transform

Extract-Load

ODI Agent

Application & SOA

Business Intelligence

& Data Warehouse

ODI Agent may be deployed in

any part of the architecture

How ODI Works: E-LT Architecture

12

(13)

Integration Between GoldenGate and ODI 12

c

• Support real-time and bulk from the same

interface

• Deploy GoldenGate processes via ODI

Bulk Load Scenario

Real Time Scenario

• Enable GoldenGate journalizing on

interfaces without restructuring.

• Manage GoldenGate Capture/Delivery

instances in ODI Topology

• Set common GoldenGate parameters

graphically

(14)

Oracle GoldenGate

Log-based Change Data Capture and

Delivery

Active-Active Replication

Reliability and Transaction Integrity

Heterogeneous Source and Targets

Big Data/Java/FlatFile Integration

• High-performance,

low-impact real-time data

integration and

replication

• Timely data for

improved business

insight

• Continuous availability

for 24/7 operation

Big Data

Cloud

Apps

Database

(15)

(16)

Oracle Enterprise Data Quality

Profile, Explore, Audit

Parse, Standardize, Cleanse

Governance

Match, Merge, Enrich

Case Management

• Improves data accuracy,

usability and ‘fitness for

purpose

• Unified interface for

ease-of-use, lower TCO

• Depth of capability in

multiple data domains

reduces project risk

Big Data

Cloud

Apps

Database

Oracle Enterprise Data Quality

(17)

Data Quality Issues – A simple Customer Table

Name

Address

City

State

Zip

Phone

Email

Bob Williams

36 Jones Avenue

Newton

MA

02106

617 555 000

[email protected]

Robert Williams

36 Jones Av.

MA

02106

617555000

Burkes, Mike and Ilda

38 Jones av.

Nweton

MA

02106

617-532-9550

[email protected]

Jason Bourne, Bourne & Cie.

76 East 51

st

_Newton

_MA

_617-536-5480

_6175541329

…

Mis-fielded data

Matching Records

Typos

Mixed business and

contact names

Multiple Names

Non Standard formats

Missing Data

(18)

• Solution

–

Profile business data to find the bad data and assess quality over

time

–

Cleanse, match and merge data before it gets loaded into the Data

Mart or Data Warehouse

–

Repair data during batch processing flows

• Benefits

–

Stop the garbage-in-garbage-out cycle!

–

Improve the trust of data within business marts and data

warehouses

–

Enable IT to deliver value to the lines of business during every data

integration flow

–

Improve business efficiency with match and de-duplicate redundant

records

Profile, Cleanse and Govern Business Data

(19)

Profiling

– Understand the data first

Interactive exploration of data,

identifying distribution and

outlying values with

drill-downs

Identify and quantify issues in

data

(20)

Oracle Enterprise Data Quality – Standardization

• Standardize, Transform and Parse

• Split names and name elements

• Identify individuals and businesses

• Derive additional attributes

Name:

Dr Ellen Van Der Heijde

Title:

Dr

First:

Ellen

Last:

Van Der Heijde

Gender:

Female

Name:

Jalila Abdul-Alim (Do Not Call)

First:

Jalila

Last:

Abdul-Alim

Gender:

Female

Note:

Do Not Call

Title:

Mr

First:

R

Middle:

J

Last:

MacDonald

Gender:

Male

Name:

Mr RJ & Mrs FB MacDonald

Title:

Mrs

First:

F

Middle:

B

Last:

MacDonald

Gender:

Female

20

(21)

Oracle Enterprise Data Quality - Matching

• Match & Merge data from disparate sources

• Create ‘best’ record based on survivorship rules

Title:

Mr

First:

Robert

Last:

Fulmar

Gender:

Male

DoB:

12/05/1978

Phone:

555-120-1329

Address:

9405 Main St

Fairfax

Virginia

22030

First:

Bob

Last:

Fulmar

Gender:

Male

Email:

[email protected]

Title:

Dr

First:

R

Last:

Fulmer

DoB:

01/01/1978

Email:

[email protected]

Address:

9407 Main Street

Fairfax

VA

22031-4001

Title:

Dr

First:

Robert

Last:

Fulmar

Gender:

Male

DoB:

12/05/1978

Email:

[email protected]

Phone:

555-120-1329

Address:

9407 Main St

Fairfax

VA

22031-4001

(22)

Oracle Enterprise Metadata

Management

Report to Source Lineage

Impact Analysis, Model Versioning

Supports Metadata Standards

Annotations and Tagging

Big Data Enabled

Big Data

Cloud

Apps

Database

Oracle Enterprise Metadata Management

Trust your data

• Exploring Metadata

• Sarbanes-Oxley Act (SOX)

Compliance

• Stewardship Collaboration

• Lifecycle Change

Management

• Data Standardization & Data

Governance

(23)

Metadata Management

“

Metadata Management involves

managing data about other data,

whereby this ‘other data’ is

generally referred to as content.

- wikipedia

Oracle Confidential – Internal/Restricted/Highly Restricted 23

(24)

Value of Enterprise Metadata Management

Oracle Confidential – Internal/Restricted/Highly Restricted 24

ETL

BI

Dashboards

App

ETL

How was sales

figure calculated?

What will happen

if I change this

table?

What reports use

the mainframe

data?

Sys Admin

Executive

BI Developer

Where did

this data

come from?

Application User

Which reports

use this

customer data?

CDC

Hadoop

Data Lake

Data Steward

Can I trust the

sources of this

customer data?

ETL

Developer

Solves a significant pain point for a wide variety of business consumers and technical staff

I want to design an

experiment to measure the

success of a signup page.

What data do I have?

Data Scientist

(25)

Data Discovery

Analytics

Oracle Metadata Management Logical Architecture

Oracle Confidential 25

Actionable Information

Da

ta

Li

neage

Impac

t

A

n

aly

si

s

ETL

Data Quality

Data Integration

Data Streams

Integration

Data Enrichment

File Based

_Cloud

Data Sources

Big Data

Relational

Applications

Business

Glossary

Metadata

Repository

Semantic

Linking

(26)

Exploring Metadata



Text Search and Metadata Browser,



Model/Diagram Visualizers, Birds-eye

View, Metadata Reporting



Data Flow and Lineage Analyzer

Sarbanes-Oxley Act (SOX) Compliance



Versioning and Model Compare



URI Based Call back from BI reports or

Business applications

Stewardship Collaboration



Feedback Comments and Review Board



Metadata Tagging with Labels



Multimedia attachments (documents,

videos, presentation, code)



URL link references for e-mail, blogging,

social networking

Detailed Feature List

Lifecycle Change Management



Metadata Harvesting from any DM, DW,

ETL/DI, BI Tooling and Big Data/Hadoop



Enterprise Architecture Management

(Metadata Configuration Management)



Impact of Change

Data Standardization & Data Governance



Supports Variety of Metadata Standards



Annotations and Tagging



Business Glossary with Semantic Lineage

Oracle Integrated



Oracle Database



Oracle Data Integrator, Oracle GoldenGate



Oracle Business Intelligence Enterprise

Edition



Oracle Business Intelligence Applications

(ODI and older Informatica based OBIA)

(27)

Oracle Data Services Integrator

Federated Data Services

Data Virtualization

Bi-directional Reads & Writes

Integrates with Oracle SOA

Integrates with Oracle Business Analytics

• Easily develop data

services and ad-hoc

queries without

infrastructure changes

• Open and standards

based—SOA, Web

Services, Java, Service

Data Object

Big Data

Cloud

Apps

Database

Oracle Data Services Integrator

(28)

Oracle Service Oriented Architecture (SOA) – Product Mapping

(29)

• Consolidation

: Move data from distributed sources to central hub; consuming applications query hub database

• Data Virtualization / Federation

: Query and update distributed sources in-place and provide combined result

• Hybrid Approach

: Federation and Consolidation are combined based on best approach for individual use cases

Different Data Access Techniques

Source A

Consolidated Hub

Source B

ODI/OGG

Consumer

Consolidation only

Source A

Source

B

ODSI

Consumer

Data Virtualization only

Source A

Consolidated Hub

Source B

ODSI

ODI/OGG

Consumer

Source C

(30)

When to Use Which Integration Technology

Data Virtualization

Service Bus / Business Process

• Combine & normalize data access services

(Create, Read, Update, Delete)

• Orchestrate services to business processes

• Service Routing / Mediation

Data Virtualization

Consolidation

• Data cannot be replicated (security, ownership)

• Consolidation is not efficient for access pattern

• Real-time access is required and replication is not efficient

• Transactional updates

• Need to change model frequently

• Performance is priority

• Data needs to be offloaded from original source

• Replication is used for redundancy

(31)

Data Virtualization Use Cases

• Data Services Layer

–

Provide a clean data abstraction for SOA and application developers

–

Shield consumers from changes in data topology

• Augmenting Business Intelligence

–

Extend access to sources outside the data warehouse

• Rapid Prototyping of Data Integration

–

Quick creation of an interface to integrated data without building a consolidated store

• Cloud Data Integration

(32)

• Native SOA Data Integration

Native XML type system allows processing and passing of hierarchical & SOA data.

Based on open standards (XML, WS, JDBC/SQL, XQuery)

Tight integration in Oracle SOA: Fast-path with OSB, Governance with OER

• Normalization and Reuse through Data Services

Multi-layer approach to normalizing and transforming source data

• Highly Scalable, Industrial-Strength Performance

Linear scalability through WLS Clusters

Performance through caching, streaming and query optimization

• Fine-Grained Security and Auditing

Row and column-based data security

• Comprehensive Update Capabilities

Automated CRUD operations on data services

Distributed XA transactions through WLS Transaction Manager

(33)

Oracle Active Data Guard

Snapshot Database

Open for Read-Only Operations

Continuous Recovery

Using for GoldenGate Extract

Automatic Block Repair

Big Data

Cloud

Apps

Database

Oracle Active Data Guard

High Availability of the Database

• Main Component of

Maximum Availability

Architecture (MAA)

• Role Transition between

Source and Standby

Database in Case of

Failure

(34)

Note: A single DR copy may be

multi-purposed for different combinations of

the use cases described

SYNC

or

ASYNC

Data Guard &

Active Data Guard

Use Cases: Data Guard and Active Data Guard

Real-Time Data Protection and Availability for Oracle Database

New DB

Version

Standby First Patching,

Database Rolling Maintenance

Exact copy

of primary

Query & Report Offload

Open Read-Only

Snapshot

Standby

Convert to Test Database

(open read-write)

Single Command Refresh

Exact copy

of primary

Offload RMAN Backups

Exact copy

of primary

Disaster Recovery

Manual or Automatic Failover

Redo

Far Sync,

GoldenGate Downstream

Exact copy

of primary

Source for thin snaps/clones

Exact copy

of primary

Extract offload, source for

GoldenGate ALO mode

(35)

Production

Database

Continuous redo

shipping, validation & apply

Real-time Reporting

Fast Incremental

Backups

Physical Standby

Database

Read-write

Workload

Data Guard

(36)

Production

Database

Continuous redo

shipping, validation & apply

Real-time Reporting

Fast Incremental

_Backups

Active Standby Database

(physical standby open read-only)

Read-write

Workload

Active Data Guard

(37)

Best Data Protection, Auto-Repair, and High ROI

Active Data Guard

Zero data loss at any distance

Primary Data Center

DR Data Center

Automatic Block Repair

Enterprise Manager Cloud Control

Offload Fast

Incremental

Backups

Offload read-only

workload to open

standby database

37

(38)

Data Guard and Active Data Guard Feature Sets

Data Guard

(included with EE)

Data Protection with HA

• Zero or near-zero data loss protection

• Transparent – all datatypes, workloads

• Continuous data validation

• Detect silent corruption

• Automatic database failover

• Dual-purpose DR as test system

• Simple migrations and upgrades

• Oracle Enterprise Manager integration

Active Data Guard

(Option)

Advanced Protection with High ROI

• Zero data loss at any distance

• Automatic corruption repair

• Auto-replay of inflight transactions

• Offload transport compression*

• Offload read-only workload

• Offload read-mostly workload

• Offload fast incremental backups

• Automation for rolling upgrades

(39)

Active Data Guard: Disaster Recovery and Data Protection



High-performance, simple, drop-in physical replication for HA and DR



Zero data loss, unique data corruption protection, automatic failover, offload read-only workloads



Disaster recovery for all data types, applications and workloads

GoldenGate: Flexible HA, Maintenance and Migrations, Real-Time Data Integration



Logical replication for advanced replication requirements, target open read-write



Enable minimal/zero downtime upgrades and application migration using 2

nd

_{database copy}



Heterogeneous platforms, bi-directional replication, transformations, subsetting, multiple topologies

(40)

Active Data Guard

Oracle Products for Data Movement

• Optimized Set-based transformation

• Highest latency for real-time feeds

• Heterogeneous

• Supports highly complex transformation

• Not invasive on source

• Fastest read and write

• Oracle Database only

• No transformation; no partial sets

• Unidirectional

• Not Invasive on source

• Very fast read and write

• Heterogeneous or homogeneous

• Basic transformation and partial sets

• Uni- and Bi-Directional

GoldenGate

Data

Integrator

(41)

Resources

41

Oracle Data Integration Oracle Data Integration

ORCL DataIntegration

blogs.oracle.com/dataint

OracleGoldenGate

egration

Oracle Data

Integrator

Oracle

GoldenGate

Oracle

Enterprise

Data Quality

Oracle Enterprise

Metadata

Management

Oracle Data

Services

Integrator

http://www.oracle.com/us/products/middleware/data-integration/overview/index.html

Data Integration

Oracle OpenWorld 2014

(42)

Questions and Answers

42 Oracle OpenWorld 2014

(43)

(44)