The Evolution of Business Analytics and Big Data Integration on zenterprise

(1)

The Evolution of

Business Analytics

and Big Data Integration

on zEnterprise

[email protected]

Executive Architect

(2)

Agenda of Today

(3)

Topics in this Session

1. The critical need for Business Analytics

2. The challenges that arise from Business Analytics Environments

3. zEnterprise as a Business Analytics Hub

4. Integrating Big Data with zEnterprise, today and in the future

(4)

Analytics has evolved from a

Business Initiative to a Business Imperative

Respondents who believe analytics

creates a competitive advantage

57%

increase

More organizations are using

Analytics to create a competitive

advantage

2010

58%

2011

37%

Source: The New Intelligent Enterprise, a joint MIT Sloan

Management Review and IBM Institute of Business Value

analytics research partnership.

1.6x

Revenue growth

2.0x

EBITDA(*) growth

2.5x

Stock price appreciation

Source: Outperforming in a data-rich, hyper-connected

world, IBM Center for Applied Insights study conducted

in cooperation with the Economist Intelligence Unit and the IBM Institute of Business Value. 2012

Analytics Leaders are outperforming

their competitors in key financial

measures

4 (*) EBITDA = Earnings Before Interest, Raxes, Depreciation and Amortizaton

(5)

[There is a growing desire to “weave” Analytics into

the fabric of core business applications]

Sales Marketing Customer service Finance IT Operations Product development Human resources

Manage regulation,

compliance and risk

Grow, retain and

satisfy customers

Transform processes

in finance

Increase efficiency

in operations

5

Goal: achieve better business outcomes

(6)

Companies want to focus on high-value initiatives

in core Business Areas

• Advanced client

segmentation

• Leveraging customer

sentiment

analysis

• Reducing

customer churn

• Optimizing the

supply chain

• Deploying

predictive maintenance

capabilities

• Transform

threat & fraud identification

processes

Operations

• Enabling

continuous

planning and

forecasting

• Automating

financial and management

reporting

• Improving

visibility, insight and control

Finance

• Making risk-aware decisions

• Managing financial and operational

risks

• Reducing the cost of compliance

(7)

Shifting expectations impact severely the Business

Analytics Requirements

Customers

(e.g. external, Web)

Customer Service & Support

(e.g. call centers, sales personnel)

(8)

Data must be brought back under control in order

to meet their key Business Analytics objectives

Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, October, 2012 Base: 207 Global enterprise data analytics professionals

Our data is scattered in

too many places

(top response)

57%

What are the 3 largest

problems with your

firm’s Business

Analytics?

Above all else, our data

must be secure

(tied for top response)

89%

Please rate the importance of the following

with regard to your informational needs

When we access data it

should appear as a single

source of the truth

(9)

[The top Business Analytics solution priorities]

Select the top 3 improvements you would make to your current

business intelligence and analytics strategy in order of importance.

(Select up to three

)

Make the security of our data iron-clad to put it beyond the reach of malicious users

Combine data in new ways to improve customer service and create new business opportunities

Create immediate, normalized, and transparent views of our data

Reduce the effort / speed our ability to comply with regulatory reporting requirements

Reduce the total cost of ownership of the environment.

60%

54%

52%

51%

28%

(10)

The common approach to Business Analytics

Business Analytics Solution DW & BA Administrators

& Report Authors

Data Warehouse / Data Mart / ODS Division ‘A’

Marketing & R&D Users

Division ‘B’

Marketing & R&D Users

(11)

A look at the core elements for a Business

(12)

Today’s typical Enterprise Analytics Architecture

(13)

Business Analytics Life Cycle – Asynchronous and

Distributed

13 Scoring _Rules Operational Systems Staging Area Transformation Server Data Mover Departmental Data Marts Win/Unix/Linux server Batch Process Data Mining Segmentation Prediction Statistical Analysis Multi- Dimensional Analysis MIS System Budgeting Campaign management Financial Analysis Selling Platforms Customer Profit Analysis

CRM

Batch Process

Analytical Foresight

Online Queries & Reporting

Optimized Business Processes

Customer Support Claims Processing Underwriting Fraud Management Sales Effectiveness Marketing Staging Area Analytics Server Bulk Intel/Unix/Linux Server Win/Unix/ Linux Server Win/Unix/ Linux Server

(14)

Deployment Topology must be designed for each

individual Application  “Operational Model”

This architecture uses the loadbalancing and failover features that are built into the Cognos BI software architecture.

Reporting load is spread across two Reporting servers and should one fail, the other will take over. This is the basis for a scaleable architecture.

50 Concurrent Users

14

• On which node do we deploy the software (function)?

• On which node do we deploy the data?

• What are the physical characteristics of each node?

(15)

The Operational Model must be multiplied across

(16)

The Operational Model may be multiplied across

(17)

Potential issues which may limit Analytics ROI

Data transfer is limited to off peak times Insufficient processing power to support large or complex queries

(18)

[Executives recognize that the historic multi-platform

approach for Analytics impedes moving forward]

Please evaluate each of the following statements with regard to your analytics platform:

We are looking to cloud as a way to provide greater agility in meeting analytic requirements

Having different technical environments for transactional systems and analytic systems increases cost

Data sharing across platform silos makes it difficult to get an enterprise view

Data sharing across platform silos is excessively expensive

5 = Strongly Agree 4 = Agree

24% 44%

21% 49%

20% 42%

18% 42%

18

(19)

System z: from Data Hub to Analytics Hub

19

Cognos

Differentiate with {DB2 for z/OS, System z Hardware} to integrate analytics with real time OLTP

Superior End-To-End Analytics Lifecycle Integration

Better business response,

reduced data movement, reduced complexity, reduced configuration resources,

more accurate, more secure, more available

InfoSphere

Infoserver

InfoSphere

Warehouse

SPSS

CPLEX

WebSphere

CICS/IMS

SAP

ISV Bus. App.

ILOG

Leverage System z

Operational Data Store

Operational Transactional data

Operational Analytical Data

(20)

Data In

Business System / OLTP

Data Warehousing

• Data Warehouses

• Operational Data Stores

• Data Marts

Business Analytics

• Business Intelligence

• Predictive Analytics

IBM zEnterprise Analytics Hub 

End-to-End Integrated Foundation for modern analytics

Insight Out

Bring your analytics to your data:

• 70% of the data used for

analytics originates on

zEnterprise

Easily delivers on modern analytics

requirements for:

• Timely, accurate and

secure

• Superior availability, scalability

and performance

• Rapid deployment

and

expansion

• Reduced cost and complexity

Evolves with your business:

(21)

 A large percent of the data that is accessed for

analytics originates/resides on IBM zEnterprise

 2/3 of business transactions for U.S. retail banks

 80% of world’s corporate data



Businesses that run on zEnterprise

 66 of the top 66 worldwide banks

 24 of the top 25 U.S. retailers

 10 of the top 10 global life/health insurance

providers



1,300+ ISVs run zEnterprise today, more than 275 of

these selling over 800 applications on Linux

 The downtime of an application running on System z

equates to approximately 5 minutes per year

 The System z mainframe can run over a thousand

virtual Linux images on a single frame the size of a

refrigerator

[The role of zEnterprise in Business Analytics]

(22)

© 2014 IBM Corporation Real-Time Score/ Decision Out ETL Data In R-T, min, hr, wk, mth Business System / OLTP Data Warehouses, Operational Data Stores, Data Marts

Business Analytics •Business Intelligence •Predictive Analytics

zEnterprise: an

end-to-end,

integrated,

rock-solid foundation

for business

analytics

Analytics Out Customer Interaction

What Sets zEnterprise Apart for Analytics?

(23)

23

[What Sets zEnterprise Apart for Analytics?]

 High security (EAL5+)  High availability (99.999% )  Performs at 100% capacity  Prioritization of critical

queries & workloads

 Integrated disaster recovery

 Centralized, scalable infrastructure

 Virtualization  Start with your final

architecture

 Processors, disk,

memory added dynamically without

outage

 Pre-install then activate as needed  Flexible deployment options  Co-location of data warehousing, business analytics, transactional data  Reduced data movement

 Lower latency and near

real time data

 Rapid acceleration of complex queries

Superior availability, scalability and performance

Timely, accurate and secure information

Rapid deployment and expansion

Reduced costs and complexity

zEnterprise:

makes the technical challenges the easy part!

(24)

IBM Cognos Business Intelligence for zEnteprise

An analytics offering that provides the breadth of business intelligence

capabilities required to supports all users from mission critical to tactical to

strategic

 Provides the qualities of service required for mission critical analytics

 Enables organizations to deliver

BI as a service

 Provides access to the Big Data within:

Analyze the Variety of data the Business demands

24

– IBM BigInsights, Apache Hadoop

and Cloudera, Amazon Web

Services’ Elastic Map-Reduce

service (AWS EMR)

– IBM InfoSphere Streams for both

real-time processing and monitoring

(25)

Smart Analytics Cloud for large enterprises

This offering transforms the delivery of business intelligence into a service that is

readily available and affordable to corporate users across and beyond the enterprise

Report Authors Cloud Administrators

BI Administrators

Smart Analytics Cloud

WAS z/VM for Dev/Test Cognos Linux DB2 WAS z/VM for Production Cognos Linux DB2 Cloud boarding application Data Mart / Warehouse Finance Line of Business ‘B’ Marketing R&D

Data Mart / Warehouse

Senior Executives

Large enterprise organization

z/VM and IBM-Wave for Cloud management Ops Mgr for z/VM VM Perf Toolkit Tivoli Automation Tivoli Monitoring z/VM Tivoli usage metering

Linux Data Mart /

Warehouse

Operations

Data Mart / Warehouse

HR

Line of Business ‘A’

Marketing R&D

Data Mart / Warehouse

(26)

Solution Edition for Enterprise Linux Server

26

 The Enterprise Linux Server (ELS) is a System zEC12, zBC12,

z196 or z114 machine configured to run Linux-only workloads :

 IFL

− 2 to 13 for zBC12, 2 to 10 for z114 − 6 to 80 for z196, 6 to 101 for zEC12)

 Hardware maintenance for three to five years

 z/VM software with three to five years of subscription and support

 Intended to allow for VERY competitive acquisition

 Suse / RedHat zBC12 Linux licences promotions!

Solution Edition for Enterprise Linux Server provides attractive pricing for

consolidating Linux Applications and particularly for Database Server workloads

=

Reduced Cost VIRTUALIZATION

+

_EFFICIENCYENERGY AUTOMATION

+

SECURITY & RESILIENECY

+

Dynamic Infrastructure

Extended with “ELS for Analytics”

Extended with ”Cloud Ready for Linux on z”



For customers who have database workloads such as Oracle or DB2

LUW on Linux on System z



Provides Cognos and SPSS in a Linux environment that can access a

customer’s database on Linux on z, z/OS, or databases on separate

UNIX/INTEL servers that are network attached to the zEnterprise

environment



This offering can be extended with a PureData for Analytics network

attached server, to offer exceptional query speed to these customers

(27)

Predictive Analytics on zEnterprise

SPSS Decision Management for Linux on System z

SPSS Collaboration and Deployment Services for Linux on System z

 Employs both predictive models and business rules to automatically generate recommended actions

 Provides role-based models and security for in scoring, job scheduling, repository services, and integration

Predictive Customer Analytics Acquire Grow Retain Predictive Operational Analytics Manage Maintain Maximize Predictive Threat & Risk Analytics

Monitor Detect Control Decision Management Modeler Statistics

Collaboration and Deployment Services IBM SPSS Statistics for Linux

on System z

IBM SPSS Modeler for Linux on System z

 Apply math to decision making and research for commercial, government,

and academic users

 Data mining tool used for generating hypotheses and scoring  Text analysis for unstructured data to model consumer behavior

IBM SPSS Modeler with Scoring Adapter for zEnterprise

 Data mining tool used for generating hypotheses and scoring  Text analysis for unstructured data to model consumer behavior

 In-Transaction Scoring with DB2 z/OS: Embeds the Scoring Algorithm

Directly within the Transactional Application

(28)

Scoring Options with an OLTP Application

1) Historical Scoring

DB

Application

Historical Data

store Scoring and _Modeling Application R-T, min, hr, wk, mth

ETL

Copy

Delivers

a static

score

Batch Copy

(nightly, weekly, monthly)

score value insert to DB

2) Real-time Scoring

2b) In-DB scoring

1a) Web Services scoring

Call/Response

Delivers a

dynamic

score

OLTP App OLTP _DB 28

• Excellent accuracy,

speed and

performance while

reducing cost and

complexity

• Improves accuracy

by scoring new and

relevant data

directly within the

OLTP application

• Scales to large

data volumes to

improve accuracy

of data models

• Delivers the

performance

needed to meet

and exceed SLAs

of OLTP

applications

• Minimizes

demand on

network, HW, SW

and resources

Delivers better, more profitable decisions, using the latest data, at the point of customer impact

• Enables more informed customer interaction

• Improves fraud identification and prevention

(29)

Can you really scale to support the volume of an

OLTP application without impacting performance?

29

 Meets most demanding workload

 Required: 3K to 5K trans./sec.

 Measured: > 12K trans./sec.

 Provides best value

 Most accurate score is calculated in real time

0 2000 4000 6000 8000 10000 12000 14000 2 4 8 Int er n al T h ro u g h p u t Rat e (IT R/se c) Number of CP Engines SPSS Scoring Adaptor for DB2 for z/OS

z196 and zEC12 Throughput

z196 score zEC12 score

 Meets stringent SLA requirement

 Small additional CPU cost to score

(30)

Degree of Complexity

Co

mpe

titive

Ad

van

tag

e

Standard Reporting Ad hoc reporting Query/drill down Alerts Forecasting Simulation Predictive modeling Optimization

What exactly is the problem?

How can we achieve the best outcome?

What will happen next? What will happen if … ?

What if these trends continue? What actions are needed?

How many, how often, where? What happened?

Reporting

Stochastic Optimization

How can we achieve the best outcome including the effects of variability?

Descriptive

Prescriptive

Predictive

Business Analytics Styles and Solutions

Increasing prevalence of compute and data intensive parallel algorithms in commercial workloads driven by real time decision making requirements and industry wide limitations to increasing thread speed

ILOG BRMS Business Events

(31)

DB2 Analytics Accelerator for z/OS

Blending zEnterprise and Netezza technologies

31

Fast

Complex queries run up

to 2000x faster while

retaining single record

lookup speed

Cost Saving

Eliminate costly query

tuning while offloading

complex query processing

Appliance

No applications to

change, just plug it in,

load the data, and gain

the value

A high performance analytics accelerator appliance

for IBM zEnterprise, delivering dramatically faster

complex business analysis transparently to all users

Times Faster Query Total Rows Reviewed Total Rows

Returned Hours Sec(s) Hours Sec(s)

Query 1 2,813,571 853,320 2:39 9,540 0.0 5 1,908 Query 2 2,813,571 585,780 2:16 8,220 0.0 5 1,644 Query 3 8,260,214 274 1:16 4,560 0.0 6 760 Query 4 2,813,571 601,197 1:08 4,080 0.0 5 816 Query 5 3,422,765 508 0:57 4,080 0.0 70 58 Query 6 4,290,648 165 0:53 3,180 0.0 6 530 Query 7 361,521 58,236 0:51 3,120 0.0 4 780 Query 8 3,425.29 724 0:44 2,640 0.0 2 1,320 Query 9 4,130,107 137 0:42 2,520 0.1 193 13 DB2 Only DB2 with IDAA IDAA V3 Enhancements • High Performance Storage Saver • Incremental Update • New optimization of DB2 unloads and table/partition refreshes • High Capacity up to 1.28 PB for a single Accelerator

• More queries eligible for acceleration

…and DB2 for z/OS ~ A highly tuned database for OLTP & Analytics

3. z Ent erpris e as a Bus ines s Analy tic s H ub IDAA V4 Enhancements • Static SQL support • Multiple encodings • Multi-row fetch

(32)

IDAA can Serve as Enterprise Analytics

Accelerator

32

DB2

BW

non-BW

BA

Legacy

A single IDAA instance can be connected to multiple DB2

subsystems running various different applications.

• Applications access data exclusively via DB2, thus there is no

need to change the interface.

• Broad range of applications ensures fast ROI

(33)

[IBM zEnterprise Analytics System 9700/9710]

Mixed Workloads for Next Generation Business Analytics

33

The next generation of System z

analytics; an integrated solution of

hardware, software and services

that enables customers to rapidly

deploy cost effective game

changing analytics across their

business.

Preselected

All the necessary

components are

identified and integrated

into an end-to-end

solution

Pretested

Over 20 different

customer typical

configurations are

pre-sized and tested

Solution Priced

Aggressively priced for a

cost-effective add-on or

new deployment for

customers with critical

(34)

Accelerator

System z z196 / EC12

z/OS Operating System Stack

DB2 for z/OS

ETL/ELT

Flexibility in Critical Data Decision Systems

Operational Source Systems Common Metadata

Organized for simplicity

and functionality

34 Data Warehouse Data Mart ODS DS8870 • Delivers a PACKAGED SOLUTION • Leverages IBM industry leading software • Customized with optional components • Priced using ”Solution Edition” concept

IBM zEnterprise Analytics System 9700

(35)

ETL/ELT

Operational Source Systems

Cost Effective Critical Data Decision Systems

Data Warehouse Data Mart ODS

System z z114/zBC12

z/OS Operating System Stack

DS8870

DB2 for z/OS

DB2 Analytics Accelerator

Organized for simplicity

and functionality

35 • Delivers a PACKAGED SOLUTION • Leverages IBM industry leading software • Customized with optional components • Priced using ”Solution Edition” concept

IBM zEnterprise Analytics System 9710

(36)

DB2 for z/OS and

IBM DB2 Analytics Accelerator

• Simplification

• Single point of entry

• Reduced Data Movement

• High fidelity data

• Competitive

price/performance

• Dynamic routing for most efficient fit for purpose runtime architecture • Combines the strengths of both System z and Netezza technology • Single environment for security, logging, back-up, and recovery

* Advanced analytics is supported by Netezza but not currently exploited by DB2 Analytics Accelerator

OLTP Transactions Operational analytics Real time data ingestion

High concurrency Advanced analytics* DB2 Native Processing Standard reports Complex queries Historical queries OLAP

OLT

A

P = Single Workload-Optimized System for OLTP, DW, Historical Data

DB2 for z/OS with Analytics Accelerator

On-Line Transactional and Analytics Processing

(37)

What is Big Data?

37

Google can give you nearly 1.73 Billion options

Vendors have even more definitions

Here is how Gartner defines “Big Data”

“Big data is high-volume, high-velocity and high-variety

information assets that demand

cost-effective,

innovative information processing for

enhanced insight and decision making

”

Big Data

… data

from web logs,

RFID, sensor

networks, social

data, Internet text

and documents,

Internet search

indexing, call detail

records, astronomy,

atmospheric

science, genomics,

biogeochemical,

biological,

interdisciplinary

scientific research,

military surveillance,

medical records,

photography

archives, video

archives, and

(38)

The biggest Information Technology

inflection point in 35 Years!

38

Business models

under constant

pressure

Customers are more

demanding and

connected

Great relationships

trump great products

(39)

Beyond the 3 traditional “V’s”

(40)

Examples of Big Data Applications

40

Each day  12 Billion Terabyte of tweets

Imagine you could analyze them each day to figure out what people are saying about your products

Imagine that hospitals could take the thousands of sensor readings collected every hour per patient and identify subtle indications that the

patient is becoming unwell

Imagine you can decide whether someone qualifies for a mortgage in minutes,

by analyzing many sources of data, while the client is still on the phone or in the office

Imagine if green energy company could use petabytes

of weather data along with massive volumes of operational

data to optimize asset location and utilization

Imagine if law enforcement agencies

could analyze audio and video feeds in

real-time without human intervention to identify

suspicious activity

As new sources of data continue to grow in volume, variety and velocity, so

too does the potential of these data to revolutionize the decision making

process in every industry !

(41)

The “low risk” Big Data Starting Point

Where are organizations getting the most return on Big Data projects?

Source: IBM Global Big Data Online Survey

(42)

The results of a Webcast Survey …

42

 Have you already implemented or are you planning to implement any Big Data based initiatives within the next 6 months?

 How would you rate the value of being able to integrate insights from social media, telemetry, unstructured data into your analytics and decision making processes?

 Do you see the IBM System z platform as pivotal to the success of Big Data initiatives? Yes 31% No 69% Yes 90% No 10% High 50% Medium 36% Low 14%

You do not

need to build

your Big Data

(43)

Enterprise Integration and Governance…

The key to success of incorporating Big Data

43



Information Integration

–

Insights from Big

Data must be

incorporated

into the

Data Warehouse and

Analytics/Decision

engines



Information Governance

–

Companies need to

govern

what comes

in, and the insights

that goes out

Big Data Platform

Data Warehouse

Enterprise

Integration

Traditional Sources

New Sources

(44)

Big Data Exploration

Find, visualize, understand

all Big Data to improve

decision making

Enhanced 360

o

_View

of the Customer

Extend existing customer

views (MDM, CRM, ERP)

by incorporating additional

internal and external

information sources

Operations Analysis

Analyze a variety of machine

data for improved business

results

Data Warehouse Augmentation

Integrate Big Data and data warehouse

capabilities to increase operational

efficiency

Security/Intelligence

Extension

Lower risk, detect fraud

and monitor cyber

security in real-time

Typical Current Big Data Use Cases

(45)

Hadoop, the Popular Big Data Technology

45

 Open Source project of the

Apache Foundation

 Written in Java, originally developed by Doug Cutting

who named it after his son’s toy elephant

 Optimized to handle...

– Massive amounts of data through parallelism

– A variety of data (structured, unstructured, semi-structured) – Using inexpensive commodity hardware

 Great performance



HDFS  Reliability provided through replication

across different computers



MapReduce Engine  Perform Calculations

• Not to process

transactions (random

OLTP access)

• Not good when work

cannot be parallelized

• Not good for low latency

data access

• Not good for processing

lots of small files

• Not good for intensive

calculations with little data

(OLAP)

Hadoop complements

existing RDBMS technology !

• Compression applied

when data are shipped

(46)

46

Accelerators

Information Integration & Governance

Data Warehouse Stream Computing Hadoop System Discovery Application Development Systems Management

BIG DATA PLATFORM

InfoSphere Data Explorer Find, navigate, visualize all data

Accelerators

Speed time to value with analytic and application accelerators

InfoSphere BigInsights Bringing Hadoop to the enterprise

InfoSphere Data Warehouse

Delivers deep insight with advanced database analytics & operational analytics

Information Integration and Governance Governs data quality and manages the information

lifecycle

InfoSphere Streams

Analytics for data in-motion exploration

IBM BIG DATA PLATFORM

Logical platform with many physical deployment options

A Big Data Platform is an essential component of a

broader Big Data and Analytics Strategy

(47)

Highest Quality Data Services for Big Data

47

InfoSphere BigInsights

for zBladeCenter Extension

for exploration & online archiving

DB2 Analytics Accelerator

powered by Netezza technology

for reporting & analytics and operational analytics

DB2

for SQL & NoSQL transactions with enhanced Hadoop integration

in DB2 11 (beta)

zEnterprise

IMS

for highest performance transactions with enhanced Hadoop integration

in IMS 13 (beta)

• A NoSQL or “Not Only SQL” database

provides a mechanism for storage and

retrieval of data that use looser consistency

models than traditional relational databases

in order to achieve horizontal scaling and

higher availability

• Highly optimized for retrieval and

appending operations and often offer little

functionality beyond record storage

• Aimed at scalability and performance for

certain data models

• Useful when working with a huge quantity

of data (Big Data) when the data's nature

does not require a relational model

(48)

Enhancing Big Data Analytics with IMS and

DB2 for z/OS

48

Unstructured data sources

are growing fast

• New user-defined

functions and generic table UDF capability

Relational projection of IMS model

• Two significant needs:

1. Merge this data with trusted OLTP data from zEnterprise data sources

2. Integrate this data so that insights from Big Data sources can drive business actions

 DB2 is providing the connectors and the database capability to allow DB2 applications to easily and efficiently access Hadoop data sources

 IMS & DB2 provide the connectors and the database capability to allow BigInsights to efficiently access each data source