• No results found

Next Generation Data Warehousing Appliances

N/A
N/A
Protected

Academic year: 2021

Share "Next Generation Data Warehousing Appliances"

Copied!
37
0
0

Loading.... (view fulltext now)

Full text

(1)

Next Generation Data Warehousing

Appliances

23.10.2014

Presentert av:

Espen Jorde, Executive Advisor

(2)

3.12.2014 2

(3)

Agenda

Affecto’s new Data Warehouse architecture

-

Pains and gains

DW/BI/BA Appliance

-

Why

-

What does it do

-

How does it solve your issues

(4)

At least 30% shorter projects

100 times faster response

50% less operational costs

(5)

Best practice – until now…

Data

Integration Stage Layer System Z System Y System X Enterprise Layer DM DM DM Visual Storytelling Performance Management Ad-hoc Analysis Reporting Data Sources ETL

(6)

Typical Business Intelligence Challenges

• Poor query performance • Long data load window • Refresh rate too rare

• Long project delivery time • Large backlog

• Heavy maintenance • Technical debt • Too complex solutions

• Non-integrated tools • Lack of documentation • Outdated architecture and

legacy solutions

• Business work-around • Temporary solutions • Manual workload • Quality issues

Quality and Risk

Performance

(7)

Affecto’s Reference model

Data Virtualization ,, Real-time Analysis Visual Storytelling Performance Management Ad-hoc Analysis Reporting Data Integration Streaming Real-time Cloud Big Data System Y System X Enterprise Layer DM DM DM Analytical Modeling Analytical Sandbox MDM Appliance(s)

#1

#2

#3

#3

VDM Cache Stage Layer Hadoop Stage Layer

Integrated Development Environment

(8)

Agenda

Affecto’s new Data Warehouse architecture

-

Pains and gains

DW/BI/BA Appliance

-

Why

-

What does it do

-

How does it solve your issues

(9)

 Specialized

 Built for a purpose

 Complete solution

 Easy to use

 Standardized interface

 Reasonably prized

Something:

What is an appliance?

(10)

Technology Is the Driving Force

Shaping the Future

(11)

Rapid and accelerating pace of change -

Those who lag behind will quickly disappear

(12)
(13)

Typical Business Intelligence Challenges

• Poor query performance • Long data load window • Refresh rate too rare

• Long project delivery time • Large backlog

• Heavy maintenance • Technical debt

• Too complex solutions • Non-integrated tools • Lack of documentation • Outdated architecture and

legacy solutions

• Business work-around • Temporary solutions • Manual workload • Quality issues

Quality and Risk

Performance

(14)
(15)
(16)

Typical Business Intelligence Challenges

• Poor query performance • Long data load window • Refresh rate too rare

• Long project delivery time • Large backlog

• Heavy maintenance • Technical debt • Too complex solutions

• Non-integrated tools • Lack of documentation • Outdated architecture and

legacy solutions

• Business work-around • Temporary solutions • Manual workload • Quality issues

Quality and Risk

Performance

(17)

Inside the IBM PureData System for Analytics

Optimized Hardware

+ Software

Hardware accelerated

AMPP

Purpose-built for high

performance analytics Requires no tuning

Snippet Blades ™

Hardware-based query acceleration with FPGAs

Blistering fast results Complex analytics

executed as the data streams from disk

Disk Enclosures

User data, mirror,

swap partitions

High speed data

streaming

SMP Hosts

 SQL Compiler  Query Plan  Optimize  Admin

(18)

Typical data load improvements

Acceptable throughput using ODBC (ETL)

-

2-4x

High throughput using Direct Loader (ETL)

-

10-75x

Extreme throughput using SQL Push-Down (ELT)

-

30-200x (approaching 1.5 mill trans/sec on a small appliance)

(19)

Query performance

Mid size tables – 10-100x query improvement

Queries on large data volumes – 100-1000x improvements

(20)

Sweet spot

Loading HUGE tables

Playing around with HUGE tables

-

Adding columns

-

Changing data

ELT

Querying on large volumes of detailed data

In-database Analytics (R, SPSS, SAS, Phyton, m.fl.)

In-database Geospatial

(21)

PureDat

a

Imp

act

Drive Productivity with In-Database Analytics

 Easy to Govern

 Lower infrastructure cost  Improved Analyst productivity  Simpler – No data movement

 Accurate - No sampling  Faster – In-Db scoring

Reduced

Effort

(22)

Typical Business Intelligence Challenges

• Poor query performance • Long data load window • Refresh rate too rare

• Long project delivery time • Large backlog

• Heavy maintenance • Technical debt

• Too complex solutions • Non-integrated tools • Lack of documentation • Outdated architecture and

legacy solutions

• Business work-around • Temporary solutions • Manual workload • Quality issues

Quality and Risk

Performance

(23)

Time to market?

-

Appliance not the main solution, but…

-

Simplified data modelling

-

Ease of creating new databases

-

Ease of duplicating data

-

Decreased time used on development and testing due to improved

performance

(24)

Agenda

Affecto’s new Data Warehouse architecture

-

Pains and gains

DW/BI/BA Appliance

-

Why

-

What does it do

-

How does it solve your issues

(25)

3.12.2014 Kilde: Kristian Ramsrud, GOBI 2014 25

(26)

3.12.2014 Kilde: Kristian Ramsrud, GOBI 2014 26

(27)

3.12.2014 Kilde: Kristian Ramsrud, GOBI 2014 27

(28)

3.12.2014 Kilde: Kristian Ramsrud, GOBI 2014 28

(29)

Appliance demands

(30)

Norsk Tipping - Goals

A flexible DWH which is easily loaded during the available time period.

A scalable solution enabling growth without tuning and refactoring.

A DWH providing good response times to end users without using aggregates. Thereby

reducing the number of scheduled standard reports and moving towards self-service BI.

Data that are easily accessible for the business users and analysts.

A DWH where data quality issues can be corrected automatically after the problem has been

identified and solved in the source system (easy to implement ETLs that can correct errors).

A DWH requiring little effort to operate (DBA

, system administration…)

At the end of the day:

Better decision support

Shorter time to market

(31)

Norsk Tipping - Requirements

Minimal effort to operate.

Minimal effort (migration) to get started and see gains, thereby creating room for

removal of complexity, refactoring etc. Gradual migration must be possible. NT

choose when to switch source/target for the different jobs.

Minimal effort to convert today’s Oracle relational database to the new format.

New environment must support several parallel test and production instances.

Backup and restore must be easy.

We need good failover solutions.

We must be able to access tables from e.g Toad.

We want to keep ETL developed in Informatica PowerCenter.

Possible to do import/export db objects to/from systems in a standard format.

Must support mixed workload, inserts simultaneously as analytical queries run.

Must support external workload scheduling.

Must cope with parallel execution of jobs.

(32)

Is Converting its Data Warehouse from Oracle to

IBM PureData for Analytics

(33)

What is the main trend evolving?

-

Consider the many new architectures that boost performance. If

your EDW is still on an SMP platform, make migration to

MPP a priority. Consider distributing your data warehouse

architecture, especially to offload a workload to a standalone

platform that performs well with that workload.

-

When possible, take analytic algorithms to the data, instead

(34)

Gartner: “By 2015, 15% of organizations will modernize their

strategies for IM capability and exhibit 20% higher financial

performance than their peers.

We all will have to change our data warehouse strategies.

Are you going to move while you have control, take action now –

reaping the benefits early?

or

Wait and see until the circumstances force you to fight your way

out of the problems?

(35)
(36)

Thanks!

bjorn.runar.nes@affecto.com

espen.jorde@affecto.com

(37)

At least 30% shorter projects

100 times faster response

50% less operational costs

References

Related documents

Future work will be the implementation of the parameter identification hardware block, which can improve the accuracy of SoC estimation by updating the cell model parameters in

Drawing on and internalizing the work of the DGI, the Fund created the SDDS Plus in 2012, at the time of the Eighth Review of the Fund’s Data Standards Initiatives (IMF, 2012a).”

FODPSP, the non-justiciability derives from the provision of section 6(6)(c) of the Constitution, to the effect that: “The judicial powers vested in accordance with the

List of Appendices Appendix 1: An overview of typologies of welfare states Appendix 2: Population in Mongolia, thousand persons Appendix 3: The Social Security Fund in the USSR:

Compared with CPU-OpenMP (parallel computing with multiple.. CPU cores) based analysis, GPU based FLIM analysis has significantly sped up lifetime calculations. b) A

* For health care institutions from pediatrics, occupational health medicine and medical practice for school children there aren’t any special preventive activities according to

Anna Lobosco, Ph.D., DD Program Development Specialist IV, NY Developmental Disabilities Planning Council (DDPC), 99 Washington Avenue,

If the increased risk of burning or immersion for insured vehicles compared with uninsured vehicles is due to insurance fraud, the data in Table 8 provide the basis for an estimate