Next Generation Data Warehousing
Appliances
23.10.2014
Presentert av:
Espen Jorde, Executive Advisor
3.12.2014 2
Agenda
•
Affecto’s new Data Warehouse architecture
-
Pains and gains
•
DW/BI/BA Appliance
-
Why
-
What does it do
-
How does it solve your issues
At least 30% shorter projects
100 times faster response
50% less operational costs
Best practice – until now…
Data
Integration Stage Layer System Z System Y System X Enterprise Layer DM DM DM Visual Storytelling Performance Management Ad-hoc Analysis Reporting Data Sources ETL
Typical Business Intelligence Challenges
• Poor query performance • Long data load window • Refresh rate too rare
• Long project delivery time • Large backlog
• Heavy maintenance • Technical debt • Too complex solutions
• Non-integrated tools • Lack of documentation • Outdated architecture and
legacy solutions
• Business work-around • Temporary solutions • Manual workload • Quality issues
Quality and Risk
Performance
Affecto’s Reference model
Data Virtualization ,, Real-time Analysis Visual Storytelling Performance Management Ad-hoc Analysis Reporting Data Integration Streaming Real-time Cloud Big Data System Y System X Enterprise Layer DM DM DM Analytical Modeling Analytical Sandbox MDM Appliance(s)#1
#2
#3
#3
VDM Cache Stage Layer Hadoop Stage LayerIntegrated Development Environment
Agenda
•
Affecto’s new Data Warehouse architecture
-
Pains and gains
•
DW/BI/BA Appliance
-
Why
-
What does it do
-
How does it solve your issues
Specialized
Built for a purpose
Complete solution
Easy to use
Standardized interface
Reasonably prized
Something:
What is an appliance?
Technology Is the Driving Force
Shaping the Future
Rapid and accelerating pace of change -
Those who lag behind will quickly disappear
Typical Business Intelligence Challenges
• Poor query performance • Long data load window • Refresh rate too rare
• Long project delivery time • Large backlog
• Heavy maintenance • Technical debt
• Too complex solutions • Non-integrated tools • Lack of documentation • Outdated architecture and
legacy solutions
• Business work-around • Temporary solutions • Manual workload • Quality issues
Quality and Risk
Performance
Typical Business Intelligence Challenges
• Poor query performance • Long data load window • Refresh rate too rare
• Long project delivery time • Large backlog
• Heavy maintenance • Technical debt • Too complex solutions
• Non-integrated tools • Lack of documentation • Outdated architecture and
legacy solutions
• Business work-around • Temporary solutions • Manual workload • Quality issues
Quality and Risk
Performance
Inside the IBM PureData System for Analytics
Optimized Hardware
+ Software
Hardware accelerated
AMPP
Purpose-built for high
performance analytics Requires no tuning
Snippet Blades ™
Hardware-based query acceleration with FPGAsBlistering fast results Complex analytics
executed as the data streams from disk
Disk Enclosures
User data, mirror,
swap partitions
High speed data
streaming
SMP Hosts
SQL Compiler Query Plan Optimize AdminTypical data load improvements
•
Acceptable throughput using ODBC (ETL)
-
2-4x
•
High throughput using Direct Loader (ETL)
-
10-75x
•
Extreme throughput using SQL Push-Down (ELT)
-
30-200x (approaching 1.5 mill trans/sec on a small appliance)
Query performance
•
Mid size tables – 10-100x query improvement
•
Queries on large data volumes – 100-1000x improvements
Sweet spot
•
Loading HUGE tables
•
Playing around with HUGE tables
-
Adding columns
-
Changing data
•
ELT
•
Querying on large volumes of detailed data
•
In-database Analytics (R, SPSS, SAS, Phyton, m.fl.)
•
In-database Geospatial
PureDat
a
Imp
act
Drive Productivity with In-Database Analytics
Easy to Govern
Lower infrastructure cost Improved Analyst productivity Simpler – No data movement
Accurate - No sampling Faster – In-Db scoring
Reduced
Effort
Typical Business Intelligence Challenges
• Poor query performance • Long data load window • Refresh rate too rare
• Long project delivery time • Large backlog
• Heavy maintenance • Technical debt
• Too complex solutions • Non-integrated tools • Lack of documentation • Outdated architecture and
legacy solutions
• Business work-around • Temporary solutions • Manual workload • Quality issues
Quality and Risk
Performance
Time to market?
-
Appliance not the main solution, but…
-
Simplified data modelling
-
Ease of creating new databases
-
Ease of duplicating data
-
Decreased time used on development and testing due to improved
performance
Agenda
•
Affecto’s new Data Warehouse architecture
-
Pains and gains
•
DW/BI/BA Appliance
-
Why
-
What does it do
-
How does it solve your issues
3.12.2014 Kilde: Kristian Ramsrud, GOBI 2014 25
3.12.2014 Kilde: Kristian Ramsrud, GOBI 2014 26
3.12.2014 Kilde: Kristian Ramsrud, GOBI 2014 27
3.12.2014 Kilde: Kristian Ramsrud, GOBI 2014 28