www.bileader.com 1
Best Practices in Creating a
Successful Business Intelligence
Program
Wayne W. Eckerson
•
Wayne Eckerson
•
Wayne Eckerson
•
BI thought leader
•
Founder, BI Leadership Forum
•
Director, BI Leadership
•
Former Director of Education and
Research at TDWI
www.bileader.com 3
BI Success Framework
Agenda
CULTURE
Data Treated as a Corporate Asset
P er for man ce M ea su reme n t Fact -b ased D ecision s PEOPLE An aly st s
Casual and Power Users
D at a D ev elo p er s ORGANIZATION B u sin ess -or ien ted BI Emb ed d ed An aly st s
Analytical Center of Excellence
ARCHITECTURE B ott om -up Top -d own Sandboxes PROCESS Cross-functional Collaboration D ev elo p ment M et h od s Pr oj ect M an ag eme n t DATA Unstructured Structured In te rn al Exte rnal
Appendix
•
Self-service BI
•
Evolving DW architecture
•
Designing dashboard
displays
Business value of BI
Personalized recommendations based on history Personalized online games based on playing habitsBest time to buy ;
average fare by airline, date & market
Customized energy management for customers
Proactive health
insurance that identifies at-risk patients
Optimize the siting of wind turbines by mining larger volumes of data
Analyzes data from viral “listening posts” to prevent pandemics.
Progressive
Custom auto premiums based on actual drivingwww.bileader.com 5
•
Strategic view
–
Use data to make smarter decisions
•
Tactical view
–
Reporting and analysis
•
Process view
–
“Data Refinery”
What is business intelligence?
BI workflow
DATA ACQUISITION
ETL, data modeling, data quality, data warehousing
Reports, analysis, dashboarding, predictive modeling,
DATA DELIVERY
DATA
INSIGHTS
ACTION
“DATA IN”
www.bileader.com 7
Data
Warehousing
Business
Intelligence
Performance
Management
1990s 2000s 2010 2015“Get the data”
“Use the data”
“Improve the business”
Analytics
Desktop query/reporting Extract, transform, load tools Data warehouses
Business intelligence suites Web query/reporting
On-line analytical processing (OLAP)
Packaged analytic applications Data virtualization
Dashboards and scorecards
Visual discovery Operational BI
Data integration suites
Cloud BI Predictive analytics Mobile BI Hive/Pig Hadoop Text analytics “Drive the business”
Bus
iness
V alue High2010’s
Reporting Analysis Prediction Monitoring Query, Excel, OLAP, Viz analysisDashboards, Scorecards
Statistics, data mining, optimization Static & Interactive
Reports “What happened?” “Why did it happen?” “What’s happening?” “What will happen?” Low = Reporting = Analysis A ll users B usi ness anal y st s E xecs, M grs, W orker s S tat ist ici ans Tools U sers
Waves of BI
www.bileader.com 9
Business Intelligence
Analytics Intelligence
Co
n
ti
nu
ous
In
tellig
ence
Co
n
ten
t In
tellig
ence
Data WarehousingAnalytic Sandboxes Ev en t-d riv en
Reports and Dashboards
MAD Dashboards Data Ware- housing End-User Tools Ev en t-Driv en Alerts an d Dashb oar d s Dashb oar d Alerts Ev en t d et ec tion an d c or rela tion C EP , S tr eam s Analytic Sandboxes Design Framework Architecture Reporting & Analysis
Excel, Access, OLAP, Data mining, visual exploration
K eyw or d sear ch , B I t oo ls, X q u er y, Hiv e, Ja va, et c. Map R ed u ce , XML sch em a, K ey -v alu e pair s, gr ap h n ot ation, e tc . HDF S, N oS Q L d at ab ses Exploration Power Users Ad hoc SQL, MDX, Java, Perl, Python
BI Framework 2020
Top down Bottom-up
www.bileader.com 10
Reporting & Monitoring (Casual Users)
Predefined
Metrics
Corporate Objectives and Strategy
TOP DOWN- “Business Intelligence”
Processes and Projects
Analysis and Prediction (Power Users)
Ad hoc
queries
Analysis Begets Reports Reports Beget Analysis Pros: - Alignment -Consistency Cons: - Hard to build - Politically charged - Hard to change - Expensive - “Schema Heavy” Pros: - Quick to build - Politically uncharged - Easy to change -Low cost Cons: - Alignment - Consistency - “Schema Light” Data Warehousing Architecture Non-volatile Data Analytics Architecture Volatile DataTop-down vs. Bottom-up BI
www.bileader.com 11
•
Requires strong leaders!
•
Who deliver value fast!
•
And manage change
•
Requires purple people!
Analytical Leaders
Dan Ingle, Kelley Blue Book
1. Incremental development 2. Teamwork
3. One size doesn’t fit all
Amy O’Connor, Nokia
1. Data is a product 2. Create an ecosystem 3. Change management
Darren Taylor, Blue KC
1. Create the right team 2. Get executive support 3. Deliver a quick win
Eric Colson, Netflix
1. Eliminate coordination costs
Tim Leonard, USXpress
1. Talk language of business 2. Let business present
3. Deliver quick wins
Kurt Thearling, CapitalOne
1. Curate the data
2. Statisticians are craftsmen 3. Manage model production
Ken Rudin, Zynga
1. Questions, not answers 2. Impacts, not insights 3. Evangelists, not oracles
www.bileader.com 13
People
Reporting & Monitoring (Casual Users)
Predefined Metrics
Corporate Objectives and Strategy
“Business Intelligence”
Data Warehousing Architecture
Casual Users
Processes and Projects
Analysis and Prediction (Power Users)
Ad hoc queries Analytics Architecture Power Users
BI/DW Developers
(Centralized)Analysts
(Decentralized)Data architects, ETL developers, report developers, data administrators, DW
administrators, technical architects, requirements specialists, trainers, etc.
Super users, business analysts,
statisticians, data scientists, data analysts
TOP-DOWN
Casual Users
•
Executives/Managers
•
Salespeople
•
Operations staff
•
Customers & suppliers
Users
Power Users (Bottom up)
• Super users
• Business analysts
• Analytical modelers
• Data scientists
Explore data
Model data
Source data
80%
Monitor metrics
Analyze anomalies
Drill to detail
80%
Reports/DashboardsExcel, OLAP, Visual Analysis, Mining 61% 24% Top Down Bottom up Top down Bottom up
www.bileader.com 15
80/20 rule
80% of the time
20% of the time
CASUAL USERS Task Tools Task Tools
Executives Monitor
MAD Dashboard
Create queries Super users
(Excel, BI search, voice-based BI)
Create plans Managers Analyze
Create reports Workers Drill to detail
POWER USERS Task Tools Task Tools
Super users Ad hoc reports Self-service BI
Monitor Analyze Drill to detail
MAD Dashboard Business analysts Explore, plan, viz Viz, Excel, SQL
Statisticians Create models Data mining tools Data scientists Explore Hadoop Java, Perl, Hive, Pig
EXERCISE: Map your users to tools
80% of the time
20% of the time
CASUAL USERS Task Tools Task Tools
POWER USERS Task Tools Task Tools
www.bileader.com 17
Organization (BICC)
Departments BOBI Team Business team - Evangelizes BI/analytics- Coordinates super users and depts - Defines best practices
- Defines and document metrics - Gathers requirements
- Governs reports Business sponsors
Executive team (Business sponsors)
- Approves roadmap - Secures funding - Prioritizes projects
Technical team
- Builds and maintains the EDW - Builds semantic layer for BI tools
- Creates complex reports and dashboards - Develops model management platform - Coordinates databases and servers w/ IT
Data governance User support Director of BI Super Users/ Analysts Purple team Data developers Statisticians
•
BI is a program, not a project
–
Unique people, organization, and processes
•
Multi-level organization
–
Sponsors: executive committee
–
Business: BI director, BOBI, Super
users/analysts
–
Technical: BI/DW developers
•
Federated organization
–
Centralized – BI director, BOBI, statisticians
–
Decentralized - Supers users and analysts
BICC organizing principles
www.bileader.com 19
Process
Reporting & Monitoring (Casual Users)
Predefined Metrics
Corporate Objectives and Strategy
“Business Intelligence”
Data Warehousing Architecture
Casual Users
Processes and Projects
Analysis and Prediction (Power Users)
Ad hoc queries Analytics Architecture Power Users
BI/DW developers
Analysts
1. Start with a business process 2. Gather requirements
3. Build reports/dashboards 4. Test and deploy
1. Business problem or opportunity 2. Hypothesize
3. Explore 4. Publish
TOP DOWN Monitor the Business
BOTTOM UP Explore the business
Architecture
Machine Data Web Data Hadoop Cluster Power User BI Server Casual User Operational System Operational System Free-standing Analytical sandbox Logical or Physical Data Mart Data Warehouse Virtual Sandboxes Top-down BI Bottom-up BI External Data Audio/video Data Streaming/ CEP Engine ETL Visual discovery tools ODS In ter act iv e d ash b oa rd swww.bileader.com 21
Power User Sandboxes
Machine Data
Web
Data Hadoop Cluster
Operational Systems (Structured data) Power User BI Server Casual User Operational System Operational System
Documents & Text
Free-standing sandbox or analytical data mart Data Mart Data Warehouse Virtual Sandboxes Top-down BI Bottom-up BI External Data Audio/ video Data Streaming/ CEP Engine Analytic platform or NoSQL database In-memory Sandbox ODS ETL
Analytical workflows
Analytical database (DW) Source Systems Analytical tools 6. Parse, aggregate “Capture in case it’s needed”1. Extract, transform, load
“Capture only what’s needed”
5. Explore data
8. Report and mine data
www.bileader.com 23
BI Tools Market
Visual Discovery BI FUNC TIO NALIT Y Enterprise Department SCOPE OF DEPLOYMENT Top-down Bottom-up R ep ort ing Dash boar ds An aly sis Mi ning Pixel Perfect Reporting Ad hoc Reports/ Dashboards Analyst Data Mining Workbench Operational Reports/ Dashboards Big Data Analytics Relational OLAP Multi-dimensional OLAP Desktop Analysis (e.g. Excel) Po w er User s Cas ual User s TYPE S OF USER SBI Tools Market
Visual Discovery BI FUNC TIO NALIT Y Top-down Bottom-up R ep ort ing Dash boar ds Ana ly sis Mi ning Pixel Perfect Reporting Ad hoc Reports/ Dashboards Data Mining Workbench Operational Reports/ Dashboards Big Data Analytics Relational OLAP Multi-dimensional OLAP Desktop Analysis (e.g. Excel) Po w er User s Cas ual User s TYPE S OF USER S Analyst Department Enterprisewww.bileader.com 25
Vectors
Visual Discovery BI FUNC TIO NALIT Y Enterprise Department SCOPE OF DEPLOYMENT Top-down Bottom-up R ep ort ing Dash boar ds Ana ly sis Mi ning Pixel Perfect Reporting Ad hoc Reports/ Dashboards Analyst Data Mining Workbench Operational Reports/ Dashboards Big Data Analytics Relational OLAP Multi-dimensional OLAP Desktop Analysis (e.g. Excel) Po w er User s Cas ual User s TYPE S OF USER S R eal -tim e Analy tic sData
Structured Semi-Structured Unstructured
Hadoop
(Archive, staging area for unstructured data, data pre-processing, batch reporting and mining, other)
Analytic Platform
(Terabyte data warehouses, free-standing sandboxes)
General Purpose RDBMS
(Data marts, small DWs, ODSs)
Low Latency
Summarized Data
High cost per TB
High Latency
Detailed Data
Low cost per TB
www.bileader.com 27
Challenges: Reconcile opposites
Top
Down
Bottom
Up
An aly tic al Ma tu rity R ep or tin g Dashb oar d s Mode ling “Analytical Competitor” High Business Value
A
naly
sis
“Analytical Potential” Moderate Business Value “Pockets of Analytics”
Moderate Business Value
“Flying Blind” Low Business Value
An aly tic al Cultur e C os t cen ter Tac tic al resour ce Str at eg ic resour ce Miss io n critic al Data Maturity Spreadsheets and Access Databases Independent Data Marts & Warehouses
Enterprise Data Warehouse
Big Data Ecosystem
www.bileader.com Scale and Scope 29 An aly tic al Ma tu rity
Individual Departmental Enterprise Enterprise+
R ep or tin g Dashb oar d s Mode ling A naly sis An aly tic al Cultur e C os t cen ter Tac tic al resour ce Str at eg ic resour ce Miss io n critic al Data Maturity Spreadsheets and Access Databases Independent Data Marts & Warehouses
Big Data Ecosystem Enterprise Data
•
Self-service BI
•
Evolving DW architecture
•
Designing dashboard displays
www.bileader.com 31
Self-service BI
www.bileader.com 33
Self service or self serving?
The truth about self-service BI
“Self-service BI requires a lot of
hand-holding!”
- Kevin Sonsky, Senior Director,
Business Intelligence, Citrix
www.bileader.com 35
Reporting & Monitoring (Casual Users)
Predefined Metrics
Corporate Objectives and Strategy
TOP DOWN- “Business Intelligence”
Processes and Projects
Analysis and Prediction (Power Users) Ad hoc
queries Data Warehousing
Architecture Non-volatile Data
Analytics Architecture
Volatile Data
Types of self-service BI tools
BI objects – BI mashboards (IT DRIVEN)
Visual discovery (ANALYST DRIVEN)
Self-service BI tools
www.bileader.com 37
CONSUMERS
View
Navigate
Modify
Explore
Model
PRODUCERS
Personalize
Assemble
Craft
Source
Develop
Mor e Ana ly ti cal Mor e In ter act iv e Mor e IT -ori en ted Mor e comple x Expose on DemandSelf-service hierarchies
Self service BI
View
Navigate
Modify
Explore
Model
Personalize
Assemble
Craft
Source
Develop
Ca
su
al
U
se
rs
Po
w
er
U
se
rs
Ca
su
al
U
se
rs
Po
w
er
U
se
rs
CONSUMERS
PRODUCERS
B
I
D
ev
elo
pe
rs
www.bileader.com 39
EXERCISE #2: Map users to self-service hierarchies
View
Navigate
Modify
Explore
Model
Personalize
Assemble
Craft
Source
Develop
CONSUMERS
PRODUCERS
Best practices
Adoption
Architecture
Design
Support
Change
Mgmt
Leadership Manage Expectations Marketing BI Roadmap Councils Newsletters Town Halls Campaigns Shut down legacyCertified reports
Use the tools
Training Support Feedback Tailored Super users Numeracy Help desk Monitoring Surveys Post-mortems Mentoring Requirements Ask right questions Map processes Roles Understand incentives Role mapping Tool fitting Agile
Scrums Sandboxes Prototypes Ask right people Composite Framework MAD Ad hoc Flexibility Layers of Abstraction Data Data access Metadata/ Reuse Coverage Performance Response times Query complexity User concurrency Timeliness Quality Atomic data Delivery
www.bileader.com 41
1990s
Local data warehouses, spreadmarts in each BU
2000-2007
Fully centralized enterprise data warehouses
Strategic DW evolution
BU 1 Data Whs 1 BU 2 BU 3 BU 4 Benefits: • Rapid deployment• Local control over priorities, resources
• Customization meets high % of requirements Challenges:
• Duplication of effort across BUs
• Redundant costs (HW, SW, support staff) • Silo mentality, lack of comm across Bus • Data integration difficult without scalable
Enterprise DWs Reports
BU 1 BU 2 BU 3 BU 4
Benefits:
• Reduce data redundancy
• Promotes communication between Bus • Resource efficiency (HW, SW, FTEs) Challenges:
• BUs compete over centralized DW resources • “One size fits all” solution meets lower % of
business requirements for each BU
• Data integration difficult due to limited
Spread Mart 1 Reports Reports Reports Spread Mart 2 Data Whs 2
www.bileader.com 43
2008+
Enterprise DW foundation with context-specific flexibility
Hybrid DW architecture
• Hybrid model leverages benefits of both centralized & decentralized models
• Challenges from both models still exist to a lesser degree…but
consciously accepted given the benefits
• Crucial focus on easier data integration to support growth of various businesses
• Requires a robust “social architecture” - lots of
communications and education, a strong BICC, a clear roadmap, strong business governance, and frequent meetings.
Enterprise DW DW Foundation
ODS tables, shared dimensions
BU-owned Data Marts
BU-specific data, filters, biz rules
BU 1 BU 2 BU 3 BU 4
Reports Reports Reports Reports
Ent DM 1 Ent DM 1 BU DM 1 BU DM 1 BU DM 1 Enterprise Data Marts
www.bileader.com 45
•
Less is more!
•
Make every pixel count
•
Avoid decoration
•
Set standards
•
Tell the story of the data
www.bileader.com 47
Tell the story
Tell the story (cont)
www.bileader.com 49
‘Wow, it’s so easy to see how
different patterns are selling, how
different colors are taking off, it’s so
great to have visibility into other
sides of the business, because
there’s lot of competition across our
divisions.’
Set standards
“It’s a rare type of chart, so when people see a spiderweb chart, I want
them to associate it with patient satisfaction. It creates a mental shortcut
for people if there’s some variation and a ‘personality’ that makes a metric
stand out visually.”
www.bileader.com 51
EXERCISE: Redraw this chart
What is your ROLE?
0% 10% 20% 30% 40% 50% 60% S o ft w a re v e n d o r re p re s e n ta ti v e B u s in e s s s p o n s o r o r u s e r C o n s u lt a n t o r s y s te m s in te g ra to r A c a d e m ic B I o r IT p ro fe s s io n a l Series1
