Oracle 11g and OBIEE Data Warehousing and Analytics

(1)

Analytics and DW Using Oracle 11g and OBIEE 10g

Mark Rittman, Director, Rittman Mead Consulting

(2)

Who Am I?

• Oracle BI&W Architecture and Development Specialist • Co-Founder of Rittman Mead Consulting

• Oracle BI&W Project Delivery Specialists

• 10+ years with Discoverer, OWB etc

• Oracle ACE Director, ACE of the Year 2005 • Writer for OTN and Oracle Magazine

• Longest-running Oracle blog

• http://www.rittmanmead.com/blog

• Chair of UKOUG BIRT SIG

• Co-Chair of ODTUG BI&DW SIG

• Second year of OU BI Masterclasses

(3)

Rittman Mead Consulting

• Oracle BI&DW Project Specialists • Consulting, Training, Support

• Works with you to ensure OBIEE project success • Small, focused team

• OWB, Oracle BI, DW technical specialists

(4)

The Oracle 11g Data Warehousing Platform

• The latest release of the Oracle database

• Enterprise Edition contains a large number of data warehousing features

• Partitioning

• Bitmap indexes

• Star transformations • Compression

• Options for in-database OLAP and Data Mining

(5)

The Oracle 10g Business Intelligence Platform

• Based around Oracle Business Intelligence Enterprise Edition • Dashboards, Answers, Delivers, Oracle BI Server

• Complete set of web-based query, reporting and analysis tools • Based on Siebel Analytics

• Heterogeneous database access

• Logical Business Model

• Data integration capabilities • Add-ons for data integration,

(6)

Oracle Fusion Middleware

• Middleware architecture for Oracle products • SOA-enabled J2EE environment

(7)

Hyperion Financial Management Tools

• Financial Management tools

• Hyperion Planning, Hyperion Performance Scorecard

• Hyperion Capital Expenses Planning, Hyperion Workforce Planning • Essbase OLAP Server

• Hyperion System 9 infrastructure

(8)

The Future Oracle BI&DW Platform

• Leverages the analytic capabilities of the Oracle database • In-database OLAP, data mining, partitioning, analytic queries • Scalable store of detail-level data for analysis

• Business logic and metadata layer provided by Oracle BI Server (OBIEE) • Semantic model, calculations,

time-series functions, hierarchies • Oracle BI Presentation Server

(OBIEE) plus Hyperion EPM tools provide Presentation Layer

• Three-tier BI&DW architecture leverages Oracle RDBMS and OBIEE/Hyperion toolset

(9)

The Data Services Layer

• Based around a core Oracle Data Warehouse

• Or several federated data warehouses

• Data Warehouse has three tiers

• Staging

• Atomic/Normalized

• Performance/Dimensional

• ETL process for loading DW

• OWB (default ETL tool)

• ODI (heterogenous sources /targets)

• Data Services Layer also provides real-time and SOA feeds to OBIEE

• For applications not yet in DW • For real-time reports

(10)

The Business Logic Layer

• Contains an integrated, logical model of the business • Provides an abstraction layer for the business

• Can be auto-generated by the ETL tool

• OWB 11gR2

• Maintained by the business

• Can connect to multiple sources in the data services layer

• The Oracle data warehouse • Direct application access • OLAP cubes

(11)

The Presentation Layer

• Data access layer, query tools, interfaces, MS Office integration • Oracle BI Answers, Dashboard, BI Publisher

• BI Office integration • SOAP (SOA) interface

• ODBC interface (BO, Cognos etc) • Hyperion tools

• May access Essbase directly, by-passing Business Logic Layer

(12)

Analytics and DW Using Oracle 11g and OBIEE 10g

• An in-depth look at the technologies behind Oracle’s future BI&DW platform • What’s new in Oracle Database 11g for data warehousing

• How the Oracle BI Server provides the business logic for your BI&DW platform • Creating the Enterprise Semantic Model, how the technology works and what

are its strengths and weaknesses

• Where to place calculations, summaries and hierarchies in your architecture

• How Oracle BI Presentation Server provides dashboards and reports (and what it’s missing in the current release)

• Where the Hyperion tools fit in

(13)

Seminar Agenda

• Introduction

• ETL and DW development using Oracle 11g and OWB 11g

Coffee

• Creating the BI foundation later with OBIEE BI Server

Lunch

• Reporting and Dashboards using OBIEE Presentation Server

Coffee

• Analytics and OLAP Analysis using Oracle OLAP and Oracle Essbase

(14)

Seminar Material

• Presentation slides are available in PDF format

• http://www.rittmanmead.com/presentations

• Leave a business card and I will forward on copies, answer questions

(15)

Participant Introduction

• The most important people in this room are you • It helps if I know a bit about you

• Your Oracle BI experience • Your course objectives • Where you work

• Introduce yourself to me, and to the other seminar participants

(16)

Seminar Participation

• Questions are welcome at any time

• During the session • During breaks

• At the end of the day

• Hopefully I’ll know the answer

• If not, I’ll get back to you

• Your feedback is appreciated

• What worked well, what could be better

• Flow of slides and product coverage • Fill in the evaluation

(17)

Analytics and DW Using Oracle 11g and OBIEE 10g

Mark Rittman, Director, Rittman Mead Consulting

(18)

Analytics and DW Using Oracle 11g and Oracle BIEE

Lesson 1 : ETL and DW development using Oracle 11g and OWB 11g (and ODI)

(19)

Objectives

• Understand the data warehousing features in the Oracle database • See what’s new in Oracle Database 11g for data warehousing

• Partitioning

• Cube Organized Materialized Views

• See what’s new in OWB11g

• SCD2 Handling that works (sort of) • DML Error Logging

• Get an introduction to ODI 10.1.3.4

(20)

The Oracle 11g Data Warehousing Platform

• Oracle Database 11g

• Oracle Warehouse Builder • Oracle OLAP Option

• Oracle Data Mining Option • Partitioning Option

(21)

The Future Oracle BI&DW Platform

(22)

The Data Services Layer

• Staging

(23)

Oracle 11g Enterprise Edition Code Features

• Bitmap Indexes • Materialized Views • Star Transformations • Cost-Based Optimizer

• Oracle Warehouse Builder Core Functionality

(24)

Oracle 11g Data Warehousing Options

• Partitioning • OLAP

• Data Mining

• OWB Data Quality Option • OWB Enterprise ETL Option • OWB ERP Connectors

(25)

Partitioning

• Use table partitions to

• make large (fact) tables more manageable • ‘age out’ old data by dropping partitions • add new partitions to receive new data • set unchanging tablespaces to

READ ONLY to optimize database backup

• speed data load by using partition exchange

• can reduce query time by

• partition elimination (pruning)

• parallel query Jan Mar

Feb

(26)

Oracle OLAP Option

• Adds multi-dimensional data types and a multi-dimensional engine • Stores data in indexed arrays

• Adds an OLAP API and an SQL access layer

• (Can be) Faster than star schemas for loading and querying • Supports time-series and dimensional queries

• Based on Express Server technology • Provides consistently-fast access to

(27)

Oracle Data Mining Option

• In-database data mining engine

• Accessed via Java and PL/SQL APIs • Wide range of algorithms

• Anomaly detection • Attribute importance • Association rules • Clustering

• Classification & regression

• Nonnegative matrix factorization • Structured & unstructured data

(text mining)

• BLAST (life sciences similarity search algorithm)

(28)

New Features in Oracle OLAP 11g

• The latest release of the Oracle Database OLAP Option

• Continues the objective of embedding OLAP in the database engine • SQL SELECT access

• Cubes stored in Analytic Workspaces

• Runs faster in 11g, uses SecureFiles instead of LOBs

• Metadata in the Oracle Data Dictionary

• Automatic creation of views over AW objects

• Optimizer now “AW view aware”

• CUBE_TABLE for easier direct SQL access • CUBE access now detailed in execution plans

(29)

Cube Organized Materialized Views

• Transparently enhance the query performance of BI applications • Data is managed in an Oracle cube

• Fast refresh • Fast query

• Manage a single cube instead of 100’s of individual MVs

• Applications query the base relational tables

• Oracle automatically rewrites the queries to use the OLAP cube

• Access to the summary data in the OLAP cube is fully transparent

(30)

Positioned As the Oracle DW Enhancer

• Transparent integration with SQL

• Swap-in replacement for Materialized Views

• Refreshed and loaded in the same way as Materialized Views

(31)

OLAP Cubes Now Feature in Explain Plans

• Cube operations now feature in execution plans

• CUBE_SCAN shows data being retrieved from a Cube Organized MV

explain plan for

select sum( u.units), c.market_segment_dsc, t.year_dsc, p.family_id from units_fact u, customer_dim c, time_dim t, product_dim p

where u.ship_to_id = c.ship_to_id and u.month_id = t.month_id and u.item_id = p.item_id

group by c.market_segment_dsc, t.year_dsc, p.family_id;

PLAN_TABLE_OUTPUT

---Plan hash value: 3517365018 --- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | --- | 0 | SELECT STATEMENT | | 19 | 2375 | 37 (6)| 00:00:01 |

| 1 | HASH GROUP BY | | 19 | 2375 | 37 (6)| 00:00:01 | |* 2 | HASH JOIN | | 19 | 2375 | 36 (3)| 00:00:01 | |* 3 | HASH JOIN | | 19 | 2071 | 33 (4)| 00:00:01 | |* 4 | CUBE SCAN PARTIAL OUTER| CB$UNITS | 20 | 2000 | 29 (0)| 00:00:01 |

| 5 | TABLE ACCESS FULL | TIME_DIM | 96 | 864 | 3 (0)| 00:00:01 | | 6 | TABLE ACCESS FULL | CUSTOMER_DIM | 61 | 976 | 3 (0)| 00:00:01 |

(32)

Demonstration

(33)

New In 11g - Partitioning Improvements

• Interval Partitions

• Automatically creates partitions as they are needed

• Saves scripting the creation of new partitions each day/week/month • Useful as long as you’re not using Partition Exchange Loading

(34)

New in 11g - Virtual Columns

• Define a column based on other columns in the table • Takes up no storage space

• Equivalent to calculated measures in Oracle OLAP • Partitioning can be based on these virtual columns

(35)

New in 11g - REF Partitioning

• Partition a table based on a column in another table • Partition an detail table by the master table primary key • Useful, but actually not relevant for star schemas

(36)

New in 11g - More Partitioning Combinations

• Two-dimensional partitioning schemes

• Partitions and sub-partitions

• Partition fact table primarily by date, then create list sub-partitions • Some combinations were available in 9i, 10g

(37)

Oracle Warehouse Builder

• Oracle’s Full DW Lifecycle tool

• Part of the Database Technology Stack • Now at version 11g Release 1

• Relational and OLAP Data Modeling

• Data Mapping • Process Flows

• Data Profiling, Auditing and Error Handling

• Metadata Management • Change Management • ERP Connectivity

(38)

New in OWB11g - DML Error Logging

• Table operators in mappings can have Error Tables configured • Erroneous rows (from 10.2 DB onwards) moved into error table

• Uses DML Error Logging

• Allows direct-path (set-based) loading without error roll-back • Traps constraint errors, others still fail

• Free option, no dependency on Data Quality Option

• Actually faster than (row-based) Data Rules

(39)

Demonstration

(40)

New in OWB11g - SCD2 Handling (That Works)

• SCD2 handling now works in OWB11gR1 (and OWB 10.2.0.3) • Changes to levels above base level now handled properly

• Hierarchy versioning

• All attributes in a row with trigger attributes must now also be triggers

• No ability to mix SCD1 and SCD2 in a single dimension

• Watch out for performance though

(41)

Recently Available - Experts on OTN

• XML Input, AW Import, Dimension Derivation Experts etc • Mapping from view/SQL SELECT

SELECT

DECODE(GROUPING(channel_desc) , 1, 'Multi-channel sum'

, channel_desc) AS Channel

, DECODE (GROUPING (country_iso_code) , 1, 'Multi-country sum' , country_iso_code) AS Country , TO_CHAR(SUM(amount_sold) , '9,999,999,999') SALES$ FROM sales , customers , times , channels , countries WHERE sales.time_id=times.time_id AND sales.cust_id=customers.cust_id

AND customers.country_id = countries.country_id AND sales.channel_id= channels.channel_id

AND channels.channel_desc IN ('Direct Sales', 'Internet') AND times.calendar_month_desc= '2000-09'

AND country_iso_code IN ('GB', 'US')

(42)

Alternative Data Warehouse Data Sources

• Web Services

• Advanced Queues and Change Data Capture Streams • Enterprise Service Bus

(43)

Web Service Data Sources

• Many organizations are adopting SOA architectures

• Data can now arrive through services and events, as well as from databases

• ETL process need to be SOA-aware • Web services and business events

are now potentially DW data sources

SOA Architecture Business Activity Monitoring Web Services Manager Declarative Rules Engine Enterprise Service Bus BPEL Orchestration Bulk Data Processing ETL Process Meta-data Mappings

(44)

Advanced Queues and CDC Streams

• Trickle feed of incremental data • Arrives via queues, streams

• Often need to process in real-time • Some platforms (i.e. Oracle RDBMS)

have native support

• ETL tools need to be CDC-aware

DW Sources _{DW T} arget Metadata Initial bulk load CDC for synchron ization Increme-ntal load of new data CDC for loop-back synchron ization CDC CDC ---

(45)

----Oracle Data Integrator

• Comprehensive data integration platform

• Extraction, transformation and load of data, to and from databases, events and Web services

• Batch and real-time loading • Java-based, hot-pluggable,

(46)

(47)

Extensible Knowledge Modules

• Provided “out of the box”, and are user extensible • Written for specific source and target platforms • User created example : Oracle 10g Data Pump

(48)

Event and Service-Based Integration

• Integrates data from events (files arrive, data arrives on queue) and from services (SOA, Web services)

• Perform integration in real-time (Enterprise Information Integration) • Reflects today’s reality that

not all data lives in databases • Integrates with

Oracle SOA Suite

Oracle Data Integrator

Data-oriented Integration Event Conductor Event-oriented Integration Service Conductor Service-oriented Integration Declarative Design Metadata Data Conductor

(49)

ODI Web Service Functionality

• Data Services

• Publish data stores (tables, views etc) as web services

• ODIInvokeWebService tool

• Request data from a web service as part of a package

• ODI Public Web Services

• Invoke an ODI scenario through a web service call

Oracle Data

Event-oriented

Service-Declarative Design Metadata

(50)

ODI Data Services

• Publish a data store as a web service • Insert, Update, Select, Delete

• List (multiple requests in one)

(51)

ODIInvokeWebService Tool

• Request data from a web service

• Called as a package step (in OWB terms, a process flow) • Data is fetched into an XML document

• XML document then processed and loaded into a table

• Most suited to use in initial data preparation, then loaded into staging table for access by ODI interface

(52)

ODI Public Web Services

• Invoke an scenario (a published ODI package) via a web service call • Call ODI functionality from an application, a BPEL process

• Add as a partner link in the JDeveloper BPEL editor

• Add an Invoke activity to call the InvokeScenario operation • Add an Assign activity to set operation

(53)

Possible Web Services & ODI Scenarios

• To provide bulk processing functionality to a business process • BPEL orchestration of an ETL process

• ODI Public Web Services

• Service-based access to data mart or data warehouse data

• Data Services

• Gather data from a web service prior to an ETL operation

• Call a human workflow process as part of an ETL control routine

(54)

Oracle Data Integrator and OWB

• Complements OWB by adding advanced data integration functions

• Works in the staging/integration layer

• Pre-build packaged solutions for CDC, Web Services etc

• Supports event-based, service-based and real-time integration

• Complements the BI Server by handling large data volumes and data marts

• Provides heterogeneous functionality - OWB may be more appropriate for Oracle data warehouses

• Create offline store (data mart) with history tables (SCD etc) • Perform complex integration tasks

• Integrate SOA and event-based data

• Adds the ability to target heterogeneous platforms • Supports real-time data integration

(55)

“Next-Generation” DW Objectives

• Incorporate data feeds from multiple sources

• Database tables • Files

• Services • Events

• Create a consolidated, integrated reporting database

• Incorporates batch-loaded and real-time data

• Uses database-centric and heterogeneous tools

• Tackle integration and standardization issues in a single place

• Provide an integrated store of data, indexed and optimized

(56)

Demonstration

(57)

Summary

• Oracle Database 11g comes with a number of new data warehousing features

• Partitioning improvements

• Cube Organized Materialized Views

• OWB11g features bug fixes, enhancements and support for 11g

• SCD2 handling that works (watch performance though) • Support for DML Error Logging

• Data Warehousing in the “Fusion” era is no longer just about extracting from, and loading to, Oracle databases

• ODI provides access to heterogeneous data sources

• Non-Oracle databases • Web services

(58)

Analytics and DW Using Oracle 11g and Oracle BIEE

Lesson 1 : ETL and DW development using Oracle 11g and OWB 11g (and ODI)

(59)

Analytics and DW Using Oracle 11g and Oracle BIEE

Lesson 2 : Creating the Business Logic Layer with OBIEE BI Server

(60)

Oracle BI Suite Enterprise Edition Plus

• Oracle “Next-Generation” BI tools platform • Based on Siebel Analytics

• Originally developed by nQuire • Standards (J2EE) based

• Heterogeneous connectivity • Server and Web-based

• Oracle BI Server

• Oracle Presentation Server • Oracle BI Answers

• Oracle BI Delivers

• Oracle BI Administration

(61)

The Future Oracle BI&DW Platform

(62)

The Business Logic Layer

• Contains an integrated, logical model of the business • Provides an abstraction layer for the business

• Can be auto-generated by the ETL tool

• OWB 11gR2

• Maintained by the business

• Can connect to multiple sources in the data services layer

• The Oracle data warehouse • Direct application access • OLAP cubes

(63)

Web Server

(IIS, Tomcat, Websphere, iPlanet)

Oracle BI Server

Intelligent Request Generation Intelligent Request Generation

Data Source Adapters Data Source Adapters

Navigator Navigator

Logical Request Generation Logical Request Generation

Aggregate Navigator Fragmentation Optimization Multi-Pass / Sub-Request Logic

Optimized Query Rewrites Optimized Query Rewrites

Session Management Session Management M et ad at a In te rc ha ng e M et ad at a In te rc ha ng e

Logical SQL ODBC/JDBC (Logical Business Model) Logical SQL ODBC/JDBC (Logical Business Model)

Cache Cache Services Services Oracle BI Administration Oracle BI Administration

Analytical and Operational Data Sources

Metadata Documentation Services Metadata Documentation Services Metadata Management Services Metadata Management Services Multi-User Development Services Multi-User Development Services

Server Management Services Server Management Services ODBC over TCP/IP (SSL)

S ys te m / S ys te m / P er f P er f M on ito rin g M on ito rin g

ODBC, CLI, OCI, XML, MDX

Oracle BI Presentation Services

SOAP Web Services, XML and URL Interface SOAP Web Services, XML and URL Interface

Cache Services (Web) & Connection Cache Services (Web) & Connection MngmtMngmt

XML XML Framework Framework Web Web Catalog Catalog Service Service User Profiling, Security and Session

User Profiling, Security and Session MngmtMngmt Oracle Interactive

Oracle Interactive Dashboards

Dashboards Oracle AnswersOracle Answers

Load Balancer Load Balancer S ec ur ity S er vic es S ec ur ity S er vic es Q ue ry G ov er n. Q ue ry G ov er n. TCP/IP (SSL) Javascript for Usability & Interactivity Web Browser

XML, HTML, XLS, PDF, TXT

over HTTP/HTTPS

External Applications and Portals HTML, SOAP

over HTTP/HTTPS

Execution Engine Execution Engine

SAW Bridge (J2EE/ISAPI)

TCP/IP (SSL) Oracle BI Publisher Oracle BI Publisher Delivery Server Delivery Server Layout Interfaces Layout Interfaces Data Logic Data Logic XSL Externalized Authentication LDAP DB Authentication Custom Authenticator Oracle BI Oracle BI Cluster Controller Cluster Controller

Oracle Delivers Server

Scheduling/Event Services Scheduling/Event Services Agent Execution Logic Agent Execution Logic Device Adaptive Content Device Adaptive Content

(64)

Oracle BI Server

• Calculation and Data Integration Engine

• “Virtual Data Warehouse”

• Allows reporting across multiple sources • Data accessed though metadata layer

• Connects to RDBMS and OLAP data

• Oracle Database (including Oracle OLAP) • IBM DB2

• Microsoft SQL Server • Teradata

• Microsoft Analysis Services (OLAP) • SAP BW 'info cubes'

• Security, Summary Management • Translates incoming “logical SQL”

(65)

Oracle BI Presentation Server

• “BI Application Server”

• Single application that provides Oracle BI Answers, Dashboards • Metadata stored in Web Catalog

• User Profiling, Security and Session Management • SOAP Web Services, XML and URL Interface

Web Server

SOAP Web Services, XML and URL Interface

Cache Services (Web) & Connection

Cache Services (Web) & Connection MngmtMngmt

XML XML Framework Framework Web Web Catalog Catalog Service Service

User Profiling, Security and Session

User Profiling, Security and Session MngmtMngmt

Oracle Interactive

Dashboards

Javascript for Usability & Interactivity Web Browser

over HTTP/HTTPS

over HTTP/HTTPS SAW Bridge (J2EE/ISAPI)

(66)

Oracle BI Delivers Server

• Scheduling and distribution • Uses iBots (“Intelligent Bots”)

• Integration (through Java) with BPEL

• iBot chaining, calling from BPEL, calling BPEL

• Permits alerts, through execution of Answers reports

• If report returns rows, alert triggered • Alert appears on dashboard,

via email, SMS, cell phone • Can trigger narrative and

conditionally-appearing reports Oracle Delivers ServerOracle Delivers Server

Scheduling/Event Services Scheduling/Event Services Agent Execution Logic Agent Execution Logic Device Adaptive Content Device Adaptive Content

(67)

Oracle BI Publisher Server

• New in 10.1.3.2, based on Oracle XML Publisher

• Separate application, integrated with Oracle BI Presentation Server

• Initially usernames and passwords integrated • Thereafter separately administered

• Provides high-fidelity layout, positioning of data, charts etc

• Multiple data sources

• OBIEE Metadata Layer • Oracle BI Answers reports

• Any source accessible via SQL • XML over HTTP

Oracle BI Publisher

Delivery Server Delivery Server Layout Interfaces Layout Interfaces Data Logic Data Logic XSL

(68)

Oracle BI Administration

• MS Windows application for managing BI Server

• Creates and manages Common Enterprise Information Model

• Physical, logical and presentation layers

• Manage other aspects of BI Server

• Security

• Scheduler (Delivers server) • Jobs

• Joins

• Marketing • Sessions • Cache

• Utilities and Tools

• Connect to BI Server via ODBC DSN

Oracle BI Administration

Metadata Documentation Services

Metadata Management Services

Multi-User Development Services

Server Management Services

(69)

Core BI Server Functionality

• Data Integration through Business Model View • Calculation and analytical engine

• Support for RDBMS and OLAP data sources • Intelligent Request Generation and

Optimized Data Access • Security and Data Scoping

• ODBC Interface out, native RDBMS and OLAP interfaces in

• Clustering, replication, support

for Windows and Unix/Linux hosts, 32-bit and 64-bit

(70)

Data Access

• Reporting data may be sourced from multiple databases, applications

• Oracle Database (including Oracle OLAP) • IBM DB2

• Microsoft SQL Server • Teradata

• Microsoft Analysis Services (OLAP) • SAP BW 'info cubes'

• SAP, Peoplesoft, Siebel, E-Business Suite

• Oracle BI Server joins this source data together and presents a

logical business model

• Facts

• Dimensions • Calculations

(71)

Common Enterprise Information Model

• Metadata layer for OBIEE tools • Three layers of metadata

• Physical layer, connections to sources • Logical layer, denormalized facts

and dimensions

• Presentation layer, “report-centric” folders

• Maintained using BI Administrator • Model is the “center” of the OBIEE

(72)

Data Modeling using Oracle BI Server

• Design goal for the BI Server metadata layer is to create a Star Schema

• Fact tables

• Dimension tables • Drill paths

• Source data can either be pre-integrated (data mart, data warehouse) or integrated using

the BI Server (“federated”)

• Federated can be faster to deliver (report in place) • Assumes data sources share common key values • Data marts are simpler and faster to query

• MDX (OLAP, Multi-dimensional) data is converted into RDBMS row-sets

• Logical layer is main integration point

(73)

“Model First” Development

• Business Model is centre of the OBIEE “universe” • First element of design that is produced

• Integrated view of the organization’s data

• Option 1: Star with model first, then map on physical sources

• “purest” approach, uses theoretical model as basis • Tricky to get joins and integration working

• Lots of manual logical table and column definition steps

• Option 2: Produce model on paper, then construct it using elements of physical model

• Quicker and faster, just drag and drop • May end up with redundant columns • Still needs a design phase at the start

(74)

(75)

Data Modeling Workflow Steps

1. Define logical model (on paper or using BI Administrator) 2. Import physical data sources

3. Create physical primary and foreign keys

(76)

Step 1: Create Physical Layer

• Define ODBC connection from client or server to database

• ODBC only needed to define initial connection, import metadata • Native connection (OCI etc) takes over from then

• Import tables, views, keys etc from data source

• Define keys, foreign keys if needed • Update row counts

(77)

Physical Join Types

• If source data has foreign key joins, they will be imported • If not, they can be defined in the physical model

• Two types of join

• Foreign key joins (e.g. CUSTID = CUSTID) • 1:N joins only

• Complex joins (TRAN_DATE between STARTDATEID and ENDDATEID)

• also supports 1:1, O:1, 1:O etc

• Important to create physical joins before importing in to logical layer

• Logical joins rely on physical joins to define paths

• Makes life a lot simpler

• Create joins using Join Manager, or Diagrammer

(78)

Step 2: Create Logical Layer

• Firstly, create new logical model

• Then create logical tables and columns

• Either create from scratch, then connect to physical • Or define by dragging and dropping physical tables

• Start off with single source for each table, then add connnections

• Define dimensions to create drill paths • Add calculations

• Use Rename Wizard to clean up table and column names

(79)

Calculations

• OBIEE has it’s own SQL syntax

• Cross-platform

• Same SQL across all sources • Calculations, analytics,

time-series

• Calculation Wizard for time-series calculations

• AGO, TO_DATE

• Oracle BI Answers features SQL pass-thru

• Has to be enabled per user • Allows native functions to

be used

• Future: direct support for native SQL functions

(80)

Time-Series Calculations

• OBIEE 10.1.3.2 supports limited time-series functions • AGO – value of a measure x months, quarters, years ago

AGO(Videostore."Sales fact view".Sales, Videostore.TimesDim."Month", 1)

• TODATE – value of a measure to the end of a month, quarter, year

AGO(Videostore."Sales fact view".Sales, Videostore.TimesDim."Quarter", 1)

• Requires at least one “time dimension” in the metadata layer

• Requires at least one or more Chronological keys • Monotonically increasing value (DAY etc)

(81)

Create Calculations in the Optimal Location

• Whilst the BI Server can create calculations, it may be more appropriate to do this in the underlying database

• Oracle Database analytic functions (LAG, LEAD, RANK etc)

• Oracle Database OLAP functions (time-series, forecast, allocation)

• Use the available architecture efficiently

• Perform calculations as close to the source data as possible • Use dedicated OLAP technology if available

• Use BI Server to provide features not found in the sources

• Just because the BI Server can do something, doesn’t mean it should do something

(82)

Logical Join Types

• Logical layer supports both foreign key and complex joins • Foreign key joins not recommended

• Restrict choices that BI server can make over join paths

• Only there for backward compatibility

• Complex joins are preferred

• Indicates relationship between tables

• Allows BI Server to determine best join path

• Useful when more than one Logical Table Source

(83)

Logical Table Sources

• Logical tables can have more than one physical source • Can create >1 Logical Table Source

• BI Server creates two physical queries • Useful when mapping data from

different levels

• and when using “fragmentation”

• Each Logical table source can have >1 table mapped

• BI server creates one query per LTS, joining as per physical foreign key joins

• Used when you need columns from other joined tables in your logical table

(join only happens if additional columns requested)

(84)

Consider Where to Join Data

• Again, consider where to integrate and join data

• Whilst the BI Server can join data, it may be faster to load into a data mart

• ETL tools can integrate data at time of load

• Denormalized data structures can be faster than joining at report time • Indexing, Summary Management, Query Optimization

• Creating a data warehouse still is the optimal solution

• BIEE data source joining features are most appropriate when bringing new data in that’s not yet in the data warehouse

(85)

Step 3: Create Presentation Layer

• Business models themselves are not visible to users • Users see presentation catalogs whose columns are

defined in business models.

• Presentation catalogs appear as “subject areas” in the web UI • Presentation catalogs are “databases” to client tools using ODBC

• To create a presentation catalog, drag business model to left panel. Then customize – re-order, re-folder, re-name.

(86)

Initial Reporting Scenario

• Global Company has a data warehouse, containing order and customer data • Requirement is to take the existing warehouse (normalized)

and turn it into a star schema (denormalized) • For now, OBIEE will be used to map the data

(87)

Demonstration

(88)

Reporting Across Multiple Data Sources

• OBIEE can combine and join data from multiple data sources

• Data is retrieved in separate SQL, MDX calls and joined using the BI Server • Allows you to perform federated queries

• Watch out for performance though

(89)

Federated Query Scenario

• Additional data exists in an Excel spreadsheet

• Contains quotas, plus information on products and dates • Quotas are at the product category and month level

• Existing data is at the product and day level

(90)

Joining A Second Data Source : Step 1

• Import the new schema into the physical model, ensure FK and PK constraints are set

(91)

Joining A Second Data Source : Step 2

• Use the Join Manager, or Physical Model Diagrammer, to create joins between tables in the two schemas

(92)

Joining A Second Data Source : Step 3

• Add the new dimension tables as additional logical table sources to the exisiting logical dimension tables

• Set the dimension hierarchy level at which they are applicable

(93)

Joining A Second Data Source : Step 4

• Add the additional measures to the fact table

(94)

Demonstration

(95)

Incorporating Real-Time Data

• Initial data set consists of historical Customer Data Warehouse data • Data also exists for real-time Order Bookings system

• Two logical models could be created, but ideal is to combine historical and

real-time data in one model

(96)

Fragmentation

• Logical tables have >1 Logical Table Source

• Logical Table Source 1 = Historical Data • Logical Table Source 2 = Real-Time Data

(97)

Fragmentation Step 1

(98)

Fragmentation Step 2

• Ensure mapped tables are added to new LTS

(99)

Fragmentation Step 3

(100)

Fragmentation Step 4

• Repeat process for Dimensions

• Do not add fragmentation condition if same data in both dimension versions

(101)

Demonstration

(102)

Summary

• The Business Logic layer is central to Oracle’s BI&DW Architecture

• Consists of physical, logical and presentation layers

• Foundation is properly-joined physical table sources

• Denormalized star schema built using complex logical joins

• Calculations (including time-series calculations) not available in the source database can be added to logical model

• Where possible though, use in-database analytics and OLAP

• Dimensions defined to determine drill paths

• Additional data sources can be combined in a single business model • Real-time data can be incorporated using Fragmentation

(103)

Analytics and DW Using Oracle 11g and Oracle BIEE

Lesson 2 : Creating the Business Logic Layer with OBIEE BI Server

(104)

Analytics and DW Using Oracle 11g and Oracle BIEE

Lesson 3 : Reporting and Dashboards using OBIEE Presentation Server

(105)

Oracle BI Presentation Server

• “BI Application Server”

• Single application that provides Oracle BI Answers, Dashboards • Metadata stored in Web Catalog

• User Profiling, Security and Session Management • SOAP Web Services, XML and URL Interface

Web Server

SOAP Web Services, XML and URL Interface

Cache Services (Web) & Connection

Cache Services (Web) & Connection MngmtMngmt

XML XML Framework Framework Web Web Catalog Catalog Service Service

User Profiling, Security and Session

User Profiling, Security and Session MngmtMngmt

Oracle Interactive

Dashboards

Javascript for Usability & Interactivity Web Browser

over HTTP/HTTPS

over HTTP/HTTPS SAW Bridge (J2EE/ISAPI)

(106)

The Future Oracle BI&DW Platform

(107)

The Presentation Layer

(108)

Oracle BI Answers

• Query authoring tool for BI EE • Report against business model

• Tables, Cross-tabs • Graphs, Gauges

• View and column selectors • Narratives

• Publish to Oracle BI Interactive Dashboards • Database-neutral

(109)

Answers Views

• Each report (“request”) contains a single query

• Possible to union, minus etc with similar queries

• Query has one or more views associated with it

• Table, pivot-table

• Chart, gauge, funnel chart • Title, narrative

• View and column selectors

• Views are combined in the compound view

• Compound view generally displayed in dashboard

(110)

Answers Formatting Options

• Columns can be formatted on column, or other column value • Colors, fonts,

• Images and icons

• Can replace or

supplement figures

• Images can be added to report titles

• Comes with a library of stock images

(111)

Oracle BI Interactive Dashboards

• Main method of interacting with reports • Fully interactive

• Drill on data, graphs • Switch between views • Drill out to detail

• Incorporate images, web content

• Role-based security • Lightweight, DHTML

(112)

Dashboard Interactivity

• Prompts can be added to dashboard • Drop-down lists, multi-select

• Passes parameters to reports • Switch between different views • Change columns used in reports

(113)

Demonstration

(114)

Oracle BI Publisher

• Publishes Answers reports as PDF, XLS, RTF etc

• Also direct access to metadata layer

• “Pixel-perfect” reports • Used for

distribution-quality reporting

(115)

Oracle BI Delivers

• Scheduling and distribution • Uses iBots (“Intelligent Bots”)

• Integration (through Java) with BPEL

• iBot chaining, calling from BPEL, calling BPEL

• Permits alerts, through execution of Answers reports

• If report returns rows, alert triggered • Alert appears on dashboard,

via email, SMS, cell phone • Can trigger narrative and

conditionally-appearing reports on dashboard

(116)

Dashboard Alerts & Guided Analytics

• Enables content and layout of dashboard to change based on information analyzed

• Sections or links can be defined to only appear when there is “interesting” information in the data

(117)

Types of Guided Analytics

1.Adding a link to a dashboard, that only appears if a condition is met

2.Making reports appear on the dashboard, when a condition is met • Requires two things

• A report to run, that determines whether the event has happened • A report, or link, that you want to appear

(118)

Step 1 : Create a “Trigger” Report

• Create a report that checks for your condition • Has a branch fell below its sales threshold? • Has a product’s year

on year sales fallen? • Use filters to check

for the condition

• If the report returns rows, the trigger will be met • Save to the Web Catalog

(119)

Step 2 : Add Guided Analytics Link to Dashboard

• Drag and drop from Dashboard Objects palette

(120)

Step 3 : Edit the Guided Analytics Link Properties

• Select the report that is run to determine whether the link is displayed • Select whether any rows, or no rows, is the trigger

• Enter link text • Select report,

dashboard or URL to link to

(121)

Step 4 : View Link in Dashboard (if Triggered)

• Link only appears if condition is met

• Guides user to the next action they should take

(122)

Making Sections Display Conditionally

• Individual sections can be displayed conditionally

• Select “Guided Navigation” from the section properties • Pick the report to run

(123)

Demonstration

(124)

Summary

• Oracle BI Answers and Dashboards give you many ways to display and visualize data

• Many formatting and layout options are now possible

• Best practices exist for laying out dashboards and presenting data • Reports and dashboards can be made interactive and navigable

(125)

Analytics and DW Using Oracle 11g and Oracle BIEE

Lesson 3 : Reporting and Dashboards using OBIEE Presentation Server

(126)

Analytics and DW Using Oracle 11g and Oracle BIEE

(127)

The Future Oracle BI&DW Platform

(128)

The Data Services Layer

• Staging

(129)

The Presentation Layer

(130)

Hyperion Financial Management Tools

• Financial Management tools

• Hyperion Planning, Hyperion Performance Scorecard

• Hyperion Capital Expenses Planning, Hyperion Workforce Planning • Essbase OLAP Server

• Hyperion System 9 infrastructure

(131)

Oracle OLAP Option

• Adds multi-dimensional data types and a multi-dimensional engine • Stores data in indexed arrays

• Adds an OLAP API and an SQL access layer

• (Can be) Faster than star schemas for loading and querying • Supports time-series and dimensional queries

• Based on Express Server technology • Provides consistently-fast access to

(132)

New Features in Oracle OLAP 11g

• The latest release of the Oracle Database OLAP Option

• Continues the objective of embedding OLAP in the database engine • SQL SELECT access

• Cubes stored in Analytic Workspaces

• Runs faster in 11g, uses SecureFiles instead of LOBs

• Metadata in the Oracle Data Dictionary

• Automatic creation of views over AW objects

• Optimizer now “AW view aware”

• CUBE_TABLE for easier direct SQL access • CUBE access now detailed in execution plans

(133)

Essbase Overview

• Multi-dimensional database engine

• Calculations, aggregations, forecasting, allocation • Foundation for many analytical applications

• Originally developed by Arbor, merged with Hyperion Software 1998 • Part of Oracle Fusion Middleware

• End-User focused, very popular with business users

(134)

Essbase Architecture

• ODBC connectivity to all major RDBMSs, plus SAP B/W • Multi-dimensional OLAP engine

• Java and XML/A mid-tier • Essbase query

(135)

Essbase Analytic Capabilities

• Drill, pivot, select, compare, rank, sort, filter, group, stack, calculate, annotate, write-back....

• Reporting tools enabled by database “outline”

(136)

Essbase Calculation Ability

• Wide range of functionality

• Simple aggregations to complex cross-dimensional allocations

• Results stored or calculated upon retrieval

• Relationship-aware

• Application-wide or subset-specific • Conditional and procedural logic

(137)

Essbase Storage Options

• Traditionally, Essbase uses the Block Storage Option

• Useful for dense, sparse/dense cubes

• Full range of calculations • Slower to aggregate

• Recently introduced, the Aggregate Storage Option

• Useful for sparse data sets • Generally read-only

• Fast to aggregate

• Similar storage model to MS AS,

(138)

Query and Administration Languages

• MaxL DDL for systems administration (SQL-like object definition language)

create or replace outline on database global.global

•

MDX for accessing data and metadata

With

set [CUSTOMER_DIM3] as '[CUSTOMER_DIM].Levels(2).members' set [PRODUCT_DIM3] as '[PRODUCT_DIM].Levels(2).members' set [TIME_DIM3] as '[TIME_DIM].Levels(2).members'

set [Q] as 'crossjoin ({[CUSTOMER_DIM3]},crossjoin ({[PRODUCT_DIM3]}, {[TIME_DIM3]}))'

select

{ [Accounts].[SALES] } on columns,

NON EMPTY {[Q]} properties ANCESTOR_NAMES on rows from

(139)

Essbase Development & Support Tools

• Analytic Administration Services • Analytic Integration Services • Analytic Provider Services • Hyperion Shared Services

(140)

Administration Services

• Three-tier application for administering Essbase

• Console for editing, maintaining Essbase databases • Java, launched through Web browser

• Allows you to maintain the OLAP database “outline”

(141)

Integration Services

• IDE for defining and loading an Essbase database

• Derive an Essbase cube from a relational star schema • Define dimensions, hierarchies, measures

• Populate cube

• Set up hybrid (HOLAP) storage

• Generate calculation scripts

(142)

OLAP Models and Metamodels

• Metadata used by Integration Services • OLAP Models define the relational

star schema used to source data for an Essbase database

• OLAP Metaoutline defines the dimensional model used in the Essbase database

• Administration Services can derive the metaoutline from the OLAP Model

• Outline can then be maintained in the Administration Services Console

(143)

Provider Services

• Mid-tier that provides XML/A connectivity, Java API • Fail-over, clustering

(144)

Smart View for Office

• Microsoft Office Integration (Word, Excel, Powerpoint) • General analytic capability

• Query designer, linked views

• Inclusion of Hyperion Visual Explorer

• Oracle Hyperion Planning Integration

• UI for Planning off-line

• BI+ repository • Run-time prompts • Multi-page support • Editing POV of imported objects

(145)

Essbase Within Oracle BI Framework

• Adds multi-dimensional analytic server to OBIEE BI Foundation • Calculations, fast analysis, predictive modeling

(146)

Creating an Essbase Cube from Oracle Data

1. Create new OLAP Model using Integration Services 2. Connect to the source database

3. Create fact table and associated Account dimension

4. Create other dimensions, including a time dimension if required 5. Review dimensional model

6. Create hierarchies for the dimensions

7. Create the OLAP Metaoutline, define the measures

8. Create the database using Administration Services, choose storage option 9. Use Integration Services to load members and data

(147)

Creating in Essbase Cube : Step 1

• Start Integration Services console, select New OLAP Model • Select ODBC Data Source that

provides a connection to the source schema

(148)

Creating in Essbase Cube : Step 1

• Start Integration Services console, select New OLAP Model • Select ODBC Data Source that

provides a connection to the source schema

(149)

Creating an Essbase Cube : Step 2

• Start up the Integration Services Console, show Intelligent Help (optional) • Select “Create Fact Table” from menu, select table that contains measures • Check box to create

Account dimension • Uncheck box to

(150)

Creating an Essbase Cube : Step 2

• Start up the Integration Services Console, show Intelligent Help (optional) • Select “Create Fact Table” from menu, select table that contains measures • Check box to create

Account dimension • Uncheck box to

(151)

Creating an Essbase Cube : Step 3

(152)

Creating an Essbase Cube : Step 3

(153)

Creating an Essbase Cube : Step 4

(154)

Creating an Essbase Cube : Step 5

• Create Hierarchies for all of the dimensions • Create TIME_DIM as a Time Dimension

• Validate and then Save model

(155)

Creating an Essbase Cube : Step 5

• Create Hierarchies for all of the dimensions • Create TIME_DIM as a Time Dimension

• Validate and then Save model

(156)

Creating an Essbase Cube : Step 6

• Create new MetaOutline

(157)

Creating an Essbase Cube : Step 6

(158)

Creating an Essbase Cube : Step 6

(159)

Creating an Essbase Cube : Step 7

• Drag and drop Accounts dimension over to dimension list • Set dimensions to dense or sparse, as needed

(160)

Creating an Essbase Cube : Step 7

• Drag and drop Accounts dimension over to dimension list • Set dimensions to dense or sparse, as needed

(161)

Creating an Essbase Cube : Step 8

• Use Administration Services to create a new application • Select Block Storage or Aggregate Storage

(162)

Creating an Essbase Cube : Step 8

• Use Administration Services to create a new application • Select Block Storage or Aggregate Storage

(163)

Creating an Essbase Cube : Step 9

(164)

Creating an Essbase Cube : Step 9

(165)

Creating an Essbase Cube : Step 9

(166)

Creating an Essbase Cube : Step 10

• Use Administration Services to preview the cube

• Note : if no values in cube, check that “use default calc script” was selected when performing the member and data load

(167)

Demonstration

(168)

Querying Essbase Cubes using Oracle BI EE

• Essbase is a native data source for Oracle BI Enterprise Edition • Uses XMLA compatibility feature of OBIEE

• Requires OBIEE 10.1.3.3.2, or 10.1.3.3.1 with registry edit

• HKEY_CURRENT_USER\Software\Oracle Corporation\Oracle BI\Oracle BI Administration Tool\Hidden Features

• Create new DWORD value called “Essbase”, set to “1”

(169)

OBIEE 10.1.3.3.1 Guidelines & Limitations

• Measure hierarchies not supported, all measures shown on same level

• Import process does not know that dimension member names may be unique, you will need to inform OBIEE of this in the logical dimension member

properties

• EVALUATE function can now work against MDX data sources

• EVALUATE_AGGREGATE function will need to be used for Essbase to natively handle time-series functions (not function shipped by default)

• Essbase UDAs suppressed by default, can be enabled using EVALUATE • Requires Essbase 9.3.1 stack

• Essbase Server 9.3.1

(170)

Registering an Essbase Cube in OBIEE Step 1

• Use Oracle BI Administrator to Import from XMLA

(171)

Registering an Essbase Cube in OBIEE Step 2

• Enter the connection details for the Essbase server • URL = <server_name>:13080/aps/XMLA

• Press “Update” to pre-populate the Data Source field • Username = Essbase login ID

• Password = Essbase password

(172)

Registering an Essbase Cube in OBIEE Step 3

• Select the cube that you want to import

• Essbase cubes are hypercubes, contain measures of the same dimensionality

• Cube brings across measures and dimensions

(173)

Registering an Essbase Cube in OBIEE Step 3

• Select the cube that you want to import

• Essbase cubes are hypercubes, contain measures of the same dimensionality

• Cube brings across measures and dimensions

(174)

Registering an Essbase Cube in OBIEE Step 4

• Cube appears as new database

data source in OBIEE Physical layer • Note different icons for dimensions

and measures

(175)

Registering an Essbase Cube in OBIEE Step 5

• Drag physical cube into logical layer

• All dimensions, joins, “tables” are automatically created

(176)

Registering an Essbase Cube in OBIEE Step 6

(177)

Registering an Essbase Cube in OBIEE Step 7

• Drag logical model into Presentation Layer • Delete all the Level 0 levels

• Not required, if selected will cause request to error

(178)

Registering an Essbase Cube in OBIEE Step 8

• Presentation layer should now only contain the tidied-up column names, as if it was a relational schema

(179)

Registering an Essbase Cube in OBIEE Step 9

(180)