The ESB and Microsoft BI

Business Intelligence

The ESB and Microsoft BI

The role of the Enterprise Service Bus in Microsoft’s BI Framework

Gijsbert “Gijs” in ‘t Veld CTO, BizTalk Server MVP [email protected]

About motion10

About me



CTO at motion10



20+ years experience in integration architecture, design



Working with BizTalk Server since beginning (2001)



Architected and designed parts of BizTalk Server



4-time BizTalk Server MVP



Member of the BizTalk Server Virtual Technical Specialist team



Technical Editor of the new-to-be-released BizTalk book:

BizTalk Server 2009 R2 Unleashed (July 2010)

Agenda



What is Business Intelligence (BI)?



Microsoft‟s BI Framework explained



What is an Enterprise Service Bus (ESB)?



How does an ESB fit in the BI Framework?



What is the value add of the ESB to BI?



^Summary

What is Business Intelligence?



Gartner describe it as …



A broad stack of applications and technologies for gathering, storing, analyzing, sharing and providing access to data to help enterprise users make better business decisions



...the process of extracting raw data from operational business applications and databases and analyzing the data to make long term, high level business decisions.

What is Business Intelligence?

Improving organizations by providing business insights to all employees leading to better,

faster, more relevant decisions

Delivered through a familiar environment

Integrated into a business productivity infrastructure

Built on a trusted & extensible platform

Three levels of BI

Operational Tactical

Business Impact

Strategic

# of Decisions

Timeliness Ease of Use High-Level View

Analysis Data Access Collaboration

24x7 BI Enterprise ETL

Embedded New Form Factors

Microsoft’s BI Framework

 Complete & integrated BI and Performance Management offering

 Leverages Microsoft Office investments and knowledge

 Enterprise grade and affordable

BI Topology

BI Platform: SQL Server

Integrate Store

Report Analyze

BI Layers

Source: Gartner (april 2006)

What is an Enterprise Service Bus?



An ESB is usually the center of a Service Oriented Architecture (SOA)



SOA is the architecture style, ESB is the plumbing



It is responsible for:



Business process orchestration



Protocol adaptation



Transformation



^Routing



It creates an abstraction layer that provides much flexibility



It can be used for A2A and B2B integration



In a SOA it can host your composite services



It is basically a broker

SOA revisited

Compose

User Experience and Interaction People using Content, BI,

Collaboration and Communication

Compose

Business Process Integration, Automation and Optimization, Information Integration

Expose

Existing Systems Consume

User Directed

Standards based Interoperability SOA as

mechanism to interact

SOA as mechanism

to transact

Enterprise Service Bus visualized

Transform Service

Routing Process

Orchestration

Protocol Adaptation

End Point Resolution Pub/Sub Service

Service Consumers Service Providers

1. Transform my message

2. Determine which endpoint I need 3. Route my message

4. Route the response to a second service

5. Return the final result to me

On Ramp Off Ramp

Transform my message

Resolve a service end point address for me

Typical ESB scenario: Composition

How does an ESB fit in the BI stack?

PERFORMANCE MANAGEMENT PLATFORM

Excel Services PerformancePoint Services

BI PLATFORM

SQL Server Reporting Services

SQL Server Analysis Services

SQL Server DBMS SharePoint Server

DELIVERY

Reports Dashboards Excel

Workbooks Analytic

Views Scorecards Plans

SQL Server Integration

Services BizTalk Server ESB

ESB value add 1: Operational BI



ESB is all about (near) real-time messaging in a SOA



Fast and current information



Process-driven BI



Especially in situations where the ESB is responsible for service composition and business process

orchestration, the ESB is the ideal location to record business level information on transactions: BAM

(Business Activity Monitoring)

Operational BI - Topology

PORTAL

LINE OF BUSINESS SYSTEMS

SHAREPOINT

ANALYSIS SERVICES BIZTALK SERVER ESB

B2B

REPORTING SERVICES

BAM lifecycle

Development of Business Processes

Develop new composite processes from reuse of existing systems Deployment of highly

distributed processes

Intuitive end-user task interaction through Office Interaction with

collaborative processes Effective management and control of

distributed processes Business Process

Modeling and Documentation Visual design of electronic forms Visual design of

collaborative applications

Real-time tracking of

end-to-end business process performance

Management visibility into business process performance

Real-time process optimization

BAM - How it works

Examples – Operational BI

^1. Analyst designs view template

2. BizTalk captures process events

3. SQL Server aggregates events to cubes

4. Excel requests SQL data and renders

5. Accessible through SharePoint portal

BAM is real-time Business Intelligence

with a business process context

ESB value add 2: Service Provider



Because the ESB is the center of your SOA, it can provide all kinds of services to the applications

connected to it



The ESB can provide the following functionalities as services to the Data Integration Layer in your BI stack:



Transformation services



Validation services



Access to legacy applications

Transformation Services



Building these transformation maps is done by using the graphical, drag-n-drop mapper



These graphical maps are compiled to the XSLT standard



BizTalk Server executes these XSLT maps in realtime during processing of transactions



The BizTalk Server ESB provides the transformation feature as a service that can be consumed by ESB users such as the Data Integration Layer in your BI stack

Validation Services



The ESB provides services to validate transactions against:



^Schema



Business Rules

BI’s data integration layer

• Extensive Connectivity

• High speed connectors for:

Oracle, Teradata and SAP BW

• Standards based support

• Excel, XML, flat files, binary files

• Connection to Applications via BizTalk Server ESB

• MS Message Queues

• Change Data Capture

• Transparently captures changes

• Real-time Integration

Unstructured data

Legacy data: Binary files

Application database

Change Tables

OLTP

DW

Access to legacy applications

PeopleSoft JD Edwards OneWorld XE JD Edwards Enterprise1 Oracle ODBC Siebel

TIBCO

Rendezvous TIBCO EMS SAP

EDI/AS2

X12 and EDIFACT support

Drummond Interoperability Certification MQ

MSMQ WSE HTTP SMTP File FTP POP3 SOAP SWIFT HL7

RosettaNet HIPAA SQL

WCF Adapter SharePoint

Host Applications

IBM mainframe zSeries (CICS and IMS)

Midrange iSeries (AS/400)

IBM DB2

Mainframe DB2 for z/OS

Midrange DB2/400

DB2 Universal Database for open platforms (AIX, Linux, Solaris, and Windows)

Host Files

“Pluggable Adapters In the Box”



The introduction of a canonical data model as part of the implementation of an ESB is a good design principle



It gives the following advantages to any ESB

implementation, but especially to the ESB in a BI stack:



Uniform description of what your data means



Canonical schemas, if well designed, are self-documenting



Easier to understand what the data means and how to map it to datawarehouses to create meaningful information that

really serves analysis



Data validation at the canonical level is much more efficient.

Business Rules apply to information you understand well.



Switching back-end applications does not impact your BI stack and data integration layer configuration

The use of canonical data models

When to choose what for integration?



SSIS when:



Batch oriented, scheduled, bulk data loads



Very high performance integration needed



Integration is purely needed for BI



Data is relational



Data needs to be pulled



^{ESB when:}



Near real-time data integration / low latency operational reporting



Changes to data in datawarehouse are incremental



SSIS does not provide access to the needed data source out-of- the-box and BizTalk Server does



Data structures are mixed and data coming from mixed platforms



Data is mainly pushed

Summary



There are 3 levels of Business Intelligence: Operational, Tactical and Strategic



ESB‟s first value add to BI is providing input to Operational BI



An important layer in the BI Stack is the Data Integration Layer, this is where the data is gathered, transformed and cleaned



ESB‟s second value add to BI is providing and brokering reliable services for data transformation, validation and access to legacy applications



BizTalk Server is the ESB that is part of Microsoft‟s Application Platform and perfectly ties in with the BI Stack

Contact Details



Come to our Booth



Register to win a Flip Camera



Gijs in „t Veld



[email protected]



www.indafield.com/BizTalkNotes



www.motion10.com