Challenges in Business Analytics An industrial research perspective

(1)

Zurich Research Lab

Challenges in Business Analytics

An industrial research perspective

30 October 2008

Eleni Pratsini

IBM Zurich Research Laboratory

Alexis Tsoukiàs

LAMSADE-CNRS, Université Paris Dauphine

Fred Roberts

(2)

2 Challenges in Business Analytics; An industrial research perspective 30 October 2008

60 Years of IBM Research

The Sun Never Sets at IBM Research

1945 1

st

_IBM

Research Lab

in NY

(Columbia U)

Established: 1995

Employees:

40 Established: 1972

Employees:

500 Established: 1982

Employees:

200 Established: 1961

Employees:

1750

Established: 1998

Employees:

60 Zürich

Beijing

Austin

Delhi

Tokyo

Established: 1956

Employees:

300 Established: 1995

Employees:

90 1952

San Jose

California

Established: 1986

Employees:

500 Almaden

Watson

Haifa

(3)

Evolution of Role

Corporate

funded

research

agenda

Technology

transfer

Centrally funded

1970's

1980's

1990's

2000's

Collaborative

team

Shared agenda

Effectiveness

Joint programs

Work on client

problems

Research in the marketplace

Create business

advantage for

clients

Industry-focused

research

On Demand Business

Research

(4)

Overview

Business Environment

Æ

need for analytics in a world of increasing

complexity

Technical Challenges

:

Example from the Pharma Industry

-

Solution architecture

-

Data availability – a pleasure and a plague

quality, uncertainty, missing

-

Prescriptive models depend on predictive models

Optimization under uncertainty

Business Challenges

-

Client external projects

-

Client internal projects

(5)

Business Environment

We live in a highly constrained resource world

Rapidly reaching capacity for many resources

-

natural, infrastructure, human

Study: What do you see as the primary business challenges

currently affecting your organization?

-

Improving operational effectiveness

• Analytics are essential to address the constrained resources

challenge. The historical barriers to the use of analytics are

rapidly disappearing, enabling us to tackle complex high-value

problems.

(6)

6 Challenges in Business Analytics; An industrial research perspective

Analytics Landscape

Degree of Complexity

Competitive Advantage

Standard Reporting

Ad hoc reporting

Query/drill down

Alerts

Forecasting

Simulation

Predictive modeling

Optimization

What exactly is the problem?

How can we achieve the best outcome?

What will happen next?

What will happen if … ?

What if these trends continue?

What actions are needed?

How many, how often, where?

What happened?

Based on: Competing on Analytics, Davenport and Harris, 2007

Reporting

Analytics

(7)

7 Challenges in Business Analytics; An industrial research perspective

Analytics Landscape

Degree of Complexity

Competitive Advantage

Standard Reporting

Ad hoc reporting

Query/drill down

Alerts

Forecasting

Simulation

Predictive modeling

Optimization

What exactly is the problem?

How can we achieve the best outcome?

What will happen next?

What will happen if … ?

What if these trends continue?

What actions are needed?

How many, how often, where?

What happened?

Based on: Competing on Analytics, Davenport and Harris, 2007

Descriptive

Prescriptive

Predictive

(8)

Challenges in a world of increasing complexity

Advances in computer power bring capabilities (and

expectations) for solving larger problem sizes

-

Problem decomposition not always obvious

-

Sequential optimization

-

Exact vs approximate solution or mixed

-

Hybrid methods

Information Explosion

-

There is a lot of data (too much!) but how about data quality?

Not enough of the “right” data

Missing values

Outdated information

-

Data in multiple formats

(9)

(10)

(11)

Overview

Business Environment

Æ

need for analytics in a world of increasing

complexity

Technical Challenges

:

Example from the Pharma Industry

-

Solution architecture

-

Data availability – a pleasure and a plague

quality, uncertainty, missing

-

Prescriptive models depend on predictive models

Optimization under uncertainty

Business Challenges

-

Client external projects

-

Client internal projects

(12)

Example: Risk Based Approach to Manufacturing

Post-Marketing Adverse Event Reports

156,477 191,865 212,843 247,604 278,143 _266,991 286,755 0 150,000 300,000 1995 1996 1997 1998 1999 2000 2001 Calendar year Nu m b e r

‘FDA’s GMPs for the 21st Century – A Risk-Based Approach’

-

Manage risk to patient safety – “Critical to Quality”

-

Apply science in development and manufacturing

-

Adopt systems thinking – build quality in

*Horowitz D.J. Risk-based approaches for GMP programs, PQRI

(13)

13

Risk Management and Optimization

Using company’s knowledge, industry

expertise and industry guidelines &

regulations we identify the sources of risk

that need to be considered in the analysis.

Based on historical data, industry expertise

and other sources of causal

relationships, we set the causal link

strengths (conditional probabilities) and

develop a model for risk quantification

Given the risk values, financial and

physical considerations, we determine

the best sequence of corrective actions for

minimizing exposure to risk and optimizing

a given performance metric.

Phase 1: Identify Risk Sources

Phase 2: Quantify Risk Values

Phase 3: Optimal Transformation

client

data

experts

industry

∑ ∑ ∑ ∑ ∑ ∑ ⎥− ⋅ ⎦ ⎤ ⎢ ⎣ ⎡ ⋅ − ⋅ − ⋅ − ⋅ e k j jke e p j pje pe j pje pje j pje pje j pje pe X X T Risk R , , , jke CR Rev CT C Rev ∑ ≤ j pje X 1 ∀p,e ∑ ≤ k jke Y 1 ∀p,j,e ∑ =∑ k p pje jke X Y ∀ j,e ∑ ⋅ ≤ p pje X je pj CAP a ∀ j,e pje k jke pjke pje Y X Risk =∑Haz ⋅ ⋅ ∀p,j,e pje pjeT X ≥ ∀p,j,e pje ipje pje X T X ≤ −1+ ∀p,j,e ∑ ≤ j pje T 1 ∀p,e 1 1− − ≤ pje pje X T ∀p,j,e jke jke jkeY R Y ≤ −1+ ∀ j,k,e 1 − ≤jke jkeY Y jke jke R Y ≥ 1 / 0 , , , , 0 = ≥ XYTR Risk e k j,, ∀ e k j,, ∀ 0 2000 40006000 80001000012000140001600018000

client

Revenue at Risk as a percentage of Revenue 1.6% 7% 9.7% 6.4% 11% 10.6% 5.3% 1% 1% 1.3% CAPE BLST 4 CAPE CRD 1 CAPE PCH 1 CAPE PTCH 1 CAPE BOT 1 CAPE BOT 2 CAPE BOT 3 CAPE VIAL 1 CAPE VIAL 2 CAPE VIAL 3

industry /

experts

(14)

Definition of “Risk”

Fitness for Use

-

Quality

: product conforms to specifications

-

Efficacy

: ensures product has a positive effect (drugs treats a

patient)

-

Safety

: end product is not harmful (patient does not suffer

from fatal side effects)

Compliance

: failure to comply with regulations

(quality, efficacy and safety not fully under control),

e.g. lack of documentation

(15)

Risk: Three sources of hazards

Products:

Traditional focus; Based on portfolio

analysis

-

product is unstable or contaminated

Technologies:

some technologies may not be

appropriate given risk they create.

-

a new blender does not work well and the

mixing is not done properly

Systems:

basis for regulatory inspection.

Create transfer risks between product profile

classes. Systems capabilities & technologies.

-

a biotech product is developed and there is

no control of the specification of its

components

(16)

16 Challenges in Business Analytics; An industrial research perspective 30 October 2008 16

Refactoring

Model

Data Collection

Route

map

Envisioning supported by

Re-factoring model

Products

Systems

Technology

Define Scenarios

Product/Asset

Configuration

Implementation

Strategy

Review

Cost base of

Sites

Collect key data

Establish Risk Profile

Assess readiness for change

Test scenarios:

Prioritise on impact on

Revenue Risk, and

Profitability

Adapt scenarios

Optimise route-map for

implementation

based on Impact on:

Revenue @ Risk

profile

Investment Strategy

Impact of Change

Output Input

1

3

4

2 Review route

maps

select final

scenario options

Select scenario

Define

implementation

strategy

5

6

Input

Calculate

Revenue

profile

Input

Overview of Approach

(17)

17 Challenges in Business Analytics; An industrial research perspective 30 October 2008 17

Refactoring

Model

Data Collection

Route

map

optimization model

Products

Systems

Technology

Define Scenarios

Product/Asset

Configuration

Implementation

Strategy

Review

Cost base of

Sites

Collect key data

Establish Risk Profile

Assess readiness for change

Test solution:

Prioritise on impact on

Revenue Risk, and

Profitability

Adapt model

Output Input

1

3

4

2 Review route

maps

select final

solution options

Define

implementation

strategy

5

6

Input

Calculate

Revenue

profile

Input

Overview of Approach

(18)

Effect of uncontrolled data collection

A pharmaceutical firm is assessing the risk of two of their sites: Country A

and Country X. Engineers at both sites are asked to collect non-compliant

and defective batches in the last two years. Analysis of the data indicates

that site A has 50% more defective / non-compliant batches than site X.

They deduce that they should move their manufacturing processes to X or

drastically change their operations at A.

Just before launching an audit of their site A, they receive a letter from the

FDA mentioning many recalls of batches all coming from site X.

Further investigation indicates that site X does not report the many defects

they notice every week and they are indeed more risky.

Lack of observations does not mean no events; it means no reporting of events

Extract knowledge from non statistical sources, e.g. experts

(19)

Technology, product and system failure are all linked via common root

causes. The risk model starts from building a dependency graph of all these

root causes.

Cause Effect Analysis

industry

Expertise

Client

Constraints

Past

Experiences

Different sub-models created:

• Dosage form variability

• Employee execution

• Process control

(20)

Employee Ability to Execute Sub-model

Directly linked to GMPs. Example events: reporting not done properly, quality control

protocol not implemented, etc.

(21)

Elicitation of Expert Opinion - Challenges

Network parameters can be estimated from data and manual entry of

probabilities.

Using expert knowledge in Bayesian networks can be an intractable task

due to the large number of probabilities that need to be elicited

We need

an

elicitation method

that is fast and gives not only a probability but also a

confidence in the probability

an

aggregation method

for combining the elicitated probabilities taking into

account the experts’ confidence in their assessments

A

probability estimation

method to reduce the number of probabilities to elicit

in a noisy MAX node

(22)

(23)

23

Based on location, technologies or any previously identified components involved in

the drug production process, the network model computes an estimate of the quality

risk. Changing the way a drug is produced will (a priori) induce a change in the quality

risk estimate.

Site A

Site B

Quality failure: 2.4%

Quality failure: 1.9%

Optimization model

(24)

Risk Exposure

Performance measure for exposure to risk:

(Insurance premium)

Risk value:

∑

⋅

=

j

pje

pe

Risk

sk

Revenue@ri

Revenue

_pe

∑

⋅

=

k

pje

jke

pje

Y

X

(25)

Formulation

MAX

∑

_⎥

−

∑

⋅

⎦

⎤

⎢

⎣

⎡

⋅

−

⋅

−

⋅

−

⋅

e k j jke e p j pje pe j pje pje j pje pje j pje pe

X

T

Risk

R

, , , jke

CR

Rev

CT

C

Rev

ST

∑

≤

j pje

X

1 ∀

p

,

e

∑

≤

k jke

Y

1 ∀

p

,

j

,

e

∑

=

∑

k p pje jke

X

Y

∀

j

,

e

∑

⋅

≤

p pje

X

_je pj

CAP

a

∀

j

,

e

pje k jke pjke pje

Y

X

Risk

=

∑

Haz

⋅

∀

p

,

j

,

e

pje pje

T

X

≥

∀

p

,

j

,

e

pje ipje pje

X

T

X

≤

₋₁

+

∀

p

,

j

,

e

∑

≤

j pje

T

1 ∀

p

,

e

1

1 −

₋

≤

_pje pje

X

T

∀

p

,

j

,

e

jke jke jke

Y

R

Y

≤

₋₁

+

∀

j

,

k

,

e

1 −

≤

_jke jke

Y

jke jke

R

Y

≥

1 /

0 ,

,

0 =

≥

X

Y

T

R

Risk

Transfer

Remediation

Capacity, Risk

State, Level

e

k

j

,

∀

e

k

j

,

∀

(26)

17 product families

3 systems plus one for subcontracting

14 technology platforms – most available in all systems

Planning horizon: 5 years split into 10 periods of 6 months

Restriction on rate of change

Statistical analysis gave risk values based on historical data

Revenue projections available

Aggregated costs per scenario – later disaggregated

Simulation of given scenarios (ending result); optimization

(27)

Work center Current Scenario X

Tech 1 Tech 2 Tech 3 Tech 4 Site B Tech 5 Tech 6 Site A Tech 7 Tech 8 Tech 9 Tech 10 Tech 11 Tech 12 Tech 13 Tech 14 Site D 2005 2006 2007 2008 Site B Site C Site B 2009 Site C Site A Site C Site A / Site C Site A

Example Analysis

Values in 10^ -4

Values in Thousands

0.000 0.500 1.000 1.500 2.000 2.500 3.000 3.500 4.000 4.500 Mean= 3 5 7 9 11 3 5 7 9 11 5% 90% 5% 5.7013 8.625 Mean=7208

Optimal sequence of actions for minimizing risk exposure

i i i

p

rev

E

(exposure

)

=

∑

⋅

)

p

(

p

rev

)

(p

rev

)

Var(

i i i i i i i

⋅

=

⋅

−

=

∑

2

var

∑

2

1 exposure

Revenue@risk distributions 0 5000 10000 15000 20000 Revenue@risk De n s it y Scenario 1 Scenario 2 As Is Fully Opt

Calculate Business Exposure

(28)

(29)

Data Variability

The consideration of the frequency distribution of each solution is essential when

there is uncertainty in the input parameters

Robust Optimization techniques to determine the solution that is optimal

under most

realizations of the uncertain parameters

W h ich So lu tio n / Plan is b e tte r ?

0 2 0 00 4 0 0 0 6 0 0 0 8 0 0 0 1 0 00 0 1 2 0 0 0 1 4 0 0 0 1 6 0 0 0 Re v@ Ris k fr e q ue n c y di s tr ib u ti Pla n A Pla n B 6200

λ

= 4300

λ

= 3900

area = 5%

area = 20%

0 2 0 00 4 0 0 0 6 0 0 0 8 0 0 0 1 0 00 0 1 2 0 0 0 1 4 0 0 0 1 6 0 0 0 Re v@ Ris k fr e q ue n c y di s tr ib u ti Pla n A Pla n B

0 2 0 00 4 0 0 0 6 0 0 0 8 0 0 0 1 0 00 0 1 2 0 0 0 1 4 0 0 0 1 6 0 0 0 Re v@ Ris k fr e q ue n c y di s tr ib u ti Pla n A Pla n B 6200

λ

= 4300

λ

= 3900

area = 5%

area = 20%

(30)

Possible Approaches

Stochastic Programming

-

Average Value certainty equivalent

-

Chance constraints (known CDF – continuous)

Robust Optimization

-

Soyster (1973)

-

Bertsimas & Sim (2004)

CVaR

(31)

Modeling with the approach of Bertsimas and Sim

Formulation adjusted to incorporate a constraint that could be violated with a

certain probability:

Uncertain parameters have unknown but symmetric distributions

Uncertain parameters take values in bounded intervals:

Uncertain parameters are independent

(

)

e

s

t

p

x

e

p

x

)

Rev@Risk(x

fit(x)

SecuredPro

e

s

t

p

s

t

e

s

t

p

,

}

1 ,

0 {

,

1 value

critical

s.t.

max

,

∀

∈

∀

≤

∑

∈

E

X

x

[

H

ˆ

t,

s,

e

-

H

t,

s,

e

;

H

ˆ

t,

s,

e

+

H

t,

s,

e

]

∈

e

s,

t,

Haz

(32)

Bertsimas & Sim cont’d

Provided guarantee:

(

Rev@Risk(

)

>

critical

value

)

≤

γ

Pr

x

Example Values of Γ:

γ

=

1

2 n

(

1 −

μ

)

n

ν

⎣ ⎦

⎛

⎝

⎜

⎞

⎠

⎟ +

n

l

⎛

⎝

⎜

⎞

⎠

⎟

l

=

⎣ ⎦

ν

+

1 n

∑

⎧

⎨

⎪

⎩

⎪

⎫

⎬

⎪

⎭

⎪

n

=

total number of uncertain parameters

,

ν

=

Γ +

n

2 ⎛

⎝

⎜

⎞

⎠

⎟

,

μ

=

ν

−

⎣ ⎦

ν

(33)

Application to pharma example

Robust Formulation:

(

)

0 ,

,

0 ,

,

0 }

1 ,

0 {

,

1 ,

,

ˆ

max

, , , , , , , , , , , , , , , , , ,

≥

∀

≥

∀

≥

∀

∈

∀

≤

∀

≤

⋅

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

⋅

≤

−

∀

≥

+

≤

Γ

⋅

∑

∈

Z

e

s

t

Y

e

s

t

V

e

s

t

p

X

e

p

X

e

s

t

Y

X

Y

e

s

t

Y

V

Z

V

Z

X

e s t e s t e s t p s t e s t p e s t p e s t p e s t e s t e s t t,s,e p,t,s,e

,

H

Rev

value

critical

Rev

s.t.

, , , , e s, t, e p, e s, t, p, t,s,e e p, e s, t,

+

H

fit

SecuredPro

(

)

E

x

X x

(34)

34 Challenges in Business Analytics; An industrial research perspective 30 October 2008 0 5 10 15 20 25 0 5 10 15 20 25 Theoretical Bound

Actual Probability of Constraint Violation

Ideal Performed

(35)

Observations

Provides solutions that are much more conservative than

expected

Plans satisfying risk requirement are discarded by the

optimization - suboptimal solutions

Adjustments to calculation of parameters to improve

results

-

Extreme distribution

-

Consider the number of

expected basic

variables and not the

total number of variables affected by uncertain coefficients

e.g. production of 100 products, 3 possible sites, uncertainty in

capacity absorption coefficient:

-

300 possible variables (value of n)

-

each product can only be produced on one site i.e. only 100 variables >

0 (use this value for the calculation of

Γ

)

(

Rev@Risk(

)

>

critical

value

)

<<

γ

(36)

Bertsimas & Sim approach

0 5 10 15 20 25 0 5 10 15 20 25 Theoretical Bound

Ideal

Original settings: production not restricted to a single site

Production on a single site: Γ computed with the number of active uncertain parameters Production on a single site: Γ computed with the total number of uncertain parameters

Production – Distribution

continuous variables

0 5 10 15 20 25 0 5 10 15 20 25 Theoretical Bound

Ideal

Γ computed with the number of active uncertain parameters

Γ computed with the total number of uncertain parameters

Pharma Regulatory Risk

(37)

Related work

Robust optimization Models for Network-based Resource

Allocation Problems, Prof. Cynthia Barnhart, MIT

-

Robust Aircraft Maintenance Routing (RAMR) minimizing expected

propagated delay to get schedule back on track (taken from Ian, Clarke

and Barnhart, 2005)

Among multiple optimal solutions, no incentive to pick the solution with

“highest” slack

Relationship of

Γ

(per constraint) to overall solution robustness?

-

Comparison with Chance constrained Programming

Explicitly differentiates solutions with different levels of slack

Extended

Chance Constrained Programming

-

Introduce a “budget” constraint setting an upper bound on acceptable delay

(38)

Value at Risk & Conditional Value at Risk

VaR

Maximum loss with a specified

confidence level

-

Non sub-additive

-

Non-convex

β

Density of

f

(

x

)

β-Quantile

1-

β

CVaR

β

-Quantile

α

_β

(

x

)

(

β

=95%)

E

[

f

(

x

) |

f

(

x

) >

α

_β

(

x

)]

Advantages

-

Sub-additive

-

Convex

Density of

f

(

x

)

CVaR(

x

)

α

_β

(

x

)

(39)

CVaR in Math Programming: Linear Program with

Random Coefficients

CVaR can be used to ensure 'robustness' in the stochastic

constraint:

Min

CVaR (

)

T

_b

α

≤

=

≥

c x

x

Dx

d, x

0 ξ

Consider a LP where, coefficient vector, is random:

Min

T

_≤

_b

=

≥

c x

x

Dx

d, x

0 ξ

ξ

(40)

Reformulation of Rockafellar & Uryasev (2000)

Good news: Fully linear reformulation, so presentable to any LP solver

Bad news: New variables, and New constraints =

O

(Number of Samples)

-

Reformulated LP has grown in both dimensions!

-

Large number of CVaR constraints & large number of samples => Very Large

LP

1 Each CVaR expression, say a constraint:

CVaR (

)

is reformulated as:

1 (

)

(1

)

:

sample of random vector

(out of N total samples)

T

N

T

i

th

i

b

t

b

N

i

α

+

=

≤

+

≤

− α

∑

ξ

x

-1

1 (1

)

0

N i i T i i i

t

z

b

N

z

t

z

=

+

≤

− α

≥

∑

ξ

x

-Linear Reformulation

Reformulation

(41)

Observations

Number of scenarios necessary for a good estimation of

CVaR(

α

,

x

)

and thus meaningful results?

>20,000 scenarios necessary for stable results

(Kuenzi–Bay A. and

J. Mayer. ”Computational aspects of minimizing conditional value–at–risk”,NCCR FINRISK

Working Paper No. 211, 2005).

(42)

Recent advances in CVaR

Æ

Kunzi-Bay & Mayer (2006) presented a 2-stage interpretation if CVaR appears only in the

Objective Function & their algorithm CVaRMin led to an order of magnitude speed-up

Challenges in Business Analytics An industrial research perspective

Zurich Research Lab