Improving Demand Forecasting

2nd _{July 2013}

John Tansley - CACI

Overview

 The ideal forecasting process:

 Efficiency, transparency, accuracy

 Managing and understanding uncertainty:

 Limits to forecast accuracy, including the Poisson limit

 The Forecast Value Add approach:

 From simple to more complex models

 Types of forecasting model:

 Econometric, quantitative, and combined

 Types of quantitative models:

 Time series, explanatory, combined

 Case study 1:

 Improved call volume forecasting for financial services debt management

 Case study 2:

The ideal forecasting process

 Goal of forecasting process:

 Provide the best possible forecast, given the information available

 ‘Best’ means:

 Efficiency:

 Automate data feeds as much as possible

 Transparency:

 Understandable (avoiding black box or overly complicated Excel)

 Accuracy:

 Self explanatory!

Managing and understanding uncertainty

 Best possible accuracy is outside the control of the analyst  Factors that affect accuracy:

 Lack of all necessary information:

 Only have access to limited data  Problem changes rapidly over time

 Inaccuracies in known information:

 Inaccurate data

 Incorrect mental model of the business problem

 Fundamental limits to accuracy

Poisson accuracy limit

 When forecasting counts, there is a fundamental limit of achievable

accuracy – the Poisson limit

08:00:00 09:00:00 10:00:00 11:00:00 12:00:00 13:00:00 14:00:00 15:00:00 16:00:00 17:00:00 18:00:00

100 calls in a 10h day - equally spaced

Call

08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00

100 calls in a 10h day - random

Poisson accuracy limit

 When forecasting counts, there is a fundamental limit of achievable

accuracy – the Poisson limit

08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00

1000 calls in a 10h day

Poisson accuracy limit

 Demand in a particular time period or location is generally distributed

according to a universal distribution – the Poisson distribution.

 The spread of this distribution is around the square root of the demand volume

 Understanding this limit helps to

 Set reasonable accuracy expectations Demand

Mean Spread Spread (%)

9 3 33%

25 5 20%

64 8 13%

100 10 10%

0 20 40 60 80 100 120 140 160 180 0 5 10 15 20 E rr o r

Forecast error versus model complexity

Error Best model

Naïve model performance Poisson limit performance

The Forecast Value Add approach



Start simple



Additional complexity is only worth it if it increases accuracy



Can measure by how much each incremental step improves the

forecast



No point adding complexity if forecast error increases

Types of forecasting models

Econometric Models

• Manually built

models

• Small datasets (if

any)

• Manual setup

• Based on business

knowledge

Bayesian

Econometric Models

• Manual model

structure

• Model parameters

set from user

constraints and

data

• Based on both

business

knowledge and

data

Quantitative Models

• Automatic models

• Larger datasets

• Little user control

over parameters

• Based on data

• Examples:

Regression,

Decision Trees

Types of quantitative models

 .

Time input1 input2 input3 Target

1 2 3 4 explanatory time series combined

Regression, Decision Trees, Neural Networks

Weekly profile, moving

average, ARIMA, Exponential smoothing,

ARIMA with drivers,

Decomposition Forecasting Use drivers, add insights

Good for trends

Case study 1 – Improving the forecasting process

 Improved call volume forecasting for the debt management function

 CFS were creating forecasts in large Excel sheets, populated manually  CFS had a desire to improve process, and remove single man dependency  Solution:

 Software: statistical forecasting models  Automation of data feeds

 Results:

 Reduced single man dependency

 Immediately showed increased speed (from 2.5 to 1 day) and

transparency

Case study 2 – Long term demand forecasting

 30 year water demand forecasting for a large water board

 Yearly forecasts across 10-20 geographical areas, and 10-20

business sectors

 A few years’ worth of demand, economic and weather data

 Approach: Bayesian Econometric Models

 Allows the model structure to be set by the user  Model parameter estimates are set by the user

 Model parameters are then calibrated on existing data

 Result:

 Forecasts that combine the best business knowledge and the data  Parameters can be set by the user if needed, or dictated solely by

Wrap-up

 A large number of techniques are currently available for forecasting,

the key is choosing right technique for right problem

 A good forecasting approach should add insight as well as accuracy  Forecast Value Add approach: keep it as simple as possible

 Key is to keep on top of models – keep them understandable and easy

to update