A SCIENTIFIC APPROACH TO MANAGEMENT:

(1)

A SCIENTIFIC APPROACH TO MANAGEMENT:

USING DATA TO MAKE BETTER DECISIONS

Avi Goldfarb

Professor of Marketing

Rotman School of Management, University of Toronto December 4, 2014

(2)

Who Am I?

 Professor of Marketing. At the Rotman School of Management since 2002  PhD in Economics from Northwestern University

 Research: How digital technology changes markets

 Teaching: designed and implemented a suite of courses in our MBA program

that teaches students how to integrate data into business problem-solving

(3)

What Am I Going to Talk About Today?

The Promise of Data (Big or Small)

The Reality

What’s Missing?

The Real Value of Data to Businesses

How to Get There

(4)

Now Famous Example: Target and Pregnant Women

 A couple of years ago, a man walked into Target in Minneapolis demanding to

see the manager

 He was clutching coupons that had been sent to his daughter and he was

angry

 He said, “She’s still in high school, and you’re sending her coupons for baby

clothes and cribs?”

 The manager apologized, and called a few days later to apologize again

(5)

How did Target Know?

 Target knew which of their customers had babies - for example, based on baby

shower registries and purchases of diapers.

 They then looked at the past purchases of these customers and identified 25

products that predict the second trimester (lotions, scent-free soap, cotton balls, etc.).

 This gave a score for the likelihood of a pregnancy and a predicted due date

(6)

Why Pregnant Women?

Given the wide variety of possible target groups, and the limited time for analyzing data, why focus on this particular group?

 Research has shown that habits change dramatically around the birth of a child  In other words, based on past data and insights from economics, sociology, and

psychology, they hypothesized that identifying pregnant women would be profitable

 They collected and crunched data to test this. This informed the broader decision

of which types of customers to invest in identifying

 It turned out to be very profitable, but with an important caveat: they needed to

(7)

The “Datatization” of Management

 Managers have always collected data, in varying degrees

 But technology has now greatly reduced the costs of collecting, storing and

analyzing data

 Human activity (both economic, and social) is increasingly digital… which

leaves a footprint

→_{Organizations can measure things that were previously impossible to measure}

→ Organizations can improve measurement of things that were previously measured infrequently or imprecisely

→ This has the potential to transform the way businesses operate, organize and compete

(8)

The Promise of (Big) Data (in theory)

Costs of collecting, storing and analyzing data fall Social and economic activity moves online Organizations have more data than ever before Data is an input to decision-making at all levels Organizations make better decisions Performance improves

(9)

This Promise Has Not Gone Unnoticed

“The payoff from joining the big-data and advanced-analytics management revolution is no longer in doubt” (McKinsey Quarterly, March 2013)

“The use of big data is becoming a key way for leading companies to outperform their peers.” (McKinsey’s Big Data Report, 2013)

“Data-based decision making is on the rise all around us.” (Ian Ayres in Supercrunchers) “Data are becoming the new raw material of business: an economic input almost on a par with capital and labour.” (Craig Mundie, The Economist, 2010)

“I think this revolution in measurement … is as profound as, say, the development of the

microscope and what it did for biology and medicine.” (Erik Brynjolfsson, Director, MIT

(10)

Organizations are Responding…

“In 2013, 64 percent of organizations were investing or planning to invest in big data technology compared with 58 percent in 2012” (Gartner, Inc.)

66% of respondents believe that “business analytics creates a competitive advantage in their organization” (MIT Sloan Management Review, Research Report The Analytics Mandate)

87% of respondents believe “it is important for my organization to step up its use of analytics to better make decisions” (MIT Sloan Management Review, Research Report The Analytics Mandate)

(11)

The Labour Market is Responding…

Recent emergence of many master’s programs in business analytics/big data:

(12)

Yet, the General Feeling Seems to be One of Disappointment

“…, the evidence from the past three years indicates that analytics is no longer a new path to value” (MIT Sloan Management Review, Research Report The Analytics

Mandate)

“The challenge of finding and sustaining a competitive advantage with analytics seems to be weighing heavily on many decision makers” (MIT Sloan Management Review, Research Report, The Analytics Mandate)

“Big Data: Are We Making A Big Mistake?” (Financial Times, March 28, 2014); “Eight (No Nine!) Problems With Big Data” (The New York Times, April 6, 2014); “Growing Doubts About Big Data” (ABCnews.com, April 8, 2014); and others

“Big Data’s Travails Don’t Mean It’s Derailed” (Michael Fitzgerald, contributing editor at MIT Sloan Management Review)

(13)

Why Hasn’t the Promise been Realized?

(14)

What’s Missing?

(15)

Actually, This is Not New

Some Historical Examples:

 WRITING (3,500 BC)  STEAM ENGINE  ELECTRICITY

 COMPUTERS AND INFORMATION TECHNOLOGY

→Most major innovations experienced an initial phase where the anticipated return failed to materialize

→ It takes time to learn how to understand what a new technology can do

→ The return on the investment increases when businesses/societies realize that the new technology allows you to so things differently

(16)

The Steam Engine

A Corliss Steam Engine – the symbol of the Centennial Exhibition in Philadelphia 1876

First invented in the 1600’s, it was not until the 1870s that US manufacturer produced more power from steam than from water.

(17)

What Does Data Allow Organizations to do Differently?

Descriptive Analysis -Analyses that identify, summarize and report facts and patterns -Useful for identifying trends and breaks from trends

-Identifies “what” but not “why”

-Provides the “pulse” of the business

-”Dashboarding”

Predictive Analysis

-Analyses that use past values of variables and associations between them to predict future values of a variable -Useful for resource allocation decisions that are contingent on future outcomes or unknown variables

-Anticipates the “pulse” of the business

Prescriptive Analysis

-Analyses that use data to uncover cause-and-effect relationships that affect key organizational outcomes

-Understands why things have happened and how they can be changed

-Useful for changing situations; solving problems

“What’s happening”? “What’s likely to happen”? “Why this is happening?”

(18)

A

Hierarchy

of Data Uses

Prescriptive

(“Data-Driven Problem-Solving”)

Predictive

(“Forecasting”)

Descriptive

(“Dashboarding”)

(19)

EX: A Brewer: “

We lose market share every summer

”

They were using data to describe the problem:

 How much do we lose?  Which brands lose?

 In which markets do we lose?

They were using data to predict the problem and adjust accordingly:

 Are we going to lose it again this summer?

 How should we adjust production, inventories, sales efforts and incentives?

They were NOT using data to understand and solve the problem:

 What factors influence consumer choices?

 Why are more consumers choosing my products in the winter than summer?

(20)

Prescriptive Analysis: A New Use of Data in Management

The real value of data and analytics emerges when they are used to uncover the key cause-and-effect relationships that impact organizational outcomes



Like the microscope, management data allow you to uncover why things are happening

– To understand behavior at a new level (a modern day focus group)



Allows you to use data to support or refute alternative explanations for the problem at hand

(21)

Evidence-Based Management Decisions

 Evidence-based management decisions: the conscientious, explicit, and judicious

use of current best evidence in making management decisions

 Evidence-based business builds on the scientific method:  Start with the problem, curiosity or decision

 Hypothesize explanations, causes, possible solutions

 Ask: what would be evidence in favour/against the hypothesis  Look at the evidence

 Update

 Scientific Method: A set of “principles and procedures for the systematic pursuit of

knowledge involving the recognition and formulation of a problem, the collection of data through observation and experiment, and the formulation and testing of hypotheses“ (http://www.merriam-webster.com/dictionary/scientific%20method)

Sound familiar? Evidence-based management borrows many ideas from evidence-based medicine!

(22)

We Take This for Granted in Other Settings

 The government wouldn’t approve - and you probably would not use - a drug

that hasn’t been subject to extensive clinical trials

 Engineers wouldn’t build - and you probably would not drive over - a bridge

whose design, construction and materials haven’t been modeled and tested

 Doctors wouldn’t prescribe - and you probably would not go through - a

treatment whose effectiveness and risks haven’t been analyzed and documented

So why do we allow managers to recommend actions whose effects haven’t been carefully modeled and supported with evidence?

(23)

(24)

Search Engine Marketing (Paid Search Advertising)

 If you go online and search for “Rotman”, you will see two types of search results

 If you click on the paid ad, Rotman pays Google; if you click on the organic result, it

doesn’t. Both get you to the Rotman homepage

Organic Search Results

Paid Search Results (ads)

(25)

Paid Search Advertising Makes Big Promises

In theory, paid search ads should make advertising more effective and efficient:

1. You can target your ads at people who entered relevant search terms

• Expedia ads come up when you search for “Disney vacation” but not “winter boots” • Johnson insurance ads com up with you search for “car insurance” but not “ipad”

2. You only pay the search engine when someone actually clicks on the ad

• Ensures you only pay for the ad when it attracts a genuinely interested customer

Compare to placing an ad in a newspaper… ad is shown to many people with no interest in the product and you pay regardless of who looks at the ad or acts on it

Sounds promising. But, how should you evaluate the impact of SEM spending? Is it a good marketing investment?

(26)

What’s the Challenge Here?

 It’s NOT a data challenge. Organizations should easily have data on how much

they have spent on paid search advertising and their own revenues

 Suppose your marketing department tells you that they’ve crunched the

numbers (e.g.: ran a regression of revenues on paid search spending) and found that every 1% increase in spending on paid search raises revenues by 5%

‒ Suppose this implies a ROI of ~800%

 Is this a helpful analysis?

What if the people who click on the paid ad were going to buy anyways? If the people who click on paid search ads would have shopped at the website even without the ad… then the data will show a positive correlation between ad spending and revenues even if the ad generates NO INCREMENTAL SALES

(27)

One Company’s Take on This: eBay

 In 2010, eBay was the fifth largest advertiser on Google, spending

approximately $51 million

 But, today:

‒ If you google “eBay”, you WON’T see any paid ads for eBay come up

‒ If you google “used cars eBay”, you WON’T see any paid ads for eBay come up

 What happened? Between 2010 and 2014 eBay learned that paid search ads

were ineffective

(28)

What Happened at eBay

 eBay had been using a large consulting company to advise them on and assess

their “marketing mix”

 An economist (Steve) in eBay Research Labs was skeptical of how the consulting company was measuring the relationship between marketing expenditures and revenues

‒ The marketing firm had been estimating a 1200% return on paid search advertising spending!

 Where did his skepticism come from?

‒ A deep understanding of data and statistics that allowed him to recognize the

limitation of simple correlations between ad spending and revenues

‒ A deep understanding of customer behavior and eBay’s business that led him to

hypothesize that most of the people who google “eBay” already know about eBay and therefore would find their way to eBay without the ads

(29)

Then an Opportunity for Learning Arose

For a short period of time, eBay stopped paying for ads for the search term “eBay” on Bing and Yahoo; Continued to pay for ads on Google

Steve analyzed eBay’s data during this period and found that 99.5% of the

forgone traffic from turned off ads was captured through organic search results These findings were compelling enough to convince the CFO to support a full-scale experiment on Google

(30)

The Experiment

eBay shut off all Google paid search ads in one-third of the country and kept them active in the rest of the country. Then Steve and his team measured the impact on eBay traffic and sales, for different types of keyword searches

What did they learn?

 For branded keyword searches (i.e.: “eBay”), ads generated NO ADDITIONAL

REVENUE. These people found their way to eBay anyway

 For non-branded keyword searches (i.e. “used guitar”), ads generated SMALL

INCREASE IN TRAFFIC BUT ONLY AMONG PEOPLE W/ LITTLE EBAY EXPERIENCE

 Overall, paid ads had a negative return!

(31)

Prescriptive Analytics Requires a Different Approach

1. The data, itself, is not the solution

2. Start with the problem not with the data

3. Focus on the “why” not the “what”

4. Identify the data and/or experiment that can validate or refute your

explanations

(32)

So Why Are So Few Organizations Doing It?

 It requires a new culture!

• Culture of evidence not advocacy

• Investments in learning not just growth • Executive champions

 It requires organizational changes!

• Pull vs. push approach to data • Communication and coordination • Changes in incentives, rewards

 It requires new skills and talents!

• Analysts who understand the business • Managers who understand the data

(33)

The Promise of (Big) Data (in theory)

Technological change is driving

(34)

A Scientific Approach to Management

Datatization of management presents an enormous opportunity:

For business, management and government to catch-up to many other fields in making evidence-based decisions

 Yet, the greater availability of data, on its own, will not achieve this

– Nor will new software, a Chief Data Officer, a data warehouse or simply hiring a bunch of

analysts

• The real value comes from organizations using their data to understand the

world around them… only then can they influence that world

• This requires complementary investments in people and organizational

(35)