Catherine Snyder
Supervisor – US Dealer Audit, Audit Services General Motors Company
Predictive Analytics & Predictive
Modeling
AGENDA
Overview of General Motors Company
Predictive Analytics – Definition and References
Dealer Risk Management & Optimization (DRMO) Model
Key Takeaways
DRMO Global Program Objectives
Operating Environment
Process Overview
Risk Indicators and What Could Go Wrongs
VIN / Warranty Claim Selection Process
DRMO Model Development
Keys to Success
THE NEW GENERAL MOTORS
Top Global Automaker
¶
Sales of 9.7 million vehicles
¶#1 in U.S.; #2 in China
¶
Fortune 7 company
Sales
in
120 Countries
Production
in
30 Countries
¶
396 facilities touching 6 continents
219,000
Employees
21,000
Dealerships
GM – TOP GLOBAL AUTOMAKER
Sales in 120 Countries
Percentage of deliveries by region
Production Locations in 30 Countries
GM BRANDS
Baojun
Wuling
Predictive Analytics
DEFINITION AND REFERENCES
Business Analytics Defined
Thomas H. Davenport, professor at Babson College
• Defines descriptive, predictive, and prescriptive analytics — and when to use each.
• How Managers Should Use Data
http://blogs.hbr.org/2014/09/a-predictive-analytics-primer/ http://blogs.hbr.org/2014/03/when-to-act-on-a-correlation-and-when-not-to/ http://blogs.hbr.org/2014/05/whos-afraid-of-data-driven-management/ 9
PREDICTIVE ANALYTICS
• Encompasses various statistical techniques
from modeling, machine learning, and data
mining.
• Analyze current and historical facts, capture
relationships between explanatory and
predicted variables from past occurrences,
to make predictions about the future,
unknown, events, trends and behavior
patterns.
• Accuracy and usability of results depend
greatly on the rigor of data analysis and the
quality of assumptions.
10PREDICTIVE MODELS
• Capture relationships among many factors to enable assessment of risk or potential associated with a particular set of conditions,
guiding decision for candidate transactions.
• Model the relationship between specific performance of a unit in a sample and one or more known attributes of the unit.
• Objective is to assess the likelihood that a similar unit in a different sample will exhibit the specific performance.
• Examples:
• Real-time Credit Card Transaction Predictive Model
• Credit Scoring Model
• Customer Relationship Management (CRM) Model
• Clinical Decision Support Systems
• Customer Retention/Silent Attrition Model
PREDICTIVE ANALYTICAL TOOLS
Alpine Data Labs BIRT Analytics
Angoss KnowledgeSTUDIO IBM SPSS Statistics and IBM SPSS Modeler
KXEN Modeler Mathematica MATLAB
Minitab
Oracle Data Mining (ODM) Pervasive
Predixion Software Revolution Analytics SAP
SAS and SAS Enterprise Miner STATA STATISTICA TIBCO FICO 12
Dealer Risk Management & Optimization
(DRMO) Model
KEY TAKEAWAYS
• Use audit results to drive Investment in prevention
activities – Sell Win-Win
• Model examines 100% of the population, removes
individual bias, and provides consistent results
• Combine datasets to improve risk assessment
• Integrate automated controls when you can
• No model is perfect
• Depending on the audit objectives:
•
Predictive analytic is not always the answer
•
Traditional audit approaches could be your best
friends
•
You may only need very few samples
DRMO GLOBAL PROGRAM OBJECTIVES
• Implement global, common process to improve audit objectivity, effectiveness and efficiency
• Engage global subject matter experts
• Perform advanced analytics on 100 % of the claims to aggregate dollars-at-risk per dealer
• Allocate resources to highest risk transactions/dealers • Refresh model based on audit results and changes
• Expedite identification of global issues and root cause analyses to facilitate global corrective actions and process improvement
• Consolidate risk profiles and audit results to provide insights for business and operational improvements
• Improve visibility of risk on a timely basis • Provide interactive reporting and analytics
• Develop scorecard for monitoring effectiveness of risk management /optimization program
OPERATING ENVIRONMENT IN UNITED
STATES
• 4,300 dealers
• 2.8 million vehicles delivered in 2013
• Incentive/Warranty programs by
brand/model
• Statutes of Limitation vary by states: 6, 9,
12, 18 and 24 months
• Fraud provisions and impact on audit
rights vary by states
COMMON INCENTIVE PROGRAM
PARAMETERS
• Who
• When
• Where
• What
• How
19PROCESS OVERVIEW
Aggregate by Dealer What Could Go Wrongs (WCGWs) Risk Indicators Actual Audit Results Claim-Level Risk Score Weighted At-Risk Amount (WARA) Claim DataAudit Plan Pull List
Model Evaluation % of High Risk Claims 20
RISK INDICATORS / WHAT COULD GO WRONG
What Could Go Wrong (WCGW)
• Identify common deviation reasons for claims
Risk Indicators (RI)
• Evaluate claim attributes used to identify one or more of the WCGWs • Perform calculations on each claim
• Compare against historical exceptions to identify the strength of correlation of each indicator, both individually and in various combinations (not all indicators or combinations of indicators are equal in their predictive power)
• Use in the scoring formula for a given claim if the indicators show strong historical correlation to exceptions
Example
RI-WTY-5 Abnormal claim volume:
Identification of dealers claiming an abnormal percentage of warranty claims related to a particular part when compared to the full population of dealers.
WCGW-WTY-1 Warranty claims are processed for repairs that did not occur.
WCGW-WTY-2 Warranty claims are processed for non-covered parts.
WCGW-WTY-14 Warranty claims are processed for non-covered services.
Risk Indicator Description What Could Go Wrongs RI-IC-21 Claim Date for a Program
Across all Models: Box plot which shows high-level anomalies in claim dates across an entire incentive program.
WCGW-IC-2 An incentive claim may be submitted and paid for an expired program or a program that has not yet begun.
WCGW-IC-3 An incentive claim may be submitted and paid for an inapplicable model.
WCGW-IC-5 An incentive claim may be submitted for an applicable program date where actual delivery was outside of the program's eligible period.
WCGW-IC-7 An incentive code may be added to a claim submission after the original submission date in order to take advantage of a program or pocket an incentive and not pass it on to the customer.
WCGW-IC-11 Vehicles may be reported as sold, but a sale has not actually occurred.
EXAMPLES: WHAT COULD GO WRONGS
• Sales Incentive – 40 WCGWs
• Incentives were claimed on deliveries that did not qualify • Incentives were claimed for customers that did not qualify • Vehicles not sold may be reported as sold
• Incentives may be claimed for deliveries outside the eligible period • Dealer may misreport sales to manipulate vehicle allocation
• Warranty – 30 WCGWs
• Multiple claims were submitted for the same repair using different problem codes or operation codes
• Warranty claims were processed for non-covered services • Warranty claims were processed for repairs that did not occur
• Warranty claims were processed for vehicles in dealer possession for work that is not required
• Dealers submitted more hours for a service than actually incurred
EXAMPLES: INCENTIVE RISK INDICATORS
• % of Incentive Claims to Market Share Comparison by Geographic Region • Analyze claims monthly to determine which dealers claim more incentives
than their predicted relative geographic market share of deliveries • Dealer Penetration % Compared to National Average Penetration %
• Monthly comparison of total delivery % per incentive code at a dealership to the national average % on a brand-by-brand basis
• High/Low outliers are flagged • Abnormal Customer Names
• Customer names similar to the dealer name • Deliveries made to the same customer
• Abnormal Amount of Incentives Claimed on Vehicle Deliveries
• Vehicles with an abnormal amount of incentives claimed when compared to average number of incentives claimed/vehicle for specific incentive category • Volume of Deliveries with Claims at Month-End
• Claims for deliveries where the dealer’s delivery volume in the last part of the month appeared abnormal compared to other dealers
EXAMPLES: WARRANTY RISK INDICATORS
• Frequency of problem code use
• Identification of dealers using specific problem codes at a higher percentage of total claims than the overall dealer population. Flags claims related to
those dealer/problem code combinations as high risk • Abnormal claim volume
• Identification of dealers claiming an abnormal percentage of warranty claims related to a particular part when compared to the full population of dealers • Abnormal labor hours by operation and problem code
• Box plot which shows the total hours (regular labor and other labor hours) for each operation and problem code combination to identify claims that are
outliers when compared to the population
• Labor only claims abnormal for problem/operation code combination
• Identification of specific labor-only claims where it is abnormal for that
particular claim type to be labor-only based on the total population of claims with a specific problem/operation code combination
• Warranty claims for vehicles in dealer possession
• Identification of claims for vehicles without a warranty start date and where vehicle mileage is less than 100 24
VIN / WARRANTY CLAIM
SELECTION
VIN/WARRANTY CLAIM SELECTION PROCESS
• New claims loaded into the database
• Risk indicator values calculated for all claims
• Risk score calculated for each claim based on the risk indicator values
and incentive category or warranty type scoring function (model)
• Different incentive categories / warranty types may have unique
risk score functions (models)
• Average risk score for each VIN / Warranty Claim calculated
• VINs / Warranty Claims with higher risk scores predict a higher
exception rate based on historical patterns and behaviors
• VINs / Warranty Claims with highest risk scores flagged as “selected” as these pose the highest likelihood of exception
• Model ranks dealers for audit based on having the highest aggregate Weighted At-Risk Amount (WARA) for highest risk VINs (Incentives) or having the highest % of high risk warranty claims (Warranty)
VIN / WARRANTY CLAIM
SELECTION
AUDIT PLANNING PROCESS
Model ranks dealers for audit based on number of high risk Sales Incentive/Warranty claims
VIN / WARRANTY CLAIM
SELECTION
AUDIT PULL LISTS
The Claim Risk Score and Top 3 Primary Risk Indicators are provided in the Audit Pull List
