Predictive Modeling
The Basics Plus
… What You Need to
Know
Tweet this session: #NYNP2013
Joseph Coakley, Individual Giving Manager
Food Bank For New York City [email protected]
Britt Fouks, Solution Consultant
Target Analytics, a Division of Blackbaud Inc. [email protected]
Jeff Johnson, Account Director
LW Robbins
Michele Peys, Account Director
LW Robbins
Models: Here’s what we will cover today…
– Demystify what they are and why use
them?
– Share the science behind how they
are built (but not so much that your
eyes roll up into your head!)
– Case Studies of Success with
strategies and results!
Why Use Models?
Models :
– Isolate predictive response variables and
apply to large datasets
– Reduce interpretation subjectivity
– Identify opportunities for optimizing
universe (house-file and prospect);
ensuring best return on investment
Modeling ‘Camps’
Descriptive:
– Models that classifying people by ‘like’
behaviors or characteristics
– Help you understand more about
similar groups of people based on
specific behaviors
Predictive:
– Models that predict future behaviors
based on past action
Data Here, Data There, Data, Data, EVERYWHERE!
Behavioral/Transactional
Attitudinal/Affinity
Your own vs. External
Demographic
**The right data or cuts of data
have more impact than the type of
modeling technique**
Modeling Techniques- Descriptive
Cluster Modeling:
– Grouping of similar behaviors,
demographics, expressed interests,
loyalties for the purpose of analyzing the relationships between those groups
– Descriptive, not predictive
– Example: clustering market segments to determine differences in purchasing
behavior; leveraging these differences for product testing
Modeling Techniques- Predictive
Regression:
– Compares responders vs. non-responders
using specific variables & assesses strength
of relationships between those variables
Modeling Techniques- Predictive
Decision Tree/Chaid:
– Decision Tree Analyses map
observations about data and carves large datasets down to variables used for modeling
– Detects interactions between variables – Prunes away the least significant
variables; leaving the most predictive and consistent variables for usage
Modeling Techniques- Predictive
Other Modeling Notables
• Scores-
– Output of the model built; often in numeric form – Increase in complexity based on # of variables in
the model
Dependent vs. Independent Variables-
− Dependent Variables- variable to model − Independent Variables- help explain the DV
Correlation-
the degree to which two variables are relatedValidation
- refers to the dataset or process used to validate the model’s predictive natureQ/A: Common Questions
Applications- where would I use a
model?
– Prospecting- zip models, coop acquisition lists, merge and/or list optimizations
– Lapsed/Cultivation- warm prospect models, lapsed
optimization, major and planned giving likelihood models
Guidance
– Users/Clients have important jobs! • Ask questions
• Provide intelligence
Q/A: Common Questions
Data
– Prepping is important. Best input data yields best results
– Cultivation/Lapsed modeling should use heavy doses of your own transactional data
Evaluation
– Questions to ask when evaluating results:
• Did the model do what was intended? If not, was it due to data being used?
• How did the model index against other results?
Two Basic Models
• Co-op List Models
– Built on a charities current donor pool – Look-a-like to your current donorbase – Potential donors have not given to your
organization in the past but have given to similar groups or have exhibited purchasing behaviors which makes them a good target for an
acquisition mailer.
– To access this market your organization will be required to share your list of donors and
Two Basic Models
• Lapsed Reactivation Models
– Focused on donors with no recent gift to a charity
• Definition varies by organization. In this example we will use 36+ months lapsed. – For most organizations reactivating a lapsed
donor is worth more in long term revenue than a new donor.
– Built to find donors with recent donation activity by affinity and RFM to other
Concerns
• Organization wants to protect donors from being mailed by other groups!
– This concern is invalid. On average an individual direct mail donor supports 12.74* organizations. – Your donors are actively supporting almost 13
charities other than yours and if one, two or ten of those groups share their data then your donors are already part of the Cooperative Databases.
• Some groups would need to change their privacy policy to access modeled names.
Case Study
Vietnam Veterans Memorial Fund (VVMF)
The VVMF had over one million lapsed donors
and felt that those lapsed donors were not
mined effectively.
Lapsed Reactivation Strategy
• Set up lapsed reactivation models for the VVMF donors.
• Broke the donors into two segments 25 – 60 months donors and 61+ month donors.
• Set up a data test to evenly split the data to allow for three modeling agencies to test
against each other and to allow the client to take advantage of the best performing
Lapsed Reactivation Strategy
• Set up the acquisition control package as the lapsed reactivation control package.
– Our thought was that the lapsed donors had
received 36+ house file mailings that they failed to give a donation.
– Wanted to spend less money and due to the
volume the acquisition package was the cheapest mailing.
• First tested the models to measure
effectiveness and to determine a winner among the three agencies.
• Next, aggressively tested new lapsed-focused packages against the control package.
Lapsed Reactivation Model
• Many groups mail to lapsed donors based on a standard RFM data select.
• As the recency of the donor becomes older we tend to select higher gifts.
– 37 – 48 months $25+ – 49 – 60 months $50+ – 61 – 72 months $100+
• Selections like these ignore the $25 donor that gave in 2007 who may be ready to give again?
Lapsed Reactivation Model
• RFM selects are based on past giving history
to your organization only.
• Models look at recent giving history across all
of the charities in their database as well as
Previous Lapsed Reactivation Plan
• In 2011 there were eight lapsed mailings of
varying creative.
• Selects based on RFM.
Previous Lapsed Reactivation Results
• Overall cost per dollar raised of $1.27 was solid for lapsed donors.
• However, we knew VVMF could mail
deeper into their lapsed donor pool and possibly improve their cost per dollar raised.
Year Total Qty Gifts
Response
Pct Gross
Avg
Gift CPDR
Sample Lapsed Scoring
• Segment A is the highest ranked group of lapsed donors.
• A segment A name could be any donor of any value that had given a gift
in the last 25 – 60 months.
• Measuring the performance requires you to create coding to track by
model.
• Lapsed model segments are usually larger than RFM segments which
makes analysis of a segment easier.
25 - 60 Month 61+ months
Segment Volume Segment Volume
A 25,000 A1 25,000 B 25,000 A2 25,000 C 25,000 B1 25,000 D 25,000 B2 25,000 E 25,000 C1 25,000 F 25,000 C2 25,000 G 25,000 D1 25,000 H 25,000 D2 25,000 I 25,000 E1 25,000 J 25,000 E2 25,000
Deployment
• Over the course of 2012 we mailed
seven dedicated lapsed mailings -
average 130K per mail drop.
• Each of the three agencies produced
successful models but we determined
the strongest of the three.
Increasing Lapsed Reactivation
• Increased the number of gifts by 111% • Response rate up 32%
• Gross revenue up as expected and more
importantly cost per dollar raised fell by 6%
Year Total Qty Gifts Response Pct Gross Avg Gift CPDR 2012 915,104 14,927 1.63% $234,942 $15.74 $1.19 2011 574,659 7,087 1.23% $154,317 $21.77 $1.27 Percentage Increase 59% 111% 32% 52% -28% -6%
Takeaways
• If your organization is not using lapsed
reactivation models consider it.
• If you have a limited budget consider
focusing some of your spending at lapsed
reactivation instead of acquisition.
– Can be accomplished simply by mailing modeled names inside your acquisition efforts.
– Or setup a lapsed reactivation track separate from acquisition.
Case Study
Food Bank For New York City (FBNYC)
All of the lists rented by the FBNYC for acquisition purposes are limited to the five boroughs of New York
City. We wanted to efficiently grow our program but we needed new names.
FBNYC Issue
• Performance had been solid through 2009 but in 2010 our list performance slipped.
• Over the years the FBNYC had mailed many donor and commercial lists in our acquisition mailings.
• Had only one co-op model list in our continuation list of lists.
• Wanted to grow program but felt limited thinking we reached a wall on the good performing donor lists - were not willing to
accept poorer performance levels to widen our list market.
Acquisition Strategy – Co-op Models
• Expand our modeling efforts by testing multiple segments within two additional co-op models.
• After Year I we found success with each of the two new models.
• Year II we expanded the volume on the three
working models and tested two new co-op models. • By Year III we had five strong modeled lists and
tested our sixth model.
• As we approach this fall we are in the process of adding our seventh model.
Concerns
• One of our major concerns was that by expanding within more models, they would hurt the
performance of the other models.
• Another concern was that we would be paying for many of the same names twice or three times by expanding our modeling efforts.
Concerns
• Found some cross over between models but overall we were happy with net names after the merge purge.
• Overall percentage of multibuyers is lower in 2012 then 2009.
• Last year our merge purge produced a sizeable number of multibuyers.
– However, the multibuyers were a top ten list with a cost per dollar raised of less than $1.20.
Results
• In 2008 & 2009 our list mix consisted of: Co-op Models – 13%
Rental lists (non-models) – 65% Multibuyers – 22%
• Overall cost per dollar raised was $1.50 • In 2012 our list mix now consists of:
Co-op Models – 50%
Rental lists (non-models) – 27% Multibuyers – 23%
Some models may be sexy,
but are they appropriate for you?
• Each organization is different and models can vary widely.
• Our advice is to let the numbers guide you. • Each modeling agency will be able to tell you
which model of theirs is the strongest.
• Test their strongest model – remember they have a strong incentive to develop a successful model for your organization.
What qualities and characteristics should
you look for in a modeling partner?
• If this is your first foray into modeling then go with an established group/s.
• If your organization has 100,000 or more 25+ month lapsed names you should have your lapsed names modeled immediately.
– You are leaving money on the table.
• Co-op models – test at least one model from two different modeling agencies.
Are certain models out of your league, if
you're a smaller organization with a
smaller file?
• Lapsed models are tough to develop without a large number of names.
• Co-op models may require minimums – ask for details … don’t make assumptions.
• Don’t write off models because your charity is small. Reach out to see what a modeling agency can do for your group.
In acquisition, what is the best mix of
vertical rental and exchange lists vs.
modeled lists?
• This is a question best answered by your results. • Your list broker and/or agency will make a
recommendation.
