ECLT 5810
ECommerce Data Mining Techniques
-Introduction
Data Opportunities
● Business infrastructure have improved the ability to
collect data
● Virtually every aspect of business is now open to data
collection:
◆ customer behavior, marketing campaign
performance, supply-chain management, workflow procedures, etc
● Information is now widely available on external events
◆ market trends, industry news, etc
● Broad availability of data
◆ increasing interest in methods for extracting useful
Data Opportunities
● Companies focus on exploiting data for competitive
advantage
● In the past, statisticians and analysts to explore datasets
manually
◆ but the volume and variety of data have far
outstripped the capacity of manual analysis
● Computers have become far more powerful
◆ algorithms have been developed that can connect
datasets to enable broader and deeper analyses
● Given rise to the increasing widespread business
Data Mining Adoption
● Data Mining – extraction of useful knowledge from
data
● Knowledge may refer to - models, rules, regularities,
patterns
◆ non-trivial, implicit, previously unknown
● Used for general customer relationship management
◆ analyze customer behavior in order to manage
attrition and maximize expected customer value
● Used for credit scoring and trading, fraud detection, and
workforce management
One Data Mining Case – Retail Industry
● Hurricane Frances was on its way, barreling across the
Caribbean, threatening a direct hit on Florida’s Atlantic coast. Residents made for higher ground.
● But far away, in Bentonville, Ark., executives at Wal-Mart Stores
decided that the situation offered a great opportunity for one of their newest data-driven weapons - predictive technology.
● A week ahead of the storm’s landfall, Linda M. Dillman,
Wal-Mart’s chief information officer, demanded her staff to come up with forecasts based on what had happened when Hurricane
Charley struck several weeks earlier.
● Backed by the trillions of bytes’ worth of shopper history that is
One Data Mining Case – Retail Industry
● Consider why data-driven prediction might be useful in
this scenario.
● It might be useful to predict that people in the path of
the hurricane would buy more bottled water.
● It might be useful to project the amount of increase in
sale due to the hurricane, to ensure that local Wal-Mart are properly stocked.
● Perhaps mining the data could reveal that a particular
One Data Mining Case – Retail Industry
● Besides such expected discovery, it would be more
valuable to discover patterns due to the hurricane that were not obvious.
● To do this, analysts might examine the huge volume of
Wal-Mart data from prior, similar situations (such as
One Data Mining Case – Retail Industry
● From such patterns, the company might be able to
anticipate unusual demand for products and rush stock to the stores ahead of the hurricane’s landfall.
● Indeed, that is what happened!
● The New York Times (Hays, 2004) reported that:
◆ …the experts mined the data and found that the
stores would indeed need certain products - and not just the usual flashlights. “We didn’t know in the past that strawberry PopTarts increase in sales, like seven times their normal sales rate, ahead of a
Data-Driven Decision Making (DDD)
● Data-driven decision-making (DDD) refers to the
practice of basing decisions on the analysis of data, rather than purely on intuition.
◆ For example, a marketer could select
advertisements based purely on her long experience in the field and her eye for what will work. Or, she could base her selection on the analysis of data
regarding how consumers react to different ads.
● DDD is not an all-or-nothing practice, and different
Data-Driven Decision Making (DDD)
● There are two sample types of DDD:
1. decisions for which “discoveries” need to be made
within data
2. decisions that repeat, especially at massive scale, and so decision-making can benefit from even small increases in decision-making accuracy based on data analysis.
● The Walmart example above illustrates a type 1
problem:
◆ discover knowledge that will help Walmart prepare
Data-Driven Decision Making (DDD)
● In 2012, Walmart’s competitor Target cares about
consumers’ shopping habits, what drives them, and what can influence them.
◆ Consumers tend to have inertia in their habits and
getting them to change is very difficult.
● Decision makers at Target knew that the arrival of a new
baby in a family is one point where people do change their shopping habits significantly.
● Most retailers compete with each other trying to sell
baby-related products to new parents.
● Since most birth records are public, retailers obtain
Data-Driven Decision Making (DDD)
● Target wanted to get a jump on their competition. They
were interested in whether they could predict that people are expecting a baby.
◆ If they could, they would gain an advantage by
making offers before their competitors.
● Using techniques of data mining, Target analyzed
historical data on customers who later were revealed to have been pregnant.
◆ Pregnant mothers often change their diets, their
Data-Driven Decision Making (DDD)
● The Target example is an example of Type 2 problem
● It can typically be handled by discovering a predictive
model
● Predictive model abstracts away most of the complexity
Data-Driven Decision Making (DDD)
● In both the Walmart and the Target example, the data
analysis was not testing a simple hypothesis.
● Instead, the data were explored with the hope that
something useful would be discovered.
● Also referred to as data mining, predictive analytics,
business intelligence
Video for predictive analytics
Data mining:
the core of knowledge discovery process
Data Cleaning Data Integration Data Warehouse Task-relevant Data Selection Data Mining Pattern Evaluation
Input Data Data Mining Data Pre-Processing Post-Processing
This is a view from typical machine learning and statistics communities Data integration
Normalization Feature selection Dimension reduction
Pattern discovery
Association & correlation Classification Clustering Outlier analysis … … … … Pattern evaluation Pattern selection Pattern interpretation Pattern visualization
Steps of a General KDD Process
● Learning the application domain:
relevant prior knowledge and goals of application
● Creating a target data set: data selection
● Data cleaning and preprocessing: (may take 60% of effort!) ● Data reduction and transformation:
● Find useful features, dimensionality/variable reduction, invariant
representation.
● Choosing functions of data mining
summarization, classification, regression, association, clustering.
● Choosing the mining algorithm(s)
Classification and Prediction
Finding models (functions) that describe and
distinguish classes or concepts for future
prediction
e.g., classify countries based on climate, or
identify good clients
Model: decision-tree, classification rule, neural
network
Cluster analysis
Class label is unknown: Group data to form new
classes
e.g., cluster houses to find distribution
patterns
Clustering based on the principle: maximizing
the intra-class similarity and minimizing the
interclass similarity
Association (correlation and causality)
age(X, “20..29”) ^ income(X, “20K..29K”)
→ buys(X, “PC”)
[support = 2%, confidence = 60%]
Data Mining Database Technology Statistics Other Disciplines Information Science Machine Learning Visualization
