Data Mining Association Analysis: Basic Concepts and Algorithms Lecture Notes for Chapter 6 Introduction to Data Mining

Data Mining

Association Analysis: Basic Concepts

and Algorithms

Lecture Notes for Chapter 6

Introduction to Data Mining

by

Association Rule Mining

●

Given a set of transactions, find rules that will predict the

occurrence of an item based on the occurrences of other

items in the transaction

Market-Basket transactions

Example of Association Rules

{Diaper} → {Beer},

{Milk, Bread} → {Eggs,Coke},

{Beer, Bread} → {Milk},

Implication means co-occurrence,

not causality!

Definition: Frequent Itemset

●

Itemset

–

A collection of one or more items

Example: {Milk, Bread, Diaper}

–

k-itemset

◆

An itemset that contains k items

●

Support count (σ)

–

Frequency of occurrence of an itemset

–

E.g. σ({Milk, Bread,Diaper}) = 2

●

Support

–

Fraction of transactions that contain

an itemset

–

E.g. s({Milk, Bread, Diaper}) = 2/5

●

Frequent Itemset

–

An itemset whose support is greater

than or equal to a minsup threshold

Definition: Association Rule

Example:

●

Association Rule

–

An implication expression of the form

X → Y, where X and Y are itemsets

–

Example:

{Milk, Diaper} → {Beer}

●

Rule Evaluation Metrics

–

Support (s)

◆

Fraction of transactions that contain

both X and Y

–

Confidence (c)

◆

Measures how often items in Y

appear in transactions that

Association Rule Mining Task

●

Given a set of transactions T, the goal of

association rule mining is to find all rules having

–

support ≥ minsup threshold

–

confidence ≥ minconf threshold

●

Brute-force approach:

–

List all possible association rules

–

Compute the support and confidence for each rule

–

Prune rules that fail the minsup and minconf

thresholds

Mining Association Rules

Example of Rules:

{Milk,Diaper} → {Beer} (s=0.4, c=0.67)

{Milk,Beer} → {Diaper} (s=0.4, c=1.0)

{Diaper,Beer} → {Milk} (s=0.4, c=0.67)

{Beer} → {Milk,Diaper} (s=0.4, c=0.67)

{Diaper} → {Milk,Beer} (s=0.4, c=0.5)

{Milk} → {Diaper,Beer} (s=0.4, c=0.5)

Observations:

• All the above rules are binary partitions of the same itemset:

{Milk, Diaper, Beer}

• Rules originating from the same itemset have identical support but

can have different confidence

Mining Association Rules

●

Two-step approach:

1.

Frequent Itemset Generation

–

Generate all itemsets whose support ≥ minsup

2.

Rule Generation

–

Generate high confidence rules from each frequent itemset,

where each rule is a binary partitioning of a frequent itemset

●

Frequent itemset generation is still

Frequent Itemset Generation

Given d items, there

are 2

d

possible

Frequent Itemset Generation

●

Brute-force approach:

–

Each itemset in the lattice is a

candidate

frequent itemset

–

Count the support of each candidate by scanning the

database

–

Match each transaction against every candidate

Computational Complexity

●

Given d unique items:

–

Total number of itemsets = 2

d

–

Total number of possible association rules:

Frequent Itemset Generation Strategies

●

Reduce the

number of candidates

(M)

–

Complete search: M=2

d

–

Use pruning techniques to reduce M

●

Reduce the

number of transactions

(N)

–

Reduce size of N as the size of itemset increases

–

Used by DHP and vertical-based mining algorithms

●

Reduce the

number of comparisons

(NM)

–

Use efficient data structures to store the candidates or

transactions

–

No need to match every candidate against every

Reducing Number of Candidates

●

Apriori principle

:

–

If an itemset is frequent, then all of its subsets must also

be frequent

●

Apriori principle holds due to the following property

of the support measure:

–

Support of an itemset never exceeds the support of its

subsets

Found to be

Infrequent

Illustrating Apriori Principle

Pruned

supersets

Illustrating Apriori Principle

Items (1-itemsets)

Pairs (2-itemsets)

(No need to generate

candidates involving Coke

or Eggs)

Triplets (3-itemsets)

Minimum Support = 3

If every subset is considered,

6

_C

+

_C

+

_C

= 41

With support-based pruning,

6 + 6 + 1 = 13

Apriori Algorithm

●

Method:

–

Let k=1

–

Generate frequent itemsets of length 1

–

Repeat until no new frequent itemsets are identified

◆

Generate length (k+1) candidate itemsets from length k

frequent itemsets

◆

Prune candidate itemsets containing subsets of length k that

are infrequent

◆

Count the support of each candidate by scanning the DB

◆

Eliminate candidates that are infrequent, leaving only those that

are frequent

Reducing Number of Comparisons

●

Candidate counting:

–

Scan the database of transactions to determine the

support of each candidate itemset

–

To reduce the number of comparisons, store the

candidates in a hash structure

◆

Instead of matching each transaction against every candidate,

match it against candidates contained in the hashed buckets

Generate Hash Tree

2 3 4

5 6 7

1 4 5

1 3 6

1 2 4

4 5 7

1 2 5

4 5 8

1 5 9

3 4 5

3 5 6

3 5 7

6 8 9

3 6 7

3 6 8

1,4,7

2,5,8

3,6,9

Hash function

Suppose you have 15 candidate itemsets of length 3:

{1 4 5}, {1 2 4}, {4 5 7}, {1 2 5}, {4 5 8}, {1 5 9}, {1 3 6}, {2 3 4}, {5 6 7}, {3 4 5},

{3 5 6}, {3 5 7}, {6 8 9}, {3 6 7}, {3 6 8}

You need:

• Hash function

• Max leaf size: max number of itemsets stored in a leaf node (if number of

candidate itemsets exceeds max leaf size, split the node)

Association Rule Discovery: Hash tree

1

5 9

1

4

5

1

3 6

3

4

5

3 6 7

3 6 8

3 5 6

3 5 7

6 8 9

2 3 4

5 6 7

1

2

4

4

5

7

1

2 5

4

5 8

1,4,7

2,5,8

3,6,9

Hash Function

_{Candidate Hash Tree}

Hash on

1, 4 or 7

Association Rule Discovery: Hash tree

1

5

9

1 4 5

1 3 6

3 4 5

3 6 7

3 6 8

3

5

6

3

5

7

6

8

9

2

3 4

5

6 7

1

2

4

4

5

7

1

2

5

4

5

8

1,4,7

2,5,8

3,6,9

Hash Function

_{Candidate Hash Tree}

Hash on

2, 5 or 8

Association Rule Discovery: Hash tree

1 5

9

1 4 5

1 3

6

3

4 5

3

6

7

3

6

8

3

5 6

3

5 7

6

8 9

2 3 4

5 6 7

1 2 4

4 5 7

1 2 5

4 5 8

1,4,7

2,5,8

3,6,9

Hash Function

_{Candidate Hash Tree}

Hash on

3, 6 or 9

Subset Operation

Given a transaction t, what are

the possible subsets of size

3?

Subset Operation Using Hash Tree

1 5 9

1 4 5

1 3 6

3 4 5

3 6 7

3 6 8

3 5 6

3 5 7

6 8 9

2 3 4

5 6 7

1 2 4

4 5 7

1 2 5

4 5 8

1 2 3 5 6

1 + 2 3 5 6

3 5 6

2 +

5 6

3 +

1,4,7

2,5,8

3,6,9

Hash Function

transaction

Subset Operation Using Hash Tree

1 5 9

1 4 5

1 3 6

3 4 5

3 6 7

3 6 8

3 5 6

3 5 7

6 8 9

2 3 4

5 6 7

1 2 4

4 5 7

1 2 5

4 5 8

1,4,7

2,5,8

3,6,9

Hash Function

1 2 3 5 6

3 5 6

1 2 +

5 6

1 3 +

6

1 5 +

3 5 6

2 +

5 6

3 +

1 + 2 3 5 6

transaction

Subset Operation Using Hash Tree

1 5 9

1 4 5

1 3 6

3 4 5

3 6 7

3 6 8

3 5 6

3 5 7

6 8 9

2 3 4

5 6 7

1 2 4

4 5 7

1 2 5

4 5 8

1,4,7

2,5,8

3,6,9

Hash Function

1 2 3 5 6

3 5 6

1 2 +

5 6

1 3 +

6

1 5 +

3 5 6

2 +

5 6

3 +

1 + 2 3 5 6

transaction

Factors Affecting Complexity

●

Choice of minimum support threshold

–

lowering support threshold results in more frequent itemsets

–

this may increase number of candidates and max length of

frequent itemsets

●

Dimensionality (number of items) of the data set

–

more space is needed to store support count of each item

–

if number of frequent items also increases, both computation and

I/O costs may also increase

●

Size of database

–

since Apriori makes multiple passes, run time of algorithm may

increase with number of transactions

●

Average transaction width

–

transaction width increases with denser data sets

–

This may increase max length of frequent itemsets and traversals

of hash tree (number of subsets in a transaction increases with its

width)

Compact Representation of Frequent Itemsets

●

Some itemsets are redundant because they have identical

support as their supersets

●

Number of frequent itemsets

Maximal Frequent Itemset

Border

Infrequent

Itemsets

Maximal

Itemsets

An itemset is maximal frequent if none of its immediate supersets

is frequent

Closed Itemset

●

An itemset is closed if none of its immediate supersets

Maximal vs Closed Itemsets

Transaction Ids

Not supported by

any transactions

Maximal vs Closed Frequent Itemsets

Minimum support = 2

# Closed = 9

# Maximal = 4

Closed and

maximal

Closed but

not maximal

Alternative Methods for Frequent Itemset Generation

●

Traversal of Itemset Lattice

Alternative Methods for Frequent Itemset Generation

●

Traversal of Itemset Lattice

Alternative Methods for Frequent Itemset Generation

●

Traversal of Itemset Lattice

Alternative Methods for Frequent Itemset Generation

●

Representation of Database

FP-growth Algorithm

●

Use a compressed representation of the

database using an

FP-tree

●

Once an FP-tree has been constructed, it uses a

recursive divide-and-conquer approach to mine

the frequent itemsets

FP-tree construction

null

A:1

B:1

null

A:1

B:1

B:1

C:1

D:1

After reading TID=1:

FP-Tree Construction

null

A:7

B:5

B:3

C:3

D:1

C:1

D:1

C:3

D:1

D:1

E:1

E:1

Pointers are used to assist

frequent itemset generation

D:1

E:1

Transaction

Database

FP-growth

null

A:7

B:5

B:1

C:1

D:1

C:1

D:1

C:3

D:1

D:1

Conditional Pattern base

for D:

P = {(A:1,B:1,C:1),

(A:1,B:1),

(A:1,C:1),

(A:1),

(B:1,C:1)}

Recursively apply

FP-growth on P

Frequent Itemsets found

(with sup > 1):

AD, BD, CD, ACD, BCD

D:1

Tree Projection

Set enumeration tree:

Possible Extension:

E(A) = {B,C,D,E}

Possible Extension:

E(ABC) = {D,E}

Tree Projection

●

Items are listed in lexicographic order

●

Each node P stores the following information:

–

Itemset for node P

–

List of possible lexicographic extensions of P: E(P)

–

Pointer to projected database of its ancestor node

–

Bitvector containing information about which

transactions in the projected database contain the

itemset

Projected Database

Original Database:

Projected Database

for node A:

ECLAT

●

For each item, store a list of transaction ids (tids)

ECLAT

●

Determine support of any k-itemset by intersecting tid-lists

of two of its (k-1) subsets.

●

3 traversal approaches:

–

top-down, bottom-up and hybrid

●

Advantage: very fast support counting

●

Disadvantage: intermediate tid-lists may become too large

for memory

Rule Generation

●

Given a frequent itemset L, find all non-empty

subsets f ⊂ L such that f → L – f satisfies the

minimum confidence requirement

–

If {A,B,C,D} is a frequent itemset, candidate rules:

ABC →D, ABD →C, ACD →B, BCD →A,

A →BCD,

B →ACD,

C →ABD, D →ABC

AB →CD,

AC → BD, AD → BC, BC →AD,

BD →AC, CD →AB,

●

If |L| = k, then there are 2

k

– 2 candidate

Rule Generation

●

How to efficiently generate rules from frequent

itemsets?

–

In general, confidence does not have an

anti-monotone property

c(ABC →D) can be larger or smaller than c(AB →D)

–

But confidence of rules generated from the same

itemset has an anti-monotone property

–

e.g., L = {A,B,C,D}:

c(ABC → D) ≥ c(AB → CD) ≥ c(A → BCD)

◆

Confidence is anti-monotone w.r.t. number of items on the RHS

of the rule

Rule Generation for Apriori Algorithm

Lattice of rules

Pruned

Rules

Low

Confidence

Rule

Rule Generation for Apriori Algorithm

●

Candidate rule is generated by merging two rules

that share the same prefix

in the rule consequent

●

join(CD=>AB,BD=>AC)

would produce the candidate

rule D => ABC

●

Prune rule D=>ABC if its

subset AD=>BC does not have

high confidence

Effect of Support Distribution

●

Many real data sets have skewed support

distribution

Support

distribution of a

retail data set

Effect of Support Distribution

●

How to set the appropriate minsup threshold?

–

If minsup is set too high, we could miss itemsets

involving interesting rare items (e.g., expensive

products)

–

If minsup is set too low, it is computationally

expensive and the number of itemsets is very large

●

Using a single minimum support threshold may

Multiple Minimum Support

●

How to apply multiple minimum supports?

–

MS(i): minimum support for item i

–

e.g.: MS(Milk)=5%, MS(Coke) = 3%,

MS(Broccoli)=0.1%, MS(Salmon)=0.5%

–

MS({Milk, Broccoli}) = min (MS(Milk), MS(Broccoli))

= 0.1%

–

Challenge: Support is no longer anti-monotone

◆

Suppose: Support(Milk, Coke) = 1.5% and

Support(Milk, Coke, Broccoli) = 0.5%

Multiple Minimum Support (Liu 1999)

●

Order the items according to their minimum

support (in ascending order)

–

e.g.: MS(Milk)=5%, MS(Coke) = 3%,

MS(Broccoli)=0.1%, MS(Salmon)=0.5%

–

Ordering: Broccoli, Salmon, Coke, Milk

●

Need to modify Apriori such that:

–

L

₁

: set of frequent items

–

F

₁

: set of items whose support is ≥ MS(1)

where MS(1) is min

_i

( MS(i) )

–

C

₂

: candidate itemsets of size 2 is generated from F

₁

Multiple Minimum Support (Liu 1999)

●

Modifications to Apriori:

–

In traditional Apriori,

◆

A candidate (k+1)-itemset is generated by merging two

frequent itemsets of size k

◆

The candidate is pruned if it contains any infrequent subsets

of size k

–

Pruning step has to be modified:

◆

Prune only if subset contains the first item

◆

e.g.: Candidate={Broccoli, Coke, Milk} (ordered according to

minimum support)

◆

{Broccoli, Coke} and {Broccoli, Milk} are frequent but

{Coke, Milk} is infrequent

– Candidate is not pruned because {Coke,Milk} does not contain

the first item, i.e., Broccoli.

Pattern Evaluation

●

Association rule algorithms tend to produce too

many rules

–

many of them are uninteresting or redundant

–

Redundant if {A,B,C} → {D} and {A,B} → {D}

have same support & confidence

●

Interestingness measures can be used to

prune/rank the derived patterns

●

In the original formulation of association rules,

Application of Interestingness Measure

Interestingness

Measures

Computing Interestingness Measure

●

Given a rule X → Y, information needed to compute rule

interestingness can be obtained from a contingency table

Y

Y

X

f

f

f

X

f

f

f

f

f

|T|

Contingency table

for

X → Y

f

₁₁

: support of X and Y

f

₁₀

: support of X and Y

f

₀₁

: support of X and Y

f

₀₀

: support of X and Y

Used to define various measures

●

support, confidence, lift, Gini,

Drawback of Confidence

Coffee

Coffee

Tea

15

5

20

Tea

75

5

80

90

10

100

Association Rule: Tea → Coffee

Confidence= P(Coffee|Tea) =

0.75

but P(Coffee) =

0.9

⇒ Although confidence is high, rule is misleading

⇒ P(Coffee|Tea) = 0.9375

Statistical Independence

●

Population of 1000 students

–

600 students know how to swim (S)

–

700 students know how to bike (B)

–

420 students know how to swim and bike (S,B)

–

P(S∧B) = 420/1000 = 0.42

–

P(S) × P(B) = 0.6 × 0.7 = 0.42

–

P(S∧B) = P(S) × P(B) => Statistical independence

–

P(S∧B) > P(S) × P(B) => Positively correlated

Statistical-based Measures

●

Measures that take into account statistical

Example: Lift/Interest

Coffee

Coffee

Tea

15

5

20

Tea

75

5

80

90

10

100

Association Rule: Tea → Coffee

Confidence= P(Coffee|Tea) =

0.75

but P(Coffee) =

0.9

Drawback of Lift & Interest

Y

Y

X

10

0

10

X

0

90

90

10

90

100

Y

Y

X

90

0

90

X

0

10

10

90

10

100

Statistical independence:

If P(X,Y)=P(X)P(Y) => Lift = 1

There are lots of

measures proposed

in the literature

Some measures are

good for certain

applications, but not

for others

What criteria should

we use to determine

whether a measure

is good or bad?

What about

Apriori-style support

based pruning? How

does it affect these

measures?

Properties of A Good Measure

●

Piatetsky-Shapiro

:

3 properties a good measure M must satisfy:

–

M(A,B) = 0 if A and B are statistically independent

–

M(A,B) increase monotonically with P(A,B) when P(A)

and P(B) remain unchanged

–

M(A,B) decreases monotonically with P(A) [or P(B)]

when P(A,B) and P(B) [or P(A)] remain unchanged

Comparing Different Measures

10 examples of

contingency tables:

Rankings of contingency tables

using various measures:

Property under Variable Permutation

Does M(A,B) = M(B,A)?

Symmetric measures:

●

support, lift, collective strength, cosine, Jaccard, etc

Asymmetric measures:

Property under Row/Column Scaling

Male

Female

High

2

3

5

Low

1

4

5

3

7

10

Male

Female

High

4

30

34

Low

2

40

42

6

70

76

Grade-Gender Example (Mosteller, 1968):

Mosteller:

Underlying association should be independent of

the relative number of male and female students

in the samples

Property under Inversion Operation

Transaction 1

Transaction N

.

.

.

.

.

Example: φ-Coefficient

●

φ-coefficient is analogous to correlation coefficient

for continuous variables

Y

Y

X

60

10

70

X

10

20

30

70

30

100

Y

Y

X

20

10

30

X

10

60

70

30

70

100

Property under Null Addition

Invariant measures:

●

support, cosine, Jaccard, etc

Non-invariant measures:

Support-based Pruning

●

Most of the association rule mining algorithms

use support measure to prune rules and itemsets

●

Study effect of support pruning on correlation of

itemsets

–

Generate 10000 random contingency tables

–

Compute support and pairwise correlation for each

table

–

Apply support-based pruning and examine the tables

Effect of Support-based Pruning

Support-based pruning

eliminates mostly

negatively correlated

itemsets

Effect of Support-based Pruning

●

Investigate how support-based pruning affects

other measures

●

Steps:

–

Generate 10000 contingency tables

–

Rank each table according to the different measures

–

Compute the pair-wise correlation between the

Effect of Support-based Pruning

●

Without Support Pruning (All Pairs)

●

Red cells indicate correlation between

the pair of measures > 0.85

●

40.14% pairs have correlation > 0.85

Scatter Plot between Correlation

& Jaccard Measure

Effect of Support-based Pruning

●

0.5% ≤ support ≤ 50%

●

61.45% pairs have correlation > 0.85

Scatter Plot between Correlation

& Jaccard Measure:

Effect of Support-based Pruning

●

0.5% ≤ support ≤ 30%

●

76.42% pairs have correlation > 0.85

Scatter Plot between Correlation

& Jaccard Measure

Subjective Interestingness Measure

●

Objective measure:

–

Rank patterns based on statistics computed from data

–

e.g., 21 measures of association (support, confidence,

Laplace, Gini, mutual information, Jaccard, etc).

●

Subjective measure:

–

Rank patterns according to user’s interpretation

◆

A pattern is subjectively interesting if it contradicts the

expectation of a user (Silberschatz & Tuzhilin)

◆

A pattern is subjectively interesting if it is actionable

(Silberschatz & Tuzhilin)

Interestingness via Unexpectedness

●

Need to model expectation of users (domain knowledge)

●

Need to combine expectation of users with evidence from

data (i.e., extracted patterns)

+

Pattern expected to be frequent

-

Pattern expected to be infrequent

Pattern found to be frequent

Pattern found to be infrequent

+

-Expected Patterns

Interestingness via Unexpectedness

●

Web Data (Cooley et al 2001)

–

Domain knowledge in the form of site structure

–

Given an itemset F = {X

₁

, X

₂

, …, X

_k

} (X

_i

: Web pages)

◆

L: number of links connecting the pages

◆

lfactor = L / (k × k-1)

◆

cfactor = 1 (if graph is connected), 0 (disconnected graph)

–

Structure evidence = cfactor × lfactor

–

Usage evidence

–

Use Dempster-Shafer theory to combine domain