Chap 3. Decision Tree Learning

Chap 3. Decision Tree Learning

Machine Learning Spring, 2009 Cheong Hee Park

Modified from lecture slides for textbook Machine Learning, Tom M. Mitchell, McGraw Hill, 1997

Outline

Decision tree learning

Approximate discrete-valued target functions

The learned function is represented by a decision tree

ID3 learning algorithm Entropy, Information gain

Issues in decision tree learning Avoiding overfitting the data

Incorporating continuous-valued attributes Handling missing attribute values

Decision Tree for P layT ennis

Outlook

Overcast

Humidity

Normal High

No Yes

Wind

Strong Weak

No Yes

Yes Sunny Rain

Decision Trees

Decision tree representation:

Each internal node tests an attribute

Each branch corresponds to attribute value Each leaf node assigns a classification

Decision trees represent:

Each path from the tree root to a leaf -> a conjunction of attribute tests

Tree -> a disjunction of these conjunctions

(Outlook = Sunny ∧ Humidity = N ormal) ∨ (Outlook = Overcast) ∨ (Outlook = Rain ∧ W ind = W eak)

When to Consider Decision Trees

Instances describable by attribute–value pairs Target function is discrete valued

Disjunctive hypothesis may be required Possibly noisy training data

Examples:

Equipment or medical diagnosis Credit risk analysis

Modeling calendar scheduling preferences

Top-Down Induction of Decision Trees

Main loop:

1. A ← the “best” decision attribute for next node 2. Assign A as decision attribute for node

3. For each value of A, create new descendant of node 4. Sort training examples to leaf nodes

5. If training examples perfectly classified, Then STOP, Else iterate over new leaf nodes

Which attribute is best?

Entropy

S is a sample of training examples

p⊕ is the proportion of positive examples in S p is the proportion of negative examples in S Entropy measures the impurity of S

Entropy(S) ≡ −p⊕ log₂ p⊕ − p log₂ p

Entropy(S)

1.0

0.5

0.0 0.5 1.0

p+

Entropy

Entropy(S) = expected number of bits needed to encode class (⊕ or ) of randomly drawn member of S (under the optimal, shortest-length code)

Why? Information theory: optimal length code assigns

− log₂ p bits to message having probability p.

So, expected number of bits to encode ⊕ or of random member of S:

p⊕(− log₂ p⊕) + p (− log₂ p )

Entropy(S) ≡ −p⊕ log₂ p_⊕ − p log₂ p

Information Gain

Gain(S, A) = expected reduction in entropy due to sorting on A

Gain(S, A) ≡ Entropy(S) − X

v∈V alues(A)

|S_v|

|S| Entropy(S_v)

ID3(Examples, Attributes)

Examples: training examples

Attributes: a list of other attributes that may be tested by the learned decision tree

Create a Root node for the tree If all examples are positive,

return the single-node tree Root, with label = + If all examples are negative,

return the single-node tree Root, with label = -

If attributes is empty, return the single-node tree Root, with label = majority between + and -

Otherwise Begin

See the next page Return Root

ID3 - continued

A <- the attribute from Attributes that best classifies Examples

The decision attribute for Root <- A For each possible value, v_i, of A

Add a new tree branch below Root, corresponding to the test A = v_i

Let Example_vi be the subset of Examples that have value v_i for A

If Examples_vi is empty

Then below this new branch, add a leaf node with label = most common value of target attribute in Examples

Else below this new branch, add the subtree ID3(Examples_vi, Attributes − {A})

Training Examples

Day Outlook Temperature Humidity Wind PlayTennis

D1 Sunny Hot High Weak No

D2 Sunny Hot High Strong No

D3 Overcast Hot High Weak Yes

D4 Rain Mild High Weak Yes

D5 Rain Cool Normal Weak Yes

D6 Rain Cool Normal Strong No

D7 Overcast Cool Normal Strong Yes

D8 Sunny Mild High Weak No

D9 Sunny Cool Normal Weak Yes

Training Examples-continued

Day Outlook Temperature Humidity Wind PlayTennis

D10 Rain Mild Normal Weak Yes

D11 Sunny Mild Normal Strong Yes

D12 Overcast Mild High Strong Yes

D13 Overcast Hot Normal Weak Yes

D14 Rain Mild High Strong No

Determine the best attribute in Root

Gain(S, Outlook) = 0.246 Gain(S, Humidity) = 0.151 Gain(S, Wind) = 0.048

Gain(S, Temperature) = 0.029

S_sunny = {D1, D2, D8, D9, D11}

Gain(S_sunny, Humidity) =?

Gain(S_sunny, T emperature) =?

Gain(S_sunny, W ind) =?

Hypothesis Space Search by ID3

...

+ + +

A1

+ – + –

A2

A3

+

...

+ – + –

A2

A4

– + – + –

A2

+ – +

... ...

–

Hypothesis Space Search by ID3

Hypothesis space is complete!

Target function surely in there...

It performs a simple-to-complex, hill-climbing search

through the hypothesis space, beginning with the empty tree

Outputs a single hypothesis

It contrasts with Candidate − Elimination method No back tracking

Local minima...

It uses all training examples at each step to make statisically-based search choices

Robust to noisy data...

Inductive Bias in ID3

Approximate inductive bias: “prefer shortest tree”

Preference for short trees, and for those with high information gain attributes near the root

Bias is a preference for some hypotheses, rather than a restriction of hypothesis space H

Occam’s razor: prefer the shortest hypothesis that fits the data

Occam’s Razor

Why prefer short hypotheses?

Argument in favor:

Fewer short hyps. than long hyps.

→ a short hyp that fits data unlikely to be coincidence

→ a long hyp that fits data might be coincidence Argument opposed:

...

...

Issues in Decision Tree Learning

Avoiding overfitting the data

Handling continuous valued attributes

Choosing an appropriate attributes election measure Handling training data with missing attribute values ...

Overfitting in Decision Trees

Consider adding noisy training example #15:

Sunny, Hot, N ormal, Strong, P layT ennis = N o What effect on earlier tree?

Outlook

Overcast

Humidity

Normal High

No Yes

Wind

Strong Weak

No Yes

Yes Sunny Rain

Overfitting

Consider error of hypothesis h over training data: error_train(h)

entire distribution D of data: errorD(h)

Hypothesis h ∈ H ^overfits training data if there is an alternative hypothesis h⁰ ∈ H such that

error_train(h) < error_train(h⁰) and

error_D(h) > errorD(h⁰)

Overfitting in Decision Tree Learning

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9

0 10 20 30 40 50 60 70 80 90 100

Accuracy

Size of tree (number of nodes)

On training data On test data

Avoiding Overfitting

How can we avoid overfitting?

stop growing when data split not statistically significant grow full tree, then post-prune

How to select “best” tree:

Measure performance over training data

Measure performance over separate validation data set MDL: minimize

size(tree) + size(misclassif ications(tree))

Reduced-Error Pruning

Split data into training and validation set Do until further pruning is harmful:

1. Evaluate impact on validation set of pruning each possible node (plus those below it)

2. Greedily remove the one that most improves validation set accuracy

produces smallest version of most accurate subtree What if data is limited?

Effect of Reduced-Error Pruning

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9

0 10 20 30 40 50 60 70 80 90 100

Accuracy

Size of tree (number of nodes)

On training data On test data On test data (during pruning)

Rule Post-Pruning

- converting A Tree to Rules

Outlook

Overcast

Humidity

Normal High

No Yes

Wind

Strong Weak

No Yes

Yes Sunny Rain

IF (Outlook = Sunny) ∧ (Humidity = High) THEN P layT ennis = N o

IF (Outlook = Sunny) ∧ (Humidity = N ormal) THEN P layT ennis = Y es

. . .

Rule Post-Pruning

1. Convert tree to equivalent set of rules 2. Prune each rule independently of others

- by removing any preconditions that result in improving its estimated accuracy

3. Sort final rules into desired sequence for use

Perhaps most frequently used method (e.g., ^C4.5)

Continuous Valued Attributes

Create a discrete attribute to test continuous T emperature = 82.5

(T emperature > 72.3) = t, f

Temperature: 40 48 60 72 80 90 PlayTennis: No No Yes Yes Yes No Compute information gain associated with each of (T emperature > 54) and (T emperature > 85)

Select the best

Unknown Attribute Values

What if some examples missing values of A? Use training example anyway, sort through tree

If node n tests A, assign most common value of A among other examples sorted to node n

assign most common value of A among other examples with same target value

assign probability p_i to each possible value v_i of A

assign fraction p_i of example to each descendant in tree

Classify new examples in same fashion