Detecting and Correcting Errors of Omission After Explanation-based Learning

(1)

Detecting and Correcting Errors of Omission

After Explanation-based Learning

Michael J. Pazzani

Department of Information and Computer Science University of California,

Irvine, CA 92717 [email protected]

(2)

Outline

I. Problem:

A. Detecting errors in generalization produced by EBL B. Assigning blame to rules in domain theory

C. Correcting domain theory II. Types of Errors

A. Errors of Omission- Fail to make correct prediction B. Errors of Commission - Make incorrect prediction

III. Indexing generalizations in Memory

A. Explanatory- Given result, Predict cause B. Predictive- Given action, Predict result IV. Unsupervised detection of errors of omission V. Blame Assignment & revising domain theory

(3)

Background:

OCCAM

• Performance task

• Predict outcome of economic sanction incidents • Outcome inferred by hierarchical classification

• Error of omission occurs if incident cannot be classified

• Error of commission occurs if incident is classified incorrectly

• Learning Method:

•Combines empirical and explanation-based learning

• empirical techniques learn the domain theory used by EBL • Problem: incorrect domain theory → incorrect generalizations

coerce

s1 s2 s3

s5 s6 s7

(4)

Terminology

Types of examples:

Foundational: examples from which the domain theory is learned E.g., Parent helping child

Performance: examples of the performance task E.g., Kidnapping examples

Foundational examples are subproblems of the performance task. Performance examples are examples of the performance task.

Distinction is relative to task. If task is predicting whom to sell ransom insurance to, the kidnapping examples are foundational. Types of rules (or schemata)

Domain: used by EBL to explain performance example

(5)

Problem Statement:

• Rule in domain theory are learned (or hand-coded)

Parents have a goal of preserving their children’s health • Create compiled rule with EBL

One plan to obtain money is to threaten to kill the child of a rich person.

• Detect error of omission in compiled rule from performance example

A kidnapper obtains money from grandparent of hostage. • Assign Blame for error on rule in domain theory

• Revise rule in domain theory

Members of the same family have a goal of preserving each other’s health

(6)

Why Errors of Omission?

Only type of error created by one-sided learning algorithms • Incremental, hill-climbing algorithms (Langley et al., 1987).

• Accounts for some human learning • Grammar: (Berwick, 1986)

• Concept Acquisition (Bruner, et al, 1956)

• Subject of theoretical analysis (Valiant, 84; Haussler, 87) • Hypothesis never more general than “true” hypothesis. Incremental blame assignment and revision possible:

Error of Omission: One of the domain theory rules used to create a compiled rule needs to be

generalized by dropping a condition that is not present in a performance example Error of commission: One of the domain theory rules used to

create a compiled rule needs to be

specialized by adding a condition that is present in a new performance example.

(7)

Unsupervised detection of error of omission

Problem: Find a schema in memory that would predict the outcome of a new performance example, if the schema were more general. Approach: Distinguish between two uses of schemata

Predictive: What would happen if the United States

refused to sell computers to South Korea unless South Korea stopped exporting automobiles to Canada?

Explanatory: What could cause the price of oil to rise?

coerce

s1 s2 s3

s5 s6 s7

(8)

Derivation and use of Indices Derived analytically:

explanatory: Features from consequent of domain rules. predictive: Features from antecedent of domain rules. Use during retrieval:

explanatory: Finding a schema to explain the cause of an outcome predictive: Finding a schema to predict the outcome of an event.

Determining indices: An example Australia and France, 1983

In 1983, Australia refused to sell uranium to France, unless France ceased nuclear testing in the South Pacific. France paid a higher price to buy uranium from South Africa and continued nuclear testing.

(9)

Determining indices: An example 1. Threat -> Increased demand

2. Increased demand -> Willingness to pay higher price 3. Purchase -> Possess

(ACT TYPE (SELL)

ACTOR (POLITY EXPORTS ?Y

ECONOMY (FREE)) TO ?X:(POLITY IMPORTS ?Y

ECONOMY (FREE))

OBJECT ?Y:(COMMODITY) MODE (NEG))

(STATE TYPE (DEMAND-INCREASE) ACTOR ?X

OBJECT ?Y) result

(10)

Indexing by predictive and explanatory features

coerce

(coerce actor (polity exports =OBJECT economy (free))

target (polity economic-health (strong)

economy (free)

imports =OBJECT) ...

response (act type (sell)

actor (polity bus-rel =TARGET exports =OBJECT) object =OBJECT

price (money value (>market)) to =TARGET)

outcome (goal-outcome type (failure))

actor target ... response outcome

(11)

Detecting an error of omission input: New performance example

Hierarchy of schema (domain & compiled theory) Retrieve schema by following predictive indices

If schema has outcome ,

Then If outcome of example and schema agree Then EXIT

Else “Error of commission” Else If example is explainable

Then EBL(example)

Else If retrieve schema by explanatory indices Then “attempt blame assignment” Else TDL(example) or SBL(example)

Find a schema sufficiently close to the performance example that would explain the example if generalized.

(12)

Blame assignment

(defining “sufficiently close”)

1. Reexplain the schema with EBL that maintains dependencies between constraints in the compiled rule and rules in the

domain theory. (Trade-off between storing & recomputing):

(coerce

target (polity economic-health (strong) <- Rule.013 economy (free) <- Rule.012 imports =OBJECT) <- {Rule.01 Rule.13}

2. Find differences between the schema and the new event. 3. Collect inference rules responsible for differences.

4. If one inference rule is responsible for all the differences, then assign blame to this inference rule.

(13)

Blame assignment: An example US and USSR, 1980

In 1980, the US refused to sell grain to the Soviet Union if the Soviet Union did not withdraw its troops from Afghanistan. The

Soviet Union paid a higher price to buy grain from Argentina and did not withdraw from Afghanistan.

2. Find differences:

(coerce target (polity economy (free)))

3. Collect rules: {rule.12}

(14)

Correcting the domain theory

Convert performance example to foundational example:

(COERCE ACTOR (POLITY NAME (US)

ECONOMY (FREE) ...) TARGET (POLITY NAME (USSR)

ECONOMY (CONTROLLED)...) THREAT (ACT TYPE (SELL)

ACTOR =ACTOR TO =TARGET MODE (NEG)) (ACT TYPE (SELL)

ACTOR (POLITY TYPE (COUNTRY) NAME (US)

ECONOMY (FREE) ...) TO (POLITY TYPE (COUNTRY)

NAME (USSR)

ECONOMY (CONTROLLED)...)

OBJECT (COMMODITY AVAILABILITY (COMMON) TYPE (GRAIN))

MODE (NEG)) RESULT

(STATE TYPE (DEMAND-INCREASE)

(15)

Correcting the domain theory (cont’d)

Generalize antecedent of rule to accommodate new example:

Old:

(ACT TYPE (SELL)

ECONOMY (FREE)) TO ?X:(POLITY IMPORTS ?Y ECONOMY (FREE)) OBJECT ?Y:(COMMODITY) MODE (NEG)) New:

(ACT TYPE (SELL)

ECONOMY (FREE)) TO ?X:(POLITY IMPORTS ?Y)

OBJECT ?Y:(COMMODITY) MODE (NEG))

(16)

Experimental Design

• All rules that indicate effect of threat were modified by

conjoining all preconditions. (cf. Kolodner, 1984)

• Run on economic sanction database (15, actual, 5 hypo) • Accuracy measured after 10 and 15 examples for 10 trials.

Results

OCCAM with error correction more accurate than OCCAM

p<.005, t(18)=3.16

OCCAM with correct knowledge base more accurate (few examples)

10 15 0.0 0.2 0.4 0.6 0.8 1.0 occam occam+errors occam+errors+ Number of examples Accuracy correction

(17)

Related Work

• Recovering from incorrect knowledge in SOAR (Laird, 1988) • “patches” compiled theory to avoid incorrect knowledge • doesn’t update domain theory

• may require patching for each use • ML-SMART (Bergadano et al., 1988)

• Batch system that works on complete set of examples • can handle errors of commission

• supervised

• Theory Revision (Ginsberg, 1988) • Batch system

• Can handle classification noise • supervised

(18)

Conclusion

I. Unsupervised detection of errors of omission requires distinguishing between explanatory and predictive uses of schemata.

II. Assigning blame on rules in the domain theory for errors in compiled theory can be accomplished by maintaining dependencies between conditions of