Uncertainty in Learner modelling

A typical Learner Model performs four actions: diagnosis (knowledge tracing or plan recognition), feedback selection, predicting and model updating, and curriculum advancing. The term knowledge tracing (Corbett et al. 1992; Conati et al. 2002) refers to

the process of inferring the learner’s knowledge state from their behaviour. Plan recognition is slightly different from the diagnosis task. Diagnosis is an attempt to attribute meaning to student behaviour during the performance of a task, or after it has been completed, whereas plan recognition describes an attempt to identify the plan or

goal of the learner before they complete a task (Greer et al. 1995). A fine-grained domain

model is necessary for micro level diagnosis in order to provide interactive problem solving help (while a learner is working on a solution) in procedural type learning domains.

If a CBL system makes critical assumptions based on an inaccurate Learner model and vague observations, it cannot carry out perfect diagnoses. Therefore, the potential remedial actions which are usually based on these inaccurate diagnoses could be inappropriate. Later, if the Learner model is updated in the light of having been given useless feedback, the prediction could be wrong. Furthermore, the next lesson, if selected using a new predicted Learner Model, could be untimely. Iteratively, if this process continues, the errors could accumulate, and eventually the system might behave erratically (Mayo 2001). The resultant Learner model could significantly deviate from the real user (Figure 2.5). Eventually, this would adversely affect the trustworthiness of the system and could seriously impact the motivation of the learner.

Villano (1992) suggests a range of strategies to use the Bayesian Belief Network (BBN) for typical pedagogical actions such as item selection (the best next question), updating, curriculum advancement, hint-selection (during problem solving) and feedback (after problem solving). Basically, he uses prerequisite links between various test-items using AND/OR graphs. In the Corbett et al. (1992) scheme, the Learner model is a set of LISP

state (with no forgetting). For each rule, the probability that a rule is in the learned state will be associated for a learner. Though it is not explicitly stated, the Dynamic Bayesian Network rules (Russell et al. 2003) are used to update this probability with relevant

interaction. An important characteristic of Corbett et al’s (ibid.) model is that the

programming rules are considered independent.

Reye et al. (1995) suggest the usage of statistical decision theory for post-diagnostic

pedagogical actions. Collins et al. (1996) use BBN for adaptive testing based on

granularity hierarchies (McCalla et al. 1994) and then use a utility function based on

statistical decision theory for item selection (the best next question). Mayo et al. (2001)

also use utility functions with BBN for curriculum advancing in their systems called CAPIT. As Mayo et al. put it:

“It [pedagogical module of CAPIT] performs two significant PAS tasks. Firstly, given the violated constraints, it selects the single violated constraint about which feedback should be given, secondly when Pick Another Problem is clicked, or when the student solves the current problem, the pedagogical module selects the most appropriate next problem for the student” (Mayo et al. 2001, p.14).

Figure 2.5 (a) Learner Model Usage (Mayoet al.2001)(b) Uncertainty Escalation Present lesson and problems

Select remedy Present remedy

update

Select next lesson

Learner Model Diagnose (a) Reality remedy diagnosis update 1st start

Learner model deviates widely from reality in each cycle

2nd_Lesson next lesson

(b)

Model of Learner

CAPIT is based on constraint-based modelling (Ohlsson 1994), a successful variation of overlay models (Mitrovic 2005). The structure and prior probabilities of the network in CAPIT are selected using machine learning techniques. Murray et al. (2000) also used

utility functions for curriculum advancing in their systems called DT Tutor. The backbone of DT Tutor is the Dynamic Decision Network, which is essentially a Dynamic Bayesian Network with decision and utility nodes. It also accounts for the student’s focus of attention and emotional states. Stern et al. (1999) use machine learning techniques in

MANIC to improve the prediction using BBN in user modelling. Other than DT Tutor, Conati et al. (2002) describe a CBL system for Newtonian physics, ANDES, that uses

BBN for three key tasks: knowledge tracing-inferring from student’s actions what the student knows, plan recognition- why the student did something, and prediction- what will the student do next. To enable micro level diagnosis, the knowledge in ANDES is represented at a fine-grained level using AND/OR graphs, but, unlike Villano (1992), ANDES uses aggregation links (instead of prerequisite links).

Most of the systems discussed above use belief propagation when external evidence becomes available. It is computationally a very expensive task. However, Murray et al.

are more optimistic “Unfortunately, belief network inference is still NP-hard in the worst case. However, many stochastic sampling algorithms have an ‘any time property’ that allows an approximate result to be obtained at any point in the computation”((Murray et al. 2000, p. 154).

Hawkes et al. (1990) first proposed fuzzy theory (Zadeh 1973; Hopgood 2000) for

Learner modelling in the system called TAPS. The following usages of this model other than diagnosing are enthusiastically discussed; selecting instructional routine, intervention frequency, and mentoring type. The learner’s emotional states were also included in the fuzzy rules in their model. Later, Katz et al. (1992) use fuzzy theory in SHERLOCK-II

for knowledge tracing and diagnosis. They use the term knowledge state for a possible

level of competence of a trainee on a skill or concept. In order to apply fuzzy rules, they use a kind of point-score scheme to update the strength of knowledge states. Though fuzzy logic is easy to understand and manipulate, it is not used to its full potential for uncertainty management in Learner models. Some other approaches for uncertainty handling in user models are discussed in Jameson (1996).

Maintaining optimal Learner models is challenging, mostly due to the uncertainty associated with assigning credits between various probable causes based on observed evidence. One of the solutions suggested is opening the Learner model to the users. In the next section this issue is discussed in detail.