A Simple Learner Model

Various types of Learner models and their features were discussed in Chapter 3. Firstly in this section, a very simple Learner model that provides a relevant scaffolding process and suitable feedback will be discussed, and progressively, the design for a locally-intelligent Learner model will be described. As outlined in Section 5.6, the domain knowledge in LOZ is organised in a tree format (see also Figures 5.2 & 5.4). The important entities in

concept, a series of sub-concepts, and a set of exploration concepts. A sub-concept is associated with a sequence of mental states (from a basic mental state to a target mental state  see Figure 6.1). Each mental state is considered as a scaffolding stage.

After learning the pre-condition concept and the first sub-concept, the system expects that the learner will be in the basic mental state associated with the first sub-concept. At this juncture, the system will check the learner’s strength in the basic mental state using formative assessments. If it is at the desired level, the system will lead the learner to the next suitable scaffolding stage. As mentioned previously, the MC tests are used for formative assessments in order to measure the learner’s strength in the current mental state. The distracters in the MC tests are designed in a manner so that any possible misconceptions can be identified5; and therefore, they may be easily isolated and treated.

5 Confidence-based MC tests can identify more precisely the potential misconceptions.

State Schema- An overview Specifying attribute in

UML class

Able to specify Simple State Schemas, if UML attributes and invariants are given

Figure 6.1 Lesson Organization (including Mental States)

State Schema specify the attributes (partially) and invariants

Initial values cannot be specified

Pre-Concept

Sub-Concept Main-concept

DELTA separates dependent attributes and independent attributes

Able to specify Simple State Schemas, if UML attributes (not invariants) are given

Able to specify Simple State Schemas, if textual description only given

Able to specify Simple State Schemas, if UML attributes and dependencies are given

Able to specify Simple State Schemas, if dependencies are not explicitly given

Target Mental State

Target Mental State Sub-Concepts

Sub-Concept

Explore-Concepts

A simple system, without a Learner model, may use the performance of a student on a given test item to select the feedback and the next scaffolding stage. However, these pedagogical decisions may not be appropriate: for example, the given test may not be suitable. A single performance measure is not sufficient to gauge the knowledge level of a learner. Therefore, in order to provide adaptable support, the individual learner’s state of knowledge (or strength of mental state) should be modelled. It should be noted that the knowledge level for a learner is dynamic and therefore any such model will also be dynamic.

Strength of Mental State

A numerical variable (ranging from zero to 100) known as a “Strength of Mental State” (SMS) is used in this research, in order to measure the learner’s level of potency in a certain mental state. SMS represents the increasing strength in a particular mental state as it moves from zero to 100. Naturally, for any particular learner, the current value for SMS will depend on their previous SMS values. Basically, SMS may be associated with the traditional ‘overlay’ concept (Holt et al. 1993). Notwithstanding, in this research, the

semantics of this measure are interpreted through the constructivist approach. As previously mentioned, the Learner model in LOZ will be locally-intelligent, which means that the SMS related to a main-concept will not have any impact on the SMSs related to the other main-concepts. However, the SMSs related to the same main-concept may impact on each other in a limited manner (see Figure 6.1).

SMS may be interpreted in quantitative terms or in probability terms. In a quantitative interpretation, the value SMS=50 means that the system believes that the learner is 50% strong in the corresponding mental state and alternatively, in probability terms, it can be interpreted as the system believing 50% that the learner is ‘sufficiently strong’ in the basic mental state. An initial value for SMS0 should be assigned. For convenience, it can be

assumed that 50 will be the initial value for basic mental states of all the first sub- concepts. Which means that, after learning the materials relevant to the pre-condition lesson and the first sub-concept, the strength of the basic mental state will be 50.

After answering an MC test, the system has then acquired some evidence about the learner’s strength in the current mental state. The following heuristic formula is used in

performance, which can assume two values  Correct or Incorrect. The resultant SMS values are bounded by zero and 100.

The above formulae are based on the notion of a traditional ‘point-scoring’ scheme (Katz

et al. 1994). A weak student will be awarded higher points than a stronger student if they

both perform equally well on a test. It may be a lucky guess in an MC test, but that possibility is not considered here. Similarly, a strong student will be given a larger penalty than a weak student if they both perform equally badly. This may be a slip-up, but that possibility is also not considered here. Moreover, more marks will be awarded (or deducted) in the earlier rather than in the later stages of the scaffolding process.

There are several heuristics used in this research. Stern et al discuss this issue in relation

to their MANIC system: “Many of the decisions made in MANIC are rather ad hoc, with numbers determined by domain experts and instructors. However, this is not the fault of MANIC alone: many systems rely on a priori formulas designed by instructors. But, we are not satisfied with this as a solution” (Stern et al. 1998, p.581). Heuristics are

inevitable in life. For example, we used to set pass marks for exams with 50% and above being a pass. The difference of knowledge between those students who were on either side of 50% may have been negligible, whilst the impact on their results would be significantly different. Stern et al (1998) discuss some machine learning techniques to

improve the heuristics used in MANIC. The heuristic formulae used in this research can also be tuned, using empirical methods or machine learning techniques (Kiat 2003).

After updating the SMS, the system will attempt to provide suitable feedback and curriculum sequencing. Different levels of PAS options for different performance levels have already been designated in Chapter 5. A higher level PAS option is intended for a

If P is Correct, SMS i+1 = SMS i + (100 - SMS i ) * J N + 1 If P is Wrong, SMS i+1 = SMS i - SMS i * J N + 1

Where, N is the total number of scaffolding steps associated with a sub-concept, J is the number of remaining steps to be completed,

lower level student. For any particular learner, the current SMS can be used to determine the appropriate level of PAS option in the present situation. For this purpose, SMS is divided into 5 ranges: 0–20, 21-40, 41-60, 61-80, 81-100. Table 6.1 gives the SMS ranges and their corresponding PAS levels for the performance categories Correct and Incorrect.

Depending on the range of SMS, the required level of PAS option will be selected. SMS in the lower range indicates that the corresponding learner needs a higher level PAS option. For example, if a learner with a low SMS answers a test incorrectly, detailed feedback will be given and, subsequently, the learner may also need to take another test at the same scaffolding level.

The mental states, in the series associated with the same sub-concepts, are sequentially dependent (Figure 6.1), and therefore, the following formulae are used to initialise the SMS related to the subsequent mental states in the series:

As previously explained, before moving to the subsequent mental state (if the SMS of the previous mental state in the series is less than the current SMS) all the previous SMSs in the series, that are less than the current SMS, will be updated by the current SMS, which means that the corresponding SMSs of the mental states in a particular series will always

Table 6.1 SMS and Pedagogical Actions

SMS- ranges P - Incorrect P- Correct

81-100 PASwL1 PAScL1

61-80 PASwL2 PAScL2

41-60 PASwL3 PAScL3

21-40 PASwL4 PAScL4

0-20 PASwL5 PAScL5

If SMSlast j >50 , SMSfirstj+1 = 50 + (SMSlast j -50) *

N + 1 1

Otherwise , SMSfirstj+1 = 50

Where, N is the total number of scaffolding steps associated with a sub-concept, SMS i.j denotes the SMS value of the jth mental state after the ith MC test of the

diagram without UML support, they cannot have less than 80% strength in doing so with additional UML support. Within the same main-concept, only the basic mental states of different sub-concepts are considered dependent (Figure 6.1). The following formulae are used to initialise the strength of the basic mental state associated with the subsequent sub- concepts of the same main-concept.

Nevertheless, designing and maintaining even a locally-intelligent Learner model is challenging, primarily due to the uncertainty associated with the sources of evidence concerning the learner’s knowledge state. The above described method is not efficient. Although, different levels of PAS options are designed to deal with uncertainty to a certain extent, only a single table-look-up is used as the selection mechanism in this method (Table 6.1). Moreover, SMS values are updated using a simple point-scoring scheme. Therefore, based on the above model, a more efficient strategy is designed to handle uncertainty in the Learner model. This strategy will be described in detail in the next section.