MC Tests as Scaffolding Blocks

The MC tests play a key role in the scaffolding process: they not only assess the learner in each scaffolding stage, but also identify their potential misconceptions. Different levels of feedback are designed to address each misconception for different levels of learners.

5.7.1 Drawback of Traditional MC test

Although the traditional MC test is a simple yet efficient formative assessment tool, this version is not flexible enough for a learner to express their state of knowledge. The interface of the traditional version is severely limited; it only allows the learner to mark one tick (to select a correct answer). If a learner answers a test item correctly, it cannot be concluded that they are strong in the corresponding mental state. A correct answer may be just a lucky guess, an informed guess, interacting misconceptions (for example, a mixture of two misconceptions may result in a correct answer), or mastery. Similarly, if a learner selects a wrong answer, in most cases, the system assumes that they have a certain misconception and try to treat it. This is also a weak assumption. The learner may select an answer by just a blind guess, as a result of an informed guess (eliminating wrong answers, grouping most suitable answers or by both), just a mistake, or due to the associated or some other misconception. A weak but lucky person should not be treated as an expert; it is not good for their learning process. In the meantime, those who have partial knowledge should be identified and treated properly.

There are formal techniques available for making decisions under such uncertain situations (see Chapter 6). However, in traditional MC tests, these techniques are severely

testing system should allow the learner to express their level of understanding about the subject matter (relevant to the test item and the distracters) more precisely. This drawback of the traditional version hinders the whole scaffolding process as the state of the learner cannot be identified properly; and therefore, appropriate feedback materials cannot be presented to address the need of the learner. As a remedy, a confidence-based MC test schema is designed in this research so that the learner can express their knowledge on the concept under consideration more precisely.

5.7.2 Designing a Confidence-Based MC Test Schema

Confidence Based Marking (CBM) method (discussed in Chapter 3), to a certain degree, allows the learner to express their knowledge level in the test topic (Gardner-Medwin 1995). However, his method allows the learner to select only one answer that they think correct (though with various degrees of confidence). Since the learner cannot express anything on other answers the rate of potential misconceptions related to the unselected answers cannot be totally identified. Therefore, this method is not very suitable for this research. Building knowledge in the scaffolding process is highly dependent on rich and relevant feedback.

Although an imprecise classification, the Coombs’ method (discussed in Chapter 3), which requires the students to mark all the wrong options, is claimed to be able to identify five levels of knowledge on any MC test item: full knowledge (3- all and only the incorrect options marked), partial knowledge (1 or 2- some incorrect options marked), partial misinformation (-1 or –2 - some incorrect options marked, but correct option is also marked), full misinformation (-3 – only the correct option is marked), and absence of knowledge ( 0 – not answered at all or selected all options) (Bradbard et al. 2004).

However, Coombs’ method does not have any feature to specify the degree of belief in individual options; and therefore, it cannot explicitly identify the degree of misconceptions associated with different distracters (Coombs et al. 1956). This requires a

more precise and explicit mechanism for learners to express their belief against each answer option of a test item.

Based on both methods, the CBM method by Gardner-Medwin and the Elimination method by Coombs, an answering interface (Figure 5.6) has been designed for this research in order to enable the learners to indicate their degree of belief precisely. Learners may express their belief not only on their best answer but also, if they wish, on each other answer option. Learners may just select ‘not sure’ or express their belief on either band: correctness or incorrectness (obviously not on both). Initially, all the slide bars on ‘incorrect’ side and ‘correct’ side will point to 0%, and all the ‘not sure’ radio buttons will be selected. If a learner is 100% sure that a particular answer is correct or wrong, they just need to click the related radio-button. In both cases, the related slider will move to point 100%. Otherwise, learners may specify their level of belief using the relevant slide bars for the related answers. In this case, the MC test may be considered as a group of true/false category tests (where the ‘false’ option usually has higher chance in a group). Based on the degree of misconceptions, the system is now able to provide highly relevant feedback

Marking Schema

In the traditional version, performance in a test item may be considered as a binary variable: right or wrong. But in a CBM method, it may not be either right or wrong. It

Figure 5.6 Interface for Marking Confidence

Correct Incorrect Not Sure

knowledge level on a test item (or degree of strength in certain mental state) as a single numeric value in order to select certain pedagogical actions (this issue will be discussed in detail in the next section). For example, this measure may take any value between 100 (for exactly correct) and 0 (totally wrong). CBM methods can explicitly measure different levels of misconceptions related to each answer options, but cannot explicitly provide a single numerical value for the performance on a test item.

A marking scheme that gives a single numerical value as performance measure is proposed in this research. The discussion here is restricted to the MC tests with four options and one correct answer. Gardner-Medwin employed high negative marks for strong misconceptions (for level 3 confidence, -6 marks if the answer is wrong, but only 3 if it is right), which may work well in the medical profession as improper diagnosis is critical (‘not sure’ option may be better than wrong diagnosis). A moderate method is used in this research, where the metric used to measure the performance depends on the sum of the distances between the actual and proposed positions of each of the options in the slide bars in Figure 5.6.

Firstly, for the sake of obtaining a single numerical measure, the separate bands ‘incorrect’ and ‘correct’ are joined with a third band ‘not sure’ and considered as a single continuous spectrum. As in Figure 5.7, this single spectrum is now divided into seven regions; strongly incorrect, moderately incorrect, weakly incorrect, not-sure, weakly correct, moderately correct, and strongly incorrect.

The very weak approval or very weak denial is considered as ‘not sure’. More levels may give more accuracy. There will be 74 possible combinations, though some are not practical (for example, all marked as strongly incorrect). Figure 5.7 illustrates the notion of distance (say d) when a incorrect answer is marked as strongly correct (here it is 6). On

Figure 5.7 The notion of distance used in CBM test

Incorrect 100 25 50 75 0 75 _{50 25} 100 Correct not-sure

the other hand, if a correct answer is marked as strongly incorrect, the distance will be 18 (3*6). The distance (d) may vary from 0 to 18. To get a positive metric for performance (say D) on a test item, all the distances for each answer are added and the total is subtracted from 36 (6*6). This metric varies from 0 for extremely erroneous belief, to 36 for exactly correct answer. In a real sense, the metric varies actually from 18 (selecting all ‘not sure’ or all incorrect or all correct gives 18) to 36. Therefore, any mark below 18 will be considered as 18.

CBM Method: Pros and Cons

Gardner-Medwin notes, “[Learners are] more cautious in their expression of high confidence in exams than when doing formative assessment to aid study. …excessive diffidence or unwarranted confidence might disadvantage a student” (Gardner-Medwin

2005, p. 1) The CBM method designed above will be used for formative assessment during the scaffolding process; and therefore, it may be expected, as Gardner-Medwin observed, that the students may not be extra cautious in expressing their true confidence level.

The complex testing methods such as the above may confuse learners. The following remarks of three students taken from Bush’s questionnaire about ‘liberal testing’ methods reveal this fact; “You are distracted by thinking about the best tactics for getting a high mark”, “It takes away your confidence”, and “I was scared to answer a question” (Bush

2001, pp. 161-2). However, Gardner-Medwin’s observation is encouraging, “With some students the principles involved have seemed complicated when explained, but present no problem once they gain experience” (Gardner-Medwin 1995, p. 85). Moreover, there may

be some concern about whether personality type or gender differences affect CBM methods. Gardner-Medwin denies this, “Our data shows no evidence at all” (Gardner-

Medwin 2005, p. 7)

The proposed method provides elegant facilities for the learner to express their state of knowledge. This feature yields many advantages. Firstly, the system can precisely identify the misconceptions, and therefore, it can provide rich and relevant feedback. Secondly, since the uncertainty related to the learner’s knowledge level is considerably reduced, the system could select a suitable scaffolding level (curriculum sequencing).

(Gardner-Medwin 1995) and self-regulation. This, in turn, develops a high level of meta- cognitive abilities in the learners.

Alternatively, the learner may be allowed to select their preferred method of test schema among a set of limited options. However, they should know the testing method and its implications well before attempting the actual test. It is expected that the proposed method, compared to other similar methods, identifies the misconceptions and partial knowledge efficiently. However, it needs to be evaluated from a psychometric perspective (e.g. reliability, validity).