Usage of Feedback Metadata

Each exercise is a directed graph of nodes, which represent exercise states, and edges, which represent transitions between these states. A tutorial strat- egy is a transformation that is applied to the exercise graph and produces a new exercise derived from the original graph in which the new nodes and edges might appear, and the existing ones rearranged, or even deleted.

Such a transformation does not modify the meaning of the tasks in the exercise steps, but the way the learner can navigate through the solution space. This is independent of whether he can repeat the attempts to solve the task in case of failure, whether or not he receives feedback and hints of different types and in which order.

We define a rich set of annotations for feedback having the following two usage scenarios in mind:

1. To provide the author with a set of annotations that allow him to manually define a rich exercise solution space that can be reused with various tutorial strategies, where each strategy selects a subset of the given feedback types

2. To provide a rich vocabulary of feedback types that can be generated on the fly by an elaborate tutorial strategy using intelligent domain reasoning tools

Consider the following easy example of a feedback strategy that can be applied to an exercise:

• if the learner’s input in a step is incorrect, then provide him with the feedback of a type ’Knowledge of Result’ (KR) specifying that the answer is incorrect

3.3. TYPES OF EXERCISE STATES 47 In order to apply such a strategy to an authored exercise, the feedback of the type KR should be authored in the exercise solution space for each exercise step.

Figure 3.5 shows how the graph of the easy exercise considered in the Figure 3.2 transforms when this strategy is applied to it.

The exercise on the left hand side only accepts correct answers in order to proceed to the next steps. In case of an incorrect answer, the student is directly forwarded to the terminal state, containing the feedback ”You failed!”.

The transformed exercise on the right hand side has to clone the ”Incor- rect!” feedback node as many times as there are steps in the solution, because each time this feedback is given the system forwards the student back to the corresponding solution step, thus giving him another chance.

From this example we can see that the strategy can add nodes and edges to the exercise graph, but it can also remove other nodes and edges. In the given example the nodes ”Incorrect!” are added to each step together with transition edges from and back to the corresponding interaction nodes. On the other hand, the node ”You failed!” is removed together with the transition edge from both interaction nodes.

Calculate (2x)'

You failed! Well done!

Calculate 2(x)' Calculate (2x)' Calculate (2x)' Well done! Incorrect! Incorrect! 2(x)' 2 2 2(x)' 2 2 default default default default Correct! Correct!

Figure 3.5: A simple exercise graph transformed by the strategy The feedback annotations defined above are useful for elaborate tutorial strategies. Many of the feedback types can be generated automatically using

48 CHAPTER 3. KNOWLEDGE REPRESENTATION intelligent domain reasoning tools capable of diagnosis of the user actions. For example, the knowledge of result feedback (KR) can be generated know- ing the task and the student’s answer, the KCR (correct solution) can also be generated by the generative domain reasoner for the given task. Different kinds of feedback on mistakes (KM) can be generated: the location of mis- takes in the student’s answer (KML) as well as more specific error feedback if the domain reasoner encodes buggy rules.

The exercise player in ActiveMath has a built-in mechanism for gen- erating KR feedback, in case at least one correct answer is provided in one of the transitions originating at the given interaction. This mechanism also allows (in a restricted number of cases) to generate feedback of types KNM (knowledge of number of mistakes) and KML (knowledge of mistake loca- tion). The restriction is that in such an interaction with multiple input fields there should be only one way to enter a correct solution. This should be taken as a reference to compare the student’s input.

Knowledge on how to proceed (KH) can be generated on different levels, starting from the names of concepts (rules) to be applied for the given step to actually generating the next step.

Conceptual hints can be generated by extracting definitions or explana- tions of participating concepts from the static content database. Examples of automatically generated feedback using domain reasoners will be given in the subsequent Sections.

Feedback about the student’s performance (KP) can be generated by the system with the help of the local history of student’s actions and their diagnosis, stored in the system.

Meta-cognitive feedback (KMC) can be generated on the fly using the local history of student’s interactions with the system. This can be informa- tion about the usage of hints by the student, number of trials of the single steps, learning time for interactions of different types and other data about meta-cognitive behavior of the student.

Some examples of meta-cognitive behavior of the student and correspond- ing feedback that can be automatically generated by the system are shown in the Table 3.3

The local history of the exercise process which records the relevant in- formation about the student’s performance is stored in the Local Student Model component, as discussed in Section 4.6

3.3. TYPES OF EXERCISE STATES 49

Observable Behaviour Generated Feedback

Student always asks for hints directly, without even trying to solve the task

Please, try to solve the problem first and ask for a hint only if you are in trouble Student tries to solve the task many times

without success and is not asking for hints

Perhaps asking for a hint now will give you an insight on how to proceed

Student asks for many hints in a row and then submits the correct answer

Consider trying to answer after each hint, maybe you do not need so much assistance

Student thinks too long before providing the answer to the task

You might consider asking for a hint or trying an easier problem first

Table 3.3: Generating meta-cognitive feedback from observable student be- havior