non-verbal behaviour (see Chapter 3) suggests that artificial intelligence could be used to perceive learner comprehension in near real-time during learning activities.
2.3
Evaluating learning
This section presents literature and methods for evaluating learning outcomes. Evaluation of learning outcomes is an important consideration in this research as a measure of success for a comprehension based adaptive learning system. In this section two approaches to measuring learning outcomes are highlighted: learning gain metrics (section 2.3.1) and a taxonomy of educational objectives (section 2.3.2). Both play an important role in defining objective measures of
learner performance and goal attainment.
2.3.1
Learning gain
Learning gain is a simple but widely adopted measurement of the effectiveness of tuition, for example in Colt et al. (2011); Graesser et al. (2003); Latham et al. (2014); Saadati et al. (2015); VanLehn (2011). Learning gain attempts to quantify the amount of value-add by measuring the distance between pre- and
post-tuition test scores.
There are multiple mathematical definitions of learning gain to be found in literature. Latham (2011) uses equation 2.1. This interpretation of learning gain provides an absolute measure of difference between pre- and post-tuition scores. The approach does not account for prior knowledge and the learning gain is liable to be skewed upwards by under-performance in the pre-tutorial test.
Colt et al. (2011) uses a relative measure (equation 2.2) which gives a percentage gain of those marks available in addition to the pre-tutorial test. For example, if a learner scores 80% on the pre-tutorial test then there are 20%
44 Theories for e-learning
of marks to be gained in the post-tutorial test. If the learner increases their score to 90% in the post-tutorial test then they have gained 10% and have a learning gain of 50%.
post-tutorial test score − pre-tutorial test score (2.1)
post-tutorial test score − pre-tutorial test score
100.0 − pre-tutorial test score (2.2)
In this research equation 2.2 taken from Colt et al. (2011) will be used to represent learning gain. The result of equation 2.2 can be converted to a percentage by multiplying by 100.
Learning gain has proved to be a contentious topic (Cronbach and Furby, 1970; Hake, 2010). The arguments against such a simplistic appraisal technique, discussed in (Hake, 2010), and originally presented in (Cronbach and Furby, 1970), are aimed at the reliability of the statistics produced.
Boyer et al. (2008) investigated the role of cognitive and emotional scaffolding on learning gain. A surprising finding from their work is that praise during
conversational tutoring has a negative effect on learning gain, a result of increasing confidence and misplaced self-efficacy. The result highlights the unanticipated complexity of variable interactions when motivation, emotion and social interaction are at play during learning.
With this in mind, learning gain alone is an insufficient measure of success. In this research learning gain will be used as part of an ensemble of measurements along with objective attainment, participant feedback and analysis of system function error rates.
2.3 Evaluating learning 45
2.3.2
Anderson and Krathwohl’s taxonomy of educa-
tional objectives
Definition of learning objectives is an important aspect of learner outcome evaluation. Defining objectives is no small task and, if done incorrectly, can undermine the learning process and make accurate evaluation impossible. To aid designers in creating effective objectives Anderson et al. (2000) propose a taxonomy based on Bloom’s original Taxonomy of Educational Objectives. Their model has two dimensions:
• Knowledge dimension
– Factual: Basic elements of knowledge required to solve a problem – Conceptual: The relationships between basic elements in a broader
context
– Procedural: Methods, algorithms and techniques for applying
knowledge
– Meta-cognitive: Understanding of cognition
• Cognitive dimension
– Remember: Recalling previously learned information from memory – Understand: Constructing meaning from various data sources – Apply: Using knowledge to implement a procedure
– Analyse: Differentiating, attributing and organising to distinguish
between components information
– Evaluate: Make judgements based on evaluation of evidence – Create: Manufacturing, reorganising or synthesising information
to produce a novel output
The model is principled on the view that an objective is a combinatorial statement of intent along these two dimensions. Anderson et al. (2000) suggest
46 Theories for e-learning
that an objective should contain a verb, representing the Cognitive Process Dimension and a subject word representing the Knowledge Dimension.
For example, ‘As a learner I should be able tocreate a for loop constructor
to iterate ten times’. This learning objective exposes factual, conceptual and procedural knowledge of loop constructors and iteration and allows the learner to demonstrate creativity as cognitive process.
Learning objectives can provide assessment milestones which are clearly defined and evaluable. Unlike learning gain, goal attainment does not need to map complex socio-cultural, linguistic or emotional interactions. The measure- ment of success in any given learning outcome is the student’s demonstrated competence within the bounds of the objective. Learning objective attain- ment can be measured within a tutorial by assessing competence on individual tasks, for example by marking answers to questions or appraising solutions to problems.
2.3.3
Question response complexity
In designing questions for a tutorial, it is important to consider the required complexity of response. For example, a simple question requiring only a single word answer, such as ‘true’ or ‘false’, may allow a learner to guess the answer easily. Complex questions, which cannot be guessed easily may require learners to formulate complex multi-part answers containing multiple keywords, phrases, mathematics or programming code, which can contain multiple conceptual elements.