Extra Work for the Usability Evaluation of Educational Software 68

Squires and Preece [99]were aware of the limitation of applying these usability evaluation methods to evaluate educational software. They argued that just because an interface was easy to use did not mean that it was designed appropriately from an educational perspective. There was an essential relationship between the two which must be addressed to ensure good

educational software design. This awareness was also introduced by Nielsen [84] as the definition of usefulness, which was the issue of whether the system can be used to achieve some desired goal. Usefulness can be divided into two categories of utility and usability. The utility is the question of whether the functionality of the system in principle can do what is needed, and the usability is the question of how well users can use that functionality. For example,

educational software or courseware should have utility allowing students to learn from using it. Later on, Squires and Preece [100] proposed an initial set of “learning with software” heuristics, which can be used to predict the quality of the educational software that takes account of both usability and learning issues from educator’s point of view. Three important points were:

1. When learners use the educational software to learn, the model formed by learners should be consistent with the teacher’s predictive model. 2. The education software should match with the curriculum relevance and

teacher customization, as teachers will often feel the need to match particular curriculum requirements or adapt software to the specific needs of their students. A good education application should be able to facilitate teachers to do this.

3. Strategies for helping learners to recognize cognitive errors, to diagnose and recover from them should be supported by the software.

Mayes and Fowler[9], and Hornbaek [101] also claimed that the concept of usability in the field of educational software has to be adapted to pedagogical approaches and theories of learning, due to traditional usability principles not

being sufficient to guarantee successful learning. Avouris et al. [102] applied traditional usability measures to rate the usability of one of two online learning systems. Students had to perform a test after using one of the learning systems for 15 min. They found that the group of students using the system that was rated most usable showed significantly better learning outcomes. However, later on Tselios et al. [103] claimed that educational software should not simply

support the efficient execution of a task. In specific cases, increased usability could have a negative impact on learning, since executing a task efficiently may prevent essential learning processes. For example, in the database area, the Aqua Data Studio[104] provides a Visual Query Builder, which allows students to construct sophisticated database queries without having to know the syntax of SQL statements.

At this point, it’s worth noting that determining the usability of an application is not the same as evaluating its educational effectiveness. This can be interpreted as stating that a highly usable product is no guarantee that learners will retain information. Nonetheless, it is virtually assured that an application with low levels of usability will not enable learners to access and assimilate information at all. Hence, ensuring educational software is highly usable is the prerequisite for building effective educational software.

Hollender et al. [105] conducted a literature review about research on cognitive load theory and research on usability in human-computer interaction. Based on the result of this review, they proposed two conceptual models shown in Figure 3.1 and Figure 3.2, which integrated cognitive load theory concepts and usability concepts. These models are intended to offer implications for the design of complex educational software. The core point in these two models is that educational software should focus on assisting cognitive processes of students rather than on supporting efficient execution of specific tasks.

These two models suggest ways of conducting usability testing for educational software. By deeply understanding these two models, it points out the

responsibilities of the computing researcher when designing new educational software. It also identifies the responsibilities of the teacher when using educational software.

Figure 3.1. Components of CLT in Information and Communication Technology(ICT) Supported Learning[105]

The first model in Figure 3.1 outlined previous research results of CLT, which was introduced in section 2.2.2 of chapter 2. Intrinsic Cognitive Load depends on students’ previous experience. Germane Cognitive Load refers to the cognitive learning process, how students process the information to construct new schema. Hence, when teachers design worked examples, the levels of Intrinsic Cognitive Load should be provided appropriately, based on understanding

students’ previous experiences as the starting point. The Intrinsic Cognitive Load can only be provided at a suitable level for the majority of students, not for individual student. If worked examples are too advanced for the students, the Intrinsic Cognitive Load is too heavy; because students struggle to learn from them and will not be able to generate Germane Cognitive Load for constructing new schema. If the worked examples are too simple, students may find the worked examples redundant after using them; because no new schema can be constructed after using these worked examples. This model reveals that the amount of Extraneous Cognitive Load can be influenced by the design of learning objects and the complexity of software use. If a highly usable software

application can be designed or enough training can be provided to learners, the software factor in extraneous cognitive load can be minimised. As discussed before, a good user interface does not guarantee educational value, but a poor user interface is unlikely to deliver high educational value. Hence, designing an intuitive user interface for students is essential, and evaluating the usability of this user interface by applying traditional usability evaluation methods is

necessary. The remaining amount of cognitive load should be taken into account by the teacher when designing the learning objects. [105]

Figure 3.2. User, Task and Tool in an E-Learning Context, defined according to CLT. The goals of the tool are related to the three types of cognitive load. [105]

The second model, shown in Figure 3.2, integrated the CLT concepts into the usability concepts. It described that the relationship between the learner and the educational software. CLT defines the user as a learner and the task as learning. Learners need to finish the task though using the educational software. If learners can use the software to finish a task successfully, they should be able to construct new schema after finishing the task. The goal of educational

software is to adapt the Intrinsic Cognitive Load, to reduce the Extraneous Cognitive Load and to foster the Germane Cognitive Load, through assigning tasks to users. Hence, when designing educational software, providing facilities following the best practice instructional design guidance from education

research results is essential. By providing the appropriate level of worked examples as tasks, learners should be able to finish the task successfully to adapt Intrinsic Cognitive Load under the designer expectation. Decreasing the Extraneous Cognitive Load can also be achieved through reducing the irrelevant learning of using the software by making the software easy to use. When

evaluating tool usability , traditional HCI evaluation methods to evaluate the learner user interface can be applied.[105]

To summarise, when initially designing educational software, attention should be paid to designing an intuitive user interface for students. This kind of easy to learn and use interface can reduce the factors which could increase Extraneous Cognitive Load for learners. When evaluating the usability of educational

software, attention should be paid to whether the students can engage with the contents as the designers expect or predict. The usability of the user interface needs to be good enough to be operated by learners; otherwise they may stop using it. However, it is less important to provide a highly usable interface for

teachers, as they can tolerate a sufficiently usable user interface to achieve their objective, if they are highly motivated. Later on, attention can be paid on improving the user interface for the teacher.

Based on the discussion above, the strategy of evaluating IWE can be described below, with the aim of answering the research questions:

 For teacher users, usability evaluation should be qualitative. It should focus on proving the concept, which means the teacher user should be able to create and modify interactive worked examples by using this authoring environment. A usage experience report is required that identifies the problems during the process of creating and adapting worked examples. The teacher must be aware of expected Intrinsic Cognitive Load before creating worked examples, which means that the designed worked examples need to match the students’ previous

experience. The authoring environment must provide the means for the teacher to achieve instructional designs by reducing the Extraneous Cognitive Load, and fostering Germane Cognitive Load.

 For student users, usability evaluation should focus on finding the

evidence from looking at the users’ log traces. Students’ feedback about worked examples can be another resource to identify whether the level of Intrinsic Cognitive Load is appropriate or not. All evidence aims to

demonstrate delivered interactive worked examples are useable and useful to students. It also needs to measure the satisfaction with using this tool to ensure the Extraneous Cognitive Load is being minimised.  For both teacher and student users, usability evaluation should focus on

the utility of feedback, for example, how to apply the feedback to adapt the Intrinsic Cognitive Load, which needs to be processed by the students. The aim is to evaluate whether the teacher can take action to modify the worked examples based on students’ feedback or create new examples dynamically for the ongoing course progression.