Teacher Evaluation 132

One aspect of the research is that interactive worked examples can be developed and evolved over time by individual teachers using an authoring environment rather than requiring bespoke programming. Based on the previous students’ evaluation and teacher’s pilot testing results, the second development cycle was finished. A new version IWE was released and ready to be evaluated again by suitable teachers.

6.3.1 Overview the Process of Teacher Evaluation

As mentioned in the usability review section, a usable user interface for highly motivated teachers to achieve their objective is a reasonable result. Although the second development cycle was finished, the IWE was still a prototype, so at this stage a qualitative evaluation focus on using experience and identifying problems is better than a quantitative evaluation focus on measuring satisfaction of user experience. The aim of this evaluation was to answer the following

questions:

1. Can the subjects understand the underlying model of how worked examples and their component parts are structured in IWE?

2. Can subjects create their target worked examples by themselves, with the availability of the help contents?

3. Would subjects like to use these worked examples, or even create some additional worked examples for their teaching?

4. Do subjects like or prefer to use IWE to create their own worked examples, compared with the tool they usually use to create worked examples?

6. Can subjects create worked examples in a reasonable time? The process of doing this evaluation is described below:

1. Demonstrate a simple interactive learning object, which can be created by IWE using the student’s interface. And then give a short training to the subjects to familiarise them with IWE. This short training will include five aspects which are:

o Illustrate the concepts and terminology which are used by IWE and introduce the target of IWE;

o Explain the work flow by demonstrating how a learning object is built by using the teacher’s interface;

o Emphasize help content can be used during development of their work;

o Emphasize the advantage of the IWE authoring environment, including: ease of making modifications to worked examples and support for collecting questions or comments about the worked examples from students.

o Answer subjects’ questions about the introduction and

demonstration of IWE until they understand what IWE can do for them.

2. Ask the subjects to create their target worked example or improve one, which they previously created by themselves. Leave them alone for a period and only provide extra help, if subject needs it.

3. Interview the subjects about their experience of creating worked examples using IWE.

6.3.2 Built Examples

Prof. Quintin Cutts, who taught CS1P (Computing Science Level-1 Programming) course, and Dr Marilyn McGee-Lennon, who taught the database part of CS1Q (Computing System Level-1) course, were invited to use IWE to create worked examples for students to use as part of their courses. 10 examples were created by the 2 teachers within the academic teaching semester. 4 of them were designed for the CS1P course and 6 for the CS1Q course.

 Unit 7-Task 3-I example was designed to show how to apply programming patterns to solve a problem;

 Unit 8-Task 2 was designed to show how to program a function;

 ClassTestRun was designed to show a similar question to those students would be expected to solve in the class exam.

 CinemaExPlay was an example based on a previous class exam question. This example was designed to show how a plan was created based on the problem description and then how the code was produced based on the plan.

The 6 examples created for CS1Q course were listed below:

 DB Relations scenario contains 4 different examples. All these examples are created based on the two documents, one graphical document for representing ER diagram and one textual document for representing tables. The FromERToTables example used the whole of these two documents. The other 3 worked examples only presented the relevant partial documents for different educational purposes. These examples are described below:

a. 1-to-1 Relation Demo example was designed to explain how to add a foreign key in a table to represent a 1-to-1 relationship between entities in the ER diagram.

b. 1-to-Many Relation Demo example was designed to explain how to add a foreign key in a table to represent 1-to-many relationship between entities in the ER diagram.

c. Many-to-Many Relation Demo example was designed to explain how to add foreign keys in tables to represent many-to-many

relationship between entities in the ER diagram.

d. FromERToTables example was designed to explain how to transfer a complete ER diagram into tables.

 TransferDemo was a simple example designed to explain how to design an ER diagram from a requirement description.

 RequirementToER was a complex example designed to explain how to design an ER diagram from a requirement description.

6.3.3 Experience of Building Examples

Two teachers successfully created a set of worked examples, which directly fitted their courses at the teaching point researched. These practical examples were built within their busy timetables during the academic teaching period. Their detailed experience of using IWE can be found in Appendix 7. Their experience of using IWE can be summarized below:

1. They believed that IWE was a tool for structuring existing documents to be represented as interactive worked examples with a sequence of steps and integrated explanations. The starting point of using IWE was to define document types or styles, rather than creating the contents as well as styling and structuring, as IWE assumed the contents creation were completed outside the tool. Both of them were confused about the sequence of defining document types at the beginning of using IWE. 2. They found organizing a textual document as a sequence of textual

fragments was non-intuitive. They also felt that applying fine-grained styles to different textual fragments was unusual compared with other text editors. The way of forcing a new line to format the textual document was unfamiliar. They pointed out that updating the textual documents should be more flexible, as the process was defined by using the textual documents. Building a graphical document was easy to appreciate, but lack of layering of components was a limitation for constructing graphical documents.

3. They required additional options for formatting the explanations and preferred to add hyperlinks to other explanatory materials in the

explanation panel. They required even more flexible layouts of the panels to display documents, for example, a mechanism for changing the default size of each panel in different steps. They needed multiple colours for highlighting different fragments, which could support focus on a particular fragment within a highlighted group of fragments.

4. When defining a process, they found no mechanism for selecting a group of fragments and then giving an operation to the group. If a grouping fragments mechanism were added, it could reduce the number of operations on fragments significantly. Very often, they forgot to press “Update” button to modify the explanation text of a step.

5. Although they did not receive significant feedback from students to update the worked examples, they extensively checked and reviewed their creations before delivering them to the students. These worked examples evolved based on teachers’ own critical feedback.

6. They identified a useful feature of IWE, which was to represent evolving documents. This feature was not initially anticipated by the teacher or the researcher. As the teacher wanted to model the development of a plan to solve the problem, instead of creating multiple plan documents, one for each refinement of the plan, only a single document that contains all these versions of plan was required.

7. They compared IWE with PowerPoint, a popular worked examples creation tool and concluded that PowerPoint was used effectively for small-scale worked examples. However the complex interactive worked examples created by using IWE cannot or are extremely difficult to create using PowerPoint.

8. They realised that the creation of a worked example required much more intellectual effort than structuring them, whether on paper, in

PowerPoint or using IWE. The overall experience of using IWE to structure the worked examples was satisfactory, as transforming an existing worked example into IWE was a straightforward task. However, the teacher who did the pilot test of IWE suggested that past exam questions can become ideal worked examples resources to be used for IWE. This suggestion is valuable, especially for reviewing purposes when students are preparing for their written exams.

6.3.4 Conclusions

The target of teacher evaluation was to try to answer the questions, listed in section 6.3.1. So, a group interview to ask these questions as a part of

evaluation process was carried out. The answers are summarized below:

 Teachers can understand the underlying model used to structure worked examples in IWE. Unfortunately, such understanding is not immediately intuitive. It requires subjects to engage in an initial learning phase.  Subjects can create their target worked examples by using the help

 Teachers did create some usable worked examples and would like to use IWE to create more of this type of worked example.

 Teachers preferred to use IWE to create more complex worked examples, containing more details and explanations, to be distributed to students after lectures. However, for simple worked examples used in lectures, they still preferred familiar tools, like Microsoft PowerPoint.

 Teachers would recommend IWE to other teachers.

 Teachers could use IWE to create worked examples in a reasonable time period. They found the difficult part was the designing of the target worked examples, but implementing them in IWE was straightforward.