Chapter 4 described an evaluation that was performed at the University of Warwick with six volunteer course authors and designers. Each participant was experienced in the process of both teaching and web-design, and had experience with adaptive hypermedia. This was a qualitative study, where the opinion of experts was elicited, thus the number of participants is not essential. They were asked to explore the
143 system, and answer 45 questions, which we then grouped into 10 categories of basic functions, as below:
1. … browsing other author’s materials 2. … editing with MOT3.0
3. … changing hierarchies of material via drag & drop
4. … copying and linking functionality
5. … editing HTML using the editor
6. … importing Wikipedia content
7. … importing Presentation content
8. … functionality of importing content
9. … authoring for adaptation support
10.… Semi-Automatically Creating and Linking Content for adaptation
Figure 6.6 shows that the designers found most of the basic functions ‘Easy’ (or ‘Very Easy’) to use. To establish the statistical significance of these results, we have mapped the answers {‘Very Easy’, ‘Easy’, ‘Difficult’, ‘Very Difficult’} onto the values {2, 1,-1, -2}. This assumes equidistance between these labelled values, as well as monotonicity, an assumption which is widely used in literature, and also conforms to the natural language use of these words.
144 We have then applied a one-sample T-test to compare the answers against the average of 0, corresponding to ‘Neither Easy nor Difficult’, to establish if the positive average is statistically significant.
Figure 6.6: Histogram display of the responses to the MOT3.0 questionnaire
Table 6.1 shows that most answers are statistically significant with 95% confidence (P<0.05). Thus browsing, editing,changing hierarchies (Q1,2,3), and editing HTML
(Q5) are significantly easy. Also (semi-)automatically creating content and linking
are significantly useful.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 browsing and using other author's materials editing with MOT3.0 changing hierarchies of material in MOT3.0 via drag & drop
copying and linking functionality in MOT3.0 Editing HTML using the editor authoring for adaptation as supported by MOT3.0 Very Easy Easy Difficult Very Difficult
145
Question Mean StdDev T P
1. browsing other author’s materials 1.167 0.408 7 0.001
2. editing with MOT3.0 1.667 0.516 7.91 0.001
3. changing hierarchies of material via drag & drop
1.333 0.516 6.32 0.001
4. copying and linking functionality 1 1.095 2.24 0.076
5. editing HTML using the editor 1.667 0.516 7.91 0.001
6. importing Wikipedia content 1.833 0.408 11 0
7. importing Presentation content 1.833 0.408 11 0
8. functionality of importing content 1 1.095 2.24 0.076
9. authoring for adaptation support 1 1.095 2.24 0.076
10. Semi-Automatically Creating and Linking Content for adaptation
1.5 0.548 6.71 0.001
Table 6.1: The MOT3.0 questions and response frequencies
The copying and linking functionality (Q4), importing content (Q8) and general authoring (Q9) are only statistically significant with 90% confidence. We have thus further analysed the (sub-)questions that formed these categories, in order to establish the reasons for the lower confidence intervals. Below we show the one sample T-test values for question Q4, which was formed of three sub-questions:
146 Q4a: Dragging domain concepts between trees when copying/linking;
Q4b: Inserting goal map sublessons from domain concept attributes;
Q4c: Inserting other goal map lessons as sublessons.
Question Mean StdDev T p
Q4a 1.5 0.548 6.71 0.001
Q4b 0.5 1.225 1 0.363
Q4c 0.833 0.983 2.08 0.093
Table 6.2: One-Sample t-test Q4a, Q4b, Q4c (Test of mu = 0 vs not = 0)
Table 6.2 shows that whilst dragging domain concepts is significantly easy, inserting goal maps from domain concept attributes or other sublessons is not. Looking at the qualitative comments, the experts noted that: “Inserting […] domain map attributes needs improvement […] partial goalmaps cannot be inserted” and “it is easy, but a bit inconsistent: for GM you have to click add, for DM you have to drag & drop. I would like it not to refresh back, as I may want to add more than 1 attribute”.
The questions including question 8 have already been presented and discussed in Chapter 6.
Question Q9 on general authoring was composed of questions on the issues of:
Q9a: Being able to create adaptive presentations with MOT3.0 (as compared with programming adaptation from scratch);
147 Q9c: Being able to (semi) automatically link content for adaptation;
Q9d: Using graphical drag & drop interfaces in authoring for adaptation.
The one sample T-test results for Q9 are shown in Table 6.3. Thus, creating adaptive presentations with MOT3.0 is preferred (in a statistically significant way) to
programming adaptation from scratch, and also using graphical drag & drop interfaces in authoring for adaptation is considered beneficial. Looking at why the experts are not convinced about (semi)automatically creating and linking content, the comments were as follows: “The physical manipulation is easy, but you have to understand what you are doing”, “Linking automatically is only possible in a
hierarchical way. It would be interesting to see different types of automatic linking.”
Question Mean StdDev T p
Q9a 1.167 0.408 7 0.001
Q9b 0.667 1.366 1.2 0.286
Q9c 0.5 1.225 1 0.363
Q9d 1.5 0.548 6.71 0.001
Table 6.3: Responses to question 9
Thus, whilst clearly some improvements can be done (and the experts have given us some very good pointers towards this), the overall evaluation shows that people like our imaginary Professors Smith and Jones can expect to be able to author with a reasonable degree of ease personalised courseware with a system such as MOT3.0.
148