Preece et a l . (2002) define evaluation as "the process of systematically collecting data that informs us about what it is like for a particu lar user or g roup of users to use a product for a particular task in a certa in type of environ ment" (p. 3 1 7). In an interview from the same book, Shneiderman d i stingu ishes between an experiment and usabi lity testing. The goal of an experiment is to confirm or refute a hypothesis using repeatable, q uantitative mea ns, while the goal of u sability testing is to refine a prod uct as quickly as possible, t h rough identifying frequent problems (ibid. , pp. 457-458).
I n Chapter 3 it was hypothesised that to successfully provide a computer-based university-level distance-learning environme nt on an anywhere , a nytime basis, three student-centred design strategies should be emphasised. They are: a distributed , rather than a centralised , network a rchitecture; a user-initiated adaptable and col labo rative, rather than system-in itiated adaptive, approach to presentation and content; and a special-purpose , rather than a general-purpose, G U I environ ment. In Chapter 4 a learn ing computer was proposed as a conceptualisation of these strategies, which was then partially specified in Chapter 5 as the I M M E D IATE system . T h e successful prototyping o f I M M E D IATE was summarised in Chapter 6. This included testing the system on a peer-to-peer LAN. This experience fu rnished a more detailed specification for a learning computer implementation a nd provided provisional
Chapter 7 : User evaluation 1 76
support for the unde rlying hypothesis. To provide more conclusive evidence, it was important to evaluate I M M E D IATE under conditions more closely a pproximating those faced by its target u se rs .
l t was argued in Chapter 3 that extramural e-lea rning environments should be evaluated along the three dimensions of functionality, accessibi lity and usability (3.2).
The goal of the evaluation phase was to confirm or refute the hypothesis, by evaluating I M M ED IATE a long all three dimensions. This would require testing with users under realistic conditions, using quantitative and qualitative measures.
7.2 The Evaluation Strategy
Severa l key issues h ad to be decided in determining the evaluation strategy - what was to be evaluated, h ow, where, and by whom . In line with the goals, the eval uation would focus on assessing the completeness, accessibility, and usabil ity of the Learning S hell functionality. The e mphasis would be on the student as user rather than as
learner.
The evaluation would com bine aspects of a field trial and of more formal usability testin g . lt would be conducted with volu nteer users representative of extram u ral students, who would work under field conditions reflecting the more challenging end of the spectru m in which d istance learning takes place. If IMMEDIATE worked and was u sable under these conditions, then it could reasonably be asserted that it would work a nd be usable anywhe re, anytime. At the same time, each volu nteer wou ld be req uired to complete the same set of prescribed tasks u nder the same condition s. In this way, their i nteractions with the system could be monito red and compared in a measurable way.
The testing with u sers would have two facets. lt would test the completeness and accessibility of the system functionality, primarily u sing quantitative data . And it would assess the usability of the Learning Shell interface , using quantitative and q ua l itative means. lt would be i m po rta nt to demonstrate with q uantitative data that the system was accessible, functional and u sable in the field. But it would also be valuable to obtain the views of the users on I M M E D IATE and how it compared with other com puter systems they have u sed .
7 . 2 . 1 Focus on student as user
The evaluation wou ld focus on the student as user, i.e. how easily he or she could learn to use the system a nd carry out all the required tasks in the Learn ing S hell
Chapter 7 : User eval uation 1 77
environment. lt would not address the issue of the student as lea rner - i .e. h ow well the user could learn the cou rse subject matter.
IMMEDIATE has been designed to overcome the usability and accessibil ity problems associated with web-based and standalone ed ucational software, by providing a system which make s minimal technical demands on the student user, both i n terms of technolog ical platform and computing experience. lt does not introduce new learning o r teaching methods that need t o be eva luated.
What needed to be eva luated was the ability of the system to deliver a wide ra nge of learn ing functionality in a usable form to remote students , and therefore that it was a workable, universal alternative to correspondence-based extramural study. The emphasis was upon the completeness of the functionality, on its form rather than its content.
7.2.2 Testing for fu nctio nal ity and access i b i l ity
Evaluating IMMED IATE for functionality and accessibility involved testing whether the three core components of the system - the Learning Shell, the Course Authoring and Management System, and the Commun ications Manager - could deliver the appropriate fu nctional ity in a timely manner to remote users, and assessing the reliabil ity and perfo rmance of the system u nder these circumstances. For students to use I M M E DIATE for learning, all the three subsystems had to work togethe r effectively.
This requ i red insta lling and running the entire IMMED IATE system in the field with actual users, and observing the results. This wou ld be com bined with the usability testing a nd observation .
Event logs wou ld be used at both the client a n d server ends of the network to record quantitative data verifying that all functions worked and were accessible, and to identify and analyse any p roblems that might arise.
7.2.3 Usability testing
The centrepiece of IMMEDIATE is the Learning Shel l . Therefore , considerable emphasis in the evaluation phase was placed upon assessing the Shell interface for usability. Would an extramural student studying at home a lone be able to complete a set of typical tasks unaided (e . g . attend a lectu re o r partici pate in a g roup d iscussion)? H ow easily could t hey achieve th is?