The areas of interest were: the range of tasks that could be supported; the role and impact of the shared workspace. Data collected included both qualitative data (on student and teacher perceptions and language learning outcomes and processes) and quantitative data (logging user problems and identifying patterns in use of the shared workspace). Both subjective perceptions of participants and the more objective opinirais of language teacha- experts and other observers were elicited.
The target was to obtain results supported by both subjective and objective data, and repeated either across courses or, in the case of Trials 2 and 3, over the period of the trial concerned. Where this was not the case, the data were regarded only as raising questions. Additional research was then carried out to explore these and to discover whether the suggestions were substantiated. For example: a questiœ raised in the Spanish ancillary study, about studmt-teacher communication, was investigated in Trial 3; questions arising from Trial 3 student focus groups were followed up in a questionnaire. The nature of the data supporting the findings is stated in Chapter 7.
Finding design requirements for the shared workspace is the focus of this study but, as has already been stated (Chapter One, Chapto* Three), evaluation and interpretation may be complicated by the audio and/or video conditions, by usability problems, or by the fact that task performance relies on the inter working of all the conferencing tools. Reference was made to data collected about the audio and video tools, as well as the shared workspace. Data on the usability of the conferencing system was also collected and taken into account.
Conclusions have been based on the following data:
1. Checklists from observation of the system in use (Appendix 6) 2. Questionnaire responses from students. (Appendix 6)
3. Transcripts of video and audio tape recordings of focus groups and interviews. (Appendix 6 and 7) 4. Written reports from teachers and expert (language teacher) observers. (Appendix 7)
5. Screen shots of whiteboard and text editor pages. Figures 7.1-7.7 are sample whiteboard and text editor pages from Trial 2. Further examples, are in Appendix 7.
The rest of this section discusses the nature of the data, and the circumstances in which it was collected and analysed.
6.10.1. Observation of the system in use
10.10.95 B u sin e ss French OBSERV ER2 14.11.95 B u sin ess French O B SER V ER 1 1 1.10.95 A dvanced French 0B SE R V E R 1 15.11.95 A dvanced French 0B S E R V E R 2
1 3.10.95 Latin OBSERV ERS 17.11.95 Latin OBSERVERS
1 7.10.95 B u sin e ss French 0B S E R V E R 2 2 1.11.95 B u sin ess French 0B SE R V E R 2 1 9 .10.95 P o rtu g u e se O B SER V ER 1 2 3.11.95 P o rtu g u e se O B SER V ER 1
2 0 .1 0 .9 5 Latin OBSERVERS 2 4.11.95 Latin OBSERV ERS
2 5 .1 0 .9 5 A dvanced French 0B SER V ER 1 3 0.11.95 P o rtu g u e se OBSERVERS
2 6 .10.95 P o rtu g e se O B SER V ER 1 01.12.1995 Latin OBSERV ER 1
2 7 .1 0 .9 5 Latin OBSERVERS 05.12.95 B u sin e ss French 0B S E R V E R 2
3 1 .1 0 .9 5 B u sin e ss French 0B S E R V E R 2 07.12.95 P o rtu g u e se 0B SER V ER 1
0 1 .1 1 .9 5 A dvanced French 0B SE R V E R 1 08.12.95 Latin OBSERVERS
0 2 .1 1 .9 5 P o rtu g u e se 0B SE R V E R 1 12.12.95 B u sin ess French 0B S E R V E R 2 0 7 .1 1 .9 5 B u sin e ss French 0B SE R V E R 1 14.12.95 P o rtu g u e se OBSERVER 1 0 8 .1 1 .9 5 A dvanced French 0B SE R V E R 1 28.11.95 B u sin e ss French 0B SE R V E R 2 0 9 .1 1 .9 5 P o rtu g u e se O B SER V ER 1
Table 6.6: List of teaching sessions and observers, Trial One
Observation generated both quantitative and qualitative data. Observation took place both in the room and over the network. Each of these had advantages. In the room, the observer was close enough to die participant to see usability problems, particularly those related to the physical situation. On the other hand, the observer in the room had a rather distant view of the tutorial as a whole, which might limit
understanding of events. It was possible to listen in, but this was potentially quite intrusive, since it involved sitting close enough to the workstation to attach a headset.
W b r k s t a t i o n
Headphone cable (Teacher
)bserve|
Figure 6.7: Teacher's room - Trial 1
Observing over the network was less obtrusive. It enabled all participants to be observed, but did not provide as much informatiœi about any single participant as did observation in the room. The observer's view was limited to what was shown by a small camera on the remote workstation (a head and shoulders view of the individual). This meant that it might sometimes be difficult to interpret events. On the otho" hand, the observer was "in" the tutorial, in the same way that an observer can be in a classroom and therefore shared the view of the participants, to an extent. However, there was one further complication of remote observation. A feature of networked multimedia communications is that no two participants' experiences are exactly the same because of network variations, variations in computer processing power and quality of peripherals. For instance, the quality of the audio may vary considerably.
Observers in the room concentrated on whole system design and usability. Questions addressed were: Did nominated language learning activities take place? Which shared workspace features were used? How much did studœts use the shared workspace? What usability problems occurred? What other problems did teachers and students have? In addition, observers recorded comments made by the participants and their own comments and questions.
A standardised checklist (see Figure 6.8) covered technical settings and hardware, language learning tasks, use of audio, video and shared workspace tools, including usability problems, with space for observer comments. The original checklists have been retained.
Three different obsowers were used (see Table 6.6). They filled in their checklists on paper and then enta-ed this data into an html form. Figure 6.8 is a sample return from entering the data in the html form, annotated in order to explain the coding. To try to ensure reliability over identification of tasks, the observers met before the first session and after a few sessions had been logged. Secondly, observers exchanged sessions, so that all courses were watched by at least two of the three observers. Thirdly, screenshots of shared workspace pages were available as a cross check of the task data (see 6.10.5). The set of checklists was analysed using a Perl script, which, as appropriate, counted instances that were logged and aggregated comments. A printout of the output is in Appendix 6. The aggregated observer and participant comments were categorised in order to identify a list of issues, which could thai be compared with the issues that emerged from analysis of the student and teacher focus groups, interviews and the expert observers' report.
Conditions: The name and date of the class has been deleted since it would enable identiHcatbn of tutor and
students. In addition to identifying class, date and observer, this section records whether the session was recorded
(using a prototype multimedia data recorder), whether the audio tool's packet repair mechanism, "redundancy", is switched on, whether the frame rate is set at ottier than the default of 2 frames per second, machine name, whether extra light is provided and any other conditions that may arise.
Date=Deleted Class=Deleted Observer=2
Recorded=N Redundancy=na Frame_rate=na Machine= Heniy Headset=na Light=y Other conditions=0
Teacher-identified Tasks are coded, T.
T1-T8 correspond with the list of tasks in section 6.9.1.
T9 is for observers to list activities that fall outside these categories.
T10 was a project, not PhD, attempt to indicate the extent to which students were interacting with one another, as opposed to responding individually to the teacher. It was not used in this research.
Tl_Conversation=l T2_Graminar=0 T3_Spelling=0 T4_Gap-filling=0 T5_Reading=l T6_Writing=0 T7_Vocabulary=l T8 Pronunciation^ 1 T9_Other=Translation o f single words and short phrases T10_Interaction=Intermittent
\ndeo-reiated items are coded, V:
V1_Camera - observed participant adjusts camera: each instance logged.
V2_Posltion - observed participant adjusts own position in relation to camera: each instance logged.
Numbers V3 and V4 relate to posstle user problems, commonly found in video-conferenchg environments: V 3 - a participant moves out of shot (instances counted). V4 Participant image suggests lighting is inadequate 1 or 0 = Yes or No).
V5 - observed participant enlarges one of the Images (see Figure 6.1 for an example of doing this) Instances counted.
V6 - complete loss of video image(s). Each instance logged.
V7 is for events that fall outside the given categories (described briefly by obsen/ers).
Vl_Camera=0 V2_Fosition=0 V3_Out-of-shot=2 V4_Poorly-lit=0 V5_Large-pic=0 V6_ioss=0 V7_0thei=Studentl responded to question with a nod. She is well-lit so this works and you can see her expression. Student 1 tends to give visual as well as verbal responses and therefore appears at first sight to be more fully involved than Student2.
Shared workspace-related items are coded W:
W1 to W7 refer to the main functions of the shared workspace. The observer records whether each is used in the
session (0 or 1). To clarify 2 of these: W5 = creation of a new, blank page during the session. W6 = change between
pages created, usually prior to the session, through file imports.
W8-W10 differentiate between importing Postscript and Text files, and record whether files were imported during the teaching session.
W11 and W12 count instances of problems: W11 -cras h or freeze of the tool; W 12-failu re of the click-to-type mechanism.
W_Other is for events that fall outside the given categories (described briefly by observers).
W l_Type=l W2_Draw=0 W3_Erase=l W4_Colour=0 W5_Blank-page=0 W6_Change-page=l W7_Move=0 W8_lmportedPS=l W91mportedPlain=0 W10_Lesson-import=0 W ll_Freeze-crash=l W12Type-failure^2
W13_student_use=fi"equent W14_Other=Start plus 14 minutes TUTOR cannot get whiteboard to respond. OBS. also tried. Worked after quit and restart. In the meantime students wrote the words which TUTOR said and appeared to enjoy doing it. Problem when TUTOR wanted to erase one [of[ the student-typed words. TUTOR also appeared frustrated by inability to write. Some problems navigating between pages especially remembering which way to scroll. Page numbers not noticed so all found it hard to reference them. Student2 continued to use wb after problem was fixed. Scanned text in small font. Studentl leans forward to read it and is therefore only visible in the bottom o f the window. Scrolls too fest and skips pages 3 times.
Audio-related items are coded A:
A1 - observed participant adjusts the headset (instances counted).
A2 and A3 - observed participant adjusts different volume controls (instances counted).