A user study was conducted to assess the usability and functionality of the XSS Framework. Fur- thermore, the study should show whether the framework is flexible enough to support different types of collaboration scripts. During the development of the framework regular tests were con- ducted using the M.U.R.D.E.R. script. However, as a proof of concept we needed to see if the framework works with other collaboration scripts as well.
3.4 Case Study 51
7 MEDIA INFORMATICS
DEPARTMENT FOR COMPUTER SCIENCE
Group Administration Tool
Group Administration ToolNew group Load user data
Groups
Unassigned Users
Username Sex Age Nationality Dummy Col Dummy Col More male female female Tom Kate Tanja 19 21 18 UK US German dummy data dummy data dummy data dummy data dummy data dummy data i i i i i Group 1 Group 2 Close group Start
Username Role More
Tom Kate Sex Age male female 19 21 i i i Play ˅ listener ˅ reviewer ˅ testscript UK US Nationality drag&drop .csv
_
܆Figure 3.15: Group Administration Tool for tutors.
10 MEDIA INFORMATICS
DEPARTMENT FOR COMPUTER SCIENCE
Automatic Group Formation
# 10
Assign all users
Random Under conditions
homogeneous heterogeneous random Gender
Age Nationality
cancel OK
_
܆52 3 From Graphical Learning Designs to Scripted Learning Applications
3.4.1
Method
Participants and Designs
We invited four doctoral students of the learning sciences who worked with different collabo- ration scripts. That way the participants could be asked to use their own collaboration scripts. Three participants were female. Their age ranged between 26 and 34.
Learning Task and Procedure
The subjects modelled their own collaboration scripts using MoCoLADe and exported an IMS- LD document. Using the XSS Framework the IMS-LD was then imported to create the according learning application (using the Simple Starter). Furthermore, a simulation was run with the Multi Client Tool and the Group Administration Tool was used for the group formation. However, the Dummy Clients were not finalized. Thus, the complete development process (see Figure 3.11) was run through except for steps four and six.
As the subjects were not familiar with MoCoLADe the study was divided in two parts. In the first session the experimenter gave a MoCoLaDe training. In the beginning of the training session there was a lecture on MoCoLaDe’s functionality and components. Afterwards the participants were given laptops and modelled an exemplary collaboration script following a step-by-step tu- torial given by the instructor. Finally, the participants modelled their own collaboration scripts and exported it as IMS-LD document.
The second part of the study was scheduled for the following day. At first there was a short intro- duction to the framework. Then the development process was demonstrated using the exemplary script from the day before. Afterwards each participant imported the IMS-LD file that they had created in the last session, ran the application and tested the Multi Client as well as the Group Administration Tool. In the end, there was a questionnaire followed by an open discussion about missing features as well as other suggestions for improvement.
Measures
At the end of the second day, the participants were handed out questionnaires to examine the usability of the XSS Framework. The questions were split into usability assessment of the server and of the Group Administration Tool (cf. Figure 3.17). In addition, the questionnaire contained open questions regarding features that should be added to the XSS Framework. Their ideas were then discussed in more detail in interviews which were conducted after the questionnaires had been handed in.
3.4 Case Study 53
3.4.2
Results
Functionality
The doctoral students modelled a peer-questioning (King, 1997), two peer-review (Weinberger et al., 2005b) and an argumentative collaboration (Stegmann et al., 2007b) script. All of these learning scripts were successfully modelled in MoCoLaDe, imported and run as simulation using the XSS framework. However, IMS-LD currently does not support iterative cycles of activity sequences or conditional behavior (if ... then do activityX else do activityY). Both occurred in the user study. While this shortcoming is not caused by the XSS Framework, it limits the frame- work’s functionality. To solve the problem the participants’ collaboration scripts were simplified. Multiple cycles were reduced to one cycle and instead of conditional behavior a default activity was defined.
The participants were overall satisfied with the functionality. However, they asked for additional features during classroom operation. Similar to the Multi Client, which provides an overview during simulations, an Observation Tool could provide an overview of all running client windows and update changes in real time. The Observation Tool should also visualize which clients are in the same group. Furthermore, in the final questionnaire three participants suggested adding the functionality to export data. In the open discussion this suggestion was further discussed. As the individual steps as well as the results of the learning session often need to be evaluated, it should be possible to export a protocol and the outcome of the learning session.
Usability Assessment
To assess the usability of the user interfaces the final questionnaire contained several questions regarding the server and Group Administration Tool. The questions (for example ‘The operations are clearly labelled’) had to be answered on a Likert scale from 1 (completely disagree) to 5 (completely agree). All questions were answered very positively with averages of 4.5 or higher (see Figure 3.17).
In the discussion one participant proposed to combine the three windows (Simple Starter, Group Administration Tool and Multi Client) into one single window. Whether this is still feasible once additional features (e.g. the Observation Tool) are implemented, and how the windows could be merged to maintain a clear arrangement needs to be considered carefully.
3.4.3
Summary
Our case study provided evidence that the XSS Framework in combination with MoCoLaDe could be a helpful tool for educators. Doctoral students in the field of educational sciences were able to author their collaboration scripts with MoCoLaDe and test them using the framework’s simulation tool. The XSS Framework not only provides tools for the development of collabo- ration scripts but also serves as runtime environment for collaboration scripts. Our results also
54 3 From Graphical Learning Designs to Scripted Learning Applications
22 MEDIA INFORMATICS
DEPARTMENT FOR COMPUTER SCIENCE
Server Group Administration Tool
4.75 4.50 4.75 5.00 5.00 4.50 5.00 4.75 1 2 3 4 5 The configuration of the server is easy. The operations are clearly labeled. It is easy to start the server. The operations are clearly labeled. It is easy to assign learners to groups. It is easy to manage user profiles. It is easy to manage groups. It is easy to switch between manual or automatic assignment completely disagree completely agree
Figure 3.17: Usability assessment of server and Group Administration Tool.
showed some limitations of the current approach. So far IMS-LD does not allow cycles of ac- tivities or conditional sequences of activities. As cycles and conditions are fairly common in collaboration scripts the framework functionality should be further extended beyond the capabil- ities of IMS-LD. In addition, the participants wished for additional functionalities. During the learning phase they would like to monitor the activities of the learners. Finally, it should also be possible to archive process data and outcomes.