An evaluation session was designed to guide students through each of the areas available on Aeolus. The students were Undergraduates at the School of Computer Science in the University of St Andrews who responded to a lab announcement for volunteers. A worksheet was provided (see Appendix 1.9). In this worksheet the students start with familiarisation of virtual worlds and the movement within such an environment. Next, students are guided towards the lecture theatre and asked to find some information on CSMA and the Aloha protocol.
After this, the worksheet advises students to review the problems and solutions section of Aeolus. This section provides the animation of aspects of CSMA and the hidden and exposed node problems. Once students have worked along this section, there is a question about frame formats. The frame format section of the island provides an interactive frame structure and display board containing information. The students gather information on the different fields here and answer questions such as which components of the frame provide support for fragmentation. Once these sections have been covered, the students can construct their own experiments using WiFiVL. This enables the display of the mechanism for transferring data, the ability to modify the MAC protocol and toggling the RTS/CTS frames.
The students were a mixture of 3rd and 4th year honour students studying for Internet Computer Science degree at the University of St Andrews. The students were not able to use any external resource such as a web search engine to provide answers to the question sheet.
151
6.13.1
Observer Notes
The session took place in February 2010 and lasted two hours, with five students using the same computer configuration of CentOS release 5.3 (Final) 2.6.18-128.7.1 x86_64 Intel Core 2 Quad CPU Q6600 @2.4GHz. The client viewers were the Second Life viewer 1.23.5 (136274) October 14 2009 13:31:00 Release Candidate. The server that hosted an instance of the WiFi Island was a Windows 7 Professional, Intel Xenon CPU E5504 @2.00GHz (2 processors) 16GB RAM, 64bit CPU and OS. The island was loaded using a single shareable OAR file.
The default configuration of Second Life viewer was modified to increase the maximum resolution and draw distance. The default draw distance limited the navigation of users around Aeolus as they were unable to see structures further than 50 metres away. It appeared as though some slides were slow to load but did not hinder progress by a great degree.
Aeolus had all permission restrictions removed. This allowed any student to modify or create any object on the island. While some slides and video displays were deleted, this was mostly done by accident and easily remedied. The students enjoyed having freedom over their progress through the worksheet and their environment. Although not specifically mentioned in the worksheet, a great interest by the students was shown in video content as well as the WiFi museum.
Three of the group organised themselves whilst reviewing the slides so that they only moved on when everybody was ready. The organisation was done through a mixture of chatting in the laboratory and
in-world messages.
An Open Simulator bug [302] that would not fully delete objects from the environment was observed during this session. The issue can be fixed by users reconnecting though the students did not find it to be a large impediment. This was fixed by updating to the latest release of Open Simulator.
One student, new to the study of wireless networks, having progressed through the problems section, spotted an error in one of the slides. The student went on to hypothesise, correctly, about what the slide should say. This demonstrates that the problems section was readily absorbed and understood by the student.
Four out of five students were able to use the virtual laboratory through the use of only the worksheet and helper pages provided. This demonstrates that there is adequate information to enable students to perform tasks in the virtual laboratory. However, more information can be provided in the helper documentation especially with respect to more advanced settings and simulations available.
6.13.2
Structured Feedback
There were three forms of evaluation used, a System Usability Scale (appendix 1.5), Perceived Educational Value (appendix 1.6) and a more general feedback form with open ended questions (appendix 1.11). The SUS results are shown in full in Table 28 and summarised in Table 29.
152
Question
Student ID
1
2
3
4
5
I enjoyed using this system
4
4
4
4
4
I found the system unnecessarily complex
2
2
2
1
2
I thought the system was easy to use
4
4
4
4
5
I think that I would need the support of a technical person to
be able to use this system
1
3
2
2
1
I found the various functions in this system were well
integrated
4
4
5
3
4
I thought there was too much inconsistency in this system
1
2
1
1
2
I would imagine that most people would learn to use this
system very quickly
4
4
5
4
5
I found the system very cumbersome to use
2
2
2
1
1
I felt very confident using the system
3
4
4
4
5
I needed to learn a lot of things before I could get going with
this system
2
2
1
1
2
Table 28: Full System Usability Scale results showing the question and individual student response where 1 is strongly disagree and 5 is strongly agree
The results shown in Table 28 is summarised in Table 29.
Student ID
SUS Score as a Percentage
1
77.5
2
72.5
3
85
4
82.5
5
87.5
Table 29: Summary of the results from an evaluation session using the System Usability Scale, the average score was 81%
The usability feedback from users shows all of the students were very comfortable with using the system. The overall average results are the highest results received in a usability test in this dissertation.
153
Question
Student ID
1
2
3
4
5
I feel that I have learned something by using this system
4
5
4
4
4
The wireless simulation provides believable information
4
5
5
5
4
I found it easy to find out information about the MAC layer
protocols
3
5
5
5
5
The quality of the material presented was consistent
4
5
5
5
4
Reviewing the hidden node terminal problem helped me to
understand it
4
5
4
5
5
I feel that using this system helps develop my understanding
of wireless protocols
4
5
5
5
4
I found the system educationally stimulating
4
5
5
5
4
I was able to easily create different scenarios
4
5
4
5
5
The tools provided by the system allowed me to practice the
theory that I have learned relating to wireless networks
4
5
5
5
4
The animation was intuitive and easy to understand
4
4
4
4
4
Table 30: Full perceived educational value responses from individual students where 1 is strongly disagree and 5 is strongly agree
The results shown in Table 30 are summarised in Table 31.
Student ID
Average Perceived Educational Value as
Percentage
1
78
2
98
3
92
4
96
5
86
Table 31: Summary of the feedback from users on the Perceived Educational Value, the average score was 90%
The user was provided with four questions which enabled an open anonymous response. These questions are summarised here and available as they were supplied to the students in appendix 1.11.
1. Is there any aspect of the system you particularly liked? 2. Was there anything you found difficult to do in the system?
3. Are there any features not already present you would like to see added? 4. Do you have any comments about this session in general?
The comments received were very positive, with students highlighting parts of the laboratory session they enjoyed and suggesting improvements in other areas. An example of the positive feedback is demonstrated in Table 32.
154
1. I liked the slides and simulations available, they were easy to use and understand 2. It was difficult to remember the colour coding during the simulation
3. Some sort of portable colour coded chart for the simulation (object you can drop on the ground in front of simulation)
Prevent users from deleting the slides 4. It was fun and interesting.
Table 32: Example open feedback from students
Most students commented that they enjoyed using the system as shown in the example feedback provided in Table 33.
• Other than some underlying parts, bugs in the world simulator for example, the session was quite effective
• It was fun and interesting
• An enjoyable experience overall, OpenSim was a bit buggy
Table 33: Selection of user feedback to Question 4 on the open feedback form
6.13.3
Informal Unguided Interview
The users expressed the view that the session was very enjoyable and that they could see many possibilities with a WiFi island. All users did say that they felt they had learned something through using the software. An interesting discussion with a student was the number of students on an island. They expressed concern that having 30 students on an island would cause great difficulty with activities such as reviewing lecture slides. The student, when told that they could have an island of their own, responded that this was not ideal either as they found it valuable to learn from seeing and interacting with what others were currently doing on the island.