6.9 WiFiSL Evaluation
6.9.1 First Evaluation Session
The first evaluation session was performed with the participation of a post-graduate student in the School of Computer Science at the University of St Andrews researching networks. The student was under direct supervision and asked to perform various tasks in-world using the virtual laboratory. Before attempting these tasks, an introductory talk was provided detailing the uses of the components and ways to construct a scenario.
Initial problems involved the complicated rights management in Second Life. The default behaviour of Second Life is to restrict copying an item that has been set for sale. When the node and orb were set for sale, both had to be explicitly marked as available for copying by the student.
The student was quickly able to join nodes together. The notion of scaling traffic time to real time was seen as confusing.
This was the first use of WiFiSL and despite problems, the user was able to deploy networking components, construct a scenario and review the resulting animation. The version of WiFiSL used was one where all actions and durations in the virtual world were scaled from the user-perceptible time frame of minutes to a simulator-suitable timeframe of milliseconds. The first set of results from this session is the process-data which provides us with information on the user’s processes. The process-
136
data presented here highlights problems with this version of WiFiSL which were addressed before continuing evaluation with other users.
6.9.1.1 Process Data
• Multiple permission problems when transferring the networking components into the user’s inventory. The networking component in their inventory was unable to be copied more than once.
• Lack of textual support. The user was unable to review previous signals transmitted on the network in the chat history box.
• The user expressed difficulty in relating to the simulation time. This was due to the lack of simulation time being displayed when the signal is animated. WiFiVL I and II had shown the time at which a signal was transmitted in a text console.
• Difficulties in traffic construction expressed and a misunderstanding of how to express duration in a link.
• The console interface was used and became a powerful way of expressing more complicated scenarios. To construct a scenario in the console requires the sender and receiver identifier. There was no way to find a networking component’s identification number in the virtual world.
• User expressed confusion during long gaps in the playback. The user was unsure if this was due to gaps in the simulation or if all the results had been shown and playback was finished.
Task Name
Status at end of session
Deploy networking components into the environment
Successful
Construction of a scenario involving two nodestransferring a file
Successful
Alter the routing protocolUnsuccessful
Alter the MAC protocol and describe the changesobserved
Unsuccessful
Modify the sensing range and observe the difference in
Unsuccessful
Modify the RTS/CTS threshold and describe, with usageof supplied statistics, any changes observed
Unsuccessful
Table 21: Task results – bottom data for first evaluation session137
Question
Strongly
disagree
Strongly
agree
I think I would like to use this system frequently
I found the system unnecessarily complex
I thought the system was easy to use
I think that I would need the support of a technical person to be able to use this system
I found the various functions in this system were well integrated
I thought there was too much inconsistency in this system
I would imagine that most people would learn to use this system very quickly
I found the system very cumbersome to use
I felt very confident using the system
I needed to learn a lot of things before I could get going with this system
Table 22: SUS results from the first evaluation session, the score for this response is 52.5%
This SUS score is 52.5/100 and reflects the difficulties the user had in interacting with the WiFiSL. The perceived educational value was 66%.
6.9.1.2 Unstructured Interview
The unstructured interview was used to reflect on the difficulties encountered, suggested improvements and areas of the interface that were enjoyable to use. The user expressed a real sense of engagement and immersion into the virtual world. They found the interface for placing networking components in- world and creating connections to be “easy to use”. Another aspect that was discussed was the advantage and interest in observing the signal interactions and the ability to enjoy different angles during playback. A focus point for the interview was the difficulty in understanding the translation of in-world time to simulation time.
6.9.1.3 Analysis
This version of WiFiSL scaled the length of user interaction in the virtual world to simulation time. As an example, if a user were to leave two networking components connected for 10 seconds before closing the connection, this would translate to 0.5 seconds of traffic communication in the simulation. Another example of the scaling was that the time between two connections was also scaled. So if a user were to wait 2 minutes between closing the first connection and creating a second one, then the
138
time between the two connections in the simulator would be scaled to 2 seconds. These notions confused the user and created difficulties in understanding the scenario they were creating.
The notion of translating in-world time to simulator time was confusing and so has been removed from WiFiSL. It was as a result of this session that users now construct connections between nodes and a fixed file size is transferred between nodes. The times between transfers are scheduled by the All Seeing Orb to occur 0.002 seconds after each other. This evaluation session has shown that the bottom data would not have shown up the problems that the user was encountering. It is therefore important to maintain both process and bottom data.