6. An OpenSim Training Environment
6.2 An Evaluation of OpenSim User Training Methods
6.2.3 Results and Analysis
The first analysis was performed on the times that each user spent in reaching the first set of sensors on the walking path. The distributions of the user times at each sensor are shown in box plot graphs in Fig. 6.4 (for Group-Island) and Fig. 6.5 (for Group-Doc). The distributions generally indicate a higher amount of time for Group-Doc than Group-Island. Also the variance of times per sensor is considerably higher for users from Group-Doc compared to Group-Island.
The distribution of average times for each sensor for the two groups is shown in Fig. 6.6. A clear difference in the mean times can be seen during the early stage of walking. Until the 5th sensor the participants from Group-Doc showed much longer times to reach sensors while walking on the middle of the path. From the 5th sensor to the 14th sensor the time distributions are somewhat similar in their trends but with a reasonable time difference. This may have been due to the fact that the Group-Doc users gained an opportunity to train further during their walk within the first few sensors; yet, the time taken is higher compared to the other group, although the difference is in the range of a few seconds.
135
Figure 6.4: Boxplots of walking times to each sensor for the island users
Figure 6.5: Boxplots of walking times to each sensor for the document users
For the 15th sensor, a further difference is noted in between the two sample trends with a deviation. One of the important observations during the experiment helped to explain the reason for this. The users from Group-Doc often got slowed down at the end of the walk trying to figure out of how to fly for the next phase of the task. In contrast the users from Group-Island confidently finished walking and started their flying without hesitation. Both groups showed increased times for passing the sensors 6, 10 and 13. These sensors are located at the bends of the path that require extra avatar control effort to maintain their trajectory.
136
Figure 6.6: Average times taken by each group to reach sensors
A time comparison of avatar flying between the two groups is shown in Fig. 6.7. The Group-Island users showed lower times compared to the users from the other group; we use pairwise comparison based on the user codes assigned (i.e., User1-GDOC with User1-GIsland; User2-GDOC with User2-GIsland; and so on) only for presentation purposes (the figures 6.7 - 6.10); we do not infer on pairwise comparison. Moreover, the users of Group-Island show a lower variance in time against the other group; in fact, it was the expected result as the flying path was a straight line, these users showed more confidence in doing their task quickly and on a straight path while the users who trained from the guide document showed some tremble in all directions during their flying.
137
Figure 6.7: Average time taken by the two groups – Task Flying
Fig. 6.8 and 6.9 show the times spent by each user for object creation and the required editing tasks, respectively. For both tasks, all users from Group-Island showed much less time than the other group of users. For object creation the time difference was slightly lower since it comprised a single task. However, for object editing, which included a few tasks, the time difference between the two groups is high. Also the variance of times is lower with users from Group-Island. For example, most Group- Island users carried out the object resize and re-position task in a single try through typing the exact figures, while the other group (Group-Doc) tried to drag and resize using the mouse showing a poorer understanding of the system which resulted in a trial and error approach.
138
Figure 6.8: Average time taken by the two groups – Task object creation
Figure 6.9: Average time taken by the two groups – Task object editing
At the end of the object-editing task users were asked to move along the exit path and carry out IM and chat to finish the task session. Fig.6.10 shows the times used by each participant in the two groups. User3-GDoc and User4-GDoc show a lesser time than User5-GIsland, which is an interesting observation. However, when closely examined it was identified that these two users from the Doc group did not
139
complete the IM messaging and abandoned it. Further details about task completion will be discussed later.
Figure 6.10: Average time taken by the two groups – Task IM and finishing
Table 6.1 summarises the average times spent by each group for the four tasks recently discussed. Object edit shows the highest difference between the times while IM and chat shows the lowest difference. In fact, IM and chat are relatively generic tasks that are implemented according to similar functionalities that are used in popular IM & chat applications. On the other hand, object edit is an OpenSim system specific functionality that requires a certain level of know-how to practice well. These may have been the reasons for this result; nevertheless, it implies that when the tasks are complex we can expect a further difference in times for the two groups. Table 6.2 shows the descriptive statistics of times taken by the two groups to complete the entire task. A clear difference in the total mean time can be observed between the groups; however the times were compared with a reference scale for an accurate interpretation. An expert user with extensive OpenSim/SL experience was asked to complete the experiment tasks in a similar manner (as soon as possible completion) for 3 times. The mean total time was 259.41s, which is considered as a reference value to compare the test results of the two groups. In addition to the considerable difference in the total times taken by the two groups to complete the task, a clear difference between the total times can be observed with reference to the expert user time. Group-Island showed a +31.597s of overhead time (12.18%) while Group-Doc had +124.035s of overhead time (47.81%). It is observed that the participants trained using the training island have been able to rapidly apply their skills gained through the training session for their activities compared to the group that trained with the document. In summary the participants that used the island showed
140
competencies closer to the expert user in completing the given tasks than the participants from the document group.
Group Flying (s) Object Creation (s) Object Edit (s) IM and Finish (s) Group - DOC 36.78 25.90 140.04 70.75
Group - Island 26.16 18.91 111.49 64.70
Table 6.1: Average time comparison for the four task phases
Group Mean Total Time (s) Std. Deviation Std. Error of Mean G-Island 291.07 5.76 2.58
G-Doc 383.44 11.07 4.95
Table 6.2: Descriptive statistics of the entire task for the two groups
There was a noticeable difference in the completion of tasks between the groups. Table 6.3 summarises the key observations about the task completion. Only one participant from the Group-Doc could complete all the tasks as expected. In particular, the complex tasks such as object manipulation seem to be the most challenging. In contrast, only one participant from the Group-Island use failed to complete all the tasks. However, all the participants successfully engaged in the activities of walking, flying and chat messaging.
Participant
Key observations about the completion of the tasks
User1-GDOC No object surface colour change, object positioning is not accurate
User2-GDOC No object surface colour change
User3-GDOC No object surface colour change, object size & object position are not
accurate, No IM message
User4-GDOC No IM message, the whole object is red, object position is not accurate
User5-GDOC All tasks were done
User1-GIsland All tasks were done
User2-GIsland All tasks were done
User3-GIsland No object surface colour change
User4-GIsland All tasks were done
User5-GIsland All tasks were done