• Users can rate the available self-assessment questions in terms of their difficulty and quality.
Besides increasing of the learning process quality, social activities improve the quality of the created learning material. Even when answering a quiz, users can contribute by analyzing the quality of the questions and making suggestions of how to improve them. Thus, the knowledge is being created not only explicitly by contributors, but also implicitly through discussions, answering the questions of assessment tests, or in other words through native learning activities.
5.2 Usability Evaluation
In order to evaluate the implementation we have conducted three usability studies. The first two in form of user survey aimed to evaluate the usability of the system at different stages of development. The third one in form of single-user observation aimed to define bugs and particular user interface issues. Based on the study results, we have found that the concepts behind SlideWiki are clear to the users, even though the performance and usability of the system have to be improved in order to support the large-scale collaboration more efficiently.
5.2.1 Study at the Chemnitz University
In order to evaluate the usability of the SlideWiki platform at early development stages we used the platform for accompanying the Information Systems course at Chemnitz Technical University. We informed students about the e-learning functionality of the system, in partic- ular, how to use it to prepare for the examination. The experiment was not obligatory but students actively contributed by creating additional questions and fixing mistakes in slides. The experiment was announced to 30 students of the second year and 28 of them registered in the system.
After the end of semester, we have asked participants to fill out a questionnaire which con- sisted of three parts: usability experience questions, learning quality questions and open ques- tions for collecting the qualitative feedbacks. We have collected 9 questionnaires that were filled out completely. They show us emergent problems and directions for the future.
In the first part of the questionnaire we included questions recommended by System Usability Scale (SUS) [84] system to grade the usability of SlideWiki. SUS is a standardized, simple, ten-item scale-based questionnaire29giving a global view of subjective assessments of usability.
It yields a single number in the range of 0 to 100 which represents a composite measure of the overall usability of the system. The results of our survey showed a mean usability score of 67.2 for SlideWiki which indicates a reasonable level of usability.
The second part of the questionnaire aimed to determine whether the SlideWiki helps to improve the quality of learning. It consisted of four questions with five options from “absolutely agree (1)” to “absolutely disagree (5)”. The evaluation results for these two parts are presented in Figure 5.14.
Although the positive answers prevailed, we were not satisfied by the fact that for many questions a third of participants chose the neutral value. The final part of the questionnaire helped us to understand the reasons. We included here four open questions:
1. What did you like most about SlideWiki?
Figure 5.14: Results of SlideWiki evaluation survey: mean µ and standard deviation σ.
2. What did you like least about SlideWiki?
3. What can we do to improve the SlideWiki’s usability? 4. What features would you add to SlideWiki?
Within the answers we found repeated complaints about several bugs, that interfered the working process. We consider this fact to be the main reason of neutral and contradictory values. However, we collected also positive opinions, especially about features and possibilities that SlideWiki allows. Three of the recipients mentioned that they mostly liked that SlideWiki is easy to use, four of them noted, that they liked the idea of collaborative work and sharing the presentations itself. Within the collected answers we also got important suggestions, which could be roughly divided into two groups:
• Suggestions about desired improvements of existing features such as displaying the test results graphically, supporting more import formats, improving the SVG editor etc. • Suggestions about totally new features, several of those were later implemented, e.g.
translation, templates for presentation structure, etc.
The results of the initial evaluation were promising and we have continued the features de- velopment, trying to optimize the system performance and usability at the same time. As well, we have collected statistics of the system usage. Its analysis allows us to claim that usage of SlideWiki platform positively influenced the quality of learning, as discussed in subsec- tion 6.3.2.
5.2 Usability Evaluation
5.2.2 Study at the University of Bonn
After implementing the whole SlideWiki functionality we have conducted another study. This time the goal was to have a detailed view at the problematic aspects of the implementation discovered through the previous study. Namely, questions of this survey were designed to evaluate the following three aspects:
1. responsiveness of the SlideWiki website; 2. design and usability of the SlideWiki website;
3. functionality and awareness of the features and usage of the SlideWiki website.
The survey was divided into two parts. The first part consisted of 17 multiple choice questions and the remaining 5 questions were left for comments and ideas. We have received 23 anonymous responses out of 80 students registered for the “Semantic Data Web Technologies” course.
Question Average
SlideWiki is easy to use. 2.39
I find it easy to navigate through SlideWiki pages. 2.3 SlideWiki provides a good user interface for learning slides. 2.35
I find SlideWiki clear and well organized. 2.57
SlideWiki loads quickly on my computer (3-4 sec). 2.39
SlideWiki loads quickly on my mobile (3-4 sec). 2.0
I can immediately find response to my actions (mouse click). 2.57
I can easily use SlideWiki on my mobile/tablet. 2.17
I can use SlideWiki with different browsers. 3.62
I can easily download the slides. 1.95
I found all the images, text, tables, etc. in the slides correctly rendered. 2.32 I can easily navigate through the slide pages using the navigation tree. 2.96 I had no problem finding the information I needed. 2.48
I could find some bugs while using SlideWiki. 3.78
I find all the links in the slides valid and active. 3.38 I tried the Questions section of the slides and I found it helpful 3.68 I tried the Discussion section of the slides and I found it helpful. 2.95
Table 5.5: Results of the usability study at the University of Bonn.
The results of the study, as presented in Table 5.5, proved our previous concerns about usability of the trial SlideWiki implementation, especially on mobile devices. The table consists of the MCQ and the average of the answers (1 = strongly disagree and 5 = strongly agree), categorized by the cell colors: dark-gray referring to “Responsiveness of the website”, (2)
light-gray referring to “Design and usability” and (3) white referring to “Functionality and awareness of the website features and usage”. As well we have noticed even less satisfaction from SlideWiki performance than in the previous study, even though we have spent efforts to improve it. We consider there are two main reasons for even lower SlideWiki user satisfaction we have found out through the second evaluation:
• Actual decrease in SlideWiki performance due to substantial growth of its database. Although the platform architecture was proven to be scalable with increasing amount of content, the increased amount of users simultaneously working on the same presentation was not taken into account.
• Increase of users expectations. During the development process new technological stacks and architectural patterns have appeared and spread widely, improving the possible per- formance of the systems with the similar scale. Thus, what have been considered to be relatively good performance at the beginning of development, has become not satisfying after the development has been finished.
However, the concepts underlying SlideWiki were still accepted by the users. This was proven by open answers we have collected in the second part of the survey:
• If you were to review your overall SlideWiki experience, what score would you give it out
of 10? 13 student gave it 5 and more, while 9 gave it less that 5.
• What do you find most frustrating about SlideWiki? Most of the comments concerned navigation, especially on mobile devices and slowness of loading.
• If you could change one thing about SlideWiki what would it be and why? Many students suggested improving the performance of loading slides.
• What do you like best about SlideWiki? As it can be seen from the Table 5.6, the students mentioned all main concepts we have based SlideWiki and the thesis except multilin- guality:
Feature # of students
Collaboration on the educational material 4
Different types of educational content, in particular, self-assessment items 4
Tree-structure of the content 2
Accessibility of the content from everywhere 2
Table 5.6: The most liked aspects of SlideWiki according to the final user evaluation.
• How can we improve SlideWiki? Please tell us your ideas and suggestions? Mostly students recommended changing layout (e.g. structure, minimalist design) and improving performance.
Based on the final evaluation results, we have concluded that the chosen architectural pattern and technological stack do not allow large-scale implementation of our concepts with sufficient performance. We have analyzed the technical issues and have found recently emerged tech- nologies to solve them, as proposed in section 5.3. However, before the implementation of SlideWiki 2.0 we also needed to design the interface which considers all, even minor, UI issues indicated by the users.
5.2 Usability Evaluation
5.2.3 Single-user Observation
In order to spot minor usability issues a single-user observation study [60] was conducted. Three users, whose demographical data is described in Table 5.7, were chosen.
# Gender Age Occupation Computer Skills
1 Female 19 Student Engineer Good
2 Male 25 IT helpdesk Excellent
3 Male 22 Economics Graduate Average
Table 5.7: Demographic data of users participating in the single-user observation study.
In order to perform interaction with the SlideWiki, users were asked to complete the fol- lowing sequence of actions:
1. Register on SlideWiki.
2. Create a new empty presentation. 3. Edit the presentation description.
4. Add content to the first slide, using different text colors. 5. Add an additional empty slide to your presentation. 6. Add an image to the slide.
7. Add an additional sub-deck to your presentation.
8. Move the second slide you’ve created (with an image) to the new sub-deck. 9. Change the title of the slide.
10. Create a question to one of your slides. 11. Download the presentation.
12. Initiate the translation of the presentation into a different language. 13. Comment on the translated version.
14. Change the slides theme and transitions of the translated version. 15. Edit the presentation description of the translated version.
16. Play the translated version.
17. Find and open the deck “Open Educational handbook”. 18. Open a user page of any of the collaborators.
19. Message him/her.
20. Contact the website owners.
21. Search using the search field for the presentation “Semantic Web Lectures”. 22. Examine yourself on the course.
After finishing all the stages, the users were asked to give comments and suggestions. A screen recording tool was used in order to keep the video of the user interaction with the product for further exploit. The found issues and possible solutions are summarized in [60]. For example, while completing the task #5 (Add an additional empty slide to your presentation) all three users made at least 3 wrong actions and the fastest of them carried out the task within 47 seconds. In the current component implementation a user is supposed to complete the task by right-click on the deck title in the tree and choosing the necessary action from the drop-down list. However, this interface appeared to be not intuitive even for experienced users. In order to solve the issue, we propose to add two small buttons for slide and deck adding, as depicted at Figure 5.15.
Figure 5.15: The deck tree navigation component: current state (left) and proposed improvements (right).
Another common issue found is the presentation of search results. The current state of the component is presented at Figure 5.16. All three users called their created deck using the word “test” and it was difficult to find the exact presentation within dozens of others. As well, the tag cloud to filter the content is over-fulled, and at the same time does not have to be always visible for the users. Our solution (see Figure 5.17) for these issues is to (1) enhance the search functionality by adding additional scope for search by authors and tags, and (2) hide those advanced options until user needs them.
Figure 5.16: The search component: current state.