Scenario Steps 149

MOT 2.0 (third prototype) can recommend content as an Adaptive Hypermedia system, but at the same time it can recommend peers who can help with the learning process (e.g., recommending more experienced peers, who could help the student with the current questions). The scenario created to determine the appropriateness of the hypotheses is designed as follows, see Figure 37.

Figure 37 Scenario Steps

The participants (students) take a pre-test to determine their knowledge level (for a selected domain) out of:beginner,intermediate,advanced. Based on the knowledge level extracted from the pre-test, the participants are given different sets of privileges, as per Strategy 3 in Section 8.2, which can be expressed, in natural language, as follows:

 Beginnerusers can only read the learning material.

 Intermediate users can read the learning material, as well as add comments (feedback).

 Advanced users are allowed to read the learning material, edit the tags, rate the content and add comments (feedback).

After this initial classification, the participants are asked to accomplish a learning goal using MOT 2.0 (third prototype) (i.e., to learn a specific lesson on “Collaborative Filtering”). In order to achieve the learning goal, the participants are divided into two sub- groups: Group one: the first group (see Figure 38), acting as a control group, has to perform the learning activity by using MOT 2.0 (third prototype) without any help from thecontent recommender, or the(expert) learners recommender. This group however can still benefit from all Web 2.0 social environment facilities, depending on their set of privileges, as above. Figure 38 shows a screenshot of an intermediate learner of group G1. He can give feedback but cannot rate or add tags (right frame). The centre displays the

learning content of the current item. The menu and recommendations are displayed on the left frame. As this user belongs to the control group, all items are shown as recommended.

Figure 38 Screenshot of learning environment of Group 1

Group two: the second group (see Figure 39) has to learn by using MOT 2.0 (third prototype), supported by both the help of recommended learning content(per item; i.e., the content related to the current item is recommended, as per Strategy 1 in Section 8.2), as well as with the help of therecommended learners(per item; i.e., the expert for a given content item is recommended to learners, as per Strategy 2 in Section 8.2).

Figure 39 Screenshot of learning environment of Group 2

Similarly to the previous prototype, the division into groups makes sure that participants of each of the three knowledge levels, beginners, intermediate and advanced, are distributed as evenly as possible between the two groups. This is done to ensure that no group outperforms the other two, due to its ‘lucky’ division of students.

Figure 39 shows a screenshot that is similar to the screenshot for group G1 (see Figure 38), with item content in the middle frame, a menu and recommendations on the left frame, and Web 2.0 features on the right frame. The difference is that this user is an advanced learner, so he can rate, tag and provide feedback. Additionally, as the learner belongs to group G2, only the recommended items are displayed in the left frame. Moreover, peer recommendations are displayed in the right frame.

The two screenshots above show the different views of the learning environment that are available to the two groups. Social actions such as rating, tagging and feedback (typical of Web 2.0 settings) are available to all users in all groups (as long as their knowledge level permits it). These features provide the benefits of e-learning 2.0 (as have been explained previously Section 6.5.2 and Section 8.2), and therefore represent the minimal functionality offered by MOT 2.0 (third prototype), even in the version stripped of all adaptation. After learning, the participants take a post-test(which is identical to thepre- test) to determine the learning outcome, by comparing the pre-test and post-test answers for each learner.

The effectiveness of the two groups (in terms of learning outcome) is to be determined by comparing the results of the pre-test and the post-test, and calculating an average for each group. We are aware that this only evaluates the short-term learning effect, and longer term evaluations would be needed to compare the long-term learning effect. However, it does accurately compare this effect for students who have benefited from rich adaptation (the combination of content-based adaptation and peer recommendations) with students who were provided with no adaptation, in an otherwise similar setup.

Moreover, both environments are fully functioning modern learning environments, featuring social Web 2.0 functionality (such as tagging, commenting, etc.) customized to the experience level of the user. Previous chapter research has shown that learners find such functionality helpful so these features did not need to be isolated in the current case study.

Finally, the participants answered a questionnaire about their perception of the system functions, their relevance, and finally, the system usability. Based on the answers to the post-test, the learner’s knowledge level is updated accordingly. After updating the user profile, the learner’s privileges can be updated as well. This means that – just as in the introductory scenario in Section 8.1 – the learners can see their rights updated, based on the number of the tests they are willing to take during their learning process.