Case Study: Context-based recommendations of Java LOs

Some of the challenges faced in the design of further research activities to validate my framework are derived from the novelty of this field, especially in the design and evaluation of context-aware suggestion mechanisms. At the time of writing, only one publication (Martin and Carro, 2009) has been located which contained two case studies and results findings of an evaluation of a context-based suggestion mechanism.

In order to validate my mCALS framework, I designed a further research activity to answer two research questions relating to the framework. The first question is how appropriate are the modified suggestion rules of Cui and Bull’s (2005) work,

which are incorporated into my framework, for suggesting Java learning objects to

students based on their contexts. The second question isfor what reasons do students

choose particular time slots to study in, in order to validate my novel concept of using

a learning schedule to retrieve users’ learning contexts.

This activity was an evaluation of a number of Java LOs by first year computer science (and other related courses) undergraduate students, primarily at our university. The LOs were obtained from the Codewitz LOs repository www.codewitz.org. Since these LOs are primarily oftutorialtype, I have adapted the

suggestion rules for students to study particular Java LOs. This was based on the learners’ level of motivation, their knowledge/proficiency level of Java and their amount of available time. For example, when a student had a lower level of motivation, easier LOs to study were suggested, and vice versa. The proficiency level of the LO and the length of time it requires to be completed are matched with the knowledge level of the student and the amount of available time that they have.

Participants of the study were asked to complete a feedback form after they had completed each online Java learning object. The feedback form was divided into three sections. The first section provided some basic information about the LO that was studied, the location it was studied in, length of time required, the rated motivation level of the participant at the time of study, the participant’s year and course of study and name of university. The second section provided information about studying the LO in those contexts, as follows:

1. How useful it was to study the LO in the set of contexts, i.e. motivation level, Java knowledge level, amount of available time;

2. Whether their learning experience of the LO was enjoyable;

3. Whether their experience was more enjoyable as a result of studying the LO in the proposed contexts;

4. Whether the LO was appropriate to be studied in those contexts;

5. How feasible, in their opinions, it would be to study the LO in any other contexts;

6. Which other learning activities, in their opinions, CAN be studied effectively and enjoyably in the same contexts;

7. Which other learning activities, in their opinions, CANNOT be studied effectively and enjoyably in the same contexts;

8. Whether they are aware of any LS that they may have; 9. Whether they know what these LS are;

10. Whether they would have benefitted from studying a LO suited to their LS. The final section a) related to learning content – 1) how useful they found the LO to be; and 2) would they use it again; and b) related to the time slot – 1) why they

chose the particular time slot to study the LO; and 2) whether the time slot was a good time for them to study in.

My online experiment can be viewed in Yau (2010). Appendices C and D show some screen shots of the online experiment, and the user feedback form for each LO respectively.

3.4.1 The validity and reliability of the Java LOs study

This experiment is valid as one of the means to validate whether the proposed learning contexts and suggestion rules are appropriate for use within my framework because it is a proof of concept regarding these two aspects that I wish to validate within the proposed framework. I have set up the online Java experiment using a selection of procedural and object-oriented topics taken from the Codewitz learning object repository including If-statements, arrays, while-loops, exceptions, methods, classes, arithmetic and object-oriented programming. The experiment was set up to allow participants to first select their available time (10, 15 or 20 minutes), followed by their current motivation level (high, medium, low), followed by their knowledge level of Java (high, medium or low). A choice of a few LOs that are appropriate for the context appears for the participant to select to learn/study. These suggestions are based on 1) formed general suggestion rules are as follows, and 2) the established proficiency levels of Java. Note that the difficult, medium and easy levels of tasks are in terms of cognition. See Appendix C for some screen shots of this experiment.

 If motivation = high and available time > 30 min then difficult tasks are selected.

 If motivation = medium and available time > 30 min then medium tasks are selected.

 If motivation = low and available time > 30 min then easy tasks are selected.  If available time < 30 min then easy tasks are selected.

In order to assign particular Java topics to students based upon their proficiency level of Java for my case study, I needed to first determine an order of difficulty of Java topics. A fellow student and I were not aware of any previous work that had been completed on this at the time, so we conducted two experiments – 1) a literature review of currently deployed Java textbooks at our university, and 2) a questionnaire completed by students to indicate their perceived difficulty levels of Java topics. The results of these experiments are in Yau and Joy (2004), and the topics and their levels of difficulty (in brackets) were established as follows – assignment (1), expressions (2), output (3), input (4), if-statements (5), for-loops (6), arrays (7), methods (8), classes (9). For example, when participants have a lower level of motivation, easier LOs will be suggested to them to study, and vice versa. The proficiency level of the LO and the length of time it requires to be completed are matched with the knowledge level of the student and the amount of available time that they have. In summary, I had used my previous knowledge of Java and the difficulty levels of topics within this (from my masters studies – Yau (2004)), to assign some of the appropriate Java LOs to particular contexts of use.

Participants were recruited via lectures and emails within our university as well as in other universities via HEA-ICS (Higher Education Academy – Information Computer Sciences). A total of 14 university students participated in our study – Warwick (6), Nottingham Trent (2), Coventry (2), Greenwich (2), Bradford (1) and Dundee (1). It was not necessary to record gender and age information. Participants were asked to complete an online feedback form after they had finished studying/learning an LO. Feedback required from participants primarily related to 1) how useful they had found the study of the LO in the contexts, 2) whether their learning experiences of using the LO was more enjoyable as a result of studying it in those contexts and 3) whether the suggestion rules were appropriate in the recommendation of LOs.